Originally named Khok Samet Chun (โคกเสม็ดชุน, "shore eugenia knoll"), Hat Yai was a small village until the southern railway was built there, making it one of the major rail hubs of the line. The junction which connected the town of Songkhla with the main route was formerly in the U Tapao area, but was later moved to Khok Samet Chun in 1922 when the U Tapao area turned out to be flood-prone. At that time, Khok Samet Chun had only four residents, but thanks to the investments of Khun
Niphat Chinnakorn (謝枢泗, Jia Gi Si, 1886–1972, Hakka Chinese), the railway contractor for the railway line from Nakhon Si Thammarat to Pattani, it quickly grew into a small town.
In 1928 Hat Yai was made a community (chumchon), which was upgraded to a sanitary district (sukhaphiban) on December 11, 1935. It covered an area of 4.4 km², and was administered by the first mayor, Udom Bunlong. In 1938, the municipal administration building was completed. On March 16, 1949, Hat Yai was granted town status (thesaban mueang). On May 10, 1961, the area covered by the municipality was increased to 8 km². As a result of the town's continuing growth, on August 13, 1968, a larger, new municipal administration building was opened. On April 24, 1977, the total area of the municipality was enlarged for the second time to 21 km². Finally, in 1995, the town was upgraded to city status (thesaban nakhon).
The name "Hat Yai" is a short version of "mahat yai", meaning big mahat (Thai: มะหาด) tree, a relative of jackfruits in genus Artocarpus.