Settlement began in the 1870s when Georgia native
H.H. Peaster bought 160 acres (0.65 km2) of land and built a house. The community that developed near the site of his home was originally called Freemont; in 1885 the name was changed to Peasterville. Throughout the 20th century, Peaster has served area farmers as a school and church community. During the mid-1920s the town's population grew to approximately 300, but this was soon followed by a steep decline after the Great Depression. From 1975 to 1990, Peaster had around 80 residents. Current estimates put the population at approximately 100.