The original site of the Brown County seat of Brownwood was east of Pecan Bayou. A dispute arose over land and water rights, and the settlers were forced to find a new location. Greenleaf Fisk donated 60 acres (24 ha) to relocate the county seat to the west side of the bayou, on what is now the current site of Brownwood, and 100 additional acres for county use. The town was incorporated in 1884.
During the Second World War, Brownwood was the location of U.S. Army Camp Bowie, which had a peak complement of over 80,000 soldiers. Camp Bowie serves as a training camp today at the intersection Farm-to-Market roads 45 and 2126.
On April 19, 1976, an F5 tornado struck near Brownwood, causing extensive damage, with 11 reported injuries, but no fatalities.
Brownwood's census figures were re-evaluated after the 2000 census from 18,813 to reflect more accurate figures of 20,407. The city was categorized in 2003 as a Micropolitan Statistical Area. The federal Office of Management and Budget has issued a definition of Micropolitan Statistical Areas as "an urban cluster of at least 10,000, but less than 50,000 population, plus adjacent territory that has a high degree of social and economic integration with the core, as measured by commuting ties."
A dispute over water rights created the adjacent city of Early. The two cities are separated by the Pecan Bayou, one of five major tributaries of the Colorado River of Texas.