In 1686, much of Manhattan, including the future Rockefeller Center site, was established as a "common land" of the city of New York. The land remained in city ownership until 1801, when the physician David Hosack, a member of the New York elite, purchased a parcel of land in what is now Midtown for $5,000, equivalent to $74,000 in 2017 dollars. In terms of the present-day street grid, Hosack's land was bounded by 47th Street on the south, 51st Street on the north, and Fifth Avenue on the east, while the western boundary was slightly east of Sixth Avenue (also known as Avenue of the Americas). At the time, the land was sparsely occupied and consisted mostly of forest. Hosack opened the Elgin Botanic Garden, the country's first public botanical garden, on the site in 1804. The garden would operate until 1811, when Hosack put the land on sale for $100,000 (equivalent to $1,471,000 in 2017). As no one was willing to buy the land, the New York State Legislature eventually bought the land for $75,000 (equivalent to $1,103,000 in 2017).
In 1814, the trustees of Columbia University (then Columbia College) were looking to the state legislature for funds when the legislature unexpectedly gave Hosack's former land to the college instead. The gardens became part of Columbia's "Upper Estate" (as opposed to the "Lower Estate" in Lower Manhattan), on the condition that the college move its entire campus to the Upper Estate by 1827. Although the relocation requirement was repealed in 1819, Columbia's trustees did not see the land as "an attractive or helpful gift", so the college let the gardens deteriorate. The area would not become developed until the 1830s, and the land's value did not increase to any meaningful amount until the late 1850s, when St. Patrick's Cathedral was built nearby, spurring a wave of development in the area. Ironically, the cathedral was built in that location because it faced the Upper Estate gardens. By 1860, the Upper Estate contained four row houses below 49th Street as well as a wooden building across from the cathedral. The surrounding area was underdeveloped, with a potter's field and the railroad lines from Grand Central Depot located to the east.
Columbia built a new campus near its Upper Estate in 1856, selling a plot at Fifth Avenue and 48th Street to the St. Nicholas Church to pay for construction. Shortly afterward, Columbia implemented height restrictions that prevented any taller buildings, such as apartment blocks or commercial and industrial buildings, from being built on its property. Narrow brownstone houses and expensive "Vanderbilt Colony" mansions were built on nearby streets, and the area became synonymous with wealth. By 1879, there were brownstones on every one of the 203 plots in the Upper Estate, which were all owned by Columbia.
The construction of the Sixth Avenue elevated line in the 1870s made it easy to access commercial and industrial areas elsewhere in Manhattan. However, the line also drastically reduced property values because the elevated structure obstructed views from adjacent properties, and because the trains on the structure caused noise pollution. Columbia sold the southernmost block of its Midtown property in 1904, using the $3 million from the sale (equivalent to $64.8 million in 2016) to pay for newly acquired land in Morningside Heights even further uptown, to replace its Lower Estate. Simultaneously, the low-lying houses along Fifth Avenue were being replaced with taller commercial developments, and the widening of the avenue between 1909 and 1914 contributed to this transition. Columbia also stopped enforcing its height restriction, which Okrent describes as a tactical mistake for the college because the wave of development along Fifth Avenue caused the Upper Estate to become available for such redevelopment. Since the leases on the Upper Estate row houses were being allowed to expire without renewal, the university's real estate adviser John Tonnele was tasked with finding suitable tenants, who could net the university more profit than what the row houses' occupants were currently paying.