Ormandy was born in Budapest, Austria-Hungary, as Jenő Blau, the son of Jewish parents Rosalie and Benjamin Blau, a dentist who was also an amateur violinist. Ormandy began studying violin at the Royal National Hungarian Academy of Music (now the Franz Liszt Academy of Music) at the age of five. He gave his first concerts as a violinist at age seven and, studying with Jenő Hubay, graduated at 14 with a master's degree. In 1920, he obtained a university degree in philosophy. In 1921, he moved to the United States.
Around this time Blau changed his name to "Eugene Ormandy"; "Eugene" being the equivalent of the Hungarian "Jenő". Accounts differ on the origin of "Ormandy"; it may have either been Blau's own middle name at birth, or that of his mother.
He was first engaged by conductor Erno Rapee, a former Budapest friend and fellow Academy graduate, as a violinist in the orchestra of the Capitol Theatre in New York City, a 77-player ensemble which accompanied silent movies. He became the concertmaster within five days of joining and soon became one of the conductors of this group. Ormandy also made 16 recordings as a violinist between 1923 and 1929, half of them using the acoustic process.
Arthur Judson, the most powerful manager of American classical music during the 1930s, first heard Ormandy when he conducted (as a freelancer) for a dance recital at Carnegie Hall by Isadora Duncan; Judson later said, "I came to see a dancer and instead heard a conductor".
Judson greatly assisted Ormandy's career, and when Arturo Toscanini was too ill to conduct the Philadelphia Orchestra in 1931, Judson asked Ormandy to stand in. This led to Ormandy's first major appointment as a conductor, in Minneapolis.