The initial sessions for what became the Heaven and Hell album actually began with Ozzy Osbourne after Black Sabbath's Never Say Die! Tour, when the band moved to Los Angeles for eleven months in an attempt to record a new album, a process that guitarist Tony Iommi described in his autobiography as a "highly frustrating, never-ending process". In his own autobiography, Osbourne states that he had become fed up with the experimentation on the band's previous albums Technical Ecstasy and Never Say Die!, preferring the band's earlier, heavier sound, but also admits "it was clear they'd had enough of my insane behaviour". In his memoir, Iommi revealed that he still has a tape featuring Osbourne singing an early version of what would become the song "Children of the Sea" with a different lyric and a totally different vocal melody.
Ronnie James Dio was first introduced to Iommi in 1979 by Sharon Arden, who would later marry the band's then-recently fired vocalist Osbourne. Initially, Dio and Iommi discussed forming a new band, rather than a continuation of Black Sabbath. The pair met again by chance at The Rainbow on Sunset Strip in Los Angeles later that year. Both men were in similar situations, as Dio was seeking a new project and Iommi required a vocalist. Said Dio of the encounter, "It must have been fate, because we connected so instantly." The pair kept in touch via telephone, until Dio arrived at Iommi's Los Angeles house for a relaxed, getting-to-know-you jam session. On that first day, the duo finished the song "Children of the Sea", a song Iommi had abandoned prior to Osbourne's firing.
Black Sabbath's line-up was in chaos leading up to the recording of Heaven and Hell. Not only had the band recently replaced its lead vocalist, but drummer Bill Ward was battling personal issues that would see him eventually leave the band. Original demo recordings for the album featured Geoff Nicholls on bass, as longtime bassist Geezer Butler was going through a divorce and his future with the band was in question. In fact, when Dio first joined the band he doubled as bassist and vocalist, having played bass in the band Elf in the early 1970s. At one point Iommi contacted close friend Frank Zappa for help finding a bassist; Zappa offered his bassist for the Heaven and Hell sessions but Iommi preferred a permanent member. Eventually Butler returned and Nicholls stayed on as the band's unofficial keyboardist. Former Elf and Rainbow bassist Craig Gruber also rehearsed with the band during this period, though the extent of Gruber's involvement is unclear. In a 1996 interview, Iommi stated that Gruber rehearsed with the band only "for a bit". Gruber has stated that his contribution was much more substantial; he says he co-wrote most of Heaven and Hell's songs and that it was he and not Butler who played bass on the album. Despite not being credited for his contributions, Gruber asserted, "we came to a suitable financial arrangement". Iommi later stated in his 2011 autobiography that Gruber did in fact record all the bass parts on the Heaven and Hell album, but that Butler re-recorded the parts upon his return to the band without having listened to Gruber's bass tracks.
Black Sabbath performing in Cardiff
Heaven and Hell was recorded at the Criteria Studios in Miami (the same studio in which the band had recorded Technical Ecstasy) and Studio Ferber in Paris. Dio suggested the band hire producer Martin Birch for the album. Birch was the first outside producer the band had used since 1971's Master of Reality, as Iommi had primarily produced the band's albums since that point on his own. Iommi stated that the band felt that they were creating something very special, writing in his memoir that, "Ozzy would sing with the riff. Just listen to 'Iron Man' and you'll catch my drift: his vocal melody line copies the melody of the music. There was nothing wrong with that, but Ronnie liked singing across the riff instead of with it, come up with a melody that was different from that of the music, which musically opens a lot more doors. I don't want to sound like I'm knocking Ozzy, but Ronnie's approach opened up a new way for me to think ..."
Drummer Bill Ward has stated that he has "no memory" of making the album, due in large part to his alcoholism. His behaviour became quite erratic; when the band began touring in support of Heaven And Hell, Ward began dictating long and rambling press releases to the band's public relations representatives after every show, instructing them to "get that out on the news wires tonight". Ward's personal issues, which included the deaths of both his parents, would soon force him to leave the band. Dio reportedly answered the telephone in his hotel room one morning mid-tour to hear Ward say "I'm off then, Ron", to which Dio replied "That's nice Bill, where are you going?" "No, I'm off mate. I'm at the airport now", indicating that he was incapable of completing the tour with the band. American drummer Vinny Appice was quickly brought in to replace him. As such, the Heaven and Hell album represents the only material recorded by Dio-era Black Sabbath/the Heaven and Hell band that does not have Appice on drums.