In January 1980, the members of Talking Heads returned to New York City after the tours in support of their 1979 critically acclaimed third album, Fear of Music, and took time off to pursue personal interests. Singer David Byrne worked with Brian Eno, the record's producer, on an experimental album, My Life in the Bush of Ghosts. Keyboardist Jerry Harrison produced an album for soul singer Nona Hendryx at the Sigma Sound Studios branch in New York City; Hendryx and the studio were used during the Remain in Light recording on Harrison's advice.
Drummer Chris Frantz and bassist Tina Weymouth, a married couple, discussed leaving Talking Heads after Weymouth suggested that Byrne was too controlling. Frantz did not want to leave, and the two took a long vacation in the Caribbean to ponder the state of the band and their marriage. They became involved in Haitian Vodou religious ceremonies, practised native percussion instruments, and socialised with the reggae rhythm section of Sly and Robbie.
Frantz and Weymouth ended their holiday by purchasing an apartment above Compass Point Studios in Nassau, the Bahamas, where Talking Heads had recorded their second album, More Songs About Buildings and Food. Byrne joined the duo and Harrison there in early 1980. The band members realised that it had been solely up to Byrne to craft songs even though they were performed as a quartet. They had tired of the notion of a singer leading a backup band; the ideal they aimed for, according to Byrne, was "sacrificing our egos for mutual cooperation". Byrne additionally wanted to escape "the psychological paranoia and personal torment" he had been writing and feeling in New York. Instead of the band writing music to Byrne's lyrics, Talking Heads performed instrumental jams, using the Fear of Music song "I Zimbra" as a starting point.
Eno arrived in the Bahamas three weeks after Byrne. He was reluctant to work with the band again after collaborating on the previous two albums. He changed his mind after being excited by the instrumental demo tapes. The band and Eno experimented with the communal African way of making music, in which individual parts mesh as polyrhythms. Afrodisiac, the 1973 Afrobeat record by Nigerian musician Fela Kuti, became the template for the album. Weymouth said that the beginnings of hip-hop music made Talking Heads realise that the musical landscape was changing. Before the studio sessions began, longtime friend David Gans instructed the band that "the things one doesn't intend are the seeds for a more interesting future". He encouraged them to experiment, improvise and make use of "mistakes".