Johansson was a native of Söderhamn, in the Hälsingland province of Sweden. Studying classical piano as a child, he would also go on to master the guitar, organ and accordion, before turning on to swing and bebop as a teenager. He met saxophonist Stan Getz while at university. He abandoned his studies to play jazz full-time, and worked with many American jazz musicians, becoming the first European to be invited to join the Jazz at the Philharmonic package.
The years 1961 to 1968 produced a string of classic albums, which would help define his style of re-imagining traditional European folk tunes via jazz and the avant garde. These included Jazz på svenska and Jazz på ryska (Jazz in Russian) which are both available in an expanded form on CD. Jazz på ungerska (Jazz in Hungarian) together with Danish Jazz violinist Svend Asmussen is the third album in that series. Jazz in Swedish comprises variations on sixteen Swedish folk songs with Georg Riedel playing double bass. During this period, Johansson also made several recordings with Radiojazzgruppen.
The Grammy award winning albums Musik genom fyra sekler (Music from the past four centuries) builds on traditional Swedish melodies, but this time uses larger groups of musicians. There were also 300.000 and two trio sets, 8 Bitar and Innertrio, which have been reissued as a single CD.
With his career including film & TV music, Johansson is also best known as the composer of "Here Comes Pippi Longstocking" ("Här kommer Pippi Långstrump"), the theme song of the famous Swedish TV series, Pippi Longstocking. With lyrics by character/series creator Astrid Lindgren and sung by the series' young star Inger Nilsson, it would also be one of Johansson's last works.
In November 1968 Jan Johansson died in a car crash on his way to a concert in a church in Jönköping, Sweden. His sons, HammerFall drummer Anders Johansson and Stratovarius keyboardist Jens Johansson, run Heptagon Records which keeps their father's recordings available.
American hip hop group Non Phixion sampled "Bandura" for their song "Skum". The Swedish band Opeth has claimed him as an influence on the title track for their album Heritage.