John Baldwin was born in Sidcup, Kent. He started playing piano at age six, learning from his father, Joe Baldwin, a pianist and arranger for big bands in the 1940s and 1950s, notably with Ambrose and his Orchestra. His mother was also in the music business which allowed the family to often perform together touring around England as a vaudeville comedy act. His influences ranged from the blues of Big Bill Broonzy, the jazz of Charles Mingus, to the classical piano of Sergei Rachmaninoff.
Because his parents often toured, Jones was sent to boarding school at a young age. He was a student at Christ's College, Blackheath, London where he formally studied music. At the age of 14, Jones became choirmaster and organist at a local church and during that year, he also bought his first bass guitar, a Dallas Tuxedo solid body electric followed by multiple basses in which he part exchanged until he finally bought his 1962 Fender Jazz Bass which he used until 1976. The fluid playing of Chicago musician Phil Upchurch on his You Can't Sit Down LP, which includes a memorable bass solo, is cited by Jones as being his inspiration to take up the instrument.
Jones joined his first band, The Deltas, at 15. He then played bass for jazz-rock London group,
Jett Blacks, a collective that included guitarist John McLaughlin. Jones' big break came in 1962 when he was hired by Jet Harris and Tony Meehan of the successful British group The Shadows for a two-year stint. Shortly before hiring Jones, Harris and Meehan had just had a Number 1 hit with "Diamonds" (a track on which Jones' bandmate-to-be Jimmy Page had played.) Jones' collaboration with the Shadows nearly prevented the future formation of Led Zeppelin, when the parties engaged in talks about the possibility of Jones replacing their bassist Brian Locking, who left the band in October 1963, but John Rostill was ultimately chosen to fill the position.
In 1964, on the recommendation of Meehan, Jones began studio session work with Decca Records. From then until 1968, he played on hundreds of recording sessions. He soon expanded his studio work by playing keyboards, arranging and undertaking general studio direction, resulting in his services coming under much demand. He worked with numerous artists including the Rolling Stones on Their Satanic Majesties Request (Jones' string arrangement is heard on "She's a Rainbow"); Herman's Hermits; Donovan (on "Sunshine Superman," "Hurdy Gurdy Man," and "Mellow Yellow"); Jeff Beck; Françoise Hardy; Cat Stevens; Rod Stewart; Shirley Bassey; Lulu; and numerous others. As well as recording sessions with Dusty Springfield, Jones also played bass for her Talk of the Town series of performances. His arranging and playing on Donovan's "Sunshine Superman" resulted in producer Mickie Most using his services as choice arranger for many of his own projects, with Tom Jones, Nico, Wayne Fontana, the Walker Brothers, and many others. In 1967, Most, as music supervisor, also tabbed Jones to arrange the music for Herman's Hermits' theatrical film Mrs. Brown, You've Got a Lovely Daughter, released in January 1968. Such was the extent of Jones' studio work – amounting to hundreds of sessions – that he said years later that "I can't remember three-quarters of the sessions I was on."
It was during his time as a session player that Jones adopted the stage name John Paul Jones. This name was suggested to him by a friend, Andrew Loog Oldham, who had seen a poster for the 1959 film John Paul Jones in France. He released his first solo recording as John Paul Jones, "Baja" (written by Lee Hazlewood and produced by Oldham) / "A Foggy Day in Vietnam", as a single on Pye Records in April 1964.
Jones has stated that, as a session musician, he was completing two and three sessions a day, six and seven days a week. However, by 1968 he was quickly feeling burnt out due to the heavy workload: "I was arranging 50 or 60 things a month and it was starting to kill me."