James was born Vivian Leopold James in Kogarah, a southern suburb of Sydney. He was allowed to change his name as a child because "after Vivien Leigh played Scarlett O'Hara the name became irrevocably a girl's name no matter how you spelled it". He chose "Clive", the name of Tyrone Power's character in the 1942 film This Above All.
James's father, Albert Arthur James, was taken prisoner by the Japanese during World War II. Although he survived the prisoner-of-war camp, he died when the American B-24 carrying freed allied POWs he was on ran into the tail of a typhoon, en route from Okinawa to Manila, and crashed into the mountains of southeastern Taiwan. He was buried at Sai Wan War Cemetery in Hong Kong. James would later state that his life’s works originated in his father's death.
James, an only child, was brought up by his mother (Minora May, née Darke), a factory worker, in the Sydney suburbs of Kogarah and Jannali, living some years with his English maternal grandfather.
He was educated at Sydney Technical High School (despite winning a bursary award to Sydney Boys High School) and the University of Sydney, where he studied English and psychology from 1957 to 1960, and became associated with the Sydney Push, a libertarian intellectual subculture. At the university, he contributed to the student newspaper, Honi Soit, and directed the annual students' union revue. He graduated with a Bachelor of Arts with Honours in English in 1961. After graduating, James worked for a year as an assistant editor for the magazine page at The Sydney Morning Herald.
In 1962, James moved to England, which became his home for the rest of his life. During his first three years in London studying in Cambridge, he shared a flat with the Australian film director Bruce Beresford (disguised as "Dave Dalziel" in the first three volumes of James's memoirs), was a neighbour of Australian artist Brett Whiteley, became acquainted with Barry Humphries (disguised as "Bruce Jennings") and had a variety of occasionally disastrous short-term jobs – sheet metal worker, library assistant, photo archivist and market researcher.
James gained a place at Pembroke College, Cambridge, to read English literature. While there, he contributed to all the undergraduate periodicals, was a member and later President of the , and appeared on University Challenge as captain of the Pembroke team, beating St Hilda's, Oxford, but losing to Balliol on the last question in a tied game. During one summer vacation, he worked as a circus roustabout to save enough money to travel to Italy. His contemporaries at Cambridge included Germaine Greer (known as "Romaine Rand" in the first three volumes of his memoirs), Simon Schama and Eric Idle. Having, he claimed, scrupulously avoided reading any of the course material (but having read widely otherwise in English and foreign literature), James graduated with a 2:1—better than he had expected—and began a PhD thesis on Percy Bysshe Shelley.