James was born in a Goodnestone, Dover, clergy house in Kent, England, although his parents had associations with Aldeburgh in Suffolk. His father was Herbert James, an Evangelical Anglican clergyman, and his mother, Mary Emily (née Horton), was the daughter of a naval officer. He had two older brothers, Sydney and Herbert (nicknamed "Ber"), and an older sister, Grace. Sydney James later became Archdeacon of Dudley. From the age of three (1865) until 1909 James's home, if not always his residence, was at the Rectory in Great Livermere, Suffolk. This had previously been the childhood home of another eminent Suffolk antiquary, Thomas Martin of Palgrave (1696–1771). Several of James's ghost stories are set in Suffolk, including "'Oh, Whistle, and I'll Come to You, My Lad'" (Felixstowe), "A Warning to the Curious" (Aldeburgh), "Rats" and "A Vignette" (Great Livermere).
In September 1873 he arrived as a boarder at Temple Grove School, one of the leading boys' preparatory schools of the day. From September 1876 to August 1882 he studied at Eton College, where he claims to have translated the Book of Baruch from its original Ethiopic in 1879. He lived for many years, first as an undergraduate (1882–1885), then as a don and provost, at King's College, Cambridge, where he was also a member of the Pitt Club. The university provides settings for several of his tales. Apart from medieval subjects, James toured Europe often, including a memorable 1884 tour of France in a
Cheylesmore tricycle, studied the classics and appeared very successfully in a staging of Aristophanes' play The Birds, with music by Hubert Parry. His ability as an actor was also apparent when he read his new ghost stories to friends at Christmas time.