Graves was born into a middle-class family in Wimbledon, then part of Surrey, now part of south London. He was the third of five children born to Alfred Perceval Graves (1846–1931), who was the sixth child and second son of Charles Graves, Bishop of Limerick, Ardfert and Aghadoe.
 His father was an Irish school inspector, Gaelic scholar and the author of the popular song "Father O'Flynn", and his second wife, Amalie Elisabeth Sophie von Ranke (1857–1951), the niece of the historian Leopold von Ranke.
At the age of seven, double pneumonia following measles almost took Graves's life, the first of three occasions when he was despaired of by his doctors as a result of afflictions of the lungs, the second being the result of a war wound (see below) and the third when he contracted Spanish influenza in late 1918, immediately before demobilisation. At school, Graves was enrolled as Robert von Ranke Graves, and in Germany his books are published under that name, but before and during the First World War the name caused him difficulties.
In August 1916 an officer who disliked him spread the rumour that he was the brother of a captured German spy who had assumed the name "Carl Graves". The problem resurfaced in a minor way in the Second World War, when a suspicious rural policeman blocked his appointment to the Special Constabulary. Graves's eldest half-brother, Philip Perceval Graves, achieved note as a journalist and his younger brother, Charles Patrick Graves, was a writer and journalist.