The Shadow Out of Time indirectly tells of the Great Race of Yith, an species with the ability to travel through space and time. The Yithians accomplish this by switching bodies with hosts from the intended spatial or temporal destination. The story implies that the effect, when seen from the outside, is similar to spiritual possession. The Yithians' original purpose was to study the history of various times and places, and they have amassed a "library city" that is filled with the past and future history of multiple races, including humans. Ultimately the Yithians use their ability to escape the destruction of their planet in another galaxy by switching bodies with a race of cone-shaped plant beings who lived 250 million years ago on Earth. The cone-shaped entities (subsequently also known as the Great Race of Yith) lived in their vast library city in what would later become Australia's Great Sandy Desert (22°3′14″S 125°0′39″E / 22°3′14″S 125°0′39″E / The Shadow Out of Time)).
The story is told through the eyes of Nathaniel Wingate Peaslee, an American living in the first decade of the 20th century, who is "possessed" by a Yithian. He fears he is losing his mind when he unaccountably sees strange vistas of other worlds and of the Yithian library city. He also feels himself being led about by these creatures and experiences how they live. When he is returned to his own body, he finds that those around him have judged him insane due to the actions of the Yithian that possessed his body. While he was experiencing a Yithian existence in Earth's ancient past, the Yithian occupying his body was experiencing a human one in the present day.
The narrator at first believes his episode and subsequent dreams to be the product of some kind of mental illness. His initial relief at discovering other cases like his throughout history is withered when he discovers that the other cases are too similar to his own to be without a connection. The narrator's dreams become more vivid, and he becomes obsessed with archaeology and ancient manuscripts (as was the Yithian) - but lacks any sort of proof that would demonstrate whether he was (or is) simply mad.
He discovers that the Yithians on Earth died out eons ago (their civilization destroyed by a rival, utterly alien pre-human race described as "half-polypous" creatures); but the Yithian minds will inhabit new bodies on Earth after humanity is long gone. His tenuously held sanity is challenged when he discovers the proof he seeks—and that not only do remains of the Yithians' past civilization still exist on Earth; but also still remaining are those who destroyed them. It is also mentioned that the current appearance of the Yithians is not the original; but one acquired during a previous mass-projection of the minds of their race when disaster beckoned, leaving the original inhabitants to die in the bodies of the Yithians.