Christmas feels like the right time for ghost stories. What makes them work?
There are many types of ghost story but people generally agree that the “English” version is the best. It’s something we do very well! And M R James is the undisputed master of the form. Many of his stories were written to be read around Christmas to a select group of friends. He understood the dual nature of the season – the cosiness of sitting round the fire, but at the same time the need to banish the dark.

What makes the English ghost story so distinctive? Perhaps it’s because of our interest in repression: James is a master of this, of letting fear seep through the cracks. The terror he creates is monstrous but because it’s contained within a sort of fusty academic carapace, it feels both authentic and strangely comforting. As the creator of the “antiquarian” ghost story, he insisted on a historical setting – not necessarily hundreds of years old but 30 or 40. It needs to have some distance. A story told to someone told to someone else . . .

Do you think the best ghost-story writers, such as James, were psychologically troubled?
Often in James’s stories, something from the past creeps forward that eventually ensnares the protagonist through his or her own avarice, clumsiness or stupidity, with terrible consequences. We can see the roots of this trope in James’s own life. Born in 1862, he was brought up in parsonages and rectories, educated at Eton and went to King’s College, Cambridge as an undergraduate. He was most comfortable in academia, acting as director of the Fitzwilliam Museum in Cambridge, and returning to his old college as provost in 1905. He stayed there until the last year of the First World War, when he moved back to Eton, again as provost. He was incredibly well-read – he knew the apocrypha of the Bible as well as his own language – but wore it lightly.

James was also a deeply conservative man: he didn’t like change and he increasingly feared the future and technology. That must explain why so many of his stories are set within a cosy environment but also in the past, where he felt at home. He actively campaigned to stop women becoming members of his college, for example. Indeed, his fear of women – and particularly of female genitalia – is a recurring theme in his work. His heroes are fusty academics, like him, middle-aged bachelors living a sheltered life, “pent ‘mid cloisters dim”. Whereas the manifestation of the monster often takes the form of the folklore idea of the vagina dentata – a terrible moment of putting your hand under the pillow and finding something wet and hairy with teeth, say. In “The Diary of Mr Poynter”, there is a figure made of hair:

. . . absolute stillness greeted his touch, made
him look over the arm. What he had been
touching rose to meet him. It was in the
attitude of one that had crept along the floor
on its belly, and it was, so far as could be
collected, a human figure. But of the face . . .
no feature was discernible, only hair.

That idea recurs in “Canon Alberic’s Scrap- Book”, with a haunted picture, showing a terrible creature where “at first you saw only a mass of coarse, matted black hair”. This horrible apparation manifests to haunt the academic Dennistoun after he buys the scrapbook. In the television adaptation of the story, you see bits of it when you reach the climax: there’s a terrible suggestion of a wet, hairy leg. But that’s all.

This fear of human contact, meant that for years I had assumed James to be asexual – a kind of “damn any feeling for anything other than dusty books” attitude. But the new edition of his work makes clear this wasn’t the case. He used to “rag” male students for fun during Christmas parties, wrestling them on the floor and tickling them. He also seems to have been desperately but chastely in love with a fellow academic from whose death he never recovered.

A colleague of James’s once said, perjoratively, that his was a life untroubled – a smooth progression from Eton to Cambridge and then back to Eton. He never experienced real life; it was in every sense academic. Nothing rippled the surface except these things we now know about. So why did he choose to write ghost stories? It’s a fascinating question.

Was the ghost story the form that allowed him to challenge the rational world he inhabited?
I think he would have fought the idea that he was exorcising some sort of demon. In his mind, he was creating these things as entertainment and he also wrote essays on how he performed the tricks. But I keep coming back to the idea that things were smuggled into his ghost stories, consciously or not, that he wasn’t prepared to show to anybody else, in any other way.

Is there a story or an image that particularly haunts you?
When I wrote my own ghost stories for BBC4, I kept returning to the work of M R James to study how he set up moments of shock. But as a writer you have to find what terrifies you – and for me, it tends to be the same things that scared me as a child, whether you find a new version of it or refine it. I was always terrified of faces in the window. Or the idea of being in the back of a car, going down a narrow road where the hedgerows are growing over it, and there is only the glow of the headlights. I was always convinced I’d look out and see a face. My worst fear, though, is even more simple: the idea of waking up with the pressure of someone sitting on the end of the bed, which to this day – I’m going cold now – terrifies me.

Mark Gatiss is co-creator of the BBC TV series “Sherlock”