A crash pad, a haven, a residence, a home . . . The 18 different buildings I’ve lived in to date have been, between them, all of these things to me. Most of my frenetic itinerancy has taken place during my adult years, though by the time I left home for university I’d already notched up six different addresses.

The first house I can barely remember; the second comes back to me in hazy snapshots – a sip of orange juice by the open door of the kitchen, a go on the tyre-swing in the garden (or was it a proper plastic seat?); but the third, where I lived between the ages of three and six, is a different matter. I can still take a mental walk through the rooms of that house in Newtown Road, Southampton. I can view it from the street – the rough, pebble-dash walls, the little bay window, the narrow tarmac drive – or I can walk through its interior, taking note of the layout, pausing at the turn on the stairs: right to my parents’ bedroom, left to my brothers’ and mine. I know where the windows are, the furniture – I can even see the fabric of the armchairs and curtains. And I can do the same with all the homes that came afterwards.

There is nothing exceptional about this. Why is it that we remember our houses with such uncanny clarity? The French philosopher Gaston Bachelard believed that there is a dynamic interplay between the mind and its surroundings. Each is shaped by and responds to the other. At the simplest level, a house may reflect something of our character through the furnishings we choose to adorn it, the colours we paint its walls, the number of locks we put on its doors and so on. But there is more to it than that.

The psychologist Carl Jung spent over 30 years building and extending Bollingen Tower, a second home for him and his wife, Emma, on the banks of Lake Zurich. He believed that the towers and annexes of this castle-like structure represented his psyche. After Emma’s death, he added a second storey, which he said symbolized the expansion of consciousness attained in old age. If our consciousness does expand, our sense of home begins to grow the moment we are born – from the womb, to our parents’ arms, to the cot, and eventually to the building where we spend our earliest years.

Since our association with houses is a lifelong affair, it’s hardly surprising that our fascination with other people’s houses grows as we mature. Bachelard maintained that candlelight in a window was enough to bring a street to life. He saw the house, animated by the mind’s activities, as a kind of theatre. Many artists – painters, poets, film-makers – have shared this vision. Wes Anderson’s recent Moonrise Kingdom opens with the camera tracking across the rooms of a home whose façade has been cut away to reveal what is, in effect, a life-size doll’s house. It is a compelling prelude: in a matter of moments the basic tenor of one family’s life – its habits, quirks and make-up – is laid bare.

What makes Anderson’s device so potent is that the narratives played out behind the screening facade of a house are not intended for public view. But sometimes the players have no say in the matter, as the former poet laureate Ted Hughes (an avowedly private man) knew only too well. His final collection, Birthday Letters, was written secretly over many years, following the suicide of his first wife, Sylvia Plath.
Several of the poems in it recall specific houses and other buildings that served as the backdrop to the couple’s difficult union. In “18 Rugby Street”, Hughes calls to mind the house in which he once waited for his future wife, at the start of a relationship that was to be exposed to the cruellest level of public scrutiny:

So there in Number Eighteen Rugby Street’s
Victorian torpor and squalor I waited for you.
I think of that house as a stage-set –
Four floors exposed to the auditorium.
On all four floors, in, out, the love-struggle
In all its acts and scenes, a snakes and ladders
Of intertangling and of disentangling
Limbs and loves and lives . . .

While the voyeurism here is imagined, a flick through the television guides of recent years reveals the scale of our appetite for looking into other people’s homes: Through the Keyhole; Come Dine with Me; How Clean Is Your House? It’s an obsession that seems to take little account of class or temperament or political persuasion. Even the most serious-minded are susceptible – witness the success of the Guardian’s “Writers’ rooms” series, where readers can scrutinise at leisure the minutiae of an author’s private workspace. There is the mandatory desk, the chair, the teetering piles (or neatly arranged shelves) of books. But what is that bizarre-looking trinket on the mantelpiece? The open hip flask in the corner? We are a species of nosy parkers.

Walk down any street after dark and it’s hard to stop your gaze from being drawn sideways wherever there are undrawn curtains framing a lit interior: few of us can resist these fleeting tableaux of domestic life. Edward Hopper was another artist to exploit this impulse in his work. The New York photographer Gail Albert Halaban recently took pictures of 16 houses in Massachusetts that were first made famous in Hopper’s paintings. Many of the originals – Night Windows, Cape Cod Morning, House at Dusk – offer tantalising glimpses of interior scenes, said to have been suggested by Hopper’s rides through New York on the elevated or “El” train. In Rear Window, Hitchcock poses an uncomfortable question by subtly implicating the viewer in the action: how many of us, finding ourselves in the place of James Stewart’s housebound character, would withstand the temptation to observe the goings on in our neighbours’ apartments?

It has been ever thus. The heroine of Wuthering Heights is unable to resist the lure of lights in her neighbours’ house. One of the book’s most enduring images is of Cathy and Heathcliff as children clinging to the window ledge of the drawing room under cover of night to spy on the wealthy Linton family through half-closed curtains. The book’s two contrasting houses are clearly symbolic of the two different sides of Cathy’s nature – one wild and windswept, the other (Thrushcross Grange) full of decency and order. She is not unusual in this: again it was Bachelard who observed that we all have our cottage moments and our palace moments.

Our infatuation with houses stems, perhaps, from a conviction that in some way they both contain and express an essential part of us; as if our presence might seep into the very fabric of the walls that surround us. Over time, that presence is absorbed deep into the stone and brickwork – and what we are really talking about here is memory. Several of Thomas Hardy’s poems (especially those about his late wife) express this idea. “At Castle Boterel”, for instance, tells us how the presence of the speaker and his nowdead sweetheart is preserved forever in the surrounding rocks: “what they record in colour and cast/Is – that we two passed”.

The thought is not so very odd: houses have long served as monoliths for the dead. English Heritage puts up around ten new blue plaques each year in London and there are similar schemes across the UK, from Aberdeen to Zennor, from the British Film Institute to the Plastics Historical Society. The process by which a proposed residence is deemed worthy of recognition is lengthy and elaborate but in every instance it’s the worthiness of the inhabitant, and not the building itself, that is assessed.

It would seem, then, that houses remember us just as we remember them; that memories become embodied by the places we live in. The poet Christopher Reid’s award-winning collection, A Scattering, is a moving tribute to his late wife, Lucinda. Towards the end of it, in the long poem “Lucinda’s Way”, the speaker recalls a moment one afternoon when husband and wife:

. . . crossed on the stairs.
Unprompted, you announced, “I love
our house” –
an outburst of the plainest happiness
that the high stairwell
enshrines still.

By the same token, to revisit an old house and discover that there’s no outward evidence of our ever having been there can leave us with an unsettling sense of our smallness. In “55 Eltisley” Ted Hughes registers such a shock:

Our first home has forgotten us.
I saw when I drove past it
How slight our lives had been
To have left not a trace.

Our sense of past self is often so closely connected with the house we lived in at the time as to be inseparable from it. Seeing that house inhabited by strangers is disquieting – like bumping into an ex who’s found a completely new way of dressing or doing their hair, on the arm of someone we’ve never seen before.

At the root of our disquiet is the queer but strongly felt notion that if we live inside our houses, they also live inside us – often long after we’ve left them. In his semi-autobiographical The Notebooks of Malte Laurids Brigge, the poet Rilke describes a house he had lived in many years before as “quite dissolved and distributed inside me; here one room, there another, and here a bit of corridor. […] Thus the whole thing is scattered about inside me: the rooms, the stairs that descended with such ceremonious slowness; others, narrow cages that mounted in a spiral movement, in the darkness of which we advanced like the blood in our veins.”

Philip Larkin combines the notion of the internalized house with that of the house as theatre in his unflinching portrayal of ageing, “The Old Fools”:

Perhaps being old is having lighted rooms
Inside your head, and people in them, acting.
People you know, yet can’t quite name; each looms
Like a deep loss restored, from known doors turning,
Setting down a lamp, smiling from a stair, extracting
A known book from the shelves; or sometimes only
The rooms themselves, chairs and a fire burning,
The blown bush at the window, or the sun’s
Faint friendliness on the wall some lonely
Rain-ceased midsummer evening. That is where they live:
Not here and now, but where all happened once.

Given the symbiotic nature of the relationship, it’s hardly surprising that the boundary between house and human should sometimes blur into non-existence. While Bachelard perceived a lit window as a portal to the action inside a house, for the French poet Rimbaud a lit interior is an eye looking outwards on to the street: the lighted lamp “watches in the secret heart of night” (veille au coeur secret de la nuit). And in the contemporary poet Jen Hadfield’s “Still Life with the Very Devil”, other, more covert body parts are in evidence in a house where the dishes are “stacked like vertebrae./ Under the broiler,/turned sausages ejaculate”.

This way of thinking is by no means confined to poetry and philosophy. The house/body analogy is so woven into our everyday language that we hardly notice its presence – in medical as well as other phrases. We talk of the “roof” of the mouth, the “wall” of the womb, the muscles of the pelvic “floor”; and – more curiously – our eyes are said to be the “windows” of the soul. On the flip side, every chimney has a “breast”, every cave a “mouth”, and walls (as many a gossip has learned to their cost) have “ears”.

I think about this last phrase in relation to the house I live in now. What conversations must it have been witness to? Set in a pretty Somerset village, it’s a far cry from the modern red-brick house of my teenage years, which was perched on a dead-end road with a huge chemical factory at one end and a smaller electroplating factory at the other. My current house is almost 400 years old; it was put up during the only time in English history when there was no monarch on the throne. Here and there, the flagstone floors are worn to a hollow by the many hundreds of feet that have travelled over them. So it is that the house serves as a reminder of my own impermanence – a thought I find every bit as consoling as it is discomfiting: more often than not our houses both precede and outlast us. And if they do retain some sense of our existence – even if only fleetingly – so much the better. Life rushes on, before, behind, around us; for the time being, at least, I’m more than happy to stay put.

Julia Copus’s latest collection is “The World’s Two Smallest Humans” (Faber, £9.99).