Just not so

Rudyard Kipling's detractors dismiss him as a mere apologist for Empire. But his latest biographer,

I like to think the Liverpool rock band Space knew they were quoting Rudyard Kipling in "The Female of the Species", their 1996 hit. Over a plangent guitar riff, the lead singer, Tommy Scott, mockingly wailed: "The female of the species/Is more deadly than the male."

Instinctively Space got the point of Kipling - which is that, underneath a forbidding facade, he had a Gothic imagination and was witty, edgy and raw. As a young man, his Barrack-Room Ballads had made him a superstar. Over a lifetime, his apophthegms were more arresting than Noel Coward's.

Not that Kipling is usually seen this way. Even I, who enjoyed his stories as a child, admit I later found him off-putting: not just the ecclesiastical turn of phrase in his more polemical work, such as "Recessional", but the moustachioed face, the men's club dialectics and the insistent nationalism. I was not bothered when Kipling studies were chased off the curriculum by critics of his orientalism. His "Gods of the Copybook Headings" hardly seemed worth aspiring to.

Over the years, my views changed. I noted how this anachronism refused to lie down and die quietly. His resilience was apparent in the enduring popularity of Kim, in the Disneyfication of the Jungle Books and in the disinterring of such complex stories as "Mrs Bathurst" and "They" by a new school of academics such as Craig Raine. I kept hearing about unlikely fans - from the Russian nationalist leader Aleksander Lebed to the Indian novelist Arundhati Roy. And that was before BBC listeners voted "If" the nation's favourite poem. Andre Maurois seemed to have been right that Kipling had "a permanent natural contact with the oldest and deepest layers of human consciousness". As a biographer, I wanted to discover more about the man who achieved this and - unfinished business - what this could tell us about a generation that had rejected him.

Initial research suggested someone more interesting than I had realised. On his mother's side Kipling was plugged into the Victorian intellectual establishment - nephew of both the leading pre-Raphaelite Sir Edward Burne-Jones and the more stuffy Royal Academician Sir Edward Poynter, and cousin of Stanley Baldwin, later Conservative prime minister. Because his parents were in India, young Ruddy spent much time at the Burne- Joneses', where he was exposed to literary and artistic figures from Browning to Morris (his "Uncle Topsy"). By extension he knew families such as that of the scholar Sir Leslie Stephen, whose daughter Virginia Woolf was more influenced by him than she cared to admit: an irony, since the long march from Kipling had started with the introversion of the Bloomsbury set.

Typically Kipling did not make a biographer's task easy. He burnt his parents' papers and had many of his own destroyed. He penned a memoir called, appropriately, Something of Myself. Yet he also wrote a stack of letters that he could not recover and he had such a direct manner that it is impossible to read these and not get a sense of his personality.

Along the way I made exciting biographical discoveries, such as letters from his parents to his headmaster at Westward Ho!, his grim school in north Devon, and his flirtatious billets-doux to an Indian army officer's wife, the original for Mrs Hauksbee, the formidable older woman from Plain Tales from the Hills. Persistent inquiry fleshed out his doomed relationship with Florence Garrard, the lesbian artist who provided the model for Maisie in The Light That Failed, the underestimated novel in which, returning from India in 1889, he tried unsuccessfully but illuminatingly to tackle the big themes of his author's trade.

This new material essentially involved filling in gaps in a well-documented story. More fortuitous was finding about Kipling as a thinker. This dawned as I researched his time in India, where his intellectual maturing is clearly documented in his writings. Kipling arrived on the sub-continent a clever, stroppy, sensitive youth, with no great prospects in back-of-beyond provincial journalism.

As was his nature, he set to learning about his environment. In the words of the Elephant's Child in the Just So Stories, he was full of "satiable curiosity". He studied Urdu and Persian, dabbled with opium and gained a reputation for "going for a mucker" (a euphemism for cavorting with prostitutes) in the bazaars. He became both fascinated and alarmed by what it meant to live in such a teeming country, with its profusion of religions. His story "The Strange Ride of Morrowbie Jukes" reflects his paranoid sense of living on the edge of an abyss. Indeed his sense of the precariousness of civilisation informs the rest of his work, giving it a contemporary existential connotation.

Such insights forced him to think deeply about his fellow Britons' purpose in India. A utilitarian manque, he decided the most important people were the doctors, engineers and administrators who had devoted their lives to bringing the physical benefits of western civilisation to India. His admiration for "doers" persisted till his death.

More interestingly, as a student of sociology he thought deeply about what holds a community together. In his research into the dynamics of society, he became convinced of the need for shared ideals and sense of purpose. He explored this - inter alia - in stories about fringe elements in society. In The Man Who Would Be King, for example, Carnehan and Davot establish a far-off kingdom by developing their own rituals (loosely based on freemasonry). But when they depart from their own rules (a sense of "the law" now featured in Kipling's writings), their Himalayan state falls apart.

Periods in London and the United States negatively reinforced such ideas. The 1890s aesthetes had lost the plot, he felt. His heroes remained Tommy Atkins and McAndrew, the ship's engineer. After marrying an American and settling in Vermont, Kipling was appalled by the wilder manifestations of popular democracy. When, partly with his children in mind, he wrote the Jungle Books, the Bandar-Log represented the unruly American masses, living without a concept of "the law".

He was happier when he returned to England and felt at ease with the traditional, hierarchical nature of its rural life. At Bateman's, his new home in Sussex, he took a more relaxed approach to his study of society. In his Puck stories, he explored how a culture evolves - through a relationship with the land, through storytelling, through religion and the law, through a sense of history, through that elusive mixture of playfulness and mystery that he was so good at identifying.

In the process he went a long way toward defining the nature of Englishness. His version of the national myth celebrates earthiness, individualism, fantasy and freedom through the law. I found this compelling, partly because it was unexpected and partly because it tallied with a late-20th-century need to understand what it means to be English. This discovery explained why Kipling had once caused me problems: in the 1960s global village, there was no need to contemplate such matters.

Kipling was not afraid to write about the opportunistic, piratical elements in the British character, nor was he the xenophobe his detractors imagine. His Britain (sometimes this word is interchangeable with England, sometimes not) was able to absorb Romans, Vikings and Normans. Despite its odd ways - "For Allah created the English mad - the maddest of all mankind!" - it was prepared to assume its responsibilities and to fight oppressors. Give or take a detail, this agenda is not very different from Tony Blair's. The two men would differ over political strategy, of course: having identified his ideal, Kipling was determined to maintain it. Drawing on his Indian experiences, he thought that any liberal chipping away endangered the whole edifice.

What made his vision more moving was the way it was conceived. As far as is possible for an author, Kipling was a humble man. He proudly befriended his sovereign, King George V, but refused all political honours - in particular, a knighthood or the Order of Merit. According to form, Kipling might have joined several writers (including his long-term associate Rider Haggard) who received gongs for their literary trundlings on behalf of Empire. But while he could be cringeingly self-abasing towards the aristocracy, Kipling genuinely coveted his freedom to see the world his own way. He stuck to the precepts of his own poem "The Last Rhyme of True Thomas", where a harpist refuses a king's offer of preferment because he thinks he has a nobler calling in his art.

This desire to be his own man identifies Kipling. He had his particular vision of the world, but when he wrote he gave free rein to his "demon". As he put it in Kim, he had "two separate sides" to his head and would sooner "go without shirt or shoes/Friends, tobacco or bread" than lose that perspective - one that married rationality and fantasy, western and oriental philosophy and the male and female sides of his personality.

The debate on Kipling's position remains open. Although he raises many questions, he provides no easy answers. But there is much to be learnt from the way he stood outside cultural convention and asked awkward questions. In our world of dumbed-down media, that quirky, cussed spirit is much needed. I set out looking for a historical character and was delighted to find a man of genius.

Andrew Lycett's "Rudyard Kipling" is published on 9 September by Weidenfeld & Nicolson, £25