The darkness at noon for Arthur Koestler was in his heart. Yet his early work, inspired by his disillusionment with communism, will survive the memory of his unlovable personality

Arthur Koestler: The Homeless Mind

David Cesarani<em> Heinemann, 646pp, £25</em>

Not all extraordinary writers lead extraordinary lives - some shelter happily in ivory towers of the imagination - but Arthur Koestler's own story might seem implausible even in one of his novels. Born in 1905 into a prosperous Jewish family in Budapest, he studied engineering in Vienna as a teenager. There he fell under the spell of Vladimir Jabotinsky, the charismatic leader of right-wing revisionist Zionism, described by David Cesarani as "one of the most chillingly logical and aggressive forms of Zionism".

Startlingly precocious, Koestler dropped out, went to Palestine, and was a successful newspaper correspondent by the time he was 22; he later supplemented his income by writing "sexology" books somewhere between pop scholarship and soft porn. In 1931, he took part in an expedition to the Arctic in the airship Graf Zeppelin. Then, having joined the Communist Party, he went to Russia, where he met Radek and Bukharin, little knowing that their subsequent fate would inspire his most famous book.

Back in the west, he worked in Willi Munzenberg's Comintern apparatus and, when the Spanish civil war broke out, he made three trips there, the last almost fatally: he was imprisoned by the Francoists, and for weeks listened from his cell to the firing squads at work as he awaited his own execution. But he was released and wrote Spanish Testament, which first established his name. In 1938 he left the party for reasons he explained in a devastating letter which has only recently been discovered.

When the second world war broke out, Koestler found himself interned once more, this time in France. But he made his way to England and wrote Darkness at Noon, which made him famous and then rich. He was caught up again in the Jewish story as it reached its horrible climax in Europe, revisited Palestine, but then turned his back on Israel. After 1945 he became a leading light in "the intellectual cold war", even associating with Senator Joe McCarthy, to the disdain of liberal anti-communists. In the 1950s Koestler took up a liberal cause of his own, campaigning eloquently against capital punishment, while he pursued an array of dubious new intellectual enthusiasms.

As well as dozens of mistresses and far too many shorter liaisons to count, Koestler married three times, treating his wives wretchedly. The last was Cynthia Jefferies, who nursed him with pathetic devotion through Parkinson's disease and then leukaemia. In 1983, they were found dead together. He had chosen suicide deliberately; the suspicion remained that he had not discouraged Cynthia - healthy and 22 years younger - from joining him, in a form of suttee. It was very damaging to what was left of his personal reputation.

Until now, Koestler has been unlucky in his biographers. He acrimoniously intervened when Iain Hamilton was writing his inadequate life of 1982, and Koestler's Dictionary of National Biography essay is a characteristically rubbishy piece by Woodrow Wyatt, of all people, an example of the DNB's lamentable "pick a chum" tendency. To say that David Cesarani's book is an improvement would be poor praise. It is an absorbing biography and a stimulating exercise in intellectual history.

Cesarani, whose parents fell under the spell of communists, does not dismiss communism as a foolish aberration, but then neither did Koestler. He wrote that, "in the 1930s conversion to the communist faith was not a fashion or a craze - it was a sincere and spontaneous expression of an optimism born of despair". The God that Failed, the famous 1950 collection of palinodes by former communists, may have made him a hero to the American right, but it is his earlier writings which carry the mark of that sense of optimism and despair. They are thus classic texts for the 20th century's great Kulturkampf.

If Cesarani is even-handed about communism, he has his own axe to grind: this book might almost have been called "Koestler the Jew". Koestler was a Jew, and a very interesting one, exhibiting all sorts of Jewish complexes before, during and after his involvement with Jewish nationalism. "Jewry is a sick race," says the point of view character of his Zionist novel Thieves in the Night. "I became a socialist because I hated the poor; and I became a Hebrew because I hated the Yid."

Cesarani understands, as he revealingly puts it, that those words can be characterised as Jewish self-hatred "only in so far as the entire Zionist enterprise may be described as an act of 'Jewish self-hatred' ". But his own position only becomes clear when he writes of "the rationality of the Zionist idea" or when, in a formulaic phrase (to smile at which you don't have to be a member of the PLO), he describes Zionism as an "anti-colonial movement".

The buzz of publicity surrounding this book, however, isn't to do with Koestler the anti-communist or Koestler the Jew, but with Koestler the sexual athlete, serial rapist and misogynist. Koestler's personal life was certainly hair-raising: not just endless drunken evenings (often culminating in fights with Sartre or anyone else to hand) but repeated drunken car crashes; not just endless seductions but horrific treatment of all the women in his life, even when he didn't actually rape them. His philandering was plainly neurotic. A man driven to copulate with as many women as possible not only has difficulty establishing happy relations with women, or regarding them as equals, he doesn't actually like them.

And the story will make doubly sorry reading for feminists. Koestler believed in every sexist saloon-bar prejudice, and did his best to justify them empirically. Treat 'em mean to keep 'em keen - and they'll always come back for more. I'm afraid they did. Cesarani convincingly sees in Koestler's frantic sex life signs of self-hatred. Perhaps there is also a link here to "Koestler the Jew", with Koestler as a Philip Roth character avenging the age-old humiliation of his people by shagging every shiksa in sight.

So there is the artist who fascinates the public for what he was. What about what he did, what about the work? How good are Koestler's books? One answer is less and less good as the years went by. In the last 30 years of his life he turned ambitiously but embarrassingly to philosophy, Jewish history (with his cranky attempt to prove that the European Jews were descended from the Khazars), science and the paranormal.

His important books were all written before he was 40: Spanish Testament, The Gladiators, Darkness at Noon, Scum of the Earth and Arrival and Departure. As George Orwell observed in his 1944 essay on Koestler, still one of the best things ever written about him, there was "in England almost no literature of disillusionment about the Soviet Union" - which in turn was connected with the reality that no Englishman could have experienced the things Koestler did. Three of those five books are set in prison, and "none ever escapes for more than a few pages from the atmosphere of nightmare".

In The Gladiators, Koestler used Spartacus's revolt around 65BC to explore the search for the just city, the inevitable compromises of revolution, the conflict of ends and means, the question of whether and when it is justifiable to sacrifice lives for an abstract ideal. And in Darkness at Noon he addressed one of the darkest and deepest questions of the age: why had Bukharin and the other Old Bolsheviks confessed at the Moscow show trials to crimes they had not committed? In Orwell's gloss, they did so out of a kind of warped loyalty, and because their lives as devoted communists had left them with no further purpose: "Rubashov ultimately confesses because he cannot find in his own mind any reason for not doing so. Justice and objective truth have long ceased to have any meaning for him."

Although greater novels have been written this century, and although great novelists generally stick to Sam Goldwyn's principle, "Leave messages to the Western Union", the central lesson of Darkness at Noon is as important as any other in our time. It will survive the memory of its creator's unlovable personality and conduct. Arthur Koestler matters not because of what he did but what he did and told us.

This article first appeared in the 20 November 1998 issue of the New Statesman, A prejudice as American as apple pie