The problem faced by those who want to build a new future for humanity is that they have to start in the flawed world that exists at present. Revolutionaries cannot help being compromised by the power structures they aim to overthrow. If they are to pose a challenge to the prevailing order, they need to protect themselves against repression and subversion by the state. When they organise to defend themselves, they soon come to resemble the state in secrecy and ruthlessness. The revolutionaries' dilemma is clear: either they remain high-mindedly pure and impotent, or they end up as repressive as the regime they are fighting, if not more so.
Nothing illustrates this quandary more clearly than the anarchist movement in the late 19th and early 20th century. "Anarchism's ultimate aim was to usher in a society of human beings, a heaven on earth in which harmonious coexistence was achieved without coercion or the imposition of distant authority, but rather arose out of each individual's enlightened recognition of their mutual respect and dependency." This is the anarchist ideal, as described by Alex Butterworth in one of the most absorbing depictions of the dark underside of radical politics in many years.
The early anarchists looked forward to a stateless, egalitarian society, including all of humankind, in which no one ruled over anyone else. They believed this was what human beings really wanted - a condition of harmony and equality that would exist everywhere, had it not been obstructed by exploitative and manipulative elites. But how to deal with such obstructions, which have in the past so often thwarted humanity's best impulses?
This is where Butterworth comes in. He begins his story at a meeting in the Paris apartment of Boris Savinkov, poet and novelist, anti-tsarist terrorist and later minister in Alexander Kerensky's provisional government, in October 1908. The meeting, which included the celebrated anarchist Peter Kropotkin, had been convened to investigate allegations made by Vladimir Burtsev, head of counter-intelligence for the Socialist Revolutionary Party of Russia (often called the SRs, or Esers), that the party's most revered hero, Evno Azef, was an agent of the Okhrana, the tsarist secret police.
A feature of the inquiry, which went on for several weeks, was that the head of security was himself under suspicion for spreading what many believed were unfounded accusations. Under these circumstances, how could anyone be sure of anyone? The situation had more than a little in common with the one depicted in G K Chesterton's beguiling metaphysical thriller The Man Who Was Thursday, first published in 1908, which features an Anarchist Council whose members are named after days of the week, six of whom turn out to be agents of Scotland Yard.
After long confabulations, Burtsev managed to persuade his fellow revolutionaries that Azef was working as a spy (as had been the case, in fact, for many years). It was decided that Azef would be executed by hanging in a cave just across the Italian border, which could be reached from a secluded villa that had been equipped with a tunnel. Having refused to attend the meeting, Azef escaped when he learned of its verdict, hiding in a monastery for a time. In a postscript to the affair that Butterworth does not discuss, the informer ended up in Germany, where he pursued a career as a stock-market speculator and died a natural death in 1918.
Being themselves conspirators, revolutionaries are bound to view events in conspiratorial terms. But their perceptions often have a basis in reality; conspiracy is part of the territory in which they live. As Butterworth recounts, Burtsev was for several years the target of a honey-trap operation run by the head of tsarist intelligence in Paris and later chief of police for the Russian empire, Pyotr Rachkovsky. Using as bait a rich young widow with whom Burtsev was carrying on an affair, Rachkovsky attempted to induce his intelligence counterpart to leave London for trysts in continental Europe, where the Okhrana could kidnap him more easily.
The operation was bungled when Burtsev's lover locked him in his cabin on the ship in which he had travelled to meet her, only to discover that she had no way of transporting him back to Russia. The "Sherlock Holmes of the Revolution" carried on his campaign against infiltration of the revolutionary movement for many years. Later, after being arrested and imprisoned in Russia on Trotsky's orders, he worked for a time for British intelligence. He died in poverty in Paris in 1942.
A pivotal figure in what Butterworth calls "the first international 'war on terror'", Rachkovsky was a master of deception who inflated the terrorist threat by orchestrating several provocative bombings, while playing a still obscure part in the production of The Protocols of the Elders of Zion, the anti-Semitic forgery that became a mainstay of Nazi propaganda. By controlling some of the major figures in the anarchist movement, the tsarist spymaster was able to ensure that a handful of fantasists was seen as a vast conspiracy.
The anarchists did pose a threat - Azef was implicated in the assassination of a number of public officials, for example. "A genuine conspiracy of sorts" did exist, as Butterworth says. But it never had a chance. When they imagined they were mounting a challenge to state power, the anarchists were wandering in a wilderness of mirrors. Except for a few years in Ukraine during the Russian civil war and later in Spain, where anarchism was destroyed by Stalinism, it never became a serious political force. Its role in history was to provide an invisible, all-pervasive enemy against which the forces of the status quo could be solidified and strengthened.
Butterworth has opted to present the anarchists in a mode that emphasises narrative over analysis. The result is a riveting account, teeming with intrigue and adventure and packed with the most astonishing characters. One cannot help wishing there were more extended analysis, however, for when Butterworth does offer broader observations, they are exceptionally astute. Noting how the anarchists "inherited the attributes of radical religion", including a highly developed martyrology, he comments that they viewed themselves as "oppressed heroes in a Manichaean struggle for progress". For all its enmity to religion, anarchism was a faith, rather than the science-based world-view that it pretended to be.
A feature of the anarchist faith was the conviction that only the machinations of sinister forces stand in the way of an earthly paradise. The forces were real enough, but the anarchists never asked themselves how it was that the mass of humankind could be so easily misled. Could it be that most human beings do not want to live in a world without rulers? Might it even be that they prefer the flawed actuality? For those - and there are many - who still think like Butterworth's dreamers and schemers, these are forbidden questions, leading into a wilderness more frighteningly impenetrable than any hall of mirrors built by spymasters.
John Gray is the New Statesman's lead book reviewer. His latest book, "Gray's Anatomy: Selected Writings", is published in paperback this month (Penguin, £10.99)
The World That Never Was: a True Story of Dreamers, Schemers, Anarchists and Secret Agents
Bodley Head, 384pp, £25