In the 1980s, when F A Hayek was one of the intellectual icons of the New Right, some of the more doctrinaire members of that complicated and fractious movement used to say that for him a minimal government was one that provided three things: national defence, law and order, and a state opera. It was an observation made only partly in jest. The Austrian-born economist and philosopher may have been the thinker who, more than anyone else, articulated the free-market ideology that came to power along with Margaret Thatcher; but his view of politics was formed not in Britain, his adopted country, but in the Habsburg empire, where the ­Vienna Court Opera was a department of government whose existence no one would dream of questioning.

Born in that city in 1899, Hayek came from an upper-middle-class background – his father was a medical doctor with a passion for botany who always wanted to be a professor, while his mother came from a wealthy land-owning family. The Hayeks enjoyed the prosperity of the closing decades of what the Austrian author Stefan Zweig described as “ the age of security”: the long period of stability provided by the 68-year reign of its last-but-one emperor, Franz Joseph. Hayek witnessed the collapse of an imperial regime that for generations had been more civilised and more liberal than most of the nation states that replaced it in interwar Europe. It was this Habsburg realm, as he experienced it in its final years, which shaped Hayek’s thinking about freedom and government.

My interest in Hayek, which began in the early 1970s, was as much to do with intellectual life in the Vienna of his youth as with the condition of British politics at the time. One of the first questions I asked after we had met through one of the right-wing think tanks that were proliferating around the end of that decade was whether he had known Karl Kraus, the incomparable Viennese satirist, who in 1909 had written, with some prescience: “Progress celebrates victories over nature. Progress makes purses out of human skin.” Hayek replied that he had not talked with Kraus, though he remembered seeing him crossing the road to enter a coffee house some time during the First World War. Hayek had little in common with Kraus. Cool and reserved, he had nothing of Kraus’s wit. Although he was academic in his manner, Hayek’s most striking intellectual trait was one that is uncommon in academic life – independence of mind, which enabled him to swim against some of the most powerful currents of the age.

I was also keen to learn something of Hayek’s connection with Wittgenstein, a relative of his about whom he had written a biographical fragment, “Remembering My Cousin, Ludwig Wittgenstein”, published in Encounter in 1977. Hayek met Wittgenstein by chance, on a railway station in August 1918, when they were both in the uniform of the Austro-Hungarian army. Travelling on together, they talked throughout the journey – a conversation Hayek told me had influenced him deeply, though not because of any philosophical exchange that he could remember. The two would never become close and their paths crossed only occasionally; but there seems to have been a meeting of minds between the two artillery ensigns on their way back to war. At the time both were ardent socialists who attributed the disaster that had befallen Europe to the malign impact of capitalism.

At the start of the 20th century, Vienna was one of the world’s great cosmopolitan cities. Though not without grievous bigotry – in 1897, after repeated attempts by the emperor to block the appointment, the city elected a virulently anti-Semitic mayor – the population was not divided, as much of central Europe soon would be, into violently hostile groups. The antique structures of the Habsburg state supported a society that was remarkably modern, not only in its embrace of technology (railways and trams, electric lighting and public sanitation) but also in enabling people with widely ­differing cultures to coexist and work productively with one another. The destruction of this order after the Great War by the forces of nationalism – which the US president Woodrow Wilson inflamed by insisting that Europe could be rebuilt only on the basis of popular self-determination – framed a dilemma with which Hayek struggled for the rest of his long life (he died in 1992).

How could liberal values be renewed in a time of political tribalism? It was a question Hayek could not answer. Instead, he came up with a mix of evolutionist pseudo-science and rationalistic designs for an ideal liberal regime. Having abandoned his youthful socialism under the influence of the doctrinaire market economist Ludwig von Mises (1881-1973), Hayek came to believe that a process of social evolution would impel humankind in the direction of the values he favoured. His legacy to liberal thinking has been a type of scientism – the mistaken attempt to apply the methods of the natural sciences when examining the human world. It’s an ironical outcome, given that he was a forceful critic of scientism in economics. In his speech on receiving the Nobel Prize in 1974, Hayek described the efforts of economists to mimic the methods of the natural sciences as having produced a “pretence of knowledge”.

***

One of the oddities of Hayek’s career is that while his professional standing was secured through his work as an economist, he had by the mid-1940s given up economics as his central intellectual activity. A major reason for Hayek’s shift into social philosophy was that he believed – correctly – that he had lost the debate with John Maynard Keynes about the causes of the Great Depression. There can be no doubt that his encounter with Keynes was the most important event in his intellectual life. Yet he had little insight into Keynes either as a thinker or a human being. He told me that during their acquaintance he never realised that Keynes had been homosexual – a surprising admission, as it was hardly something Keynes concealed within his circle of friends. The two men had quite different kinds of minds – Keynes’s swift and mobile, with an almost clairvoyant power of entering into the thinking of others; Hayek’s slowly probing, inwardly turned and self-enclosed. They were nonetheless on cordial terms.

Keynes found Hayek rooms in King’s College when the London School of Economics (where Hayek became a professor of economics in 1931) moved to Cambridge for the duration of the Second World War, and for a time the pair shared fire-watching duties on the roof of the college when it was feared that Cambridge might be bombed. With characteristic generosity, Keynes – while firmly rejecting its claim that government management of the economy is bound to lead to totalitarianism – heaped praise on Hayek’s anti-socialist tract The Road to Serfdom when it appeared in 1944.

The differences between the two thinkers were as much in their underlying philosophies as in their economic theories. Both were sharply aware of the limits of human knowledge. But whereas Hayek invoked these limits to argue for non-intervention in the economy, Keynes recognised that bold action by governments is sometimes the only way in which the economy can be lifted out of depression – as when Roosevelt (to whom Keynes had written an open letter in 1933) successfully adopted some aspects of Keynesian thinking in the New Deal.

Hayek was most original when he ­argued that the market is a means of discovering and transmitting information that is dispersed throughout society. It was this insight into the knowledge-creating function of markets that enabled him to formulate a decisive argument against central economic planning.

Generations of socialists have maintained that the failings of the Soviet economy were because of historical causes extraneous to the planning system: a lack of democracy rooted in tsarist traditions of despotism, the underdevelopment of the Russian economy when the Soviet system came into being, and Stalin’s deformation of Lenin’s supposedly more benign inheritance.

As Hayek perceived, none of these factors can account for the universal failings of planned economies, which have followed a similar pattern in countries as different as  Czechoslovakia and Mongolia, East Germany and Cuba. The fundamental reason for the failures of central economic planning is that economic knowledge cannot be centralised. More than the love of power or the inevitability of corruption, it is the limitations of human knowledge that make socialist planning an impossible dream. Here Hayek’s argument was unanswerable.

The trouble is that it also applies to unfettered market capitalism. No doubt markets transmit information in the way that Hayek claimed. But what reason is there to believe that – unlike any other social institution – they have a built-in capacity to correct their mistakes? History hardly supports the supposition. Moods of irrational exuberance and panic can, and often do, swamp the price-discovery functions of markets.

When considering how to overcome the Great Depression, Hayek opposed Keynes-style fiscal stimulus for the same reason he opposed monetary expansion of the sort later advocated by his friend the American economist Milton Friedman (1912-2006). In attempting to generate recovery by macroeconomic engineering, both monetarism and Keynesianism required a knowledge of the economy that no one could possess. Unlike monetarism – with which it has sometimes been confused – the Austrian school of economics that Hayek promoted insists that the quantity of money cannot be measured precisely, and that expanding the money supply cannot reflate the economy in a sustainable way.

For Hayek, the causes of the Depression lay in earlier central bank policies of cheap money, which resulted in large-scale misallocation of capital. Because no central authority could grasp the shifting pattern of relative scarcities and prices, only the market could determine the right allocation. Accordingly, believing that misguided investments had to be liquidated, Hayek argued in the 1930s for policies that were more contractionary than those that were actually pursued. The task of government was to get out of the way and let the process of adjustment run its course.

If they had been adopted while the crash was under way, Hayek’s prescriptions would have made the Depression even worse than it proved to be – a fact he later admitted. But he never accepted Keynes’s core insight that large-scale economic discoordination could be the result of the workings of the market itself. For him it was always government intervention that accounted for market disequilibrium. More sceptical as well as more radical in his turn of mind, Keynes questioned the self-regulating powers of the market. His work on the theory of probability disclosed insuperable gaps in our knowledge of the future; all investment was a gamble, and markets could not be relied on to allocate capital rightly. There were booms and busts long before the emergence of modern central banking. Left to its own devices, the free market can easily end up in a dead end like that of the 1930s.

***

Keynes’s own experience told against Hayek’s theories. As one of the 20th century’s most successful speculative investors, playing the markets on behalf of his college from a phone at his bedside before he got up for the day, he understood – in a way that the inveterately professorial Hayek did not – the ineradicable uncertainty of economic life. As a member of the British delegation at the Paris Peace Conference in 1919, Keynes had been horrified at the punitive conditions imposed by the Allies, which he forecast would destroy the German economy and lead to an upheaval that would “submerge civilisation itself”. Keynes had an acute sense of the risks posed to social stability by misguided economic policies. In contrast, Hayek consistently ignored these hazards.

Hayek’s blind spot with regard to politics was clear in the early 1980s when the first Thatcher government, in an attempt to reduce inflation and bring the public finances closer to a balanced budget, was raising interest rates and cutting public spending. As he had done during the 1930s, Hayek attacked these policies as not being severe enough. It would be better, he told me in a conversation we had around this time, if Thatcher imposed a more drastic contraction on the economy so that the wage-setting power of the trade unions could be broken. He appeared unfazed by unemployment, which was already higher (more than three million people) than at any time since the 1930s, and would rise much further if his recommendations were accepted.

Fortunately Hayek never had any influence on Thatcher’s policies. (Her chief economic adviser in these years was Alan Walters, a Friedman-style monetarist.) Equally, and perhaps also happily, Thatcher had no understanding of Hayek’s ideas. If it was true that she carried about with her for a time a copy of Hayek’s magnum opus, The Constitution of Liberty (1960), she cannot have read its postscript, “Why I am not a Conservative”, in which Hayek explains that he rejects conservatism because it lacks a vision of human progress. A case can be made that Thatcher was no conservative, either – at least if being conservative includes an aversion to policies that impose deep changes on inherited social institutions. But this is a view that goes only so far. Unlike Hayek, Thatcher understood and accepted the political limits of market economics.

Though he witnessed at first hand the collapse of liberal civilisation in interwar Europe, Hayek had little sense of the fragility of freedom. He observed how the Habsburg regime was destroyed, first by war and economic ruin and then by nationalism, but his response was to look for what he called in his book Individualism and Economic Order (1949) “a permanent legal framework”, which could serve as a guarantor of liberty in the economy and society. Here Hayek disregarded the principal lesson of the interwar years, which is that a liberal regime cannot be secured by legal diktat.

Geopolitical conflict and war, economic upheavals and new social movements have repeatedly damaged or destroyed liberal regimes. No ideal constitution can overcome the permanent threats to liberal values.

Yet throughout his writings Hayek invoked the mirage of a legal order in which vital freedoms are protected by being insulated from the political process. Something like this protection was provided by the Austro-Hungarian empire during the reign of the emperor Franz Joseph, and it is almost as if Hayek were trying to reconstitute the Habsburg realm in a new form that would last for ever. He was always sympathetic to the attempt to build a European federal union – a fact that only confirms his blindness to political realities.

***

Hayek’s attempt to fashion a regime in which the freedoms he cherished would be invulnerable to political challenge led him to some curious proposals. In The Political Order of a Free People (1979), the third volume of his last major work, Law, Legislation and Liberty, he outlined a scheme for a bicameral legislature in which the upper chamber is composed only of people elected at the age of 45 for a 15-year term by an electorate also consisting only of 45-year-olds. When they reached 60, members of the upper house would be retired and given a lifelong sinecure.

Hayek liked to ridicule the idea that institutions could be designed on the basis of abstract models – a view he criticised as embodying a philosophy of “constructivist rationalism”. Yet his scheme for an ultra-liberal constitution was a prototypical version of the philosophy he had attacked.

It may have been a half-conscious awareness of the limitations of this rationalistic philosophy that fuelled his evolutionary speculations. Underpinning his defence of the free market was a belief in what he called “spontaneous order in society” – the idea that, if only human beings were not subject to oppressive governments, they would evolve in ways that allowed them to live together in peace and freedom. This was not a view held by Hayek’s friend and LSE colleague Karl Popper, who gently demolished it when I talked with him, or by the conservative philosopher Michael Oakeshott, also a colleague at the LSE, who dismissed it – accurately – as “rubbish”. A type of unplanned order may well emerge in society but there is no reason why it should respect liberal values. There is nothing particularly liberal about the Mafia.

The fallacy that a process of social evo­lution is at work that will promote the spread of some version of liberal values goes back a long way. Propagated by Herbert Spencer, the prophet of laissez-faire who first coined the expression “survival of the fittest”, it was widespread in the late 19th century. There are many similarities between Hayek’s and Spencer’s theories, not least the idea that capitalism will prevail over other economic systems because it is more productive and can support a larger human population. Hayek assured me that he had never read Spencer, and I’m sure this was the case. Very similar ideas had been popular in fin-de-siècle Vienna. Hayek was doing no more than reviving a recurrent modern delusion – the belief that history obeys evolutionary laws, which somehow underpin a process of progressive social development.

The spread of capitalism over the past decades is a result of human decisions, not the workings of some imagined evolutionary process. Communism collapsed in the former USSR not because it was less productive than capitalism (though this was certainly the case) but because the Soviet state became embroiled in an Afghan war it could not win, while losing control of parts of eastern Europe and the Baltic states. Another important factor was the unintended impact of Mikhail Gorbachev’s reform policies, which, rather than strengthening the regime as he intended, exposed how little popular legitimacy it possessed.

A variety of capitalism came to China through the policies of Deng Xiaoping, who pulled down the curtain on the Maoist era. None of these developments resulted from the operation of evolutionary laws, and we are now seeing a reassertion of state power in both Russia and China.

Hayek’s belief that vital freedoms can be enshrined in law and thereby taken out of politics is ultimately delusive. But it is not an aberration peculiar to the brand of right-wing liberalism that he professed. An anti-political liberalism is the ruling illusion of the current generation of progressive thinkers. Philosophers such as John Rawls and Ronald Dworkin had views of justice very different from Hayek’s. Whereas Hayek rejected any redistribution of income beyond that required by a minimum level of subsistence, Rawls and Dworkin demanded different versions of egalitarianism. What all these thinkers had in common was the idea that reasonable people will converge on a shared conception of what justice requires. In this view, politics isn’t a rough-and-tumble in which rival interests and ideals contend with one another unceasingly, but a collective process of deliberation that leads to a common set of values. Some such vision seems to have possessed Ed Miliband, until he discovered that his ideal of equality was not widely held and the parliamentary road to predistribution was closed.

***

Hayek may still have lessons to teach us. The policies he recommended during the Great Depression may have been badly flawed but his insight that prosperity cannot be restored by unending expansion of debt may have some value at a time when the limits of “Keynesian” quantitative easing are becoming clear. It is in any case far from obvious that Keynes would have supported a continuation of QE once a disastrous collapse had been averted. “Keynesianism” is a confection of Keynes’s more mechanical disciples, not an indication of how this mercurially brilliant mind would have responded to our present dilemmas. Again, Hayek’s claim that nothing can be done to mitigate the impact of free markets on social cohesion was dangerously misguided. But he was right to point out that capitalism cannot be remodelled to fit some conception of an ideally fair distribution of resources. Whether any kind of social democracy can be reconciled with the anarchic energies of global markets is an open question.

Hayek may have shown the unreality of left-liberal visions of egalitarian capitalism, but it was Keynes who understood fully the vanity of liberal rationalism. In “My Early Beliefs” (1938), a talk later published as a memoir, Keynes mocked the philosophy held by himself and his friends before the First World War: “We were not aware that civilisation was a thin and precarious crust . . . only maintained by rules and conventions skilfully put across and guilefully preserved.”

Hayek watched the interwar collapse with horror, as Keynes did, and shared many of Keynes’s liberal values. What he failed to understand is that these values cannot be renewed by applying any formula or doctrine, or by trying to construct an ideal liberal regime in which freedom is insulated from the contingencies of politics.

John Gray’s latest book is “The Soul of the Marionette: a Short Inquiry into Human Freedom” (Allen Lane)