The logic of Marxism demanded that America, the most advanced capitalist country, should also be the country that produced the strongest socialist movement. The British Marxist H M Hyndman wrote in 1904 that "just as North America is today the most advanced country economically and socially, so it will be the first in which socialism will find open and legal expression". The leader of the American Socialist Party, Daniel De Leon, proclaimed in 1906: "If my reading of history is correct, the prophecy of Marx will be fulfilled and America will ring the downfall of capitalism the world over."
It didn't happen. The US Socialist Party has never received more than 6 per cent of the vote in presidential elections, and has never won more than a couple of seats in Congress. The US has been the only western democracy dominated exclusively by parties sympathetic to liberal capitalism.
The reasons for this can be understood by careful historical comparison of the United States with other western societies, and by examining variations in socialist support within the United States. The distinctive elements of American culture - anti-statism and individualism - negated the appeal of socialism for the mass of American workers. The electoral system for selecting a president, which aggregates votes throughout the country, magnified the penalties of voting for a third party, while the permeability and ideological flexibility of the established parties provided opportunities for organised labour to exercise influence without establishing an independent party. Further, the American working class, from its inception in the post-civil-war decades, was exceptionally diverse, ethnically, racially and in terms of religion. These cleavages were more powerful sources of political identity for most of the American lower classes than was their commonality as workers. Finally, American socialists failed to build an alliance with trade unions. Relations between socialists and mainstream unionists were generally hostile from the mid-1890s. Efforts to create a labour party came to nothing.
But do socialist or social democratic parties really make a dif- ference? What were the consequences of the failure of socialism in America? And if other countries, such as the UK, move closer to American-style politics, can we expect similar consequences in their societies?
The American left is no longer so exceptional. Over the past two decades, across western democracies, parties that were established as socialist, social democratic or labour have gradually dropped the statist elements of their programmes. Some have distanced themselves from labour unions. None of the major socialist parties advocates more public ownership; most accept market principles even in areas of the economy that were formerly nationalised, such as transport, telecommunications and utilities. The great breach between progressive socialisation of the economy and laissez-faire has narrowed into a debate between regulated capitalism and neo-liberalism. At issue is the character and degree of regulation of the economy, not the future of capitalism.
Just how general this shift has been is evident from expert studies of social democratic parties in the European Union. Every such party (except Pasok in Greece) moved from left to right between 1984 and 1995 (as evidenced by growing support for market liberalism), and the UK Labour Party moved most of all. From being the most left-wing social democratic party in Europe in 1984, Labour became one of the most right-wing in 1995, with only Italy's PSI, Austria's PSO and Greece's Pasok further to the right. Put another way, the most left-wing European social democratic party in 1995 (Germany's SPD) was further to the right than 12 out of 14 such parties in 1984.
In Britain, the Labour government has built on, rather than reversed, previous Conservative government policies on markets, unions and welfare. Likewise, the Labour parties of Australia and New Zealand, when they held national office during the 1980s, abandoned protectionist policies, deregulated the economy, privatised state enterprises, and moved from centralised wage-fixing through arbitration to a market system. Indeed, in both countries, the shift to neoliberal market principles and the dismantling of state controls took place under labour, rather than conservative, governments. In New Zealand, the Labour government followed what has been described as the most Thatcherite policy among western governments, including Britain's. The prime minister David Lange believed that economic equality conflicted with economic growth, and that the latter should have priority: "Social democrats must accept the existence of economic inequality because it is the engine which drives the economy."
To the extent that these labour parties have tended to be radical, it has been on non-financial issues. The British Labour Party has created legislatures in Scotland and Wales and is reforming the House of Lords. The Australian Labor Party favours cutting ties to the British monarchy. Both the Antipodean parties have given priority to environmental protection and women's issues, tried to confront the legacy of past oppression of indigenous minorities and, particularly in New Zealand, opposed nuclear power and weaponry. Such policies try to retain support among public- sector professionals and the left intelligentsia; they place the parties in a tradition of liberal radicalism or populism that owes little to socialism.
It is no longer possible to say that the US is the only western society without a socialist party, because such parties no longer exist in most western societies. American political exceptionalism has run its course.
Yet the unique failure to create a viable socialist or labour party still casts its shadow on American society. The table below (click here to view the table) gives, in its first three columns, an overview of the relative economic and political strength of the lower class in 17 western European and English-speaking democracies. The US is at the low extreme for lower-class political power, together with Canada. In the US, social democracy is simply absent. In Canada, the social democratic NDP has held power in several provinces, but not at the national level. In terms of union organisation, the US is again at the low extreme, this time alongside France. In neither country were unions united behind a labour or socialist party. (In France, they were divided into different political camps, but the vast majority of workers in France are covered by collectively bargained wage contracts because agreements for unionised workers are extended to unorganised workers by law.)
So when one compares the US with other western democracies, the picture that emerges is one of continued lower-class weakness. No other western democracy remotely approximates America in this regard.
Now consider the columns to the right. The fourth column underpins the conventional wisdom that the US is remarkable for its low level of taxation and government spending. In 1996, it was the only western country in which government extracted less than 30 per cent of GDP. One has to go outside the western world to find societies with a smaller state.
Again, spending on social welfare as a proportion of GDP (column five) is low in the US. It is the only developed nation that does not have a government-supported, comprehensive medical system and it is the only western democracy that does not provide child support to all families.
Further, no western democracy has as unequal a distribution of wealth as has the US once tax and transfer payments are included in the calculation. The standard scale for measuring inequality, the Gini coefficient (in which 0.0 would represent total equality where everybody received the same income), is almost 10 per cent higher in the US than in the next most inegalitarian country, the UK (see column six).
In both these countries, there was a sustained rise in economic inequality in the 1980s and 1990s; in the UK, the Gini coefficient went from 27.0 in 1979 to 34.0 in 1991 (an unparalleled increase for a single decade in both absolute and relative terms) to 34.6 in 1995.
Data on relative poverty tell a similar story (see column seven). The US stands out as the society with the greatest income differentials: 11.7 per cent of the population has an income less than 40 per cent of the median income, a figure that is almost double that of Australia, the next most unequal country. It may be objected that the US is a rich country: someone on an income of less than 40 per cent of the US median could be relatively well-off by the standards of other western countries. But the last column of the table shows that, even by US standards, the US has a higher proportion of its population living in poverty than any other country except Ireland.
Although there is evidence that US poverty has fallen in recent years, the conclusion from international comparisons is unambiguous. The US combines an extremely high standard of living with exceptionally low levels of taxation and social spending and equally exceptional levels of income inequality and poverty.
Is there a causal link between these distinctive characteristics of American society and the inability of socialists to establish a viable social democratic party in the US? There exists a methodo-logically sophisticated literature concerned with public policy outcomes in western Europe and English-speaking democracies that helps us to answer precisely this question. It concludes that societies in which social democratic parties have consistently played a role in national government and in which unions are strongly organised tend to have extensive welfare systems and greater economic equality.
Social democratic governance over the period 1945-94 bears little relation to comparative rates of economic growth - either posi-tively or negatively. But it is strongly associated with indicators of total taxes, social security transfer expenditure, equality and relative poverty. When we examine the combined effect of social democratic governance and trade union membership on these variables, the associations are yet stronger. They remain strong even when other variables - such as a country's political institutions, its overall income level, its vulnerability to international economic pressures or its economic structure - are taken into account.
Prior to the war, social spending was no weaker in the US than in the most advanced European countries. In 1938, the Roosevelt administration spent 6.3 per cent of GDP on its social policy programmes (employment assurance and public employment) - a greater proportion than that in Sweden (3.2 per cent), France (3.5 per cent), the UK (5.0 per cent) and Germany (5.6 per cent). But American commitment to social policy evaporated during and after the Second World War, at the very time that social democratic parties pushed ahead with ambitious state and welfare policies in Europe.
The absence of social democracy in the US has not only reduced state spending as a whole, but has tilted public policy towards strongly represented groups, such as the upper and middle classes, business and unions, farmers and the elderly, and away from weakly represented groups, including non-unionists, single mothers, young people and the poor. Government spending in the US on education (5.3 per cent of GDP) and on pensions for the elderly (7.2 per cent) is not much below the means for all OECD countries, whereas spending on other social programmes (1.2 per cent), most of which goes towards less privileged groups, is less than one-quarter of the OECD mean.
From the 1980s, however, the effect of social democratic governments on welfare policy markedly declined. Over the past two decades, the only broad budgetary component on which social democratic parties have made a palpable difference is the kind of expenditure that includes daycare for children and parental leave. The difference is particularly significant in Scandinavia, where social democrats have taken the lead on a variety of women's issues and have sought to bring women into the labour force. But in other areas of social policy, including transfer payments, health and public pensions, all governments, irrespective of their ideological stripe, have tried to cut back to balance their budgets. Intense fiscal pressures arising from international financial markets, unusually high levels of unemployment and, in Europe, the efforts of governments to meet the Maastricht criteria for monetary union have constrained government spending no matter what the goals of the party in power.
Yet the effects of social democratic governments before the 1980s live on. Once a government policy is in place it is likely to be defended by those who benefit from it. A policy legacy may also shape expectations about what a government is able to do. Even if a new administration wishes to abolish a policy and has the support of a large majority of the public, its efforts may be torpedoed by those who mobilise to defend the status quo. Social democrats, echoing conservatives, may say that taxation is too high and that government spending should be reined back. But government spending has steadily increased throughout the 20th century, and while the rate of increase slackened during the 1990s, it was not reversed. In 1913, government spending in western capitalist societies (and Japan) averaged just 8.3 per cent of GDP. By 1980, it had soared to 43.3 per cent. In 1990, after a decade of intense effort to cut government budget deficits, it had increased to 46.1 per cent, and by 1996 it had inched up to 47.1 per cent.
So, as social democratic parties the world over shift away from their traditional moorings toward the free market, should we expect the gap between the US and other western democracies to narrow? Or will institutional legacies and values continue to shape a country's response to external events?
There are signs that the influence of social democracy as a distinct approach to policy is not exhausted. The seemingly universal shift to support for the free market may be of short duration. Even its advocates, including Joseph Schumpeter from the 1930s and Irving Kristol from the 1970s, have noted that capitalism does not advance the same pretensions to solve major human problems that socialism and communism once did. Capitalism, the free market, is not a utopia even when limited to economic considerations. At best, it holds out the promise of a lottery, but like all such awards, the jackpots go to a relatively small minority of players.
At the centre of free-market ideology is an emphasis on self-interest - in invidious terms, on greed. The argument has been put, from Adam Smith to Milton Friedman, that the uninhibited pursuit of gain results in a growing economy which benefits all, regardless of status or wealth. But as we know, not only do some individuals fail to benefit, but also countries differ enormously in economic performance. And the business cycle, which seems inherent in market economies, not only fosters growth, it implies downswings - periods of economic recession and increased unemployment - as well.
Moreover, capitalism - which, unlike socialism, does not promise to eliminate poverty, racism, sexism or war - appeals only weakly to the idealism of many young people and intellectuals. Nor does it address environmental concerns; classic free-market liberals resist policies that interfere with the market.
The US once more stands out politically among western democracies in that it lacks even a minimally effective green party. Green parties are represented in national parliaments and/or the European Parliament in every one of the 13 richer countries that are members of the European Union. In 1999, they participated in government coalitions in Belgium, Finland, France, Germany and Italy.
The struggle between the left, the advocates of change, and the right, the defenders of the status quo, is not over. In the once communist-dominated countries, the terms "left" and "liberal" have been used to describe free-market and democratic tendencies that seek to reduce the power of state bureaucracies; the terms "right" and "conservative" usually refer to groups that defend state controls. Ironically, this is the way these concepts were first used during the 19th century. In the west, following the rise of socialist movements, "left" came to mean greater emphasis on communitarianism and equality, on the state as an instrument of reform. The right, linked to defensive establishments, has, particularly since the Second World War, been identified with opposition to government intervention. The rise of green parties in western Europe is merely one indication that the contest between these two orientations has not ended.
S M Lipset is Hazel Professor of Public Policy at George Mason University; Gary Marks is professor of political science at the University of North Carolina. This essay is adapted from their book It Didn't Happen Here: why socialism failed in the United States, published by Norton on 5 July (£19.95)