Special report 3 - The city that forever resists the rational

Since it burst its stone banks in medieval times, London has defied central planners. Only Margaret

I wonder what Lewis Mumford, the great doyen of urban studies, would have made of the Oscar Niemeyer pavilion currently squatting in front of the Serpentine Gallery in London's Hyde Park. Every year, the gallery - which has plotted a reputation for itself at the very point on the grid where chic contemporary art, evanescent celebrity and the London beau monde all intersect - asks a prominent architect to design a teahouse. This little semi-folly graces the parched grass for a few months: a porte-cochere to the inner sanctum of the gallery. There is always considerable interest in the pavilion. I was there last week and no fewer than three stills photographers were hard at work, together with a brace of camera crews. But why?

Niemeyer, a nonagenarian Brazilian, best known for his contributions to that monumental nationalist folly, the capital city of Brasilia, isn't much more than an emulator of Le Corbusier, an architect of whom Mumford wrote in the late 1950s: "[His] imagination is deeply in harmony with the negative tendencies at work in contemporary society, [he] has for a whole generation been the single most powerful influence over architecture and city planning in every part of the world." The Niemeyer pavilion is so small as to evade opprobrium - after all, this is a pavilion, not an office block or a militaristic citadel - but how are we to read its enthusiastic embrace by London's wannabe aesthetic intelligentsia?

Can it be that they are disposed to regard this dinky concrete awning as in some sense a model of modernism? That, by placing it on their lawn, the Serpentine people are acknowledging a fact already enacted on a big scale by Tate Modern: namely, that modernism is now securely of the past? Or is the pavilion a still more ironic comment, one understood intuitively by its local contemplators? Because, of all the world cities that came of age in the first half of the 20th century, London is arguably the least brutalised by Le Corbusier and his bully boys. I'm not claiming that London is unaffected by modernist architecture and urban planning; rather that London is so immemorially messy, that despite their best - or worst - efforts, modernist architects have proved unequal to the task of significantly altering its fabric.

However, I doubt all of the above conjectures. The sad truth is that the Serpentine people, together with the febrile fashionistas who flock to their fundraising parties, actually believe the Niemeyer pavilion is cool. Noisily ignorant of their own history, these epigones sum up the argument that the World Spirit of Urbanity has moved on. London had the Hip Olympics in the 1960s and - arguably - the 1990s (as New York had it in the 1970s, Paris in the 1920s and Vienna in the 1890s). Now this sacred celebration of the evolutionary role of cities has upped the stakes.

Mumford, again writing of the depredations of 20th-century urban planners, said that "[they] attempt to cover over their initial mistakes by repeating them on a wider scale". But in the case of London, you'd have to say that this process has been undertaken by that unhappy antheap, the city itself. The English psychogeographers Iain Sinclair and Peter Ackroyd have it right when they ascribe to the city its own persona, for the way London has resisted the rational over the centuries, while continuing to transcend its own very raison d'etre, is suggestive of a refractory genius. Ever since the early medieval city burst its stone banks and ran all over the place, London has displayed all the orderliness of an inundation. Wren, Nash, Unwin - planners have come and gone. Only Margaret Thatcher seemed to understand the anarcho-dynamic of London, and hailed it by abolishing city-wide institutions of planning and administration. Continuously elaborating on its own waywardness, the city has spread and spread - and now even that complacent girdle, the green belt, has been broken.

The past 20 years saw a steady stretching of its definition, but even if "the green belt" had held, London's field-eating disorder had already leapfrogged beyond it. The garden cities, intended to relieve the pressure of population, contained within themselves the seeds of new urban growth. Now, each Stevenage or Milton Keynes stands at the hub of a ribbon development, or forms part of an edge city, as London continues to cover its initial mistakes. Even were nimbyism to acquire the muscle of national socialism, it would not be able to stop the stealthy formation of an uber-M25 240 miles in circumference.

For Lewis Mumford, the prospect of contemporary London would be quite horrifying. Schooled in the progressivist sociology of Patrick Geddes, he viewed the urban environment as subject to its own evolutionary tendencies, from village to metropolis, to conurbation, to megalopolis and finally . . . yes, you guessed . . . to necropolis. "That cycle," Mumford wrote, "has described the course of all the historic metropolises, including those that arose again out of their own ruins and graveyards." The kind of therapy he and his mentor would have recommended for this hypostasising city is far too late; indeed, as long ago as the interwar period, the garden city movement seemed to recognise that the patient was finished and it might be better to save the tumour.

With its crazy patchwork of development, its wilfully undercapitalised and inefficient transport system, and its astronomical cost of living, London has reverted to type. Its growth was originally predicated on the stretched teat of the Strand, running between the cash cow of the City and the King Baby at Westminster. Over more than a millennium, the city tried on the guise of a trading entrepot, then the hat of manufactory, until in the latter half of the 20th century it shed all pretence of being anything other than the cult centre of capitalism itself.

Now, in 2003, 33 per cent of the workforce labour in the plastic mines of finance and business, tapping their way through a ritual seam; while a further 25 per cent are there solely to service their sleep and feeding. Never strictly a conurbation in the sense that Geddes coined the term (namely an undifferentiated urban wasteland growing in the wake of coal and steel), London has also managed to defeat his expectation of doom: by all rights the city should be dead, but it won't lie down.

Why should this be? Mumford was writing under the shadow of what he undoubtedly expected to be a nuclear Armageddon. Singling out the Pentagon in Washington for particular derision, he wrote: "Nuclear power has aggravated this error and turned its huge comic ineptitude into a tragic threat." Certainly, he would have had little hesitation in identifying the repeated and strident warnings of spooks and politicos that London is about to be irradiated by a "dirty bomb" as the latest manifestation of "the [Bronze Age] citadel . . . come back to life once more, with every ancient dimension magnified, every error raised to the tenth power". And, ceteris paribus, station some tanks at Heathrow: that'll remind them that this isn't a series of boulevards for strolling flaneurs, but the very reification of pyramidical power.

However, lurking behind the pessimism of Mumford and Geddes is a far deeper strain of Promethean optimism. The very dialectic they sought to impose on urban growth analogises the Hegelian concept of the World Spirit alighting within successive cultures. It is this buried supposition that animates the private Niemeyer viewers at the Serpentine just as much as the suits on the Olympic bid committee, or even Ken Livingstone's transport planners. All of them in their different ways seek to recapture the genius of a Rome that they feel has departed.

The Niemeyer worshippers rightly realise that modernism has never properly alighted in London, let alone touched the city with its prestressed fire. The Olympic committee and its backers are the promoters of the eternal flame of urban renewal: this time the infrastructure, the economic upturn and the popular feeling will coalesce so that London will swing yet again. As for Ken Livingstone and his aborted foetus of an assembly, it is worth considering exactly why the introduction of the most modest of traffic-control levies should have aroused as much popular hysteria as it did. Surely it's because the congestion-zone cameras were at least an allusion to the very rational planning that London has always succeeded in shucking off its great whale's back of masonry, brick and tar? Faced with the possibility of genuine and encompassing civic renewal, the cockneys resorted en masse to the most bizarre conspiracy theories: Ken had phased all the traffic lights and/or roadworks so as to paralyse the city and justify the charge. Predictably, in the wake of its introduction, traffic is imperceptibly grinding down to its traditional, halting anomie.

I hope I can be forgiven for founding this essay - the conclusion of which could be summarised as London is dead, long live London - on the physical substance of the city. I leave it to others to gas on about this artist, that musician, or those writers. I could also have hypothesised as to why the necropolis continues to bestride the earth like a zombie city. At least part of the explanation must lie in the black economic underbelly of London, and with its immigrants, illegal or otherwise. By 2007, a third of the population of inner London will belong to ethnic minorities. In the past, the Moloch depended on fresh babies from the English provinces; now it savours more international fare. This development, the city as a global entrepot for human cargo, was anticipated by neither Geddes nor Mumford. But in the last analysis, only the heirs of a particular strain in Enlightenment thought - progressive, Promethean, dialectical - can be bothered to anticipate whither the city goes; for those who live here, it's simply enough that it is at all.