When will those trains run on time?

In the 1970s, Anthony Crosland wanted "a smaller, sensible little railway". Such poverty of ambition

Ever since the mid-1970s, when I was the Financial Times's transport correspondent, a question has troubled me: why have Britain's politicians, almost alone in western Europe, failed to create the conditions in which railways can thrive? Now, as the rail system lurches from crisis to crisis, and as the Commission for Integrated Transport reports that Britain has the worst commuter congestion and very nearly the highest rail fares in Europe, the question troubles me more than ever.

To get at the answer, three periods demand examination: James Callaghan's union-accommodating social democracy of the 1970s; Margaret Thatcher's union-bashing neoliberalism of the 1980s, to which John Major added a coda in which the railway was privatised; and, finally, new Labour. The differences between these political eras could scarcely be more marked but, for the railway, the outcome has been the same - stagnation. Between 1975 and 2000, rail's share of passenger journeys fell from 8 per cent to 6 per cent, despite the soaring demand for travel in general.

What went wrong in the 1970s is pretty clear. The Callaghan government failed to gain mastery of either the public finances or the trade unions, both critical factors for the railway.

Callaghan's transport secretary was Bill Rodgers, later a founder of the Social Democratic Party, and British Rail was chaired by a top businessman, Peter Parker, himself a former Labour candidate. They had real achievements to their credit. There is no theme in John Prescott's 1999 transport white paper that is not anticipated in Rodgers's 1977 version. Parker brought TV advertising to BR, with Jimmy Savile's "Age of the Train". However, his marketing successes were undermined by labour disputes and management failures. A sorry symbol of the era is the tilting, 140mph Advanced Passenger Train, designed to cut long-distance journey times on Britain's bumpy, bendy track. Plagued by engineering faults, it was cancelled in 1981 at Treasury insistence, after 14 years and £40m. Today it sits in Crewe, a venue for children's parties. Yours for £40 a day.

There was a more fundamental problem, common to the Callaghan government and all its successors. Whatever the rhetoric, no government has behaved as if the railway was really capable of increasing capacity and so taking pressure off our ever more congested and environmentally damaging roads. Rodgers's predecessor, Tony Crosland, gave the game away when he told Parker that he saw BR's future as "a smaller, sensible little railway".

Probably Parker's biggest achievement was his success in persuading Margaret Thatcher, who did not travel by train, that he should be succeeded by his own chief executive, Bob Reid, a career railwayman. Reid promptly made friends with Nicholas Ridley, transport secretary from 1983-86. It was Ridley who convinced Thatcher to keep the railway off the privatisation list. Reid also had the good fortune to run BR at a time when Thatcherite belligerence encouraged the unions to keep their heads down, and when the Lawson boom was doing to railway demand what booms always do - making it grow.

This growth was kept in check by the Treasury's well-practised trick of insisting that BR push up its ticket prices above inflation, which had for the Exchequer the pleasant double effect of reducing subsidy and undermining any proposals to invest in greater system capacity. That is why the 1980s ended with the railway run down, but financially in better shape than at any time since nationalisation.

Many critics of rail privatisation - including John Prescott, who, after shadowing transport, took overall responsibility for it in 1997 - saw Reid's period as a golden age which demonstrated that nationalisation was on the point of proving itself. In reality, the golden age rested squarely upon Crosland's self-limiting assumption about the sensible little railway. Nobody in government, then or later, tried to design a railway capable of soaking up extra demand from choking roads. Privatisation - which, if properly designed, might have eased the investment shortage - gave incentives to the train operators to win more business, but denied them to Railtrack. Incredibly, the possibility of growth was raised nowhere in the tonnes of privatisation documents and consultants' reports that preceded privatisation. Everyone assumed that the sensible little railway would stay small or get smaller.

You had only to look at routine transport statistics to see the folly of this. I still have a piece of paper Gerald Corbett, then chief executive of Railtrack, gave me in 1998. It shows demand rising from 27 billion passenger kilometres a year in 1982 (recession) to 34 billion in 1989 (the Lawson boom). Demand then falls in the recession of the early 1990s, picking up for the subsequent boom, the longest since the Second World War.

By 1996, when Railtrack was floated on the stock exchange, demand was very nearly back to its peak of the Reid years. So, if you believed the economic forecasts of Kenneth Clarke, then the Chancellor, you knew that it would soon be well above it. By 1999, passenger demand was indeed up by one-third from its low point, and freight traffic by nearly 40 per cent. Because it was unforeseen, this increase in demand had not been matched by increased investment. It was this miscalculation that imploded after the Hatfield derailment in October 2000 as speed restrictions aggravated the system's countless bottlenecks.

Corbett's figures envisaged the possibility that, by 2007, demand could reach 45 billion passenger kilometres, an increase of almost 60 per cent on the level assumed at privatisation. Yet the railway was neglected as wantonly by Tony Blair in his first term as it was by Thatcher during hers. Blair's lack of commitment is illustrated by the seriousness with which he took the job of transport minister, a position held since 1997 (under various formulations) by Prescott, Gavin Strang, John Reid, Helen Liddell, Gus Macdonald, John Spellar and Stephen Byers. In the past 20 years, we have averaged one transport minister a year, and Blair, famous for his reluctance to shuffle the cabinet, has exceeded the average.

It is also evident that the railway has been damaged by infighting between Prescott, Downing Street and the Treasury. Dismissed as dangerously anti-motorist by the Downing Street "teeny-boppers", Prescott was made a monkey by the Treasury in his attempts to devise a way forward for the London Underground and to see beyond the design flaws in the privatised railway.

Just recently, the Commons transport committee protested about the Treasury's refusal to give evidence on Railtrack. That fits a pattern. Bob Kiley, Ken Livingstone's director of transport in London, complains that he has failed to secure a single meeting with Gordon Brown. So does Sir Alastair Morton, the outgoing, highly experienced business operator brought in by Prescott to run the Strategic Rail Authority.

In Morton's view, the culture of the Treasury is deeply implicated in Britain's railway problem. "It's a culture that is averse to investment. The Treasury manages the economy for cash. Historically the Treasury has always regarded the cancelling or reinstating of capital projects as one of its major regulators of the cash flow of the economy. The Treasury does not have the concept that you invest for the long term." In other words, to see what Britain run by the Treasury would look like, imagine yourself on any railway platform any day since 1948.

All this may seem a long way from Byers and his attempt to put Railtrack on a more solid footing, perhaps by turning it into a non-profit-distributing trust, but it isn't. Treasury short-termism and cabinet factionalism remain central to the railway problem. The railway will prosper only if the cabinet can agree that it needs to grow and, as this growth will not be self-financing, that it must be supported by long-term public or government-guaranteed capital. This is an industry where assets last for decades.

Whether this can be best achieved with the railway in the public, private or public-private sector is, frankly, a secondary point. As someone who favours greater mutual ownership, I would be delighted to see Byers go down that road. But thus far, his recipe, like Major's privatisation ragout, looks like one cooked up in haste and designed to be spat back at the next unhappy transport minister but one.

Ian Hargreaves presents A Secret History of Rail, a Juniper production for Channel 4, on 1 December at 7.10pm

This article appears in the 03 December 2001 issue of the New Statesman, Who needs 12 when one will do?