My secret life began, as if scripted by P G Wodehouse, with an invitation to tea at the Ritz.

The call came at the end of the first week of May 1992. I was the Observer's home affairs correspondent, and at the other end of the line was a man we shall call Tom Bourgeois, special assistant to "C", Sir Colin McColl, the then chief of the Secret Intelligence Service. SIS (or MI6, as it is more widely known) was "reaching out" to selected members of the media, Bourgeois explained, and over lunch a few days earlier with McColl, my editor, Donald Trelford, had suggested that I was a reliable chap - not the sort, even years later, to betray a confidence by printing an MI6 man's real name.

Would I like an informal, off-the-record chat? You bet I would. "I make no apologies for the cliché," Bourgeois said, "since we do need a way to spot each other. I will be in the lobby, with a rolled-up copy of the Times."

Over the eclairs and Darjeeling a day or two later, Bourgeois explained that while the service - "the Office", as it is invariably termed by insiders - had always had a few, very limited contacts with journalists and editors, it now felt the need to put these arrangements on a broader and more formal basis. After eight decades in which the very existence of MI6 had been an official secret, the Tory prime minister, John Major, had just avowed it in the House of Commons for the first time - part of a process of incipient glasnost, Bourgeois said.

From time to time, he went on, it might be possible to "give me a steer", and if things worked out we might progress from meeting for tea to luncheon. Of course, he would be extremely constrained as to what he might ever be able to say about real, individual spy cases. If potential MI6 sources started to think their handlers might start blabbing about them to the papers, the Office's work would soon become impossible. Nevertheless, there would be things I might find interesting that would not compromise sources or security. Anyway, here was his number.

As a youngish, ambitious hack, I was enthralled. Bourgeois, a tall, slim man with an air of effortless urbanity, seemed to exude clandestine glamour - and future scoops. He was also refreshingly upfront about why the Office was taking steps to open up; as I put it in a somewhat breathless Observer feature that weekend, with the end of the Cold War, it recognised "its place in society" was going to change. "For the first time, the service is aware that it needs to protect its image, and that as it prepares to move into new and expensive postmodernist offices on the south bank of the Thames, it needs public relations." Or, to put it more cynically, it needed the media to trumpet its continued usefulness, lest the Treasury respond to the vanishing of the Soviet threat by slashing its budget.

Even then, the conditions that Bourgeois laid down struck me as odd, and perhaps a little onerous. Our conversations would not merely be off-the-record, and hence attributable in print to an unnamed MI6 official. In public I would have to pretend they had never happened, and if I wanted to quote or paraphrase anything Bourgeois said, I would have to use a circumlocution so vague as to make it impossible for any reader to realise that I had spoken to someone from the Office at all. Should I breach these conditions, Bourgeois made clear, I could expect instant outer darkness: the refusal of all future access. MI6, in other words, would maintain a priceless advantage, a quality regarded as essential in intelligence operations of many kinds - what spies call "plausible deniability". And if, heaven forfend, the service told me something that turned out to be mistaken, or even tried to plant sheer disinformation for who knows what purpose, there would be no comeback, no accountability. I could put up, or shut up.

At the time, I pushed my misgivings to the back of my mind, accepting Bourgeois's assurance that eventually MI6 would like to have an ordinary public press office like the Home Office or Department of Health. After all, as he pointed out, "the friends" across the Atlantic, the US Central Intelligence Agency, had long had such a bureau - an entire public affairs division - without apparent harm.

Fifteen years later, this promised development has not come to pass, and both MI6 and MI5, the Security Service (whose deniable press officer I also came to know), maintain exactly the system that Bourgeois described at the Ritz.

Every national paper and broadcasting outlet has one - and usually, only one - reporter to whom each agency will speak, provided they observe the niceties. For these fa voured few, there will be access likely to grow as the journalist proves his or her "worth", along with considerable perks.

One of the things that made me uneasy about my lunches with MI5 and MI6, which usually took place at very expensive restaurants, is that, in a reversal of usual journalistic practice, the agency men insisted on paying, often with wads of cash, presumably to protect their "cover". Later, there were boozy dinners at headquarters with C or MI5's director general, flanked by their brightest and best; briefings not just from the deniable PR man but officials involved with operations; and, most useful of all, a mobile phone number in case of urgent need at evenings and weekends. (To my chagrin, I never got as far as one reporter colleague who was plied with champagne and strawberries as a guest of MI6 at the centre court for the Wimbledon men's semi-final.)

Underpinning the link between the spies and coalface hacks is further contact with editors. As befits editors' status, the spooks try a little harder to impress them. One editor told me how, a few weeks after he first occupied his chair, he was asked to lunch with the then MI6 boss, Richard Dearlove. Not for him a taxi, or a frisk by security in order to get in: his hosts sent a limo to pick him up, then whisked him from Britain's most secret underground car park direct to C's suite.

Full disclosure: both agencies decided to stop speaking to me several years ago, in circumstances that at first I found infuriating. (Quite why MI6 cut me off, I never found out, but I have been told that MI5 objected to several interviews I carried out with Britons released from Guantanamo Bay who said that MI5 staff had been complicit in their treatment and interrogation while in US custody. It wasn't that this was untrue, but it was apparently regarded as "deeply unhelpful".)

This article is not, however, the product of sour grapes. It is my honest belief that the way Britain's spooks deal with the media has simply become untenable, gravely damaging journalists and spies alike.

Questionable motives

In 2004, the parliamentary intelligence and security committee, the only independent body with powers to call the agencies to account, announced that it planned to hold an inquiry into the relationship between the spooks and the fourth estate, suggesting it was time for a wide-ranging debate. In the event, few journalists offered to give evidence, and the committee's conclusions, published in its 2005 annual report, were disappointingly bland: "The government is trying to balance the need to inform people about issues that affect them, such as the terrorist threat to the UK, whilst still protecting the agencies' work. This is a difficult balance, which requires further thought." Indeed, it does.

Twenty years ago, the Independent, led by its now much-missed political editor, the late Tony Bevins, began a campaign to reform the Westminster lobby by withdrawing from the twice-daily briefings to correspondents by the then prime minister's spokesman, the doughty Bernard Ingham. Since then, political reporting has changed beyond recognition. When Labour came to power in 1997, Alastair Campbell's comments on behalf of Tony Blair became attributable to "the prime minister's spokesman" and, eventually, to him and his successors by name. But in 1987 the lobby rules were essentially the same as those that govern briefings from MI5 and MI6 today. Like them, lobby meetings were then not merely off the record, but deniable, and those who broke the rules risked expulsion from future sessions - so making it impossible, it was believed, for transgressors to do their jobs (though Bevins and his colleagues soon demonstrated otherwise).

The old system's drawbacks had long seemed obvious, and were often canvassed, especially in magazines such as this. The lobby rules were a licence to manipulate coverage and a way of settling political scores, a game in which journalists and voters held few cards. "Lobbies of all kinds are a conspiracy against the customer, the reader," says Peter Preston, who as editor of the Guardian also campaigned for reform. "They enable the reporter to say, 'Look how clever I am. I've got this amazing source, but I'm not going to tell you who it is, so you're just going to have to trust me.' The trouble is, the in formation may well not be trustworthy at all - from either a prime ministerial spokesman or MI6."

By definition, a reporter cannot publicly question information from a deniable briefing. They must swallow it whole, or not at all. As Andreas Whittam Smith, the Independent's editor when its campaign began, pointed out in an article he wrote looking back in 2002, the old lobby rules tended "to enforce a consensus". This suited everyone: while the PM's spokesman got his message out unmodified, "When a repor ter writes along the same lines as everybody else, he or she cannot be blamed if things turn out differently." Unfortunately, he noted, "Reporters as a group are often completely wrong." As spies can be . . .

My unhappy part

To my everlasting regret, I strongly supported the Iraq in vasion, in person and in print. I had become a recipient of what we now know to have been sheer disinformation about Saddam Hussein's weapons of mass destruction and his purported "links" with al-Qaeda - claims put out by Ahmad Chalabi and his Iraqi National Congress. I took these stories seriously because they were corroborated by "off-the-record" intelligence sources on both sides of the Atlantic. I am certain that those to whom I spoke at MI6 acted then in good faith. I remember one particular conversation I had with an official in the early summer of 2003, not long before Andrew Gilligan's BBC broadcast about the government having "sexed up" its dossier on Iraqi WMDs in September 2002. Already it was becoming apparent that the threat had probably been a chimera. "Don't worry," my source said soothingly. "We'll find them. We're certain they're there. It's just taking longer than we expected. Keep your nerve."

Since then, the cloak of plausible deniability has allowed those same spooks to claim they never believed in WMDs at all, and that they were the victims of neocon and Blairite pressure. One source in particular I find particularly hard to forgive - a very senior US official who told me time and again that Saddam really did have operational links with al-Qaeda, only to state very publicly much later that the CIA had never properly endorsed this view, and that its dissemination was all the fault of the Bush administration and Chalabi.

MI5 also told me deniable codswallop in the febrile weeks after 9/11. At one lunch, an official insisted that the preachers Abu Hamza and Abu Qatada - now said by the same agency to have been Britain's most dangerous men throughout the 1990s - were "harmless rent-a-gobs" who might have a high public profile, but had no hard links with jihadist terrorism.

More recent media briefings seem equally questionable. After the 7 July 2005 London bombings, MI5 told its stable of reporters that the bombers had all been "clean skins" who had been completely unknown to them; later they said there appeared to be "no connection" between the 7/7 cell and the failed 21/7 group who tried to repeat the atrocities a fortnight later. Only two years later, thanks to evidence given in criminal trials, did it become clear that both claims were false. In fact, the two leaders of the 7/7 gang, Mohammad Sidique Khan and Shazad Tanweer, had been observed by MI5 surveillance officers at least four times, and were known to be connected to another, now convicted, terrorist cell. But MI5 had decided to leave them alone while both men had apparently trained in Pakistan, at the same time as the 21/7 group.

By misleading journalists, and thereby delaying these disclosures, MI5 bought time. Had the truth come out in the immediate wake of the attacks, the Security Service might well have been much more sharply criticised, and the demand for a public inquiry might have become irresistible.

On other occasions, spook briefings have often seemed related to questionable policy goals, such as tougher legislation in the name of counter-terrorism. There may well be arguments for such measures, just as there are strong civil libertarian arguments against them. But, for years now, the agencies have tried to load the scales of this debate with a torrent of deniable briefings about blood-curdling threats from al-Qaeda which, thankfully, have yet to materialise, from dirty bombs to plots to "take down the internet", to say nothing of a long series of stories about Iran's nuclear weapons programme.

The 7/7 attacks proved that the terrorist threat was not chimerical. Yet, for many months before, it seemed barely a week passed without the BBC's security correspondent Frank Gardner broadcasting items that had clearly been briefed by MI5 and could be reduced to a single sentence: "Be very afraid."

Spy agencies, Preston says, "are always on a war footing, and almost nothing they ever say is checkable or necessarily true". In this sense, their present relationship with the media is more dangerous than the old lobby system: under that, at least, there were other accessible sources through which Downing Street's claims could be measured. Usually, Preston says, with spies there is none. "That doesn't mean it's always all spin and propaganda. But it could be, and some of it will be. It's often there for a reason we know nothing about - internal feuding; inter-agency rivalry; being pissed off over something with the Americans - it could be anything."

Pure disinformation

It's easy to see why deniable briefings hurt the cause of reliable journalism, and make it much easier for the agencies to manipulate the media. Less obviously, they can damage the agencies themselves. It doesn't take long for a journalist to pick up the codes through which the comments of MI5 and MI6 are attributed, such as "Whitehall security sources". Over the years, I listened as the spook spokesmen expostulated about national reporters who used such tags and attached them to quotations and stories that, they insisted, were pure fiction, saying that their authors had never spoken to officers at all. Alas: unable to confirm anything on the record, the agencies could not issue denials, either.

Why have the media put up with this situation without protest for so long? One reason, aside from the lunches and limos, is that editors are extremely reluctant to lose the access they have: the spooks' stories may be unreliable, but they often make good copy, and if everyone else is peddling the same errors, it doesn't much matter if they turn out to be untrue. Another, as a seasoned BBC correspondent put it to me, may be a judgement that if MI5 and MI6 sometimes peddle disinformation, many viewers and readers may not very much care, as "we're all on the same side".

Yet there are powerful counter-arguments and, says John Lloyd, the former NS editor who is director of journalism at the Reuters Institute at Oxford, they have become stronger as the agencies have shifted roles. "During the Cold War, their main concerns were remote from most people's lives," says Lloyd. "Now, when they're concentrating on domestic terrorism and subversion . . . their public exposure is much greater. This kind of secrecy becomes more objectionable the more they become part of daily life. It will always be difficult for reporters to verify or refute secret intelligence, but we should at least be able to state openly what the source of a story is. Increasing their accountability will also increase the confidence the public has in the agencies."

Time for a change

Some in the intelligence world itself agree. One senior, recently retired MI6 officer says that, from the spooks' point of view, the disadvantages of the present system now far outweigh any benefits: "The need is for a properly staffed, formal press office. Then, if there is a trashy story, we can put our heads above the parapet and say: 'This is rubbish.' It's a basic professional service that we should be providing, and it can be very easily done in a way that eliminates the risk of revealing damaging information about our sources and methods.

"As for talking only to a select few reporters, it's simply wrong in a democratic society where we're supposed to be accountable - and also counterproductive, because it's going to make the rest instinctively hostile. America, Australia and Canada have managed with on-the-record press officers for years. Unfortunately, the current chief [Sir John Scarlett] seems to take the view that no news is good news. Given our current preoccupations, that's not only wrong, but naive."

A normal press office might, the former officer reflected, have saved the life of David Kelly, the weapons expert who killed himself after being exposed as having talked to Gilligan before he broadcast the claim that Alastair Campbell had "sexed up" the Iraq dossier. "What people forget is that Kelly briefed a lot of reporters about WMDs, not only those we thought were held by Iraq, by tacit agreement with his employers. Much of the fuss and speculation that followed the invasion, and maybe his death, might have been prevented if there had been an on-the-record press officer available."

Adopting the US model may not be the answer. Milton Bearden, the former head of the CIA's Soviet and eastern Europe division who also led the CIA's covert campaign against the Soviets in Afghanistan, warns that journalists' relationships with spooks leave much to be desired. "The energy in the UK seems to be devoted to keeping the media at bay," Bearden says. "In the US, our style has been to spin them into submission. You don't want to get starry-eyed about the way we do business [at the CIA headquarters] in Langley, Virginia." He says he knows of cases where reporters have been taken into agents' confidence - and spun pure disinformation, no less pernicious for being on the record. "There is a structural problem here. The interests of journalists and those of secret intelligence agencies just don't always coincide."

What seems certain is that the debate heralded by the Commons intelligence committee two years ago is long overdue. Meanwhile, I would like to extend a standing invitation to the staff of MI5 and MI6 to join me at the Ritz, or any other London eatery of their choice. In the unlikely event that anyone decides to accept, I make only one condition: I'll pay.