Rowan Williams, the Archbishop of Canterbury, longed to take risks but was thwarted by Church courtiers and cronies more concerned with their own survival.

When I started working at Lambeth Palace as the Archbishop of Canterbury's principal spin doctor in October 2010, I got 15 minutes of his undivided attention and asked him what he thought and what he wanted. "I think, on balance, it's rather good news," he said in that precise, hissy voice. "I think we're rather risk-averse at Lambeth Palace. I'd look to you to do something about that."

After ten unhappy months of trying to deliver on that brief, I grabbed 15 seconds with him before I walked out, just to be sure that he wanted me to go. He sighed: "I just think it's going to get too dangerous, George."

I have dwelt on the apparent incongruity of those two statements in the months since. Take risks. But don't be dangerous. Dr Rowan Williams, contrary to some daft caricaturists, has always thought through what he means before he opens his mouth and invariably means exactly what he says. After some of his supposed "gaffes" - sharia law, his leader in this magazine - he has thrown a hysterical newspaper response aside and declared that he would say exactly the same thing again. He doesn't do self-contradiction.

So why the apparent volte-face over a high-risk public affairs and communications policy? He never wrote to explain. But I conclude two things. First and most importantly, Rowan subsumes his own opinions, prejudices and preferences, even substantially his personality, in the service of his archiepiscopacy. It's what characterises the decade of his incumbency of the See of Canterbury. The cross he bears is Anglican unity at whatever cost to himself. It's why he has disappointed both liberals (where his heart lies) and conservatives (whom his heart encompasses), why his friendship with the ambitious gay campaigner Canon Jeffrey John turned to dust and why he has tried to push through an Anglican Communion Covenant, a code of conduct for the faithful that his own Church has all but rejected for being essentially un-Anglican.

Second, in the microcosm of my infinitely easier role, I failed to subsume my relatively lowlife media and political streetfighter's personality because, unlike him, I didn't think I needed to. That was a crass mistake and entirely my fault, as was my attempt to do the job on a part-time contract. A dear colleague has made it abundantly clear to me that no one else is to blame but me for taking on a particularly intransigent faction of the Archbishop's administration - and losing.

But the trouble with not squaring up to the apparatus of the Archbishop's government machine is that it breeds, and is encouraged to develop, an internal, self-serving authority, like an overweening civil service. You should never underestimate a palace's tendency to attract courtiers. The one at Lambeth is no exception. They preen and jostle for favour (somewhat pointless, as Rowan treats everyone the same). They build professional silos and guard their sometimes limited responsibilities jealously. They meet weekly around the table in the Pink Drawing Room and there is no higher endeavour than filling the Archbishop's diary over a year in advance.

Startled or offended by the apparent vulgarity of my presence and the threat of change, they isolated me like corpuscles around a foreign body. Two key members of staff indicated that they wouldn't co-operate with me before I'd even started. Wretchedly, one briefed against me in the Archbishop's car when he had a thousand more important matters on his mind.

Another ran to the chief of staff whenever I appeared in the press and would make thinly veiled threats about an industrial tribunal.

They weren't all like this. Far from it. In particular, the international development department was a model of professional engagement with the world beyond the walls of Lambeth Palace. Also, the superb former diplomat Tim Livesey, who brought me in to fill his former job and who has since taken the role of Ed Miliband's chief of staff (as if Rowan Williams wasn't a big enough public affairs challenge), struggled against the suffocating system for six years. But the waters will have closed over Livesey's (and my lesser) efforts to reform a static system. Nor is the problem confined to Lambeth. Across the river at Church House, where the ecclesiastical civil service hangs out, a similar enthusiasm for the still millpond of administrative existence prevails, with a few honourable exceptions in parliamentary affairs and the press office.

All this would amount to nothing more than a whinge from a disaffected former contractor if it didn't have such a profound effect on the function of the Church of England in general and the Archbishop of Canterbury in particular. An obsession with process and bureaucracy is actually an avoidance strategy. You really don't have to go far in SE1 or SW1 to hear the view that life would be a whole lot simpler if the Archbishop never said anything contentious and confined himself to those safe, formal events booked 12 months in advance.

Take the New Statesman's invitation to be guest editor for one issue, which was eventually achieved last June. This had sat in an in tray for months before I arrived. I'm not even sure that Rowan knew of it. The opportunity was too good to miss - what was not to like? The worry of close association with a leftist publication was easily dealt with by commissioning writers other than the "usual suspects". So the Archbishop's interview with the Foreign Secretary, William Hague, was spread over six pages, Iain Duncan Smith wrote a column on welfare reform, and even the Chief Rabbi, Jonathan Sacks, adopted a moderately centre-right position in writing about the "big society" in the United States.

Rowan was minded to draft a leader on the threat to the foreign aid budget but, after an editorial meeting, I urged him to address domestic politics. He wrote a leader critical of the current state of British politics and not solely of the coalition, as has passed into the collective consciousness. He also wrote that the government was pursuing a "radical" agenda for which it had no mandate - an uncontroversial assertion, given that bringing together two minorities (with, one might add, different manifestos) to deliver a majority doesn't make for good democracy.

So that his office was prepared, I shared the piece with a limited number of colleagues, including the first-rate Richard Chapman, secretary for parliamentary affairs, who described it as a "depth charge". The rest is history. Many in the parliamentary Conservative Party went, frankly, berserk.
Ordinary Conservative MPs failed to get it even though David Cameron, to his credit, understood the Archbishop's responsibility to address our political standards. James Gray, the MP for North Wiltshire, called it "disgraceful drivel" and added that the Archbishop should "leave the running of the country to . . . us", apparently under the impression they were doing a good job of it. Andrew Stephenson, MP for Pendle, said that Rowan was "out of touch" and "styl[ing] himself as a politician". Andrew Murrison, for Westbury, told the Church of England to stop "bleating about the splintering of society" and claimed that Rowan should stop "suggesting that my party is in some way indifferent to poverty" (he didn't), adding
that he was "not surprised that so many are defecting to Rome" (they aren't). There were also private, scribbled letters that amounted to nasty little ad hominem attacks, of a kind that wouldn't be acceptable in corporate life but which are apparently common currency in Westminster today.

What was truly disgraceful was that elected representatives in parliament should address themselves in this way to an incumbent archbishop of Canterbury who had written a thoughtful and impersonal reflection on all political parties, as they would know if they had bothered to read it properly. You would have thought that Lambeth and Church House staff might have rallied to his support and filled the public square with further polemic.

Not a bit of it. There seemed to be a common tendency informally to apologise for "any embarrassment". Rowan had written to me that the exercise had "borne ample fruit", but at our next full meeting Lambeth staff sat purse-lipped and disapproving. They wanted to know why they had not had early sight of his copy, even if it had slap all to do with their remit. Experience had shown me that this was code for neutralising and making harmless what the Archbishop had to say, through protective editing, preferably through a series of meetings and subcommittees. They claimed that I had damaged "relationships" - for which read cosy little undertakings to push paper between the organs of Church and state. But there was never any suggestion as to what these “relationships" were meant to achieve, beyond mutual admiration.

Perhaps my only regret - beyond being at Lambeth Palace at all - is that attention to Rowan's New Statesman guest editorship was focused unduly on that leader rather than the whole issue, which was an enthralling mixture of politics, contextual theology, pastoral ministry and culture. But those who by their words and actions seem to claim that there is no higher calling than the maintenance of a quiet life perform a disservice to the Church of England, the worldwide Anglican Communion and their most senior figure.

And that is the crux of the matter. The weakest, the poorest and the most vulnerable in our society can't always rely on politicians to stand in their corner. They are blessed that they have a national church that is called by gospel imperative to do so. And anyone who has witnessed Dr Williams's interventions at local and individual level will know the power that his office wields to champion the causes of the dispossessed and the voiceless. Quite apart from anything else, Rowan replies personally to more letters pleading for his help than any member of his staff (and probably more than many of them added together).

On the international stage, he does at least as much by way of intervention on behalf of the threatened as any government minister has done. For evidence of that, look no further than last year's visit and challenge to Robert Mugabe's regime in Zimbabwe, training a searchlight on an immorality that shames our politics. The Archbishop of Canterbury can do this not just because he is a Christian leader, not just because there is an Anglican communion that can be mobilised as a global network, but also because he leads a church established in law, our state's religious and cultural heritage, with the monarch as its Supreme Governor.

This side of disestablishment, that is a potent weapon of justice that can be deployed from the Archbishop's seat in the House of Lords. Yet even a disestablished Church of England would be a thorn in the side of our body politic, the conscience of a nation that has been traduced by the corruption of MPs, police, the media and the financial markets. It's a task in which the Archbishop of Canterbury must be supported, encouraged and nurtured. Instead, he is too often hobbled by a machinery more concerned with its own preservation than with his.

There is now an opportunity for renewal. Rowan has announced his departure at the end of the year. The chiefs of staff at Lambeth Palace and Church House, too, will soon be on their way. The new Archbishop of Canterbury has a golden opportunity to streamline and to make the support structures of the Church of England and, by extension, the Anglican Communion, more effective for and better suited to the 21st century. Something similar has already been achieved in the civil service; it's high time that the administration of the English Church underwent another reformation.

Here are my suggestions, born of bitter experience but offered without rancour. The new archbishop should sweep away the top-heavy management of Lambeth Palace, discarding the courtly structures in favour of a small personal staff. He probably needs no more than a diary secretary, a chaplain and a junior press officer. All other executive functions would move to Church House in Westminster, where the Archbishop has an administrative office. There would be a single chief of staff, with oversight over both the Archbishop's and the Church of England's staff. The next most senior position is another single post that could merge all functions - call her or him, say, director of strategy and communications - to which all public affairs and media functions would report.

When it comes to the management of the Archbishop's time, only set pieces (General Synod, House of Lords, foreign trips) would be put in the diary more than three months in advance. All other requests for his time could be met with a standard reply that the Archbishop cannot commit too far in advance. That would leave him with adequate time to respond to events political and pastoral as they arise, making the new archbishop properly responsive to live issues. (By contrast, when I wanted to find time for the Archbishop to see the Health Secretary for dinner regarding NHS reform, I was given a date three days hence or in five months' time, apparently no other windows being any longer available.)

I doubt whether anything like this kind of revamp could happen, such is the system's tendency to self-preservation. But, ironically enough, probably the man best suited to achieving it would be Dr John Sentamu, Archbishop of York and the bookies' favourite for Canterbury. He maintains a relatively modest staff in York. I say it is ironic because there is a widespread view developing that Dr Sentamu has peaked too early in the Canterbury Stakes - a couple of years ago he might have been a shoo-in, but now he may be just a tad too old to have a long enough run in the job.
It has to be said also that there is a fine line between economy of human resources and autocracy. Under my suggested system, the new archbishop would be running his show through Church House, not his own show on a skeleton staff, which may be more Dr Sentamu's style.

Yet all it needs is an expression of will. A younger man with energy could do it, too. The well-fancied Christopher Cocksworth, Bishop of Coventry, could be such a reformer. For bold reform is what is needed. The Archbishop is at his best when bringing the gospel to political and economic affairs - indeed, it is his duty to do so. Those Tory backbenchers who wrote to him after the New Statesman editorship suggesting that he "stay out of politics" are relics of an older way of doing politics, when Planet Westminster lived remotely and arrogantly by its own rules and before social media took a more direct ownership of the processes of political communication.

On my brief watch, Rowan intervened in welfare reform, opposed the privatisation of our forests, expressed his unease over the shooting of Osama Bin Laden and, of course, edited the New Statesman. I wasn't responsible for all those initiatives, but I was proud to encourage them. Most of them were opposed or regretted by the Church machine.

It is difficult to know whether Rowan was comfortable at the sharp end of politics. After a warm correspondence during the travails of office, I heard not a word from him when I followed his guidance and quit. But he is a deeply unusual man. Holy, wise, kind - and sometimes remote. And his sincere humility makes him capable both of pursuing his own line and of being trampled over by apparatchiks.

The next archbishop of Canterbury could learn from his and, I would add respectfully, my experience and operate his office very differently. He should concentrate on making it nimble, professional and as daring as Christian witness at its best. Dissolving the Court of Lambeth Palace would only be the start.

George Pitcher was the Archbishop of Canterbury's secretary for public affairs from 2010-2011