I know just how Iain Duncan Smith feels this week. You give your all to your party and it turns on you and then the literary critics cannot separate the author from the book - but oh, the relief, the wonderful, blessed relief of leaving office! I simply had not realised how completely being a minister had taken me over until I was propelled into opposition in 1997 and the red boxes vanished overnight. So did the crammed official diary and the weekends yielded up to visiting 135 prisons. I had time, glorious time, freedom, glorious freedom, and my days, my views and my priorities became once more my own. Everybody said how well I looked.

Yet opposition palls, and when I found myself back in No 10 this week for the first time since the advent of a Labour government, I decided I had seen enough of it (of opposition, that is, not of No 10). The occasion was a reception hosted by Cherie Blair to mark the inauguration of the Longford Scholarships, bursaries for ex-prisoners who wish to study. Antonia Fraser, surrounded by large numbers of her family, gave a speech about Our Father, which I think referred to Frank Longford rather than the One which art in Heaven.

Too few care about the rehabilitation of prisoners, as I found out on Tuesday morning, when the home affairs select committee took evidence on education and work in prisons. The press benches, which are packed to overflowing when we are studying asylum, were empty. If we could persuade people that what we do with offenders in jail can be a crucial tool of public protection, they might take a bit more interest.

Perhaps people thought I was talking about prisons at No 10 when they saw me in animated conversation with Derek Lewis (the former head of the prison service) and his wife, Louise, but it is more likely they thought I was talking about Michael Howard. In fact, we were discussing Catholic saints, as Derek, a Presbyterian who once took refuge in a graveyard rather than venture into a Catholic Church, has just bought a vast plot of land on Mull, complete with a ruined chapel which he wishes to restore. The chapel has connections with Mary MacKillop, of whom I had never heard but who apparently has claims to sainthood in Australia, where they are somewhat irritated that she is being overshadowed by Mother Teresa. You learn the most amazing things in the most unlikely places.

If, however, we were not talking about Michael Howard, then everybody else was and had spent the week doing little else. The press reported my own position with various degrees of accuracy, some claiming I had done a U-turn despite my oft-repeated statement that I was not retracting a single word I uttered about him in 1997, and others saying I was supporting his leadership bid when what I had said I would do was support him if he became leader.

He should be glad I did not sign up to his bid. I backed Hurd not Major, Clarke not Hague and Clarke not IDS, so it's always the guy I'm not supporting who wins. But that has never stopped me thereafter giving all my loyalty to the winner. At long last, it looks as if there has been a similar outbreak of loyalty across the party as a whole and that we have learnt in six years that which took Labour 15 years: namely, that fighting for the soul of the party does nothing but keep the other lot in power.

It can often be local rather than national events that give most pleasure, and on Monday I found myself opening a new classroom at St Margaret's school in Collier Street, a village in my Kent constituency. As I arrived, the children were playing outside and their laughter floated over the countryside, merry, innocent and unrestrained. Inside were piles of fruitcake and cups of tea prepared by the village ladies, who had raised £30,000 in just a few months for their school. Thank God for England.

There is, however, another England, another Britain, where the playgrounds are full of bullies and swearing, £30,000 is a fortune, the schools are dingy and dilapidated and aspiration is a foreign word, while every responsible agency shrugs. It is about that Britain that I want to hear from our new leader, and pretty darn quick.