It is mildly ludicrous that Andrew Mitchell, the chief whip and alleged verbal abuser of police officers, won't be attending his party's annual conference next week. And yet it is not that surprising. He might indeed cause a "distraction" from the business at hand - his excuse for bunking off - and he doesn't have a departmental brief, so he doesn't need to make a speech. So the calculation for Mitchell personally is fairly simple: why bother?

A problem for the Conservative party is that he is not alone in thinking that. Tory MPs have been grumbling more or less openly about their conference and wondering aloud whether or not to show up. The complaint is a familiar one: the whole show is run by and for the benefit of David Cameron's clique; only the favoured, Osborne-groomed ministers will be allowed near a platform or microphone; the whole jamboree is really just an excuse to gouge money from corporate public affairs budgets. (Conferences are very lucrative for governing parties as they hoover up lobbyist cash.)

The same gripes can be heard on the periphery of the Labour and Lib Dem gatherings but in my experience it is the Conservative one that has been most dramatically hollowed out in recent years. (The Lib Dems have a residue of actual democracy at theirs, which makes it worthwhile for members to go and Labour numbers are bolstered by unions, which are a better at mobilising numbers than Conservative associations.)

More seasoned hacks than me were shocked last year by the absence of ordinary delegates at the Tory gathering in Manchester. Senior figures in the party were also alarmed by the sight of empty chairs in the hall when David Cameron gave his keynote address.

Cameron's leadership has accelerated the decline in grassroots participation in the conference. That was inevitable given the way the "modernisers" around the leadership sought to define themselves in explicit contrast with much of what the party had once appeared to represent. The battle-scarred infantry of the Tory wilderness years didn't exactly take kindly to the appearance of a pomaded young cavalry officer riding in and telling them their campaign medals were worth nothing and that their only hope was to march behind him to victory. (They followed him for want of a better plan and never forgave him when victory still proved elusive.) Coalition also means that ordinary Tory activists don't feel ownership of the government's programme. Lib Dems can at least cheer the basic fact of being in power; Tories can only mourn the fact that their power is diluted.

There was a peculiar atmosphere around those early Cameroon conferences. Pushy twenty-something aides and wannabe apparatchiks - barely distinguishable in appearance from their New Labour counterparts a decade earlier - darted around bewildered old gents in navy blazers and regimental ties. The apparatchiks are now ministerial bag-carriers, MPs - or in some cases ministers. The old gents are more likely to make the journey to a Ukip conference than a Tory one. It will be interesting to see how many local association Conservatives come to Birmingham next week.

A final thought on this subject. Ed Miliband has been criticised for failing adequately to challenge his party in Manchester last week. The allegation of tummy-tickling and comfort-zone-coddling is not unfair. As I wrote in my column this week, the specific claim that Miliband entirely ignores the deficit is wrong; the charge that he has yet to offer any practical mechanism for delivering better public services and reversing inequality when there is no money spare is closer to the mark.

It is certainly true that Miliband doesn't deliberately antagonise his party. This is a strategic choice he has made. He has had a look at the way Tony Blair used conflict with "old" Labour and Cameron has used "modernisation"as the device for grabbing public attention and defining themselves as leaders - and concluded that it is not a path worth pursuing. Why? Because it sows the seeds of division and future rebellion, corroding a base of support that is essential to sustain a long-term political project. Cameron must now deeply regret not securing a clearer mandate inside his party for the kind of changes he claimed he wanted to make.

The obvious downside to the Miliband approach is that it looks like weakness - leading in fear of alienating the most tribal element in the party which, by definition, makes it harder to reach across to swing voters. Miliband's "one nation" pitch is an attempt to hold the allegiance of the Labour faithful and extend an invitation to people who naturally support other parties. No wonder it is vague on policy.

There is every reason to think it can't work. Eventually, Miliband will have to confront sections of his party if he is serious about running public services on tighter budgets. There is no denying that Labour unity has been bought with evasion, or at the very least deferral, of some tough choices. But it is worth noting too that the much advertised alternative is over-rated. That is the macho confrontation with the party to prove that everything is changing and that the leader is something rather new and special. It is an approach that worked for a couple of years for David Cameron. It is also the approach that means his disloyal MPs don't feel like showing up to their own annual conference. (And the chief whip won't be there to chide any troublemakers who do go.)

Miliband is aiming for something else: defining his political project not by the dismay of Labour members but through their acclamation. Can it be done? Parties these days seem so marginalised and tribal compared to the rest of society that it seems hard to believe he can pull it off. It will certainly be fascinating to watch him try.