David Cameron has a problem with men. It is a different problem to the one he has with women, which has been much discussed as a phenomenon of opinion polls. There is a gender gap of around 10 points in the Prime Minister's approval ratings. Downing Street is worried enough to be recruiting a new, female special adviser with responsibility for casting a womanly eye over policy.

Influencing the government's agenda is, however, notoriously difficult for anyone outside the Prime Minister's tight-knit entourage, and the club does not often accept new members. Indeed, it is the impenetrability of that clique that is partly responsible for Cameron's problem with men - specifically the middle-aged, white men, possessed of great ambition and self-belief, who make up the cultural caucus of the parliamentary Conservative Party.

Tory MPs' alienation from their high command goes back to Cameron's project to "modernise" the party in opposition. That felt to many - not least because of the leader's high-handed manner - like an upstart, newly commissioned cavalry officer parading around making contemptuous remarks about the battle-scarred infantry. The insult was compounded by Cameron's handling of the expenses scandal. He was judged by many Tories to exhibit quiet leniency towards friends on the front bench, while noisily purging the back ranks of rumpled veterans, thereby vacating seats for smooth-cheeked Cameroons from the leader's "A-list" of candidates.

Carrot shortage

Suspicion of positive discrimination is a whole sub-category of Tory resentment. Last month's mini-reshuffle, triggered by Liam Fox's resignation as defence secretary, created very few vacancies. Two important promotions went to women. Justine Greening rose to the cabinet as Transport Secretary. Her replacement as economic secretary to the Treasury was Chloe Smith who, at 29, became the youngest minister in the government.

That latter appointment in particular seems to have been taken as a deliberate slight on the calibre of older Tory manhood. The polite whispering against Smith is that her lack of financial expertise makes her an inappropriate candidate for the Treasury at a time of crisis. The impolite version is that she only got the job because of her youth and her sex. Hostile reaction to the reshuffle stoked the Tory rebellion in a backbench debate on Britain's membership of the European Union a week later.

The talk of the Conservative benches on the morning of the vote was an analysis published on the ConservativeHome website suggesting that - once the imperatives of sexual equality and Lib Dem coalition quotas were met - only three Tory men could hope for advancement in any future round of government appointments. That makes it harder for senior party figures to manage the troops in the traditional fashion: the carrot-and-stick chat about loyalty advancing future career prospects. "It's hard to see what the carrots could be," says one forlorn, fortysomething, male MP. "There's a distinct lack of vegetables all round."

The irritation is not confined to the back benches. There are ministers of a certain age who feel that they gave the best years of their political life to opposition and are now being ignored by No 10. There is even a tinge of class resentment to some gripes. I have heard a self-made-millionaire Tory sneer at the absence of hard graft in Cameron's CV and his "silver spoon" background.

This is problematic for Cameron even if it doesn't turn into systematic parliamentary rebellion. The Downing Street operation and the Conservative Party are becoming two distinct political movements, with different brands appealing to different electorates. Private party polling shows the Prime Minister appeals to voters as a modern, considerate, "family man". Images of Cameron breakfasting with his children or dashing off for romantic mini-breaks with Samantha on easyJet draw cynical groans from commentators, but they are effective.

His own privileged background is known, but it is not a great electoral hurdle; but the Tories as a whole are still seen as the party of the rich. At the last election, enough voters were persuaded that Cameron was "a different kind of Conservative" for him to win the greatest number of seats in parliament. Not enough were persuaded that he had changed the Tories for them to secure a majority. Unless something dramatic happens, it will be very difficult to achieve a better result next time.

Vintage whine

In particular, the Tories need the votes of those famously unimpressed women. "It is something that needs to be in the front of our minds at all times," says a Downing Street source. There is, I am told, a recognition in No 10 that the problem is about much more than presentation. Cuts to the public sector risk hitting women's jobs disproportionately; women are generally thought to be more sensitive to inflation as they tend to make the painful choices that arise when household budgets are stretched; while mums on low incomes have been hit by cuts to childcare-related benefits. But those trends will not easily be reversed when the country is locked on course for years of austerity. Since that path is non-negotiable, Cameron is stuck trying to feminise around the margins, which convinces no one and aggravates the bruises on certain male Tory egos.

Of course, backbench sniping is a tradition in British politics and, of course, not all Conservative men are grumpy chauvinists, nor even the majority. But it is remarkable how much venom has built up so quickly when the government is just 18 months old. The sour tone that many Tory MPs use when moaning about their leader has a dangerous vintage, harking back to historic divisions - old whine in new battles.

Cameron's difficulty appealing to women is only a symptom. His real problems are policies that would make any government unpopular and a party that was already too unpopular to win an election but has not the will to change. The Prime Minister should worry that where women voters are now leading, the other half of the electorate will soon follow.