How toxic is the Tory brand? It's a question that often occupies the mind of Andrew Cooper, David Cameron's director of strategy, who has consistently warned the Prime Minister that his party is still loathed by many of the voters that it needs the support of to win a majority next time round.

With this in mind, we conducted a poll with ICD Research on the subject, and the results make for fascinating reading. Asked whether they would consider voting for the Conservatives at the next election, just 40 per cent of the public said Yes and 60 per cent said No. The latest YouGov poll [2] puts the Tories on 37 per cent (Labour is on 42 per cent), although this excludes those who don't know how they would vote and those who wouldn't vote.

At the weekend, Cameron apologised for his recent comments to Labour MP Angela Eagle ("calm down, dear") and to Conservative MP Nadine Dorries ("'I know the honourable Lady is extremely frustrated"), for fear of appearing sexist. Before this, a leaked Downing Street memo [3] revealed that the government was concerned about its plummeting support among women. Our poll suggests that such fears are well-founded.

The Tories are significantly less popular among women than men, with just 35 per cent of women saying that they would vote for the party, compared to 44 per cent of men. Little wonder when, as I've noted before, so many of the coalition's austerity measures - the abolition of baby bonds, the three-year freeze in child benefit, the abolition of the health in maternity grant, the cuts to Sure Start [4], the withdrawal of child tax credits from higher earners - hit women and families hardest.

While Cameron is under pressure to shift rightwards on tax and public service reform, our poll found that a significant percentage of voters still see the Tories as too right-wing.

40 per cent of the public believe the party is too right-wing, compared to 18 per cent who believe it is too left-wing and 42 per cent who believe it is in the right position. Women are more likely than men to believe that the party is too right-wing. 42 per cent said that the Tories were too right-wing, compared to 37 per cent of men.

The results should not come as a surprise. As Conservative MP Nick Boles wrote recently in the Telegraph [5], "After three years of modernisation, David Cameron had shifted the perception of the party back towards the centre, and of himself even more so - although both were still seen as a bit more extreme than Labour and Gordon Brown. Since 2009, however, the party has shifted back to the Right in voters' eyes and, worryingly, so has he."

30 per cent of people said that the Conservatives "are much more right-wing than me" and 26 per cent said that they are "slightly more right-wing than me."

Finally, asked how much they agreed or disagreed with the statement that "the Conservative Party is in touch with ordinary people", 31 per cent said they completely disagreed, compared to just 6 per cent who said they completely agreed. 18 per cent agreed with the statement and 28 per cent partially agreed with it.

The conclusion is clear: there are few votes to be won by shifting ever further to the right.

This exclusive poll for the New Statesman was carried out by ICD Research, powered by ID Factor, from 23-25 September 2011 and is based on a sample of 1,000 responses.