Sayeeda Warsi is under friendly fire. Her critics inside the Conservative Party have queued up in recent days to deride and denounce her as “incompetent”, “invisible” and “not up to the job” of party chair. According to the Daily Telegraph’s front-page splash on 2 April irate Tory backbenchers want “a senior MP . . . appointed as full-time Conservative Party chairman, ending the current arrangement where [the] job is shared by two peers”.

The attacks on Warsi from fellow Tories, since her elevation to the House of Lords and promotion to the party’s front bench in 2007, have been as vicious as they have relentless. A now-notorious post on the ConservativeHome website described her appointment as “the wrong signal at a time when Britain is fighting a global war against Islamic terrorism and extremism”. The website’s influential editor, Tim Montgomerie, has called her a “lightweight” and loudly urges David Cameron to sack the baroness from her post as party chair.

Judged by the intensity and sheer volume of the anti-Warsi vitriol, it is difficult to come to any other conclusion than that her critics don’t like her because she ticks three very un-Tory boxes: she is female, Asian and Muslim. Since it is 2012 and they can’t say as much in public, her right-wing opponents target instead her alleged lack of “competence” and “ability”.

Such criticisms are unfair and untenable. Consider the recent record of self-inflicted Tory wounds. Warsi can’t be blamed for the “pasty” or “granny” taxes (step forward, George Osborne). Nor can she be held responsible for the panic over petrol (step forward, Francis Maude). And it was her co-chairman, and Cameron’s close friend, Andrew Feldman, who appointed the seedy Peter Cruddas, of Sunday Times video infamy, as party treasurer. On the issue of Cruddas and the “Cash for Cameron” debacle, a source in Conservative Campaign Headquarters tells me, Warsi didn’t disappear or go to ground: she wasn’t asked to take to the airwaves on the Monday morning because the decision had already been taken to send out Maude and the party’s deputy chairman, Michael Fallon.

“Downing Street decided they needed a dull grey man in a suit to shut the story down,” says my source – though, of course, Maude’s glib reference to “kitchen suppers” did the exact opposite. Nonetheless, “Maude and Fallon are seen as dispensable; Warsi isn’t.”

Shuffled off

But for how much longer? One cabinet minister tells me Cameron is under “horrendous pressure” from his backbenchers to sack Warsi. Encouraged by Fallon and his supporters, they want a stronger, more ultra-Conservative figure, based in the Commons, not the Lords.

Ultimately, though, it was Cameron’s decision to split the chairman’s job between Warsi and Feldman, despite the last disastrous experiment with co-chairs – Liam Fox and Maurice Saatchi, under Michael Howard. And it was the PM who insisted Warsi tour the country’s Conservative associations, week in, week out, in her role as a “campaigning” chairman, rather than stay in London and tour the TV studios.

Despite the collapse of the Conservative vote in Bradford West on 29 March, Warsi has had some success at the polls. “When was the last time a party in government won local council seats?” says an aide. The Tories, according to their internal calculations, had expected to lose 300-odd seats in the local elections of May 2011 but ended up gaining 85 new ones – and winning the AV referendum.

But none of this seems to matter. Warsi irritates and infuriates right-wing, male, Westminster-based Conservatives. Her outspoken comments on Islam and Muslims haven’t helped her cause; in a much-discussed speech last year, Warsi declared that Islamphobia had “passed the dinner table test” in the UK.

At the time, Downing Street briefed reporters that she had failed to clear her speech with the Prime Minister. Cameron is said to have later conceded to Warsi that he agreed with much of what she said. But it was too late. Friends of Warsi note how the speech, and the official reaction to it, sent out a message to the right of the party, especially the neoconservatives, that she was “dangerous”. Tory commentators intensified their calls for her to be sacked and shifted to a less visible role, such as British high commissioner in Pakistan.

“It’s fucking patronising to suggest she should be shuffled off to Pakistan,” says a friend. “Perhaps they think she’ll be out of sight, maybe blown up by a bomb and that’ll be the end of her.”

Warsi isn’t without allies in the cabinet but they are few and far between, including Dominic Grieve, Kenneth Clarke, Andrew Lansley and the Deputy Prime Minister, Nick Clegg. She has repeatedly clashed with cabinet heavyweights such as Osborne and Michael Gove, the Education Secretary. I’m told that Gove tried to stymie efforts by Warsi to set up a cross-government working group to tackle anti-Muslim hatred. Warsi, however, persisted – and succeeded. But, says a senior Tory source, it was a “long, hard battle”.

So will the outspoken peer quit the cabinet? Jump before she is pushed? I doubt it. Warsi is a battler. Here is the only Tory frontbencher to have taken on both the BNP’s Nick Griffin (on BBC1’s Question Time) and the Islamist buffoon Anjem Choudary (on BBC2’s Newsnight).

Yet the attacks on her continue. Simon Heffer, writing in the Daily Mail on 31 March, claimed that the Tory peer was “incapable of motivating the party’s volunteer workers because her own ‘progressive’ views are diametrically opposed to the those of more traditionally conservative activists”.

Allies of Warsi inside Conservative HQ dispute that there is a disconnect between the chairman and and the grass roots and point out how well she is received by constituency associations around the country each week. They dismiss the hostile posts and comments on sites such as ConservativeHome as “Westminster chatter”.

But there is a much bigger issue here: will indulging “more traditionally conservative activists” win a majority for Cameron’s Conservatives at the next election? As I pointed out in last week’s issue, it will be difficult for the Conservative Party to win a parliamentary majority in 2015. Where will the extra votes come from?

The major factor in deciding not to vote Conservative is not being white. Labour continues to monopolise the votes of most black and minority ethnic (BME) communities and, in the 2010 general election, just 6 per cent of BME voters opted for the Conservative Party, compared with 37 per cent of white voters.

The Tories were their own worst enemy: Michael Ashcroft, the then deputy chairman who ran the marginal seats strategy, refused to prioritise BME voters or target seats with substantial BME populations. In 14 of the top 50 seats where the Tories narrowly came second to Labour in 2010, non-white voters made up more than 10 per cent of the population. These included key marginals such as Tooting, Birmingham Edgbaston and Luton South, all of which slipped through the Tories’ fingers.

It would be unfair, however, to ignore the progress that the Conservatives have made in diversifying their parliamentary party: the number of non-white MPs increased by 450 per cent in 2010 – from two to 11. Paul Uppal, a Sikh, now represents the Wolverhampton South West seat once held by Enoch Powell!

But there is still a mountain to climb. Take Birmingham City Council, currently controlled by a Conservative-Lib Dem coalition. In a city where one in three voters isn’t white, the Conservative group on the council is all-white. Tory strategists are worried it will fall to Labour in May’s local elections. Warsi, with the aid of Cameron’s former adviser Steve Hilton, tried and failed to persuade the reactionary Tory council leader, Mike Whitby, to select more BME candidates.

Speak plainly

To her opponents’ consternation, the Conservative chairman won’t stop pushing her agenda. She is on the verge of hiring a new adviser, to be based in CCHQ and tasked with getting the party to focus on BME communities. I’m told that at a recent meeting with backbench Tory MPs, a defiant Warsi declared that “unless and until campaigning with BME communities is institutionalised and embedded in every aspect of what we do as a political party, we cannot win an overall majority in 2015”.

Over the past two years, she has struggled to persuade her all-white ministerial allies to internalise this argument. But the appointment of Andrew Cooper, founder of polling company Populus and über-moderniser, as the Prime Minister’s director of political strategy in February 2011 has helped her cause. “Cooper brought hard data with him to help win the argument on the importance of BME voters,” says a senior Tory source. Late last year, he presented his empirical evidence at a meeting of Conservative cabinet ministers in the Prime Minister’s suite of offices behind the Speaker's chair in parliament. Cooper identified more than 30 urban seats, with big BME populations, that need to be gained to secure a Tory majority in 2015. “It was where her argument was finally won,” says my source.

It might be unpopular to say so in liberal circles, but I can’t help but admire the plain-speaking, hard-working, and “progressive” (© Simon Heffer) baroness. As a northern, working-class, Asian, Muslim woman, she is a huge asset to a Conservative-led cabinet that is seen as distant, unrepresentative and out of touch. But she has been let down by the more unreconstructed and short-sighted sections of her party – as well as by the Prime Minister.

When I interviewed her in October 2010, I called Warsi “the Conservatives’ secret weapon”. That they have failed to use this weapon says much more about the Tory high command and its dismal lack of judgement and confidence than it does about her.