The timing is awkward. David Miliband has been urged by his top mandarins to cancel our interview. They don't want us in the building, after their case against our whistleblower Derek Pasquill collapsed so ignominiously at the Old Bailey. Miliband, to his credit, politely declined their advice. We had agreed a long time beforehand to talk about his new world order, which he was preparing to set out in a speech to the Fabian Society's annual conference.

By way of an opener, we ask the Foreign Secretary to assess the state of play in British politics. He launches into a soliloquy: "The Arsène Wenger school of management says that you focus on your own team . . . and let the oppos ition take care of themselves. We've got a con viction leader who is determined to ensure that ideas as well as competence are at the heart of government. There is a genuine crisis of Conservatism and the fulcrum of it is how you reconcile a belief in markets with a belief in social order. It's unreconciled at a philosophical level and an intellectual level, and that's why you see it unreconciled at a political level."

Miliband has invented a catchphrase - the "civ il ian surge". He develops this theme: "There are 200 million Chinese learning English; there are more bloggers in Iran than any other country in the world per capita; Buddhist monks march for democracy in Burma. I got the idea of a civilian surge when I was talking to David Petraeus [the US military commander] in Iraq because, he says, 'You can't kill your way out of this problem - you need politics as well as security.'"

There are four great causes in current foreign policy, Miliband says. He lists them: tackle terrorism and weapons of mass destruction, "and that's what we're trying to do in Afghanistan"; try to reduce conflict, "and that's what we're trying to do in the Middle East, Kosovo and Sudan"; tackle inequality through low-carbon, high-growth economic aid and development policies, "and that's what we're trying to do in Bali and elsewhere"; and build durable international institutions that recognise international inter dependence, "and that's what we're trying to do with the EU and the UN". These, he says, "are all great progressive causes".

Democracy or security?

Miliband is seeking to reconcile what he calls "the old Westphalian settlement, which says we have no business being concerned with what goes on in other countries", with the mistakes of Iraq. He suggests that the present Tory position of David Cameron and William Hague is not unlike that of John Major and Douglas Hurd when the west stood back and allowed the massacres in Bosnia and Rwanda to take place.

We talk in detail about Tony Blair's criteria for "humanitarian intervention", and how it foun dered in Iraq. "People say, and I say myself, there are no military solutions. There are military victories, but then you've got to win the peace. That means building the institutions of civic society, and that's true in all the places where our military are deployed," he says. "It's the old argument of: 'Do you want democracy first or security first?' Actually, it's the wrong question because sequencing doesn't work. You can't have democracy without security, and as we're seeing in places like Pakistan, if you want true security you need democracy as well."

And was Iraq a great cause? Perhaps for fear of antagonising a US administration wary of the new incumbent in Downing Street, he is strange ly robust. "The idea that Iraqi citizens should be able to determine their own futures, in a democratic system that respects human rights - that's a progressive thing to want to do." So was it a progressive war? "The aim, which was to free the Iraqi people from a tyranny, is of course a progressive thing. Twelve million Iraqis went out to vote. Now, there's all sorts of things we could talk about - there are lessons, there are things that haven't gone right . . ." We put it again. Is he really proud that Britain went to war?

At this point he draws back. "A lot of our people have died. A much larger number of Iraqis have died. You have to have a lot of humility about what happened. I believe this was done for the right reasons - I don't believe the conspiracy theories. I believe it was done after a lot of hard thought and a lot of hard searching.

"The fifth anniversary invites us not to put a glib label on it, but to make sure Britain and the international community are more united about the next five years. There is a real opportunity, without pre judice to any of the deeply held views of New Statesman readers and others about the wisdom of the original decision, to say: 'Where we are now, what does Iraq need?' It needs political reconciliation, it needs economic reconstruction and it needs continued commitment to the security of the people there."

We press him on Iran. Miliband supports the US, but puts his own gloss on the issue. "Iran is a sponsor of terrorism. It is a potential source of conflict." He elaborates. "It's a country that should be contributing all its riches and all its people to a stable international community. That doesn't require a change of regime in Iran, it requires a change of behaviour on the part of the regime.

"The challenge is to make clear that the international community is serious about the stability we say is important, but also show that we're serious about the offer we're making [to Iran] to engage with the international community."

Whatever happened, we wonder, to the neoconservative dream that Blair seized on with such alacrity? "What do they say is the definition of a neocon? A liberal who's been mugged. People who came out of the 1960s, but who had lost their faith in progressive policy because they said we weren't hard-headed enough," he replies. "Now the PM says our foreign policy is going to be defined by hard-headed internationalism. The military can't bring you the solution alone, but sometimes you need the military. In Darfur, we need an African Union/UN force. It's the progressive position to say economics and politics and social intervention where possible, military intervention where necessary." He adds: "We shouldn't cede the ground of universal values to the neocons."

Miliband develops his challenge to the left, saying it should do more to reappraise the relationship between state and individual. "On its own, social democracy is not adequate for this changing world. It's necessary but not sufficient. On the other hand, you've got a progressive tradition of radical liberalism . . . whose defining belief is the idea of individual freedom in the market economy. But it's not enough, because it's got no answer for distributional questions that are thrown up - the inequality questions. The politics that will address the 21st century is the fus ion of the social democratic commitment to social justice through collective action, not just through the state."

He talks of combining a greater emphasis on civil liberties with the need, post the 9/11 and 7/7 attacks, for security. He produces a curious approach to pre-trial detention. Once you have agreed on the need for any length of time you have established a principle, he says. "There's no magic in any number. What there should be is robustness and integrity in the processes. In dividual liberties depend on strong checks and balances." The longer anyone is held, the greater should be the scrutiny, Miliband says, but there need be no limit.

We turn to the issue that has caused such discomfort: the collapse at the Old Bailey of the prosecution of a Foreign Office official, Derek Pasquill, under the Official Secrets Act. It is our contention that this was a malicious prosecution pushed by the Foreign Office, even in the know ledge that the case would not stand up. This is now the second instance of an OSA trial foun dering, and we ask Miliband if the act should be reformed. "You always have to be open-minded about this. Have I been persuaded of the case for change? No. Do I rule out that it might need to be changed? No."

Does he not recall that Labour advocated such a reform when in opposition, particularly the inclusion of a public-interest defence? Miliband appears not to be aware of this. "I need to go and do some further research before I get drawn into that." And what of the principles of the case? "In principle, I think the confidentiality of government discussions is absolutely essential to effective government and I think we need a very effective regime to police that."

Religion and terror

We press him on our demand for an inquiry. He bridles. "I'm not going to get into any individual case. There are internal disciplinary issues relat ed to the leaks and I'm not going to say anything about it." What if someone at the Foreign Office had misled the courts? "Any suggestion of that is a matter for investigation by the CPS [Crown Prosecution Service] on the back of a complaint.

"We are a department that always seeks to make sure it upholds the highest standards of government, and we always look at our own procedures and processes to make sure that happens." He concludes: "I've seen nothing to suggest that there weren't appropriate and high standards followed all the way through. But you're not going to tempt me into discussion on this."

We move to the broader issue of engaging with Islam. Miliband describes this as a two-strand approach involving security and "hearts and minds". He has been persuaded by studies that suggest "you don't confuse degree of religiosity with propensity to terrorism". He adds: "We're much further ahead than we were three or four years ago in understanding what we're dealing with and how it feeds off grievance, both real and alleged. What we're clear about is that we're trying to counter insurgency, not counter people's religious freedoms. We're trying to avoid a clash of civilisations, rather than prosecute one."

Amid reports of the odd disagreement with Downing Street, and with a new baby on the home front, we ask him if he is enjoying his job. "It's a fantastic job. Great job. All these jobs are very demanding, but it's a great job to do. It's a huge privilege to do this." Miliband is curious to know why we called it the "Edward Stourton question". Because, we advise him, that was the question on the Today programme that stumped the Prime Minister. He winces.