What do prisoners, the mentally ill, the elderly poor in long-term care and children at poorly performing state schools have in common? The answer is that, increasingly, they are in the charge of the private sector. Most residential care for elderly people is now provided privately. We have a higher proportion of prisoners in private prisons (7 per cent, rising to 11 per cent by 2007) than the United States does (5 per cent). Private companies provide mental-health units for NHS patients; and a state school, King's Manor in Surrey, has already been handed over to the private sector.
A cynic might point out that these people have other things in common: they are vulnerable; most of them are not able to vote; they do not usually choose the institutions they attend. They are, in short, powerless.
Yet the most powerful, and least vulnerable, members of society are also heavy users of the private sector. They send their children to private schools, go to Bupa hospitals for their hip and cataract operations, and place their elderly relatives in expensive private nursing homes. These people are not forced to go private. They actually pay for the privilege, sometimes with embarrassment - especially if they are Islington Labour voters.
So why is the private sector good enough for the most and the least powerful, but apparently not for the rest of us? The idea of handing over NHS hospitals or mainline state schools to private companies, or even to non-profit charitable trusts, remains anathema to much of the centre-left and beyond. For many people, the mere mention of "private sector" in the same sentence as state schools or hospitals conjures up images of a McDonald's sign above the local comprehensive, or nurses in Glaxo Wellcome T-shirts. So what is the right boundary between public and private? And will the resistance to business involvement in the provision of publicly funded services ever wither away?
In the "Dobsonite" view (so-called because of Frank Dobson's hostility, as Health Secretary, to health authorities commissioning services from private hospitals), the traditional public/private divide should remain. Sensitive public services simply cannot be entrusted to private operators; a public service should be provided by the public sector.
This view is supported by social theorists who argue that, because of the special difficulty of monitoring quality in most public services, private providers will just have too many opportunities for exploitation. Patients lack the expertise to assess the quality of the healthcare they receive; children (and even their parents) cannot judge the quality of their education until long after it is over. In the private sector, they are at the mercy of rapacious providers whose sole motive is to maximise their profits - "knaves", as the 18th-century Scottish philosopher David Hume called them. Public sector providers, by contrast, are "knights" who care not about self-advancement, but about the people they serve.
This view sits uneasily with our apparent willingness to use the private sector to care for prisoners, the elderly and the mentally ill - precisely the groups most vulnerable to knavish exploitation. Nor is it consistent with such limited evidence as we have. Many private providers of residential care are just as caring and dedicated to the welfare of the people they look after as the best local authority homes. Stories of exploitation and abuse occur as frequently in NHS and local government institutions as they do in private ones. One has only to look at recent exposes about malpractice in the NHS to realise that not everyone in the public sector is a knight.
Indeed, the private sector can sometimes be less exploitative than the public one. In a private prison we visited, the prisoners spoke passionately about their relief at getting away from HM prisons, particularly those in which the prison officers' union is a dominant force. An inmate with experience of 27 HM prisons said that his new, privately employed wardens "treated him with respect".
Moreover, an element of private provision within a public service can have beneficial effects on those parts still run in the traditional way. A Home Office study found that, while the costs of private prisons were at first significantly lower than those of public ones, the latter's costs are now falling sharply. Although some of the private sector cost reductions stem from controversial changes in working conditions (for example, pension contributions), there have also been improvements in prison design, sharper procurement and more flexible working patterns.
Until recently, the new Labour orthodoxy on the public-private balance in public services combined the Dobsonite view with a desire to exploit the benefits of the private finance initiative. Under the PFI, the state pays the private sector to build and maintain new hospitals and schools, and to run their ancillary services, such as security. But there is a Chinese wall between these ancillary activities and the provision of important services to the public. The idea is that the private sector looks after chore, not core.
For many, this is a neat and sensible distinction to make. It allows the public sector to focus on its central tasks of teaching or medical care without having to worry about fixing light bulbs. But, in practice, it is not always clear what makes one element of service provision "core" and another "ancillary". For instance, given the importance of diet and hygiene in a patient's recovery - a point emphasised in last month's NHS National Plan - are hospital catering and cleaning really ancillary? Moreover, to confine the role of the private sector in this way may be to lose what are supposed to be its benefits. The private sector is said to be better at innovation; but it is usually harder to innovate in ancillary services, there being only a limited number of different ways to clean floors, for example, but a multiplicity of ways to organise a week of teaching. Indeed, the emerging evidence from prisons suggests that it is precisely when the private sector can manage the whole service (as opposed to simply looking after the buildings) that impressive efficiency savings and innovations are made.
The government now seems willing to let the private sector extend its role - but usually only where there is clear evidence of public sector failure or crisis. At the education department, David Blunkett has sanctioned the use of the private sector to rescue failing schools and education authorities. After damning Ofsted reports, private consultants are going in to manage local education authorities in Bradford, Rochdale and Waltham Forest, and the proposed city academies will take the private and not-for-profit sector into underperforming schools. The Home Office has asked private sector companies to bid to manage the troubled Brixton prison. The new NHS plan, boldly announcing an end to "ideological barriers", envisages a wider role for private hospitals in helping to avert the annual winter flu crisis.
But this "rescue" approach - the private sector as Red Adair, fighting the fire created by public sector failure, only to slip away when the heat has abated - also has its limitations. The makeshift philosophy behind this approach might be described as "We don't quite know if the private sector will work, but we doubt it can make things any worse". Again, there is a fundamental inconsistency: if the private sector cannot be trusted to run normal public service institutions, why should it be better placed to run failing ones? And will the best companies really queue up to run the worst-performing and most intractable elements of the public sector?
If one buys the arguments in favour of private sector provision at all, then it might seem that the logical position is to go the whole hog and allow complete private sector provision. This view appears to have some new Labour support: for instance, the recent housing green paper implies that it will be no bad thing if directly managed public housing has disappeared completely in ten years' time. Indeed, some modernising zealots think that ever larger sticks and carrots should be used to encourage local government to deliver services through the private and voluntary sectors.
However, this unashamedly pro-private sector approach is itself fundamentally flawed. It ignores the likelihood that, in some areas, public providers could be just as innovative and competitive as private providers. The former are inhibited because the public sector has to operate under tighter accounting and regulatory controls, and these tend to create a more cautious approach to innovation.
If it is accepted that the public and private sectors cannot be neatly characterised as knights and knaves, then perhaps the only truly consistent position is to embrace diversity and try to be "sector-blind". In other words, we should drop prejudices about the inherent strengths and weaknesses of the different sectors and create a genuinely level playing field on which service-providers can compete under a common regulatory framework. Public authorities would retain responsibility for guaranteeing that all citizens have access to high-quality and equitable services - but they would be able to choose freely between different types of provider.
Is such an approach politically feasible, especially in sensitive areas such as education and health? It is true that centre-left attitudes towards the role of the private sector have moved a long way in a short space of time. As a prominent trade unionist recently observed, it is only ten years since he railed against the ideological indecency of refuse collectors being employed by the private sector. He now proclaims that there should be no more "no-go" areas for the private sector.
The political obstacles remain, however. First, most public service professionals and unions remain viscerally attached to a monopoly public sector provider model. In education, for example, Blunkett and his team appear to be warming to further experimentation with the voluntary and private sector, but they will face fierce political resistance from the National Union of Teachers.
Second, there is the danger that the centre-left presents a confused political message to the public if it argues both for higher spending on public services (as it should) and a greater willingness to experiment with non-public providers. Some contend that, once you concede a greater role for the private or voluntary sector, you give the impression that minimising the cost to the taxpayer is paramount and thus inevitably strengthen the hand of the privatisers and cutters. This is why new Labour needs unambiguously to present any extension of the private sector's role as part of an effort to reinvigorate universal public services. The myth that public-private partnerships are simply a staging post on the road to privatisation must be confronted.
A third problem concerns the capacity of the private and voluntary sectors to respond to calls on them. The small cadre of private and non-profit organisations providing educational services could probably fit most of their employees within the assembly hall of the King's Manor school, which one of them will soon be running. Similarly, it is doubtful that any current UK private health provider could run a major NHS hospital.
Finally, just as those opposed to business involvement in public services all too readily resort to scaremongering (warning that the private sector will impose draconian cuts or will charge for services that were hitherto free), so the private sector suffers from overblown claims of what it can do. The pro-private camp often defends the indefensible and claims the incredible. Honest business people must wince (at least in private) when they hear the unqualified tones in which politicians and commentators eulogise the innovative, hyper-efficient, endlessly risk-taking private sector.
These pressures leave new Labour in an ambiguous position. Ministers move towards greater private sector involvement, but seem reluctant to articulate the reasons why this might be desirable. One minister recently proclaimed that private companies should play a larger role in managing schools, then, in the next breath, asserted that business should not make a profit from doing so. Such confusion is typical.
Soon after he became Labour leader, Tony Blair launched his modernisation agenda by tearing up Clause Four, thus forcing the centre-left to rethink public-private relations. Five years on, much thinking remains to be done.
Gavin Kelly is the secretary to, and Julian Le Grand a member of, the Institute for Public Policy Research's commission on public-private partnerships. Comment on these issues at www.ippr.org