If asked to identify one of the most important discoveries of the 1990s, I wouldn't look in the fields of medical research, astronomy or physics. I would highlight complexity - the growing realisation that the problems facing society and, more narrowly, those facing business, are interconnected and amorphous. They defy tight description or neat boundaries.
The decline of UK manufacturing industry, for example, is not just a matter of uncompetitive wage rates and productivity levels. It also involves education, attitudes to research and development, public attitudes to manufacturing and our Victorian attitude to bankruptcy, to mention just a few factors. The sensible reaction to this kind of complexity is segmentation - break down the problem into manageable parts - and that is precisely what successive governments have done. But segmentation needs to be followed by integration, by a drawing together of the constituents so that somebody tackles the complexity of the whole. This is where governments, with their boundaries of departmental and ministerial responsibility, are far less effective.
Will the Prime Minister's programme for "joined-up government" overcome this weakness? I applaud the aspiration. But I fear that there are barriers in our system of national governance that will inevitably frustrate it. British government is not well equipped to tackle what are now generally called the "wicked issues".
Consider one particularly "wicked" issue for a small island such as ours: land-use planning. Throughout the 1980s, it was difficult to discern any semblance of a sustained planning policy. The UK rushed to follow the Americans in out-of-town retail developments. Developers took over disused industrial sites and converted them into shopping malls. Despite the obvious threat to town centres and to the prospects of further investment in them, the then secretary of state for the environment, Nicholas Ridley, expounded his vision of ministerial power and responsibility in the words, "This thing is too big for me to stop". The government recognised the need for inner-city regeneration, yet still pursued a laissez-faire policy on retail planning. It allowed a growing dependence on the car without considering the results for road policy or public transport systems.
If all the energy, invention and finance that have been poured into out-of-town shopping facilities (the huge malls and the thousands of retail warehouses) had been channelled into redevelopment of our towns and cities - and with the sensitivities learnt from the brutal developments of previous decades - governments now would have no need to worry about transport policy or urban regeneration. Within the right government framework, the whole thing would have been resolved by the private sector. Yet there was neither the will nor the apparatus to draw together these threads and achieve "joined-up government".
Once John Gummer took over at the Department of the Environment the policy became more balanced, with the future of town and city centres given proper consideration in planning decisions. This government has continued Gummer's policy and, indeed, tightened the rules to make an effective presumption against out-of-town development when a town-centre option is available.
So at least we have continuity within the environment department. But the government as a whole contains forces that still pull in quite different directions. On one side we have not only the environment department championing our green and pleasant land, but also the Social Exclusion Unit, which argues that the growth of out-of-town shopping prejudices the interests of disadvantaged people marooned in inner cities. On the other side, however, we have the combined forces of the Treasury and the Department for Trade and Industry, which have found common ground in their zeal for productivity and lower prices. They quote a report published last year by McKinsey (Driving Productivity and Growth in the UK Economy) which points out that, if the UK were to increase its productivity to the US average, it would increase disposable income per head by 10-15 per cent within a decade. That would add significantly to government receipts and, therefore, to the money available to the health, education and social security budgets.
And how are such desirable ends to be achieved? McKinsey uses the retail sector as an example, arguing that modern supermarkets and hypermarkets are "by far the most productive outlets for food retailing". If only food retailers could obtain the sites for such stores, "they would almost certainly invest substantially more and raise both the capital intensity and the labour productivity of the entire sector".
Here, then, we have a perfect test of "joined-up government": can it reconcile the different objectives of economic efficiency and social cohesion? And, in a democracy, can it do so when one side - by offering lower prices - seems to carry far more popular appeal than the other, which is really offering only long-term gains?
The same dilemmas emerge in other forms of development. There is no single and obvious solution to the Wellcome Trust's application to build a centre of biotechnology excellence on green-belt land at Hinxton, Cambridgeshire. The protection of our green environment has to be weighed not just against the short-term employment opportunities but also against our long-term prospects in scientific and industrial capability. This is nothing as straightforward as nimbyism, and it challenges the decision- taking capability of a democracy.
Consider our arrangements for resolving such issues. During the three years and ten months of the Heathrow Terminal Five inquiry (525 sitting days, and the inspector may take another two years to write up the results and make his recommendations), the Koreans built an island for their new airport in Seoul; Hong Kong built and opened a new airport in its entirety; the French authorised two further runways at Charles de Gaulle and built one of them; Frankfurt acquired the neighbouring US air force base, giving it the potential to double in size; Schiphol added passenger terminal facilities and began building a fifth runway.
No one would deny that airport capacity excites strong emotions as well as complicated issues, that individual rights as well as national policy are at stake. But the adversarial approach of a public inquiry, while giving an appearance of democratic openness, is neither a speedy nor an efficient means of achieving a rigorous and objective analysis. And can it be honestly democratic when everybody expects that, whichever government is in power, Terminal Five will get the go-ahead? At best, it seems to spin out the inevitable.
In Britain, there is a firm conviction that democratic accountability and adversarialism are synonymous. The British parliamentary system operates like Newton's cradle: a force applied at one end produces an automatic reaction in the opposite direction, and so on backwards and forwards with diminishing force until all impetus is lost. Government needs to eradicate the wastefulness of adversarialism if it is to tackle complexity.
This is not to deny the importance of democratic oversight and a vigorous opposition. The need for accountability becomes, if anything, keener as the issues become more complex. We rightly take pride in this country's role as the mother of democratic government, and our parliamentary institutions have been widely emulated - a sure sign of being "world class". But industry knows that "world class" has constantly to be reassessed. Parliament operates today much as it did when Britain was a predominantly agricultural nation and when MPs faced each other across the floor of the House - at a distance measured in swords' lengths - to debate the corn laws. It is still based on the accountability of ministers to the continual challenge of opposition and to the electorate at five-yearly intervals.
As an example of a well-organised, non-corrupt democracy our parliamentary system remains in the top league, but its emphasis on accountability and challenge limits its competence to analyse and build - and therefore to tackle the "wicked issues" that require consistent policies and an ability to assemble expertise. Its very strengths become weaknesses when it comes to the achievement of true "joined-up government" on long-term issues. Our Civil Service is a rich mine of intelligence and information. But its function is to support the government of the day, rather than government as such. It can provide a research facility for ministers, but as long as those ministers are part of a two-party confrontational system, there can be no open examination of options such as the proposal to hypothecate (earmark) the proceeds of a fuel tax to subsidise public transport.
In other words, we have no means of establishing consensus on long-term targets. And the result is that we are restricted to stertorous advance rather than continuous progress. Further, adversarialism, while rapidly exposing any weakness in government action, also supports a blame culture and stifles risk-taking. It reinforces the power of hindsight, heightens the downside of being found to be wrong and sets greater store on finding fault than on making progress. We see this continually in our attitudes to scientific risks and the effect of those attitudes on policy-making.
We might reasonably expect back-bench MPs to act as the lateral thinkers outside government. But they will be hesitant to put their names to fresh thinking in case their inchoate ideas are later held against them when they reach their ambition of ministerial office. Few politicians don't change their minds, and we should be ready to recognise this as a strength rather than to seize on it as a "U-turn".
A further concern relates to the sheer complexity of modern legislation. The government has embarked on the mammoth task of updating company law, which has been complicated by a plethora of amendments that were introduced to close loopholes brought to light through various scandals and company collapses. The objective is to establish a legislative framework that will allow proper company regulation without threatening international competitiveness. The review is as open as anybody could wish and ministers are committed to implementing it. But experience shows that politicians have no appetite for scrutiny of such detailed legislation. "Blair orders ministers to axe boring legislation", was the Financial Times's headline in June. The Leader of the House had told Labour MPs that the Queen's Speech would include only legislation that would help the party win the next general election. Well, legislation does not come any more boring than company law, and it is perhaps fortunate that the current review is scheduled to run well past the election date.
This does, however, raise the question about where the democratic principle lies in an area such as this, and also the competence of a House of Commons comprising full-time politicians to debate and reach a judgement on such an arcane subject. Is it sensible for parliament to continue to protect its rights to deal with all legislation?
Less government could mean better government - in two senses. A former Swiss MP told me recently that the Swiss parliament meets only four times a year for three-week sessions; for the rest of the year he ran his own business. The idea of parliament being in recess for 40 weeks of the year seems at the very least to be worth examining against the merits of the repeated cock-fight of Prime Minister's Question Time. The Swiss system allows a broad range of experience to take part in the democratic system, whereas we expect our MPs to spend their time locked in the environs of the Palace of Westminster and to avoid second careers. Further, our fears about partiality and improper lobbying are building increasing barricades between politics and business. Possibly a reformed House of Lords could provide a solution if it included occupational representation so that legislation benefited from some scrutiny by those who would be affected by it.
But we need also to rethink the role of government itself. Government has already started to withdraw from direct responsibility for specific activities and to place them in the hands of public-sector agencies or even the private sector. We could go further. Is the running of school buildings or even the employment of teachers an essential public function as long as ministers continue to set standards and guarantee a good education for all? In the US, the private Edison company runs 25 schools; it has restructured everything from the length of the school day to teachers' contracts.
There was a time when ministers were expected to be personally and constantly accountable for every aspect of policy and execution within their departments. Clearly this is no longer feasible. If the government withdrew from responsibility for schools and hospitals, delegated the responsibility for teachers' and nurses' pay to companies accredited to deliver those services, set performance standards for exam results and hospital waiting lists, then ministers could focus on the vital role that only government can perform - building a consensus about a system of education and healthcare appropriate to international competitiveness and a harmonious and civilised society. A government already speared on its own detailed manifesto commitments to specific class sizes or hospital waiting lists is cultivating future adversarial conflict about delivery mechanisms and thus creating distraction from the complexity of the "wicked issues".
It will not have the time and the breadth of vision to tackle these at one end of the scale if its feet are bogged down in the mire of routine service delivery at the other.
The challenge is to find ways of reconciling democracy and progress and showing that these need not be in tension. We should welcome Tony Blair's aim of achieving "joined-up government". But it is not enough. If complexity is to be tackled, we must think about where government begins and ends and consider the merits of a more inclusive approach of "joined-up non-government".
Sir Stuart Hampson is chairman of the John Lewis Partnership and a former Department of Trade civil servant. This article is based on his inaugural address as chairman of the Royal Society of Arts, Manufactures and Commerce, entitled "Joined-up non-government", delivered on 6 October