New Times,
New Thinking.

Why do big digital projects in the public sector fail?

From government transformation to the NHS, multibillion-pound projects can be high risk and low reward.

By Samir Jeraj

Both Labour and the Conservatives have promised better and more efficient online government services through digital transformation. But are these promises realistic? A series of embarrassing and catastrophic failures in major digital projects should be a source of caution. These range from the tortuous digitisation of the British courts system – which was delayed by three years, over budget and delivered few benefits – through to the futile attempts to save Birmingham City Council financially by using new digital systems, and the ongoing Horizon Post Office scandal. 

Recent high-profile cyberattacks prove that digital transformation is crucial, particularly for critical national infrastructure. A major cyberattack on Synnovis, an agency that manages blood labs for NHS trusts and GPs in south-east London, is still wreaking havoc across hospitals in the capital, with more than 1,000 planned operations, over 3,000 outpatient appointments and all non-urgent blood tests postponed. But major digital transformation projects need to be planned and managed appropriately to be successful.

“Traditionally there’s the iron triangle of a project, which is time-cost-scope,” explains Owen Pritchard, head of cyber, digital and technology at the Local Government Association. When one of these is not met, the project is deemed to have failed. But Pritchard’s own work on large digital projects has led him to a different view. “I think that perspective is flawed because you should take into account stakeholders, and the idea of stakeholder expectations,” he says. When private sector providers and policymakers have a disproportionate influence on a project, or unrealistic expectations, it is very likely that both the people doing the work and the end users suffer.

The success of digital projects relies not only on their execution and delivery, but also on creating the right environment to thrive following their completion. In other words, there needs to be a wider programme of change. “So, it’s technology-enabled change… but you [also] need to create the conditions around [projects] to make the most of them, but also to enable their success,” explains Pritchard. Essentially, without the right conditions, the project will fail.  

The National Audit Office published a report in 2021 called “The challenges in implementing digital change”, which argues that it is more important for these projects to “get [it] right at the outset” because of the risks of “unknown unknowns”. This includes understanding the problem that the technology is seeking to address and avoiding being over-ambitious.

Scale is a big challenge. Digital change projects in the public sector can cost hundreds of millions, or even billions. The Central Digital & Data Office is currently spending £8bn in digital, data and technology investment over three years. Projects at this scale create a risk that they become simultaneously too big to fail and too big to succeed. By contrast, private-sector organisations tend to avoid projects of that size because they are too risky.

One of the big tensions for Pritchard is the demand and expectation that digital projects and new technology should always cut costs. At both national and local government level, he has seen public-sector organisations adopting new technologies and delivering systems that do not meet the needs or expectations of workers or citizens because of the imperative to save money. Perversely, this can lead to losing money instead.

Give a gift subscription to the New Statesman this Christmas from just £49

In 2021 Birmingham City Council was trying to cut its costs. Austerity had hit its funding hard, and the council had a huge legal liability due to discriminatory pay practices going back years. One of the ways it went about saving money was a new IT system, Oracle, which the council estimated would save them £100m over five years. However, the transition went badly. The council requested lots of changes to the system that were out of the original scope so that it worked with their existing IT systems, creating further problems and costs. By the end of 2023, when the council declared bankruptcy (largely as a result of equal pay claims) the original budget of £19m for Oracle had ballooned to a cost of £100m without any of those cost savings emerging.

The Birmingham example also highlights another problem. Public-sector organisations have legacy systems and old data, which make any new system a challenge to implement. The underlying problem is that digital technology moves and develops quickly, whereas large organisations tend to move at a slower pace using bureaucratic processes. So, from the start there is a mismatch, according to Mark Thompson, professor of digital economy in Index (Initiative for the Digital Economy) at the University of Exeter.

“These large technology projects are often conceived to last for two, three, four years,” explains Thompson. Buying in technology, however, tends to contract on a ten-year cycle. The first three years work well, as the contractor invests in the technology and engages in improving services, but the following seven years are lacklustre. When the needs of the organisation change, the contractor will argue that any change to the systems are outside of the scope and will cost more.

“There is a view that says, commercially, a lot of those outsourcing deals are deliberately set up that way,” says Thompson. “And part of the problem, of course, is that you are unlikely to be around in ten years’ time to answer for that.” That turnover of staff means there is little institutional memory, so when that contract is renewed, few lessons have been learned. It’s a similar problem in local government where electoral cycles can be even shorter.

Procurement teams can be understaffed or under-skilled, too. This makes them more likely to be both process-driven and risk-averse, Thompson explains. “Maybe you have two or three of them [the team] who really have any kind of understanding or grip on what it is that they’re letting the contract for,” he says. In his experience, the most senior person in the room does not have an understanding of what is being bought.

Thompson is also critical of how the “weighting” or “scoring” system for choosing contractors incorporates important but softer metrics from prospective companies, such as environmental, social and governance (ESG), social value, and equality, diversity and inclusion (EDI), and basic business outcomes. “They [ESG, social value and EDI] can be played, and scoring criteria can be played by suppliers,” he says. “So the weighting is often way over potentially what it should be, given the huge, often billion multiple in risk of some of these projects.”

“Procurement teams will often have a favourite supplier. They wouldn’t admit it, but they absolutely will,” Thompson continues. This happens for many reasons such as personal relationships, or there might be a strategic dependency on them. He is also suspicious of relationships between suppliers and government departments where the former provide the latter with free consultancy or seconded staff, which he feels leads to that supplier having an advantage.

For contractors, asking questions or raising concerns about the feasibility of what is being asked for by their procurer is discouraged. This is because, in an open procurement process, the answers have to be shared with all the company’s rival bidders too. Once in place, an appointed contractor can use those unrealistic expectations to their advantage and extract more profit by asking for more money to make changes. 

One of the suggested ways to move beyond this problem is by developing an “intelligent customer function” (ICF). “It builds in the ability to get around a table, build collaborative relationships, and [build] some flexibility into contracts,” Thompson says. This means that, rather than having an overly specific contract, suppliers take a more “modular approach” that enables change in response to business needs.

The National Audit Office recommends that government organisations should equip their “non-technical” leaders with the right skills, and design “suitable approval and governance frameworks”. Whether the next government will take these views on board is unknown, but Thompson has one key warning: “If you’re overly risk averse, and you’re overly input- and process-driven, then the great irony is that you could generate an even bigger risk.”

Content from our partners
When partnerships pay off
Breaking down barriers for the next generation
How to tackle economic inactivity

Topics in this article : ,