The London 2012 Olympics was a blinding success. Universal Credit is struggling. Why do some government things go well and others not? The reason – complexity. This is not the garden variety “something jolly complicated” sort of complexity that most of us talk about, but complexity in the specific sense of a “complex system” – a system composed of many independent parts that behaves unpredictably as a result of interactions between those parts. Working with this sort of complexity requires a markedly different mindset.

Complexity experts distinguish between complicated and complex situations. For example, putting a man on the moon was undoubtedly a complicated task. But there was a clear objective – success was easily measured. The laws of physics pertained and sending a rocket to the moon did not change those laws. Effect can be predicted from cause. It is possible to make big, long-term plans and be reasonably certain of achieving them – London 2012, Channel Tunnel, Crossrail etc. We do well with physical-world tasks where analysis, big design, programmatic management and the “build” mindset prevail.

On the other hand, dealing with something like poverty is complex. Poverty is hard to define meaningfully. It is difficult to measure improvement. It is subject to changeable human behaviour and action is persistently met with unintended re-action. Effect can be deduced from cause, but only in retrospect. Progress is best made using emergent approaches – trying things, seeing what works and adapting accordingly. Here responsiveness, collaboration, human sensitivity and the “grow” mindset are more important.

Government investigations of significant IT failures do not seem to recognise the effects of complexity. Usually, problems are laid at the door of uncertain requirements, poor governance or inadequate management skills. Of course there have been straightforward programmes that have failed for these reasons but, where complexity applies, blaming these alone – and not getting to the root cause – will perpetuate failure.

The problem is not in plans, people or methods – it’s in mindset. Trying to build things that really need to be grown just won’t work – no matter how they are managed. Traditional programmes with big political promises and a fixed end date don’t work well in these situations. And this is the problem with Universal Credit, the government’s troubled welfare reform project.

This is not to say we should be abandoning programmatic approaches, but we should be ready to see where complexity is having an effect and respond appropriately.

Complexity effects are coming to bear in the case of digital transformation. Much of what is being done at the moment is more in “complicated” territory. The 25 digital exemplars identified by the Government Digital Service (GDS) are taking existing services, mostly within the confines of specific departments, and making them digital. This is going well.

But when it comes to the next stage – re-thinking the way government and its technology works – complexity will become more prevalent. Politics, human behaviour and the intricacies of widespread change will bite.

Digital government will need to be more grown than built. While GDS and some of the digital teams in Whitehall departments seem to embrace this, there is a long way to go for most of government to make the shift in mindset.

There is a parallel. In the mid-15th century, Girolamo Fracastoro developed a good understanding of germs and how infection was transmitted. Three centuries later, in the 1850s, people like Oliver Wendell Holmes were still up against it, arguing for better hygiene in medicine. Wendell Holmes was vociferously encouraging his grubby surgeon colleagues to wash their hands and sterilise their instruments between operations. That debate was thankfully resolved soon after but, even so, it has only been in the last few years, with the MRSA problem, that hospitals have got serious about washing hands. Germs are out of sight and out of mind; and it is hard to link particular infections to particular failures in hygiene.

Complexity is the same. Hidden in plain sight, it is causing significant problems that are not being properly managed. Let’s hope it won’t take 300 years for government to catch on.

This article was also published in Computer Weekly.