Dealing With Unintended Complexity: Tips From an Analyst

Nobody sets out to build tangled, unmanageable, vulnerable digital infrastructure. It just happens. Complexity creeps into all organizations, despite the best intentions, and complexity is the enemy of both security and resilience. Despite these challenges, the unrelenting cadence of technology development and the imperative to remain competitive always drive us towards increasing complexity. 

Growing network entropy is inevitable and occurs even in highly disciplined environments with strict processes. Environmental forces make it inevitable: large external events and continual changes from day-to-day operations. Organizational leaders have to move away from rigid regulation towards an agile approach to infrastructure. Change will happen anyway, and a flexible management approach will be significantly more resilient. Tolerant approaches based on education and risk awareness will also prevent staff from finding workarounds and trying out risky unsanctioned products. Modern tooling also helps, providing dynamic insights into what is actually happening, as opposed to relying on dusty documentation. Nothing can prevent complexity, but your mitigation approach can prevent it jeopardizing business resilience.

Use flexible strategy to counter disruption and division

Acquisitions are a common, extreme cause of additional complexity, where another company’s entirely different infrastructure is grafted onto another’s, sometimes doubling the number of products, cloud providers, interfaces, networks and applications. With that doubling comes a significant increase in the number of potential attack surfaces, along with massive growth in service management. Often the two IT teams have all too little time to plan the merging of their two services, and in most cases, they will be contractually locked into many of their competing providers. A fragmented environment is inevitable at first, with gradual consolidation resulting in continual change over several years. 

Sometimes the lack of an event is a problem. For example, not creating a well-defined cloud strategy in an organization can also lead to fragmented infrastructure. Without guidance, different teams migrating apps to the cloud or building completely new apps will inevitably choose the provider that suits their own project needs best, or even use a pick-and-mix approach to selecting technologies from multiple companies. More than half of all organizations are already multicloud despite the need for extra skills. The result is convoluted data flows across multiple networks and multiple control planes to master.

Credit cards are also responsible for uncontrolled change. Freemium online services allow individuals or departments to sign up to new and unknown products without the knowledge, let alone approval, of IT or security teams. This freedom allows small-scale experimentation and can lead to significant innovation and greater productivity. It can also lead to data loss and unplanned costs, especially when usage grows to the point of being a burden on the person who kicked it off and they want to offload it to IT. 

At the opposite end of the spectrum are everyday changes due to small operational adaptations. The impact won’t seem that important to those making the changes, but they can ripple through the system: small new pieces of hardware, changes to database fields, authentication changes, automatic upgrades. Each slight change creates risk, changes in management, and potentially breaks something else further along the data supply chain.

At this point some people are bound to cry out that their systems are fully and accurately documented, with carefully-labeled, detailed Visio charts of their networks. Unfortunately, those charts show what the network used to be like, depending on when the version to which you have accessed was created and nobody has time to keep them up to date. As cynical as this view may be, it merely represents the reality of an organically changing organization: documentation is stale and only stagnant organizations have static infrastructure. Relying on documentation without validating it with those with current knowledge is hazardous.

Use federation to master fragmentation

All this complexity, fragmentation and duplication reduces digital resilience. The traditional approach to governance would be to try and make it all stop. Unfortunately, that approach is simply not going to work. The changes are almost always the result of someone trying to do their best — not the result of some malicious conspiracy, as too many senior managers like to think. 

The answer to the conundrum lies in moving from fragmented to federated. The only practical approach is to accept that change is for the best and cannot, indeed should not, be stopped. Attempts for strict control will fail. That does not mean that IT leaders shouldn’t protect certain infrastructure, but IT will always be evolving with the business and users it serves, and leaders should act accordingly.

Achieving digital resilience needs a new, flexible, and above all, active approach to change. This positive approach must start with education, ensuring that everyone from the board down understands that digital infrastructure is the foundation of every business, and that foundation needs defending against attackers and failure. Publish guidelines on using cloud services of all kinds that are easy to find and easy to follow. 

For more technical work, including software development, server and network configurations, automation can centralize responsibility. Deploying hardware configurations and the software running on them together enables us to set common, stable security rules that do not require human intervention. Easier and safer. 

Finally, observability tools enable network and security operations to monitor changes across the infrastructure they manage. No more relying on people updating those diagrams, but a live, dynamic view reflecting the true state of the network and what is connected to it.

Put these elements in place and your organization can begin to master digital resilience. And with that, overall business resilience.

Richard M. Marshall, the Principal of Concept Gap in Scotland, has over 30 years of experience in software design and innovation. He is a battle-hardened entrepreneur, has written hundreds of thousands of lines of code, launched products, raised funding, and spent 5 years as a Gartner analyst. He writes and lectures widely on software development, mobile, low code, web tools and AI with a flair for bringing the hype down to earth and helping clients become smarter innovation adopters. Richard holds a PhD in Computer Science from the University of Edinburgh. For fun he writes fiction and practices Parkour.