Designing Complex Systems

What kind of world and nation do we want? What are the realistic possibilities from which we are constrained to choose? Already these are enormously difficult questions. Perhaps foreign relations are the trickiest questions. Our present way of living is deeply intertwined with people all around the world. One simple measure of this is that we import about two trillion dollars of goods, while our total GDP is about fifteen trillion dollars. The more our vital interests are involved with other nations, the more we will find ourselves driven to protect those interests. This activity powerfully shapes our government. On the other hand, it's not as if utter isolation is a real option.

A similar spectrum appears at the national level. Perhaps the Tea Party is a revival of the Confederate ideas that sparked the nineteenth century Civil War. Our public and private institutions are so international that the boundaries between the fifty states are almost invisible. To what extent it could be possible to reconnect institutions to smaller regions, to states or even to cities... it is not clear how this could be done and what the consequences would be. The economic principle of comparative advantage is one demonstration of the more general maxim, "United stand, divided we fall." Yet surely that maxim must have its limits.

What we have now is a way of living that is dominated by global institutions, public and private. It surely makes good sense to avoid concentration of power in any institution, but the protection of our vital interests and the regulation of interactions among such huge players surely requires government at a similar scale. In a different world a small scale government might be effective. In today's world, the only possible government is a large and complex government. The challenge is to find a way to make such a government efficient and effective.

In this project, we don't have to start with a blank sheet of paper. There are many global scale institutions and other systems of great complexity. The pragmatic approach is to study what works elsewhere and what doesn't work.

There are two aspects to building complex systems. The system itself will have some structure, e.g. incorporating hierarchy and repetition. The other aspect is the process of design. Just like there is no single system structure that can serve all purposes, similarly the design process needs to fit the specific problem at hand. Still, there are some generally useful design methods that can be combined and adapted to fit most any problem.

Complex system design generally takes place in a series of phases. Design can be thought of as a series of decisions. The idea behind the phases is to order the decisions as well as possible. The main principle is that a decision is best made when the consequences of the alternatives are as clearly known as possible. For a trivial example, the size and shape of a container is generally best decided upon after the sizes and shapes of the contents are already known, otherwise the contents might well not fit in the container, or it could be unnecessarily large. The usual first phase of designing a complex system is determining what it will be required to do and the constraints put on it by whatever external factors.

Another methodological principle is to identify and resolve risks early. The failure of a crucial component can prevent the system from performing its intended function. The less experience designers have in working with such components, the higher the risk of a design error. In such cases it may be worth while to build prototypes as a way to gain experience before scaling up to the full system.

Complex systems are generally designed by large teams with diverse expertise. Transparency and accountability are necessary in the design process in order for this diversity to remain a strength. For the designed system to cohere, effective communication must be maintained between the various design teams. Hidden decisions become hidden problems that can trigger catastrophic failures.

Any complex system will involve risk and novelty: each unique arrangement of components creates fresh opportunities for unforeseen interactions. To work effectively, a complex system must incorporate ways for its design to be changed as problems are detected, or as system requirements shift. The processes of design, implementation, and deployment cannot be kept completely distinct.

Separation of concerns is a principle that can apply both to system structure and to the design process. If every component of the system is tightly coupled to every other component, the system will be extremely brittle. Most likely some interactions will not be handled correctly and the system won't even work from the start. In any case, any small change in requirements will lead to updates rippling through the whole system, at prohibitive expense. The notion with separation of concerns is to reduce the degree to which system components constrain each other, to reduce the level of coupling between them. The detailed operation of each component can be largely independent of the details of other components, as long as a few basic requirements are met. Changes in the design details of one component can have very little or no impact on the requirements of other components. This lets design teams work more independently with many fewer iterations of rippling changes, and then during implementation and deployment problems can also be identified and fixed with minimal impact.

I starting thinking about government as a complex system when I started hearing about the problems with earmarks in legislation. It seems many such clauses are introduced into bills at the last minute. Legislators vote on bills whose contents they cannot effectively track. This is very much like computer software when configuration management tools are not used. Last minute changes are notoriously buggy but developers always want to get the latest fixes and features into the next product release. The software industry has worked out some effective methods to manage this problem. Perhaps the government can learn something!