COZYING UP TO COMPLEXITY

COZYING UP TO COMPLEXITY

I ENJOYED SAMUEL ARBESMAN’S FIRST BOOK, The Half-Life of

Facts, which was a discussion of the exponential pace of change,

as exemplified by Moore’s Law, among other things. When I

saw the title of his new book, Overcomplicated, I assumed that

it would be a warning that we technologists had gone too far in creating

complex systems. It would advocate moving to simpler systems, just as

a doctor might advise an overweight person to go on a diet. I was prepared

to argue against such a conclusion, but as I discovered upon reading

the book, Arbesman does not say that complexity is necessarily bad

or that we should seek simplicity. Instead, he maintains that systems are

now unknowably complex, that they will become even more so, and we

should…just get over it. 􀂚Much of the book is spent in discussing the reasons

why complexity is inevitably increasing. Arbesman writes that “almost

everything we do in the technological realm seems to lead us away from

elegance and understandability, and toward impenetrable complexity

and unexpectedness.” He cites three main factors driving this increase—

“accretion,” “interconnection,” and “edge cases.” Accretion is the result

of large systems being built on top of smaller and older systems, often

via the incorporation of legacy code, producing what we call kludges. As

these subsystems become interconnected, the resulting entanglement

can change what was simply intricate to truly complex. Finally, complexity

is exacerbated by the inevitable existence of edge cases—that myriad

of individually negligible exceptions and rarities that yet constitute the

long tail of cases that must all be accounted for in system design. The

complexity resulting from these factors has passed a tipping point where

no single person can fully understand a complete system. 􀂚 At this point

in reading the book I’m thinking that this is all familiar territory to us

engineers. The question is: What do we do

about it? Arbesman passes quickly by our

usual strategies to manage complexity—

the abstraction of subsystems to hide the

complexity of lower levels, and “good

hygiene” in coding. Good things to do,

but they won’t solve the ultimate problem,

argues Arbesman. So what else is there?

Arbesman suggests two approaches.

The first is that we need to create more

generalists—ideally people with “T-shaped”

knowledge bases, those with a deep specialty

combined with more superficial

knowledge of a broad area. This has

become particularly important now that

most systems involve divergent specialized

fields. However, Arbesman acknowledges

that generalists are relatively useless

unless accompanied by narrow specialists.

Moreover, the market does not at present

support generalists. It was easier to

be a renaissance man in the Renaissance!

The other approach, which is dealt with

at some length, is that we need to think

more like biologists than physicists. A

physicist would be inclined to attempt a

mathematical analysis of the system, hoping

for an elegant solution that explains

and predicts behavior. On the other hand,

a field biologist, accustomed to the overwhelming

complexity wrought by evolution,

would work from the bottom up,

cataloging the appearance and behavior

of the systems found, hoping perhaps

to identify a new species or ecosystem.

I’ve been thinking about this biological

approach, but I’m not sure how much

insight it would provide. Some of our

bugs are like distinct new species, like

the “Goto” trap in programming, but the

overwhelming majority might be one-offs,

with little general relevance.

I agree, however, with Arbesman’s final

conclusion that we should celebrate the

functionality and sophistication of our

creations. Our achievements will necessarily

move beyond the understanding of

a single human. After all, evolution has

created us, creatures with astounding

functionality and resilience alongside

esoteric fragilities. Perhaps it will be the

same in technology. _