The Yada Yada Paradox

Or why emergent anything is a cop-out

The output of a very simple computer program can be uncannily complex.

Such is the present state of AI. As we have covered before, the programs to create embeddings and transformers embody quite basic math concepts and apply them in quite simple ways. The AI jargon may be unfamiliar and the engineering details (mostly recipes worked out by trial and error) are clever. But from the point of view of their core algorithms, the way Large Language Models learn - how they use training examples drawn from the Universal Library to acquire their generative grammar, dictionary, and encyclopedia (see my previous post for the meaning of these terms) is not complicated.

But from this simplicity, iterated hundreds of billions or trillions of times, output emerges that is by any measure fantastically complex. Understanding why these machines excel at some human-like activities showing human-like or superior intelligence, while making frequent and persistent childlike mistakes at other tasks is not obvious at all. For example, surely GPT can memorize all sums of two integers between 0 and 99. But it still gets basic addition of two two-digit numbers wrong about 2% of the time. That is strange.

The core question for the study of AI interpretability is whether the cognitive gap between simple math and chaotic complexity can be filled by additional predictive or descriptive rules at some intermediate level much nearer to the observed behavior, or whether the simple rules are all there is. Then we declare the phenomena emergent and quit looking for a better model.

I first read about emergent complexity in Scientific American, in an article describing John Conway's Game of Life. This computer game starts with a simple grid of squares of any dimensions, with some random distribution of occupied and empty squares.

Apply four simple rules to each square. Repeat.

These are the rules.

From

https://playgameoflife.com

Through repetition of these simple rules, beautiful patterns with mysteriously complex and unpredictable behaviors begin to arise.

These little blobs in the middle are "spaceships" seemingly flying from left to right...while the patterns around them seem to pulsate with energy.

Watch a video with six-minutes of great examples set to dramatic music:

Cellular Automata

Cellular Automata

The class of computer programs of which the Game of Life is the most famous example are called cellular automata.

Cellular automata are discrete models for computation - meaning they are supposed to help us humans to understand computation. They were conceived of first in the 1950s by John von Neumann -- yes, the same guy who helped invent the atomic bomb and game theory, and in 1945 defined the structure of all digital computers

In his non-normative and underappreciated 2002 book, A New Kind of Science, Stephen Wolfram makes big claims for cellular automata as a model for the secret mechanism that "allows nature seemingly so effortlessly to produce so much that appears to us so complex."

"[S]ome of the very simplest programs that I looked at had behavior that was as complex as anything I had ever seen. . . . I had come to view [cellular automata] as one of the more important single discoveries in the whole history of theoretical science."

Computational Irreducibility

My fundamental frustration with cellular automata is that, by design, you cannot read their instructions and apply any sort of logic or heuristic to predict what behavior will emerge. Reading their code does not tell us what it produces when it is unspooled in time through repeated iterations, and in fact cannot tell us what the outcomes will be. Wolfram calls this cognitive gap between what the code explicitly says and what it eventually does "computational irreducibility."

The idea that there is often no shortcut but to run a program and observe its behavior in order to know what it will do is a basic discovery of mathematical logic at the heart of Turing's proof of non-decidability.

Wolfrom is mathematically correct - it is provable - that the future behavior of some rule-bound systems cannot be inferred from their rules. Instead, certain systems of code-plus-processing must be simulated step-by-step and simply observed.

We call the interesting part of the behavior emergent.

Use of this word often feels like a tremendous cop-out to me.

Emergent behavior of a system is behavior that first appears at scale. It is the result of aggregate behavior of many smaller components. What makes it emergent is that even when the small-scale behavior is fully transparent and apparently understood, the emergent behavior remains mysterious and feels unexplained. If it were explained at a higher scale, it would no longer be called emergent.

Wolfram's point is that we just need to get over it. We cannot know what the emergent behavior will be from study of the enabling algorithms alone - even when they are its sole cause. And inversely, if we observe the emergent behavior, we cannot hope to explain it more fully than with the simplest rules that cause it.

How Broadly Does the Principle Apply?

But is Wolfram reasoning by analogy in saying that nature, like a computer programs, can display emergent complexity from a few simple rules, with no explanatory rules of intermediate or higher complexity being necessary? Or is Wolfram saying that nature actually is a computer program, subject to the same logic as a Turing machine?

If it is the latter, he is not alone. Viewing nature as a computer, and physics as a science of information, did not originate with Wolfram. John Wheeler popularized the concept with his phrase "it from bit" in the 1980s, and many thinkers have since proposed some version of the idea that the universe itself is a computer whose fundamental minima is information.

Fifty years later, at the highest levels of sophistication, these questions remains open. For example, in 2015 the physicist Garardus 't Hooft, who won the 1999 Nobel Prize for his mathematical resolution of some problems in modeling the electro-weak force, published The Cellular Automata Interpretation of Quantum Mechanics, in which he suggested a path forward for an alternative theory to explain all the calculations of quantum mechanics, using the frame of universe-as-deterministic-computation to organize his ideas.

The Yada Yada Paradox

A successful law of nature can be thought of as a highly-compressed representation of many phenomena that makes accurate predictions possible, while also providing explanatory clarity.

The algorithms driving cellular automata indeed serve as a compressed encoding of the behavior, not merely predicting its broad currents, but determining it completely.

But this is not at all the same thing as providing explanatory power at the level of the emergent phenomena. The difference in scale between cause and observation is too great. We can operate the machine to produce the output, but watching and waiting to see what it will do does not feel like a satisfying form of comprehension of why and how that complexity emerges. The most compressed and comprehensive description imaginable can still fail to give insight into how and at what scale the interesting behavior will emerge.

It is not transparent, or interpretable, at the scale of phenomena we care about. It is not predictive. It leaves out what we want to know. But it is nevertheless as complete a description of the process as can be before it happens.

When an initial encoding of future phenomena is too perfect it becomes useless. This is the Yada Yada Paradox.

"We start with these four rules, Yada Yada. . . out comes life."

Towards a Science of AI

Either an intelligence we see in machines is emergent in the sense of computational irreducibility, or there are intermediate representations that can replace the black box with knowledge.

The concept of computational irreducibility can be pushed too far. It is much too soon to declare defeat in the quest for interpretability. I believe that much of the intelligence of today's LLMs can be re-implemented with architectures that are far less mysterious. What was once labelled emergent and assumed intractable will yield to more explicit models, and much of the beautiful and profound mathematical architecture of the generative grammar, dictionary, and encyclopedia of the Universal Library can and will be discovered.

This back-filling of understanding, chasing after current powers and functions, is not unusual: for example, we had steam engines and the industrial revolution before we had a coherent theory of heat. As a civilization, we are at the threshold of understanding what we have wrought, and with understanding can come a safer, more consistent, and more universal application of those discoveries.