Field Guide / Governing adaptively
Self-limiting vs self-sustaining error
In any system with propagation loops there is a threshold. Below it, errors are self-limiting: corrections outpace spread, and mistakes fade. Above it, errors are self-sustaining: each error seeds more than one successor through people and records, and contamination persists no matter how good any single day looks.
Epidemiology's most useful gift to AI governance is a single question: when an error occurs, does it on average cause less than one downstream error, or more than one? Less than one, and the system cleans itself — mistakes happen, then fade. More than one, and errors compound: people adopt them, records store them, retrieval repeats them, and the system settles into a contaminated equilibrium that no individual correction fixes.
The threshold reframes what governance is for. The goal is not zero errors — no ceiling allows that — but a self-limiting regime: correction capacity that outpaces spread. And the threshold is a property of the whole arrangement, which is why system-side levers (verification before writes, reviewer capacity, provenance labels, loop cuts) can move a deployment across it when model improvements alone cannot.
The threshold also explains why deployments surprise their owners. Growth — more users, more automation, more record coupling — adds propagation paths. A system that was comfortably self-limiting at pilot scale can cross into self-sustaining territory simply by succeeding, with no change to the model at all.
The PAN Lab's central gauge is exactly this: watch which side of the threshold a scenario sits on, and which combinations of governance choices move it — in both directions.
This page is conceptual framing — a way of seeing, not an empirical claim. Documented real-world events appear in the Domain Atlas with citations; testable versions of these ideas live in the PAN Lab.