Error propagation loops

Think of three quantities: how often the AI is wrong, how many of the people and agents around it are currently carrying one of its errors, and how much of the record system is contaminated by errors that got written down. The flows between them are few enough to name. Errors are adopted by operators. Errors are written into records — directly by the system, or indirectly by a person documenting what they believed. Contaminated records are read and believed by other people. Contaminated records are retrieved by the model and repeated as fresh output. And an operator's own framing can bias the model's next answer.

Three of these flows close into loops. Records re-infect people who read them. A model that retrieves from its own past mistakes repeats them with fresh confidence. And people who have adopted an error can prompt the model in ways that confirm it. This is the epidemiological insight at the heart of the EMU framing (EMU: Error–Memory–User): error spreads and persists the way an infection does, through contact and reservoirs.

The practical consequence is that the most effective governance points are often not at the model. Cutting a loop — verifying before a record is written, labeling unverified records, giving the reviewer enough capacity to actually catch things — can matter more than reducing the model's raw error rate, because loops are what give errors a future.

The PAN Lab makes these flows visible: you can watch where a governance choice dampens a pathway, caps it, or redirects it — and where a well-intentioned choice amplifies one instead.