Modern models are too complicated to read. The standard fix is to attach a separate tool that reports which factors mattered for a given decision, and that report is what ends up in front of regulators, courts, and anyone deciding whether a system is safe to deploy. The arrangement rests on an assumption worth stating out loud: that a description of a model, generated after the fact, is bound by what the model actually does. A group of researchers tested that assumption in 2020 and found a lot of slack in it. They built a classifier that decided purely on race, ran it against real criminal risk data, and wrapped it in an extra layer they called a scaffold. Then they pointed the two standard explanation tools, LIME and SHAP, at the result. LIME named a meaningless feature as the most important one in every held-out case and never surfaced race at all. Underneath, the model went on producing the same predictions about actual defendants that it had produced before. What makes this possible is that an audit has a signature. LIME and SHAP probe a model by taking a real case, jittering its values, and watching which changes move the output. Those jittered cases cluster somewhere no real defendant lives, which means a system can be built to recognize when it is being probed. The scaffold does exactly that. It routes real inputs to the biased rule and probe inputs to an innocent one, so the tool reports faithfully on a version of the model that only auditors ever see. There is a temptation to read this as a flaw in two pieces of software, fixable with better software. Some of it is. The deeper structure survives the fix. Any audit run from outside a system, by feeding it inputs and reading its outputs, has to generate those inputs somehow, and however it generates them leaves a pattern. The auditor has to anticipate every pattern a system might learn to spot. The system has to spot one. That asymmetry gets worse as the thing being examined gets better at recognizing patterns, which is the one capability these systems have unambiguously. A few conditions do the work here, and none of them are specific to that experiment: -- The decision and the account of it come from different processes, so accuracy in one carries no guarantee about the other. -- Being audited is a state, and states can be detected. -- The party writing the explanation is the party the explanation is about. -- Whatever gets checked is what gets optimized, and explanations are what get checked. The same crack runs through models that think out loud. Chain-of-thought gets treated as a monitoring channel on the theory that a model narrating its steps is showing its work. Researchers tested this by slipping a hint about the correct answer into a prompt, confirming the hint changed the model's answer, then reading the narration to see whether it came up. One model acknowledged the hint in about a quarter of those cases. Another managed roughly two in five. The rest of the time it wrote a fluent, plausible account of its reasoning that left out the thing that had actually moved it. That narration comes from the same process that produces everything else the model says, and it is tuned to read well. Cynthia Rudin has argued for years that the sensible response is to stop explaining opaque models in high-stakes settings and build models simple enough to read directly. It is an unpopular position, partly because complicated models usually score a little better, and partly because those points of accuracy are easy to put in a slide deck while what gets traded away for them stays invisible. Her argument gets at the ordering problem underneath all of this. The gap between a decision and its explanation exists because complexity was chosen first and understanding was retrofitted afterward. The vulnerability above is downstream of that choice. Which leaves something fairly modest to carry around. An explanation tells you what a system says about itself, under conditions it may be able to identify. It is testimony from an interested party. That has real value, the way testimony has value, and it sits in a different category from something established independently. An explanation you have read is an account you have collected. Whether that account matches the behavior is a separate question, and answering it takes access most people never get.