But How Do It Know By John Scott

Understanding the Core Problem: How Systems Make Decisions

The question "but how does it know" is the one that comes up constantly when people encounter black-box technology. Whether you are debugging a model, auditing a pipeline, or just trying to explain to your boss why a recommendation engine showed what it showed, you need a systematic way to trace the answer. That is where the framework popularized under the name But How Do It Know By John Scott becomes useful. It is not a magic tool. It is a methodology. The framework breaks down into three layers: trace, verify, and explain. Most people skip straight to explain because that is what stakeholders want to hear. That shortcut is why things fall apart. The trace layer asks which inputs actually moved the needle. You look at feature importance, attention weights, gradient signals, or whatever diagnostic your system exposes. The verify layer checks whether those signals are stable across different runs and data subsets. A feature that looks important once but vanishes on the second epoch is noise, not insight. The explain layer is where you translate the technical evidence into language your audience understands without lying about what the model actually did. I spent about six months working through this with a classification model for internal fraud detection. The model was flagging roughly four percent of transactions, and the compliance team wanted to know why one particular batch kept getting rejected. I ran SHAP values first, then cross-validated them against permutation importance. The initial feature rankings were inconsistent between the two methods, which told me the model was relying on a correlated feature cluster rather than a single causal signal. I ended up isolating the correlation by dropping the redundant features one at a time and re-running the diagnostics. That gave us a clean explanation to hand off to the review board.

When the Framework Actually Fails

It does not work in every scenario. If your system uses deeply embedded representations like transformer attention patterns or multi-head attention mechanisms, the trace layer can produce outputs that are mathematically valid but practically useless. Attention weights are not the same as causation. I encountered this with a text classification model where the attention visualization pointed at entire phrases that looked meaningful but were actually artifacts of tokenization choices rather than semantic signals. In those cases, you need alternative diagnostics like integrated gradients or occlusion-based methods. The framework is honest enough to admit that limitation, which is more than most explanations do. Another edge case is real-time systems with high dimensionality. When you are dealing with thousands of features and sub-second inference requirements, running full trace diagnostics on every prediction is not feasible. You end up sampling or approximating, and approximations introduce their own errors. I found that batching the diagnostic runs at five-minute intervals and interpolating between them gave acceptable coverage without killing latency. That workaround added maybe twenty seconds of overhead per batch, which was fine for our use case but would not work for anything latency-sensitive.

How to Apply It Without Wasting Time

Start by listing the question you actually need answered. Not the generic question. The specific one. "Why did transaction 44921 get flagged" is answerable. "Why does the model behave the way it does" is not. A specific question constrains the trace layer and prevents you from falling into analysis paralysis. Once you have the question, pick the diagnostic tools that match your system's output format. If you have access to gradients, use them. If you only have predictions, you are limited to perturbation-based methods, and those take longer to run. Be aware of that constraint before you start. Documentation matters more than people expect. I keep a simple log for each trace run: what question was asked, what method was used, what the top signals were, and whether they held up under verification. After a few weeks, that log becomes a reference that saves you from repeating the same diagnostics. It also helps when someone later asks a follow-up question that seems new but is actually the same problem in different clothing. Most of the questions I get after the initial investigation turn out to be variations on the original trace.

What Beginners Miss

The biggest mistake is assuming that one diagnostic method gives you the full picture. Feature importance from one technique rarely aligns perfectly with another, and that misalignment is usually the most informative part of the result. When two methods disagree, something about the data structure is worth investigating. Correlation, interaction effects, or distribution shift are the usual suspects. I would rather spend an hour understanding why SHAP and permutation importance disagreed than trust whichever one looked nicer on the first pass. A second mistake is stopping the trace too early. People see a strong signal for the top feature and assume the job is done. But the top feature might be a proxy for something else entirely. I once traced a hiring model and found that education level was the strongest predictor. The model was not discriminating on education directly. It was picking up on geographic proxies that correlated with the education variable. Without digging deeper into the feature relationships, the explanation would have been technically correct and substantively wrong. That distinction matters when the explanation has real consequences.

Practical Limitations You Should Know About

The framework requires access to internal model signals. If you are working with a closed API where you only get predictions and no feature-level diagnostics, your options are limited to input-output testing and perturbation analysis. That is slower and less precise. There is no workaround for that constraint except negotiating better access with the system provider or building a proxy model that approximates the black box behavior for diagnostic purposes. Proxy models introduce their own errors, so treat any findings from them as directional rather than definitive. The method also assumes your system is deterministic enough to reproduce results. Stochastic models with high randomness can produce different diagnostic outputs on different runs even with the same input. In those cases, you need multiple traces and statistical aggregation to separate signal from variance. I usually run at least five traces and look for consistency across them. If the top signals flip between runs, the model is too unstable for reliable explanation, and you should focus on improving stability before investing time in diagnostics.

Where to Go From Here

If you want to explore the framework further, search for But How Do It Know By John Scott online. The material is available through various channels depending on what format you prefer. The core ideas are what matter more than where you find them. Start with a small, concrete question from your own work. Run the trace. Verify it. Explain it to someone who knows less about the system than you do. If they can follow the explanation without you having to add qualifiers, you have done the work correctly. If they cannot, go back to the trace layer and find what you missed. That loop is the entire framework in practice.