Two questions are coming at AI decisioning from opposite directions, and most organizations are ready for only one of them. The first comes from a regulator: why did you decline this applicant? It concerns one case, and the answer has to hold up. The second comes from the board: can we answer that question every time? That one isn't about a case at all. It asks whether the answer was a one-time reconstruction or the predictable output of a system you designed on purpose.
Answering the first without being able to answer the second is where organizations get into trouble.
The Technical Limit Nobody Engineers Around
When a model explains its reasoning, it is not reading back a log. It is producing a plausible reconstruction of how it might have arrived at the output. That account may be accurate. There is no way to prove it.
That is a real limit on explainability, and it cascades: if the model's account can't be verified, a person's understanding of that account rests on the same unstable ground. No vendor is going to engineer it away in the next budget cycle.
What you can do is stop anchoring on the model and start anchoring on something governed, stable, and meaningful outside of it.
Explainability And Interpretability: A Sequence, Not Synonyms
The NIST AI Risk Management Framework lists "explainable and interpretable" as one characteristic of trustworthy AI, but the two words do different work.
Explainability: Answers how a decision was made in the system. It is the model's account of its own mechanism — a model capability.
Interpretability: Answers why, and what it means for the person on the receiving end. It is how a human makes sense of that account — a user capability.
"The model flagged a debt-to-income ratio of 42 percent against a 36 percent approval threshold" is explainability. "That ratio, not your credit history, is why the application was declined, and lowering monthly debt or documenting additional income will improve it" is interpretability. Same decision, different audience, different work, and the second does not happen just because the first exists.
Three Of The Four Requirements Are Governance Problems
NIST IR 8312 breaks explainability into four principles: explanation, meaningful, explanation accuracy, and knowledge limits.
Explanation (the system supplies evidence or reasons for its outputs) is necessary but nowhere near sufficient. Meaningful requires the explanation be understandable to its intended consumer, which takes deliberate design. So does knowledge limits, the requirement that the system operate only under the conditions it was built for. Those three are governance problems, and not one of them requires inspecting the inside of a model.
The fourth, explanation accuracy, splits in two. If accuracy means the explanation matches what the model actually computed, it is unprovable today. If it means the explanation is consistent, approved, and defensible, it is provable now, and the provable half is what regulators and courts are asking for.
Having a strong data decision helps you explain the data within your organization, with confidence on the accuracy.
The Courts Are Already Asking For The Provable Half
In Mobley v. Workday, the court allowed claims to proceed on the theory that an AI screening tool can act as an agent of the employer using it. A vendor's model does not shield the employer: you cannot point at the AI and say it wasn't you.
Then, on February 9, 2026, the Fourth Circuit revived class claims in the consolidated Navy Federal mortgage discrimination litigation. Plaintiffs allege the credit union's underwriting algorithm evaluates creditworthiness using variables and weights the company keeps secret, including from the applicants it denies. Nothing about liability has been decided; the claims were simply allowed to proceed.
Look at what that demands in discovery. Not the model's internals: a consistent, defensible account of what drove the decision. Precisely the attainable half of explanation accuracy.
The Decision Record
The artifact that produces that account is a Decision Record: documented before the build as functional requirements, referenced later as evidentiary proof. Six elements do the work.
- Sources consulted. The authoritative sources the system may draw from, and proof later that the decision was rooted in one.
- Data fields used. The permitted inputs, and by exclusion the fields never appropriate for influencing a decision.
- Model and version. Ties a specific decision to a specific, reproducible state of the system.
- Reason codes. The approved list of reasons and their definitions — what makes two applicants with the same issue get the same answer.
- Scope flag. The operating boundaries, confirming the system only answered what it was approved to handle.
- Human accountability. Who owns review and escalation, and where on the loop they sit.
Reason codes deserve attention, because the bar moves once the list becomes customer-facing. The check is no longer internal. It is not "can the decision steward trace this," it is "can someone with no inside knowledge read this and tell what to fix?" Debt-to-income ratio: 42 percent. Approval threshold: 36 percent clears that bar. We are unable to approve your request at this time does not, though both could sit on an approved list.
Where To Start
Pick one decision flow, not all of them. Write down who asks about it and what they actually ask. Try to write the answers today, before you rebuild anything. Whatever you cannot answer is the gap, and a gap you can name is one a Decision Record can close. Run that exercise on one decision flow this week. Most teams find they can explain a decision but can't prove they'd explain the next one the same way, and that gap is far cheaper to close before someone asks than after.
Schedule a discovery call with First San Francisco Partners (FSFP) and we'll walk your decision flow against the Decision Record together. Not ready to talk? Take the AI Readiness Assessment — 10 questions to see where your governance actually stands.
