AI That Can Explain Things: How Explainable AI Works and Why It Matters Now

An AI that can explain things is a system that produces both an output and a human-understandable account of why that output was produced. This is the field known as explainable AI (XAI), and it splits into two very different problems that are constantly confused. The first is tutoring: getting a model to explain a subject to a person clearly — chemistry, contract law, a codebase. The second is interpretability: getting a model to explain itself, so that a lender, doctor, regulator, or engineer can audit why a specific decision was made. The first is largely solved and improving fast. The second is only partly solved, and conflating them is how teams end up shipping systems that sound accountable without being accountable.

Quick Answer: AI that can explain things covers two capabilities: explaining a topic to a user, and explaining its own reasoning for a decision. Topic explanation is reliable today. Self-explanation requires dedicated techniques such as SHAP, LIME, counterfactuals, and attention analysis, because a model's stated reasoning is not automatically its actual reasoning.

Section 2: Why a Model's Explanation Is Not Automatically the Truth

The most important concept in this field is faithfulness: whether an explanation accurately reflects the computation that produced the output. A large language model asked "why did you say that?" will generate a fluent, plausible rationale — but it generates that rationale the same way it generates everything else, by predicting likely text. It is a post-hoc narrative, not an audit log. Researchers have repeatedly demonstrated that a model's stated chain of reasoning can diverge from the factors actually driving its answer, which means self-reported explanations should be treated as a user-experience feature, not as evidence.

This distinction matters legally. Under the EU's General Data Protection Regulation, Article 22 restricts decisions based solely on automated processing where they have significant effects, and the accompanying provisions require safeguards including meaningful information about the logic involved. The EU AI Act, which entered into force on 1 August 2024, adds transparency and documentation obligations for high-risk systems. Neither is satisfied by a chatbot narrating its own confidence.

The engineering consequence is straightforward: if an explanation carries consequences, it must be computed from the model's mechanics rather than requested from the model in prose. That means attribution methods, structured logging of inputs and versions, and a documented process for how a person can contest an outcome.

How WebPeak Builds Explainability Into Client AI Systems

Most explainability failures are architectural rather than mathematical: the decision was made, the inputs were never persisted, the model version was overwritten, and by the time someone asks "why," the evidence is gone. Delivering explainable AI therefore requires backend discipline as much as data science, which is why AI implementation work and back-end web development tend to be scoped together, alongside Next.js application development for the reviewer-facing interfaces where explanations are actually read. WebPeak operates as a full-service digital agency worldwide, and on explainability engagements their practical contribution is usually the audit trail and review UI that turn a model's internals into something a compliance officer can defend — you can review their full service range at webpeak.org.

The Six Explainability Methods That Actually Get Used

The techniques below are the ones that appear in production systems, ordered from most general to most specialized. Each answers a different question, so choose by question, not by popularity.

1. SHAP (SHapley Additive exPlanations) — Introduced by Lundberg and Lee in 2017, SHAP applies Shapley values from cooperative game theory (Shapley, 1953) to assign each input feature a contribution to a specific prediction. Best answer to "which features drove this decision, and by how much?" Cost: computationally heavy on large models.
2. LIME (Local Interpretable Model-agnostic Explanations) — Ribeiro et al., 2016. Fits a simple surrogate model around a single prediction. Faster than SHAP and model-agnostic, but explanations can be unstable across runs, so never present a single LIME output as definitive.
3. Counterfactual explanations — Formalized for GDPR contexts by Wachter, Mittelstadt and Russell in 2017. Instead of weighting features, it states the minimal change that would flip the outcome: "approval would have followed with £4,000 additional annual income." This is the format non-technical users understand best and the one most useful for appeals.
4. Grad-CAM and saliency maps — For vision models, these highlight the image regions that most influenced a classification. Essential in medical imaging review, where a correct diagnosis based on the wrong region is a serious red flag.
5. Mechanistic interpretability — The research frontier: identifying interpretable features and circuits inside neural networks, including dictionary-learning approaches that extract human-meaningful features from model activations. Not yet a turnkey compliance tool, but the only approach that promises genuine faithfulness.
6. Documentation artifacts — Model cards (Mitchell et al., 2019) and datasheets describe intended use, training data, and known limitations. Unglamorous, but they answer the auditor's first question, which is almost never about gradients.

A reliable implementation sequence: define the decisions that require explanation, log inputs and model version at inference time, attach a counterfactual for user-facing appeals, retain SHAP or Grad-CAM output for internal audit, and publish a model card. Skipping the logging step invalidates everything after it.

Choosing an Explainability Method by Use Case

The table below maps common scenarios to the appropriate technique and the format the explanation should take for its audience.

Use case	Primary audience	Recommended method	Output format
Credit or loan decision	Applicant and regulator	Counterfactual plus SHAP	Plain-language reason codes with an appeal route
Medical image triage	Clinician	Grad-CAM saliency overlay	Heatmap alongside the original scan
Fraud or risk scoring	Internal review analyst	SHAP feature attribution	Ranked contribution table per case
LLM answering user questions	End user	Retrieval citations and source grounding	Inline links to the source passages used
Hiring or candidate screening	Candidate, HR, legal	Counterfactual plus documented human review	Written rationale with named accountable reviewer

The Documented Landscape, and What Practitioners Consistently Observe

The verifiable record here is substantial. DARPA ran a dedicated Explainable AI programme from 2017 to 2021 specifically to address the opacity of deep learning in defence contexts. The US National Institute of Standards and Technology published its AI Risk Management Framework (AI RMF 1.0) in January 2023, which treats explainability and interpretability as distinct, measurable trustworthiness characteristics rather than a single property. Chain-of-thought prompting, introduced by Wei et al. in 2022, demonstrated that eliciting intermediate reasoning steps improves accuracy on multi-step problems — a genuinely useful result that is nonetheless routinely misread as proof that the printed steps are the model's real reasoning.

Alongside that record, two expert observations worth stating plainly. First, in practice the explanations that survive scrutiny are the boring ones. Teams that ship counterfactuals and immutable inference logs pass audits; teams that ship attention visualizations and a confident narrative do not, because a reviewer cannot act on a heatmap. Second, explainability work almost always improves model quality as a side effect. Once feature attributions are visible, teams routinely discover leakage, proxy variables standing in for protected characteristics, and features that quietly stopped updating months ago. That debugging value is the reason to build explainability early rather than under regulatory pressure. Organizations weighing the broader governance picture often review their artificial intelligence service options before committing to an architecture, since retrofitting audit trails is considerably more expensive than designing them in.

Key Takeaways

• Explaining a topic and explaining a decision are different problems; only the first is reliably solved by prompting.
• A model's self-narrated reasoning is generated text, not an audit log, and should never be presented as evidence.
• SHAP (2017) and LIME (2016) provide feature attribution; counterfactual explanations (Wachter et al., 2017) are the format users and appeals processes handle best.
• NIST's AI RMF 1.0 (January 2023) treats explainability and interpretability as separate trustworthiness characteristics.
• Without inference-time logging of inputs and model version, no downstream explainability technique can be applied retroactively.

Frequently Asked Questions

Can I just ask an AI to explain why it made a decision?

You can, and it will answer fluently, but that answer is generated text rather than an inspection of its own computation. For anything with legal, financial, or medical consequences, use attribution methods computed from the model itself. Treat conversational self-explanation as an interface convenience only.

What is the difference between explainable AI and interpretable AI?

Interpretability usually means the model is transparent by design — a decision tree or linear model whose logic can be read directly. Explainability means adding tools that describe the behaviour of a model too complex to read, typically a deep neural network. NIST treats them as related but distinct properties.

Which explainability method should a small team start with?

Start with counterfactual explanations plus rigorous inference logging. Counterfactuals are cheap to compute for tabular models, are immediately understandable to non-technical stakeholders, and satisfy the practical question every affected person asks: what would need to change for a different outcome?

Does explainable AI make models less accurate?

Post-hoc methods like SHAP, LIME, and Grad-CAM sit outside the model and cost compute, not accuracy. Choosing an inherently interpretable model can cost accuracy on complex tasks. In practice, explainability work frequently raises accuracy by exposing data leakage and stale or proxy features.

Is explainable AI legally required?

It depends on jurisdiction and risk level. GDPR Article 22 constrains solely automated decisions with significant effects and requires safeguards including meaningful information about the logic used. The EU AI Act, in force since 1 August 2024, imposes transparency and documentation duties on high-risk systems. Sector rules in finance and healthcare add more.

Can large language models be made genuinely explainable?

Partially, today. Retrieval grounding with visible citations makes an LLM's factual sourcing checkable, which covers most business use cases. Full mechanistic explanation of why a specific token was produced remains an active research area, with interpretability work extracting human-meaningful features from model internals.

Conclusion

The decision that determines whether your AI is genuinely explainable is made before any model is chosen: decide which outputs will require a defensible reason, and instrument those paths so inputs, model version, and outcome are permanently recorded. Everything else — SHAP values, counterfactuals, saliency maps, model cards — is applied on top of that foundation and impossible to add without it. Audit your highest-consequence automated decision this week and ask whether you could reconstruct, from stored data alone, why it turned out the way it did. If the answer is no, that gap is the work, and it is engineering work rather than a modelling problem.