Responsible AI
Interpretability & Explainability
Interpretability is understanding how a model works; explainability is communicating why a particular output happened. The need depends on risk, debugging needs, user trust, and regulation.
- Simple models can be inherently interpretable
- Global explanations are not local explanations
- Local explanation tools are approximations
- Explanations must fit the audience
- LLM explanations may be post-hoc narratives
- Evidence beats eloquence
| Type | Example | Useful for |
|---|---|---|
| Model-intrinsic | Small tree path, linear coefficients | Debugging and audit |
| Global post-hoc | Feature importance | Understanding broad behavior |
| Local post-hoc | SHAP/LIME for one prediction | Case review |
| Evidence-based | Source passages / tool traces | Grounded user explanation |
| Counterfactual | What would need to change? | Recourse and decision support |