AI/ML Foundations
Metrics & Evaluation
Metrics convert model behavior into decision-making evidence. The right metric depends on the cost of mistakes, class imbalance, thresholds, slices, and the way the output is used.
- Accuracy is only safe when classes and mistake costs are balanced
- Precision answers: when the model says positive, how often is it right?
- Recall answers: of all real positives, how many did we catch?
- Thresholds are product decisions
- Evaluate slices, not just aggregate score
- LLM evaluation is multi-dimensional
| Metric | Question it answers | Watch out |
|---|---|---|
| Accuracy | How often correct overall? | Misleading on imbalance |
| Precision | Can I trust positive predictions? | Can hide missed positives |
| Recall | Did we catch the positives? | Can hide false alarms |
| F1 | Balance precision and recall? | Assumes equal importance |
| PR-AUC | Ranking quality for rare positives? | Harder to explain |
| Calibration | Do probabilities mean what they say? | Different from accuracy |