Applied AI Engineering

LLM Evaluation

LLM evaluation turns fuzzy behavior into repeatable evidence. It uses representative examples, rubrics, automated checks, model or human graders, regression tests, and production monitoring.

Start with examples from real usage
Separate evaluation dimensions
Automated graders need calibration
Regression evals protect prompts and retrieval
Production feedback closes the loop

LLM eval dimensions
Dimension	Example check
Format	Valid JSON matching schema
Grounding	Every factual claim supported by retrieved evidence
Task success	User goal completed
Safety	Refuses or escalates risky requests
Tool use	Correct tool, valid args, authorized action
Cost/latency	Within product budget

Evaluation loop

Good evals evolve with production failures.