Applied AI Engineering
Cost, Latency & Reliability
AI features have systems constraints. Model choice, token count, retrieval size, retries, caching, streaming, batching, and fallbacks determine whether the feature is usable and affordable.
- Tokens are both cost and latency
- Model choice should match task difficulty
- Caching helps stable work
- Retries are not free reliability
- Fallback behavior is part of UX
| Lever | Reduces | Trade-off |
|---|---|---|
| Shorter context | Cost/latency | May lose useful evidence |
| Smaller model | Cost/latency | May reduce quality |
| Caching | Repeated cost | Freshness complexity |
| Streaming | Perceived latency | Client complexity |
| Reranking fewer docs | Latency | May miss evidence |
| Bounded retries | Tail failures | May return fallback more often |