Production ML & MLOps

Model Serving & Inference

Serving is the production path from request to prediction. It must satisfy latency, throughput, availability, versioning, security, observability, and rollback constraints — not just load a model artifact.
  • Batch inference optimizes throughput
  • Online inference optimizes freshness
  • Streaming improves perceived responsiveness
  • Versioning is non-negotiable
  • Serving hardware changes the economics
  • Prediction logging enables debugging
Inference modes
ModeUse whenTrade-off
BatchPredictions can be precomputedCheap and scalable, but stale
Online syncUser waits for resultFresh, but latency-sensitive
Async onlineResult can arrive laterMore resilient, more workflow complexity
StreamingPartial output helps UXClient/server complexity
Edge/on-devicePrivacy/latency constraintsModel size and update constraints
Serving path
Inference is one step inside a larger service boundary.
Sources
  • Made With ML — Serving
  • Rules of Machine Learning — ML systems
  • OpenAI API Documentation — Production best practices