Model Serving
Running a trained model as a service that answers requests quickly, reliably and affordably, which means balancing latency, throughput and cost.
Also inference · deployment · latency · throughput · batching
Cooking one meal for a friend is easy. Running a restaurant kitchen that serves thousands of people quickly, without running out of stoves, is serving.
Where it fits
+All relationships as a list
How it works
Two numbers dominate serving:
- Latency: how long one request takes. Users notice anything beyond a fraction of a second; teams track not just the average but the slowest 1% (the “p99”).
- Throughput: how many requests per second the system handles.
They pull against each other. GPUs are fastest when processing many inputs at once, so servers batch requests together: throughput rises, but each request may wait for its batch. For large language models, which generate one token at a time, smart batching and caching are a large part of what makes them affordable.
Other everyday concerns:
- Online vs batch: answer each request live, or score a whole dataset on a schedule.
- Versioning and rollback: deploy new models gradually and switch back instantly if something breaks (see A/B testing).
- Efficiency: quantization and specialized hardware cut cost and latency.
Everyday use cases
- Case 01
Chat assistants
Generating tokens for many users at once requires careful batching on GPUs.
- Case 02
Real-time fraud checks
A card payment must be scored in tens of milliseconds.
- Case 03
Overnight batch scoring
Score every customer for churn once a day, where throughput matters more than speed.
Strengths, weaknesses and when to use it
Strengths
- Good serving makes the difference between a demo and a product.
Weaknesses
- Latency, throughput and cost pull in different directions; every setup is a compromise.
When to use itAs soon as a model needs to be used by anyone other than its developer.
Check yourself
- Q01
What usually happens when you batch more requests together on a GPU?
Show answer
A: Throughput goes up, but each request may wait longer (higher latency)