Deploying LLMs in Production

Moving an LLM from a notebook to production means choosing how it gets served, how requests get scheduled, and what happens when things go wrong. This guide covers the API versus self-hosted decision, the inference servers that power self-hosted deployments, the latency and throughput numbers that matter, and the operational plumbing around them.

API versus self-hosted

Calling a provider API is the fastest path to production. No GPUs to buy, no drivers to maintain, scaling handled by someone else. The tradeoffs are cost at high volume, data leaving your infrastructure, latency you cannot control, and dependence on one vendor's uptime and pricing changes.

Self-hosting means running the model on your own hardware or rented GPUs with an inference server. You get data control, predictable cost at steady volume, and the ability to tune and customize. You also get every operational problem: GPU procurement or rental costs, driver and framework upgrades, autoscaling, and on-call duty when the server falls over at 3 AM. The break-even point varies by model size and request volume, so do the math with your actual traffic rather than rules of thumb.

Most teams start on APIs and move specific workloads to self-hosted when the bill or the compliance requirements justify it. A common pattern is API for the flagship model on hard tasks, self-hosted smaller models for high-volume simple tasks.

Inference servers

Three open serving stacks dominate self-hosted LLM deployment.

vLLM is the most widely used. Its core innovation is PagedAttention, which manages the key-value cache in fixed-size memory blocks instead of one contiguous buffer per request, cutting memory waste dramatically. It uses continuous batching, where new requests join an in-progress batch as soon as a slot opens instead of waiting for the whole batch to finish. It exposes an OpenAI-compatible HTTP API, supports quantization, speculative decoding, tensor parallelism, and runs on NVIDIA, AMD, Intel, and CPU hardware.

Hugging Face Text Generation Inference (TGI) is a Rust and Python server with continuous batching, built for tight integration with the Hugging Face ecosystem. TensorRT-LLM is NVIDIA's stack: it compiles models into optimized engines for NVIDIA GPUs, with in-flight batching, at the cost of a longer setup and compile step. SGLang is a newer option from the same research community that has gained adoption for its structured generation and serving features.

Try them with your actual model and workload. Benchmarks from other people's setups do not transfer, because performance depends on model size, quantization, sequence lengths, and concurrency patterns.

Latency and throughput basics

Two numbers describe generation performance. Time to first token (TTFT) is how long until the response starts streaming. Tokens per second is the streaming speed once it starts. Interactive chat cares about both. Batch pipelines care mostly about tokens per second across the whole run.

Generation has two phases. Prefill processes the whole input prompt at once; it is compute bound and determines TTFT. Decode generates one token at a time; it is memory bandwidth bound and determines tokens per second. Long prompts make prefill expensive. Long outputs make decode expensive. This is why caching the prompt prefix helps TTFT so much: it skips the prefill work.

Continuous batching raises throughput by keeping the GPU busy at every step. The tuning knobs are the maximum concurrent sequences and the maximum tokens per batch step. Set the concurrency limit too high and KV cache memory overflows, causing requests to be evicted and recomputed. Start conservative, load test, and watch GPU memory.

Caching

There are two distinct caches. Prompt caching happens at the API or server level: repeated prompt prefixes reuse computed state instead of reprocessing. Anthropic supports explicit cache breakpoints; OpenAI applies prefix caching automatically. Semantic caching happens in your application: store the response to a query, and when a semantically similar query arrives, return the cached answer instead of calling the model. Semantic caching helps when users ask the same questions in different words. It needs a similarity threshold tuned against your data, and it is dangerous for anything where a stale or slightly wrong answer matters.

Cache prompts by putting static content first and dynamic content last, keeping the prefix byte-identical across calls. A timestamp at the top of a system prompt silently destroys cache hits.

Fallbacks and resilience

Models fail in ways that need handling. Provider outages, rate limits, timeouts, and content filters all produce failed or unusable responses. Design for it. Set timeouts on every call and treat a slow model as a failed one. Keep a fallback chain: primary model, then a cheaper or different provider's model, then a degraded response that is honest about being degraded. Retry with backoff on rate limits, not immediately.

Guard the request path. Validate inputs before they reach the model, cap output token counts, and put a per-request cost ceiling in place so one pathological input cannot burn the budget. If your application chains multiple model calls, the chain fails when any link fails, so apply the same fallback logic per step.

Monitoring

You cannot improve what you do not measure. Track latency percentiles (p50 and p99, not just the average), token counts per request, cost per request, error rates by type, and cache hit rates. Log the prompt, the response, and the model version for every production call, with a retention policy that fits your privacy requirements. Those logs are the raw material for regression tests, judge calibration, and debugging.

Set alerts on error rate spikes, latency degradation, and cost anomalies. A prompt change that doubles output length will show up first as a cost alert. Review a sample of live traffic weekly with human eyes; distributions drift, and your evaluation suite from three months ago may not cover what users are asking today.

A sane default setup

For a team shipping their first production LLM feature: start on a provider API, pin the model version so upgrades are deliberate, add a regression test set that gates prompt changes, set timeouts and a fallback model, log everything, and monitor cost per request. Move to self-hosted only when you can name the specific reason: the monthly API bill, the compliance requirement, or the latency SLA. Each reason points to a different architecture, so pick the reason before picking the stack.