llm-d

llm-d is a high-performance distributed inference serving stack for production Kubernetes deployments. The pitch is layered: model servers like vLLM and SGLang already run models efficiently on accelerators, and llm-d adds the orchestration above them, intelligent prefix-cache and load-aware request routing (including experimental latency-predicted scheduling), advanced KV-cache management with tiered prefix caching, and disaggregated serving, to squeeze maximum throughput and minimum latency out of real-world traffic.

It is a CNCF sandbox project founded by Red Hat, Google Cloud, IBM Research, CoreWeave, and NVIDIA, which explains the enterprise-grade posture: well-tested deployment guides ('well-lit paths'), real-world benchmarks, and broad accelerator support. At 4,800+ stars it is the serious answer when a single vLLM pod stops being enough. The cost of admission is Kubernetes itself: this is infrastructure for platform teams, not a pip install for your laptop.

1. Provision a Kubernetes cluster with GPU/accelerator nodes.
2. Follow a 'well-lit path' guide at llm-d.ai/docs for your model and hardware.
3. Deploy the llm-d stack (it orchestrates vLLM/SGLang model servers underneath).
4. Configure intelligent routing and prefix-cache policies for your traffic shape.
5. Validate with the provided benchmarks before cutting over production traffic.