llm-d 1.0: Open-Source Distributed Inference for Production

The release matters because of where AI spending is going. Building a capable model is now routine; running it economically, for many users, at scale, is not. As workloads shift from single chat prompts to multi-step agents that fan out across tools and reuse large amounts of context, the cost and complexity of serving has become the binding constraint. llm-d’s pitch is that a cluster of model servers can be turned into one coordinated serving pool, so the hardware an organisation already has goes further.

What llm-d is, and what 1.0 adds

llm-d is not an inference engine. It is an orchestration layer that sits above engines. The engines run a model on an accelerator; llm-d decides which replica gets which request, how many prefill and decode workers to run, and where the KV cache, the model’s memory of what it has already read, lives. It is engine-agnostic and hardware-agnostic, running across NVIDIA, AMD and Intel GPUs and Google TPUs, on public clouds and on-premises clusters, and it is Kubernetes-native, using components platform teams already know.

The project’s founding group is a signal in itself. Red Hat and IBM bring the enterprise open-source and Kubernetes lineage; Google Cloud brings the cloud platform and the TPU fleet; NVIDIA and CoreWeave bring the accelerators and the GPU cloud. Placing the project under the Cloud Native Computing Foundation, rather than under any one vendor, was a deliberate choice to keep the control plane neutral, and it is the reason one competitor’s silicon, Google’s TPUs alongside NVIDIA’s GPUs, can sit behind the same scheduler. The 1.0 release was announced at the PyTorch Conference, which underscores where the project sits: at the seam between the deep-learning runtimes that model authors build and the cloud-native systems that platform teams operate.

The 0.x releases were about reach: proving the stack ran everywhere. Version 1.0 is about the unglamorous work of running it day to day. It adds multi-tenant quality of service, so latency-sensitive requests and bursty agent workloads can share a fleet without starving one another; zero-downtime operations, with router high availability and rolling updates that keep traffic flowing through upgrades; a stable versioned configuration API and a hardened request path; and agent-aware serving, which keeps multi-turn conversations close to their cached context so agents reuse what they have already computed rather than recomputing it.

The problem: why inference is the bottleneck

The case for the project rests on a change in the shape of AI workloads. A year ago, most inference traffic was chat: one prompt in, one response out. Today, more of it is agentic. An agent loops through tools, carries a long conversation forward and reuses the same context repeatedly, and it does so in bursts. Run that behind a generic round-robin load balancer and the same problem appears at every scale: replicas recompute prompts they have already seen, accelerator memory fills with duplicate context, and one heavy agent stalls the interactive users next to it.

The technique that fixes this, disaggregation, splits inference into two phases with different appetites. Prefill reads the prompt and is compute-bound; decode generates tokens and is memory-bound. Separating them lets each scale on its own demand, but it only works if the KV cache moves between them fast enough, which is why the project leans on high-speed interconnects.

The mechanism, and why it is a layer, not a product

Teams can compose llm-d’s components instead of adopting the entire system. An organisation can keep its engine, its model and its cluster, and take only the serving pieces it needs. Underneath, it uses NIXL, the data-transfer library contributed through NVIDIA’s Dynamo project, to move KV cache blocks between GPU memory, host memory and prefill and decode pools. The routing is cache-aware: requests are steered to the replica that already holds their context, which is what turns a cache hit into a cost saving. Under Kubernetes, this composes with the autoscaling and gateway components a platform team already runs.

The economics: open weights on hardware you already have

The most consequential claim in the release is about money. The project cites a demonstration in which researchers served GLM 5.2, an open-weight mixture-of-experts model of roughly 753 billion parameters, on widely deployed NVIDIA GPUs, reaching more than 6.6 million output tokens a minute at peak and supporting up to 3,000 concurrent coding agents with no preemptions. At current cloud rental rates, the project says, the cost per token would have been five to ten times less than equivalent commercial API pricing, with the largest savings on the input-heavy traffic typical of agents. Separately, IBM Research reported serving coding and chip-design agents at a prefix-cache hit rate of about 98 per cent and a blended cost of roughly 0.11 dollars per million tokens, which it put at about 11.5 times lower than an all-proprietary configuration.

These are project and vendor figures, produced by the organisations that build the software, and they have not been independently reproduced at scale. But the direction is consistent with a broader shift. As open-weight models close the gap with proprietary ones for many tasks, the question moves from which model to run to how cheaply it can be served. Gartner’s own advice, in a separate forecast, is that high-cost agentic tasks belong on open-weight models whose serving capacity an organisation controls. llm-d is an attempt to make that control practical without hiring a distributed-systems team.

The adoption

The release is unusually specific about who is running it. Google Cloud has built its GKE Inference Gateway on llm-d’s endpoint picker and says it cut time to first token by more than 35 per cent in Vertex AI production while doubling its prefix-cache hit rate from 35 to 70 per cent. AWS reports up to a 70 per cent increase in tokens a second versus a standard vLLM deployment on SageMaker HyperPod. DigitalOcean uses it to serve a trillion-parameter model across AMD and NVIDIA hardware. Tesla and Red Hat report a threefold increase in output tokens a second on AMD GPUs. IBM Research, JetBrains, Mistral and Roblox all describe production or near-production use. That roster, more than any benchmark, is the evidence that the stack is real.

At the PyTorch Conference keynote, Google Cloud demonstrated the approach at a scale that shows the intent: 1,000 sandboxed agents on a GKE environment hitting a single llm-d endpoint across vLLM, SGLang and a mixed fleet of TPUs and GPUs, with cache-aware routing and tiered KV cache keeping each agent’s steps on the replicas holding its context while flow control queued bursty agent loops so interactive requests stayed fast. The project addresses this problem by handling many agents, shared context, and uneven bursts.

The competitive layer

llm-d does not sit alone. NVIDIA’s Dynamo is an orchestration layer for the same job, tightly integrated with NVIDIA’s stack, and it contributes NIXL, which llm-d also uses. KServe, the Kubernetes model-serving project, now consumes llm-d rather than competing with it. The vLLM production stack is a narrower option, tied to one engine. The genuine question for a buyer is not which orchestrator is fastest, since no published figure was measured on their hardware, but which fits their fleet: Dynamo for an NVIDIA-centric stack, llm-d for a multi-vendor Kubernetes environment. And for a team serving one model on one node, neither layer earns its keep.

What buyers should ask

For a platform team weighing it, the questions are practical. Does our fleet justify an orchestration layer at all, since below a certain scale a single vLLM instance is simpler and faster? Which accelerators do we actually run, and does the stack have a validated path for them? What operational costs do the extra moving parts create, and who manages the routing configuration? How stable is the API across minor releases, and how much re-qualification does an upgrade require? And what do the cost figures look like on our own workload, rather than in a vendor benchmark?

llm-d 1.0: What it signals

llm-d 1.0 is the moment an open-source AI infrastructure project stops being a promise and becomes something a platform team can standardise on. Its importance is less any single feature than the shape of the thing: a vendor-neutral control plane, governed by a foundation, that lets many engines run on many accelerators behind one scheduler. If it works as described, it does for AI inference what Kubernetes did for containers, making scarce and expensive hardware go further and loosening the grip of any one vendor on the serving layer. The claims are the project’s own, and the honest test is the one its adopters face: whether the savings show up on their own clusters.

That is why the open-versus-proprietary framing matters less than it might appear. llm-d does not argue that open models are better; it argues that the serving layer should be neutral, so that whichever model an organisation chooses, it can run on whatever hardware it has, at a cost it can control. If that layer becomes standard, the competition moves up to the models and the applications, and the infrastructure underneath becomes a commodity, which is precisely what happened with containers.

---