IBM, Red Hat and Google release llm-d 1.0, an open system for running SI models on servers
Prepared by Sintelligent with the help of SI. Published in SI: . Updated: .
Running large models on your own servers is getting cheaper without writing a custom system for it.
According to IBM Research, version 1.0 gets a stable API, support for multiple tenants with different priorities on shared infrastructure, and upgrades without downtime. The main features are routing that is aware of the prefix cache, KV-cache offloading, batch inference, prefill/decode disaggregation and wide expert parallelism. The project is led by Red Hat, IBM and Google with CoreWeave and Nvidia taking part; it is hosted at the CNCF as a sandbox project. According to IBM, a test with agentic coding reached 97% cache reuse, and in a deployment with 3,000 concurrently running software agents the cost was 5-10 times lower than a comparable commercial service. These are the authors' own measurements.
Sources 1