IBM explains distributed inference methods for large language models

IBM Research described distributed inference through llm-d, an open-source, Kubernetes-native framework that coordinates groups of large-language-model servers rather than balancing each server in isolation. The explainer covers routing requests to servers that already hold shared prefixes, moving KV cache to CPU memory or storage, prioritizing request flow, pacing batch jobs, separating prefill from decode, and distributing mixture-of-experts components across nodes.

IBM reports that one llm-d demonstration captured 97% of available prefix reuse in its tested workload; it also cites wide-expert-parallelism demonstrations with lower expert-weight memory per GPU and three to nine times the KV-cache capacity, depending on topology. These figures belong to demonstrations cited by the company; the post is not an independent benchmark study.