Mooncake: The KVCache-Centric Architecture Powering Kimi at Scale

When your LLM service handles millions of long-context requests daily, traditional serving architectures break down. Moonshot AI’s Mooncake — the serving platform behind Kimi — takes a radically different approach: make the KVCache the center of everything.

The result? 75% more requests handled in production while meeting latency SLOs. Up to 525% throughput improvement in long-context scenarios.

The Problem with Conventional LLM Serving

Most LLM inference systems treat KVCache as a byproduct — something generated during prefill and consumed during decode, then discarded. But for long-context models and services with heavy prompt reuse (think chatbots with system prompts), this is massively wasteful:

1. Prefill is compute-intensive — Generating KVCache for 128k tokens is expensive
2. KVCache is memory-intensive — A 70B model’s KVCache for 128k tokens is ~40GB
3. Recomputation is everywhere — Same prompts, same KVCache computed over and over

Mooncake’s Three Key Innovations

1. Prefill/Decode Disaggregation

Mooncake physically separates prefill and decode clusters:

Request → Prefill Cluster (compute-heavy)
              ↓ KVCache transfer
        Decode Cluster (memory-bandwidth-bound)

This lets you optimize each cluster independently — prefill instances for raw FLOPS, decode instances for memory bandwidth and latency.

2. Disaggregated KVCache Pool

Here’s the clever part: GPU clusters have vast amounts of underutilized CPU memory, DRAM, SSDs, and network bandwidth. Mooncake builds a distributed KVCache pool from these resources:

Tier 1: GPU HBM (fastest, smallest)
Tier 2: Host DRAM (fast, larger)
Tier 3: SSD/NVMe (slower, huge)

Reusable KVCache lives in this pool. When a new request arrives with a known prefix, Mooncake fetches cached KV states instead of recomputing.

3. KVCache-Centric Scheduler

The global scheduler (called “Conductor”) orchestrates everything:

1. Check if reusable KVCache exists for the request prefix
2. Transfer cached KVCache to selected prefill instance
3. Compute only the new tokens’ KVCache
4. Transfer complete KVCache to decode instance
5. Serve the request

The scheduler balances throughput maximization against SLO constraints, with a prediction-based early rejection policy for overload scenarios — accepting that some requests won’t be served is better than degrading everyone’s latency.

Transfer Engine: The Secret Sauce

Moving 40GB of KVCache between machines quickly requires serious infrastructure. Mooncake’s Transfer Engine achieves:

• 87 GB/s on 4×200 Gbps RoCE networks
• 190 GB/s on 8×400 Gbps RoCE networks
• 2.4x–4.6x faster than TCP

Key features:

• Multi-NIC bandwidth aggregation
• Topology-aware path selection (NUMA affinity, etc.)
• Automatic failover on network errors
• Zero-copy transfers where possible

Ecosystem Integration

Mooncake isn’t just internal infrastructure — it’s now deeply integrated across the LLM ecosystem:

• vLLM: KV connector for disaggregated prefill
• SGLang: Hierarchical KV caching backend, EPD disaggregation
• TensorRT-LLM: KVCache transfer for PD disaggregation
• LMDeploy: PD disaggregation backend
• LMCache: Remote connector integration
• TorchSpec: Speculative decoding training at scale

The project won Best Paper at FAST 2025 and joined the PyTorch Ecosystem in February 2026.

Real-World Impact

Numbers from production:

Metric	Improvement
Request throughput	+75%
Long-context throughput (simulated)	Up to +525%
Weight updates for 1T-param model	53s → 7.2s (7x faster)

For the Kimi-K2 deployment on 128 H200 GPUs: 224k tokens/sec prefill, 288k tokens/sec decode.

Getting Started

Mooncake is open source with comprehensive documentation:

• GitHub: kvcache-ai/Mooncake
• Paper: arXiv:2407.00079
• Docs: kvcache-ai.github.io/Mooncake

The Transfer Engine can be used standalone if you just need high-performance data movement. Mooncake Store provides distributed KVCache management. Both integrate with existing inference stacks.

Why This Matters

As context windows grow (1M+ tokens becoming common) and prompt reuse increases (system prompts, RAG patterns), KVCache management becomes the bottleneck. Mooncake’s architecture — treating KVCache as a first-class distributed resource rather than ephemeral compute byproduct — is the right abstraction for this future.

If you’re running LLM inference at scale and haven’t looked at disaggregated architectures, Mooncake provides both the conceptual framework and production-hardened implementation to get started.