Scaling High-Performance AI: How Cloudflare Optimizes Kimi and GLM Models for Workers AI
Cloudflare has unveiled advanced optimization techniques to run demanding large-scale models, specifically Moonshot's Kimi K-series and Z.ai's GLM, on its Workers AI platform. These Mixture-of-Experts (MoE) models are known for their long-context capabilities but present significant memory challenges. To address this, Cloudflare utilizes a combination of KV cache quantization, model weight compression, and cache protection. A key breakthrough involves transitioning the KV cache from 16-bit precision (BF16) to 8-bit floating point (FP8), which effectively doubles the context capacity for models like Kimi K2.6 from 686,000 to 1.37 million tokens. By leveraging the open-source SGLang framework, Cloudflare achieves higher efficiency and lower costs while maintaining model accuracy, ensuring high-performance inference is accessible closer to the end-user.
Key Takeaways
- Memory Efficiency: Cloudflare uses FP8 quantization for the KV cache to halve its memory footprint, doubling the available context for long-context models.
- Model Support: The optimizations specifically target high-demand Mixture-of-Experts (MoE) models, including Moonshot's Kimi K-series and Z.ai's GLM.
- Framework Integration: All inference serving and benchmarking are conducted using SGLang, an open-source framework that Cloudflare actively contributes to.
- Performance Gains: By separating prefill and decode phases and layering compression techniques, Cloudflare supports more concurrent requests on shared hardware without losing accuracy.
In-Depth Analysis
Overcoming the Memory Bottleneck in Long-Context Models
Serving large-scale models like Kimi and GLM presents a unique set of infrastructure challenges, primarily centered around memory constraints. These models utilize a Mixture-of-Experts (MoE) architecture and support extensive context lengths, which are highly beneficial for users but demanding for hardware. Cloudflare's analysis reveals that in long-context scenarios, it is often the KV (Key-Value) cache—rather than the model weights themselves—that exhausts GPU memory first. The KV cache is essential for maintaining the state of a conversation, allowing the model to reference previous tokens without re-processing the entire history for every new output.
To mitigate this, Cloudflare has implemented 8-bit floating point (FP8, e4m3) quantization for the KV cache. Previously stored in 16-bit precision (BF16), this shift effectively reduces the memory required for the cache by 50%. For a model such as Kimi K2.6, this optimization increases the context capacity from approximately 686,000 tokens to 1.37 million tokens on the same hardware. This allows for significantly longer interactions and more complex data processing within a single session.
Strategic Optimization Layers: Compression and Cache Protection
Beyond quantization, Cloudflare employs a multi-layered strategy to maximize GPU utility. This includes model weight compression and the separation of the prefill and decode phases of inference. Prefill involves processing the initial input prompt, while the decode phase generates subsequent tokens one by one. By isolating these phases, Workers AI can optimize the specific compute requirements of each, leading to faster overall performance.
Furthermore, because these optimizations allow more requests to be packed onto shared hardware, Cloudflare has introduced mechanisms to protect the shared cache. This ensures that the increased density of requests does not lead to performance degradation or resource contention. These technical refinements are designed to lower the cost of serving state-of-the-art models, making it economically viable to offer high-performance AI capabilities at scale.
Collaborative Innovation with SGLang
Cloudflare’s infrastructure relies heavily on SGLang, an open-source inference serving framework. Through rigorous benchmarking, Cloudflare identified SGLang as a market leader in performance. Rather than maintaining a private fork, Cloudflare works closely with the SGLang development team to upstream patches and new features. This collaborative approach ensures that the broader open-source community benefits from the stability and performance enhancements developed for Workers AI, while Cloudflare maintains a cutting-edge platform for its users.
Industry Impact
The ability to serve massive, long-context models like Kimi and GLM at the edge represents a significant shift in AI accessibility. By reducing the hardware barriers through software-level optimizations like FP8 quantization and weight compression, Cloudflare is setting a precedent for how infrastructure providers can handle the next generation of MoE models. This efficiency not only lowers the entry barrier for developers looking to integrate sophisticated AI into their applications but also promotes a more sustainable model for GPU resource management in data centers worldwide.
Frequently Asked Questions
Question: Why is the KV cache more problematic than model weights for memory?
In long-context models, the KV cache grows with every token processed in a conversation. For very long dialogues or document analyses, the cumulative size of these stored keys and values can exceed the static size of the model's weights, eventually filling the GPU's memory and limiting the maximum context length.
Question: How does FP8 quantization affect model accuracy?
According to Cloudflare's findings, transitioning the KV cache to 8-bit floating point (FP8) allows for a significant reduction in memory usage with no change in the accuracy of the model's output. This makes it a highly effective optimization for scaling inference.
Question: What is the role of SGLang in Cloudflare's AI stack?
SGLang is the open-source inference serving framework used by Cloudflare to run and benchmark its AI models. Cloudflare contributes patches and features back to the SGLang project, ensuring the framework remains optimized for high-performance, production-grade AI traffic.
