Back to list
vLLM v0.28.0 Released: Major Performance Optimizations for Kimi-K3 and DeepSeek V4 Support
Product LaunchvLLMOpen SourceLLM Inference

vLLM v0.28.0 Released: Major Performance Optimizations for Kimi-K3 and DeepSeek V4 Support

The vLLM project has announced the release of version 0.28.0, a massive update featuring 584 commits from 270 contributors. This version introduces a comprehensive performance push for the Kimi-K3 model, including Decode Context Parallel (DCP) support, fused FlashKDA kernels, and adaptive speculative token budgets that improve Time to First Token (TTFT) by approximately 60%. Additionally, the release brings end-to-end support for DeepSeek V4, enabling sparse MLA for various decoding modes and AMD Quark NVFP4 support. Significant memory efficiency gains are also highlighted, with optional shared-expert sharding saving up to 17 GiB of memory per GPU. The update further expands hardware compatibility with enhanced ROCm support for both Kimi-K3 and DeepSeek V4 across multiple architectures.

Hacker News

Key Takeaways

  • Kimi-K3 Performance Breakthrough: Extensive stack-wide optimizations including Decode Context Parallel (DCP) and fused FlashKDA kernels.
  • DeepSeek V4 Full Integration: End-to-end support for sparse MLA across plain decode, MTP, and DSpark speculative decoding.
  • Substantial Memory Savings: New optional shared-expert sharding reduces memory usage by approximately 17 GiB per GPU.
  • Enhanced Hardware Compatibility: Expanded ROCm support for Kimi-K3 and DeepSeek V4, specifically targeting gfx11 and gfx950 architectures.
  • Efficiency Gains: Kernel-level speedups of 1.5x to 3x for combined all-gathers and a 60% improvement in DSpark TTFT.

In-Depth Analysis

The Kimi-K3 Performance Push

The release of vLLM v0.28.0 marks a significant milestone for Kimi-K3 users, as the development team has implemented a major optimization effort across the entire software stack. One of the most notable additions is the support for Decode Context Parallel (DCP), which allows for more efficient handling of long-context sequences during the decoding phase. To further enhance throughput, the update introduces fused FlashKDA decode and prefill kernels, which streamline the computational path for attention mechanisms.

Beyond kernel fusion, the update incorporates SiTU activation support for MegaMoE and GEMM-RS for sequence parallelism. These technical improvements are complemented by a 1.5x to 3x kernel-level speedup in combined all-gathers, a critical operation in distributed inference. Perhaps most impactful for user experience is the introduction of an adaptive speculative token budget, which has demonstrated a ~60% improvement in Time to First Token (TTFT) for DSpark. For organizations constrained by hardware, the optional shared-expert sharding is a game-changer, potentially saving 17 GiB of memory per GPU, thereby allowing larger models to run on standard hardware configurations.

DeepSeek V4 and Sparse MLA Support

vLLM v0.28.0 brings robust end-to-end functionality for DeepSeek V4. A core component of this support is the implementation of sparse MLA (Multi-Head Latent Attention), which is now functional for plain decode, Multi-Token Prediction (MTP), and DSpark speculative decoding. This ensures that DeepSeek's latest architectural innovations can be fully leveraged within the vLLM ecosystem.

To support high-performance execution on diverse hardware, the release includes AMD Quark NVFP4 support and specific optimizations for sparse top-k metadata kernels. The development team has also focused on reducing overhead by narrowing eager CUDA graph regions, which helps in stabilizing performance during complex inference tasks. Furthermore, the integration includes reasoning-effort prompts and mappings, allowing for more nuanced control over the model's output generation process.

Hardware Expansion and Speculative Decoding

Hardware flexibility remains a core focus for vLLM. Version 0.28.0 extends ROCm enablement to the gfx11 and gfx950 architectures, ensuring that AMD users can benefit from the latest optimizations. Kimi-K3 is also now compatible with ROCm via the V2 model runner, bridging the gap between NVIDIA and AMD performance capabilities.

Speculative decoding continues to evolve in this release with the introduction of DFlash and other advances. By optimizing the speculative decoding pipeline, vLLM reduces the latency associated with generating high-quality tokens, making real-time interaction with massive models more viable for production environments.

Industry Impact

The release of vLLM v0.28.0 significantly lowers the barrier to entry for deploying state-of-the-art models like Kimi-K3 and DeepSeek V4. By achieving a 17 GiB reduction in memory usage per GPU through shared-expert sharding, vLLM is enabling more efficient utilization of existing data center resources. This optimization is particularly relevant as the industry moves toward Mixture-of-Experts (MoE) architectures, which are traditionally memory-intensive.

Furthermore, the 60% improvement in TTFT and the 1.5-3x speedup in communication kernels signal a shift toward ultra-low-latency inference. As AI applications move closer to real-time human-agent interaction, these stack-level optimizations become the foundation for the next generation of responsive AI services. The expanded support for AMD hardware via ROCm also fosters a more competitive and diverse hardware ecosystem, reducing reliance on a single vendor for high-performance LLM serving.

Frequently Asked Questions

Question: How does vLLM v0.28.0 improve memory efficiency for large models?

Version 0.28.0 introduces optional shared-expert sharding, which can save approximately 17 GiB of memory per GPU. This is particularly beneficial for MoE models like Kimi-K3, allowing them to fit into smaller memory footprints without sacrificing performance.

Question: What specific improvements were made for DeepSeek V4?

vLLM now supports sparse MLA end-to-end for DeepSeek V4. This includes compatibility with plain decode, MTP, and DSpark speculative decoding. Additionally, it adds AMD Quark NVFP4 support and optimizations for sparse top-k metadata kernels.

Question: What are the performance gains for Kimi-K3 in this release?

Kimi-K3 sees a major performance push including a ~60% better TTFT for DSpark due to an adaptive speculative token budget. It also benefits from 1.5x to 3x speedups in kernel-level all-gathers and the introduction of fused FlashKDA kernels.

Related News

Gemini API Tool Retrieval Preview Gives Agents Access to Hundreds of Tools With Fewer Tokens
Product Launch

Gemini API Tool Retrieval Preview Gives Agents Access to Hundreds of Tools With Fewer Tokens

Google AI Blog announced that Tool Retrieval is now in preview in the Gemini API. The feature is presented as a way to give agents access to hundreds of tools while using fewer tokens. The announcement pairs those two points: the breadth of tool access and lower token use. It does not explain how Tool Retrieval works, specify how token use was measured, or quantify the reduction. It also does not provide an exact tool count beyond describing access to hundreds of tools. The stated release status is preview, and the announcement does not give a timeline for general availability.

LangSmith Tool Gateway Enters Public Beta for Secure, Governed AI Agent Tool Use
Product Launch

LangSmith Tool Gateway Enters Public Beta for Secure, Governed AI Agent Tool Use

LangChain announced that LangSmith Tool Gateway is in Public Beta. The gateway handles credentials, identity, and access for every agent tool call, with policies, approvals, and traces. The announcement frames secure, governed tool use around the individual call: its focus is on how agents access tools, rather than on what those tools or agents can do. The update identifies what the gateway covers but does not explain how credentials are managed, how access decisions are made, when approvals apply, or what information traces contain. The Public Beta is the only release status stated in the announcement.

Google’s Gemini Decision 1 Picks Answers Instead of Writing Them Across Five Input Types
Product Launch

Google’s Gemini Decision 1 Picks Answers Instead of Writing Them Across Five Input Types

Google’s Gemini Decision 1 is a model designed to pick answers rather than write them, according to Shittu Olumide’s report for KDnuggets. It reads text, images, video, audio, and PDFs in a single forward pass and supports up to 262,144 input tokens per request. The central distinction is between what the model reads and what it does with that input: it accepts several kinds of material, but its stated task is answer selection, not answer writing. The token figure describes the maximum input per request, not how much the model outputs. The report does not explain how answers are presented for selection or provide results showing how well the model performs.