NVIDIA and Princeton introduce ReplaySSM to speed hybrid-model decoding
Researchers said ReplaySSM, which buffers state-space-model inputs, made decoding kernels 1.46–1.80 times faster across four models at batch size 256.
Event / data period: 2026-10-12
Published: · Updated:
Researchers at NVIDIA and Princeton University's Dao AI Lab described ReplaySSM, a method for hybrid language models that buffers recent state-space-model (SSM) inputs instead of writing the full state to memory at every token step. It represents state as a checkpoint plus a short input buffer, separating output computation from state updates to simplify speculative-decoding rollback and limit error accumulation in low-precision reads.
In tests on four Nemotron 3 and Qwen3.5 models at batch size 256, the team reported decoding-kernel speedups of 1.46–1.80 times and end-to-end speedups of 1.20–1.47 times; the same tests showed no gain at batch size one. On two 120B-class models, speculative throughput rose 1.85 and 2.10 times at batch size 512. These are results under the researchers' own hardware and software conditions; the method has been integrated into TensorRT-LLM, vLLM and FlashInfer.
Data
ReplaySSM decoding speedup
Endpoints of speedup ranges reported across four hybrid models at serving batch size 256.
- · Occurred
NVIDIA and Princeton introduce ReplaySSM to speed hybrid-model decoding
In tests on four Nemotron 3 and Qwen3.5 models at batch size 256, the team reported decoding-kernel speedups of 1.46–1.80 times and end-to-end speedups of 1.20–1.47 times; the same tests showed no gain at batch size one. On two 120B-class models, speculative throughput rose 1.85 and 2.10 times at batch size 512. These are results under the researchers' own hardware and software conditions; the method has been integrated into TensorRT-LLM, vLLM and FlashInfer.
Methodology
Four-model results reported in the 12 October 2026 post by NVIDIA and Princeton's Dao AI Lab, compared with tuned vLLM kernels. Each measure is reported as a range in the source; only its lower and upper endpoints are stored here. Tests used one H100 or B300 GPU at batch size 256. These are the authors' performance results under their hardware and implementation conditions.
Updated:
Subjects and places
Subject proposals
- Science and technologySuggested
- Preserved editorial category/topic; broad candidate onlydeveloper.nvidia.com ↗
Geographic references
Only source-provided geographic references are shown. Opening an external map is optional. Optional story maps use OpenStreetMap. Opening a map sends your IP address and this site’s address to the map provider; location data remains readable as text without loading the map.
- United States · Place role: Coverage · Place type: Country or area · Location precision: RegionalEvidence
- Explicit preserved editorial country/region; no city or national centroid inferreddeveloper.nvidia.com ↗