# NVIDIA and Princeton introduce ReplaySSM to speed hybrid-model decoding

> Researchers said ReplaySSM, which buffers state-space-model inputs, made decoding kernels 1.46–1.80 times faster across four models at batch size 256.

- Canonical URL: https://newsverum.com/en/news/nvidia-princeton-replay-ssm-speeds-hybrid-decoding-october-2026
- Publisher: Newsverum
- Section: Technology
- Country: United States
- Published: 2026-10-15T19:00:45+00:00
- Updated: 2026-10-15T19:00:45+00:00
- Event / data period: 2026-10-12
- Other editions: [Türkçe](https://newsverum.com/tr/news/nvidia-princeton-replay-ssm-speeds-hybrid-decoding-october-2026.md), [العربية](https://newsverum.com/ar/news/nvidia-princeton-replay-ssm-speeds-hybrid-decoding-october-2026.md), [Français](https://newsverum.com/fr/news/nvidia-princeton-replay-ssm-speeds-hybrid-decoding-october-2026.md), [Español](https://newsverum.com/es/news/nvidia-princeton-replay-ssm-speeds-hybrid-decoding-october-2026.md), [Deutsch](https://newsverum.com/de/news/nvidia-princeton-replay-ssm-speeds-hybrid-decoding-october-2026.md), [Português](https://newsverum.com/pt/news/nvidia-princeton-replay-ssm-speeds-hybrid-decoding-october-2026.md)

Researchers at NVIDIA and Princeton University's Dao AI Lab described ReplaySSM, a method for hybrid language models that buffers recent state-space-model (SSM) inputs instead of writing the full state to memory at every token step. It represents state as a checkpoint plus a short input buffer, separating output computation from state updates to simplify speculative-decoding rollback and limit error accumulation in low-precision reads.

In tests on four Nemotron 3 and Qwen3.5 models at batch size 256, the team reported decoding-kernel speedups of 1.46–1.80 times and end-to-end speedups of 1.20–1.47 times; the same tests showed no gain at batch size one. On two 120B-class models, speculative throughput rose 1.85 and 2.10 times at batch size 512. These are results under the researchers' own hardware and software conditions; the method has been integrated into TensorRT-LLM, vLLM and FlashInfer.

## Data

### ReplaySSM decoding speedup

Endpoints of speedup ranges reported across four hybrid models at serving batch size 256.

Methodology: Four-model results reported in the 12 October 2026 post by NVIDIA and Princeton's Dao AI Lab, compared with tuned vLLM kernels. Each measure is reported as a range in the source; only its lower and upper endpoints are stored here. Tests used one H100 or B300 GPU at batch size 256. These are the authors' performance results under their hardware and implementation conditions.

| date | label | description | status |
|---|---|---|---|
| 2026-10-12 | NVIDIA and Princeton introduce ReplaySSM to speed hybrid-model decoding | In tests on four Nemotron 3 and Qwen3.5 models at batch size 256, the team reported decoding-kernel speedups of 1.46–1.80 times and end-to-end speedups of 1.20–1.47 times; the same tests showed no gain at batch size one. On two 120B-class models, speculative throughput rose 1.85 and 2.10 times at batch size 512. These are results under the researchers' own hardware and software conditions; the method has been integrated into TensorRT-LLM, vLLM and FlashInfer. | occurred |

Sources: <https://developer.nvidia.com/blog/replayssm-rethinking-state-storage-for-faster-hybrid-model-decoding/> · Download data: [JSON](https://newsverum.com/api/v1/articles/nvidia-princeton-replay-ssm-speeds-hybrid-decoding-october-2026/data/replayssm-decoding-speed-ranges?lang=en&format=json) · [CSV](https://newsverum.com/api/v1/articles/nvidia-princeton-replay-ssm-speeds-hybrid-decoding-october-2026/data/replayssm-decoding-speed-ranges?lang=en&format=csv)

## Sources

- [NVIDIA Technical Blog and Princeton University Dao AI Lab](<https://developer.nvidia.com/blog/replayssm-rethinking-state-storage-for-faster-hybrid-model-decoding/>) (Source checked: 2026-10-15; Published: 2026-10-12)

---

Cite the canonical URL above. [Editorial standards and corrections](https://newsverum.com/en/editorial)