# GraphTok moves long AI prompt tokenization onto the GPU

> Researchers at KTH Royal Institute of Technology introduced GraphTok, a system that tokenizes large language model requests on a GPU. In a Dynamo frontend experiment, a real H100 NVL performed tokenization while backend GPU workers were simulated.

- Canonical URL: https://newsverum.com/en/news/gpu-tokenization-agentic-llm-serving-graphtok-2026
- Publisher: Newsverum
- Section: Technology
- Country: Sweden
- Published: 2026-10-13T01:40:12+00:00
- Updated: 2026-10-13T01:40:12+00:00
- Source review period: 2026-10-09
- Other editions: [Türkçe](https://newsverum.com/tr/news/gpu-tokenization-agentic-llm-serving-graphtok-2026.md), [العربية](https://newsverum.com/ar/news/gpu-tokenization-agentic-llm-serving-graphtok-2026.md), [Français](https://newsverum.com/fr/news/gpu-tokenization-agentic-llm-serving-graphtok-2026.md), [Español](https://newsverum.com/es/news/gpu-tokenization-agentic-llm-serving-graphtok-2026.md), [Deutsch](https://newsverum.com/de/news/gpu-tokenization-agentic-llm-serving-graphtok-2026.md), [Português](https://newsverum.com/pt/news/gpu-tokenization-agentic-llm-serving-graphtok-2026.md)

*The source does not state the event date; the period shown is the source review period.*

AI agents resend earlier conversation and tool outputs with each new request. Prefix caching can skip model computation for previously processed text, but the full request still has to be converted into token IDs before the cache can be checked. GraphTok performs this work on a GPU using a CUDA graph and routes short inputs to a CPU.

In a Dynamo frontend experiment, it sustained 518 requests per second for 121,000-token traffic within a two-second P99 time-to-first-token target; gigatoken sustained 296 and the default tokenizer 18.5 requests per second. GPU workers were simulated, and the authors verified correctness only for tested tokenizer configurations, so the results are not a general serving-capacity estimate.

## Data

### Frontend performance comparison

The load test uses partly simulated GPU workers.

Methodology: In a Dynamo frontend experiment with a cached 121k-token context and a 2-second P99 time-to-first-token target, a real H100 NVL performed tokenization while GPU workers were simulated.

| Context token count | Sustained-load scope |
|---|---|
| 121000 | 518 requests/s within 2-second P99 target; GPU workers simulated |

Sources: <https://arxiv.org/abs/2610.13325> · Download data: [JSON](https://newsverum.com/api/v1/articles/gpu-tokenization-agentic-llm-serving-graphtok-2026/data/frontend-benchmark?lang=en&format=json) · [CSV](https://newsverum.com/api/v1/articles/gpu-tokenization-agentic-llm-serving-graphtok-2026/data/frontend-benchmark?lang=en&format=csv)

## Sources

- [arXiv](<https://arxiv.org/abs/2610.13325>) (Source checked: 2026-10-13; Published: 2026-10-09)

---

Cite the canonical URL above. [Editorial standards and corrections](https://newsverum.com/en/editorial)