Newsverum.
SearchK

Search stories

Find the story. Follow the source.

Explore
Explore the data3246Agriculture366Business504Climate64Culture28Economy464Education89Energy594Environment72Finance94Fiscal Policy56Government62Health141Infrastructure237Politics55Science39Security49Society35Sports121Technology480
EN
العربيةDeutschEnglishEspañolFrançaisPortuguêsTürkçe
Sections
Technology / WORLD

GraphTok moves long AI prompt tokenization onto the GPU

Researchers at KTH Royal Institute of Technology introduced GraphTok, a system that tokenizes large language model requests on a GPU. In a Dynamo frontend experiment, a real H100 NVL performed tokenization while backend GPU workers were simulated.

Newsverum · Source brief1 min read

Source review period: 2026-10-09

Published: · Updated:

The source does not state the event date; the period shown is the source review period.

Reading sizeSources ↓Share story ↗
Conceptual illustration of technology, processors and connected information.Illustration
Symbolic illustration

AI agents resend earlier conversation and tool outputs with each new request. Prefix caching can skip model computation for previously processed text, but the full request still has to be converted into token IDs before the cache can be checked. GraphTok performs this work on a GPU using a CUDA graph and routes short inputs to a CPU.

In a Dynamo frontend experiment, it sustained 518 requests per second for 121,000-token traffic within a two-second P99 time-to-first-token target; gigatoken sustained 296 and the default tokenizer 18.5 requests per second. GPU workers were simulated, and the authors verified correctness only for tested tokenizer configurations, so the results are not a general serving-capacity estimate.

Data

Table

Frontend performance comparison

The load test uses partly simulated GPU workers.

121000518 requests/s within 2-second P99 target; GPU workers simulated
Methodology

In a Dynamo frontend experiment with a cached 121k-token context and a 2-second P99 time-to-first-token target, a real H100 NVL performed tokenization while GPU workers were simulated.

Updated:

DownloadCSV ↓JSON ↓
Structured coverage

Subjects and places

Subject proposals

Geographic references

Only source-provided geographic references are shown. Opening an external map is optional. Optional story maps use OpenStreetMap. Opening a map sends your IP address and this site’s address to the map provider; location data remains readable as text without loading the map.

  • Sweden · Place role: Coverage · Place type: Country or area · Location precision: Regional
    Evidence
    • Explicit preserved editorial country/region; no city or national centroid inferredarxiv.org ↗
Taxonomy version: IPTC Media Topics 2026-Q2

Continue reading

Technology

AWS publishes an EKS guide to multi-node Ray LLM inference

An AWS guide dated October 12 demonstrates a Ray LLM container setup on EKS that splits one LLM across two GPU nodes, running prefill and token generation separately. The post focuses on deployment and configuration.

AWS Machine Learning Blog1 min read Data included
Technology

AWS reports MoE kernel optimization results on SageMaker

AWS detailed GPU kernel optimizations for mixture-of-experts models on SageMaker AI on October 13. The company described swapping matrix operands, routing FP8 operations to the Triton backend and removing unnecessary just-in-time recompilations.

AWS Artificial Intelligence Blog1 min read Data included