{"article":{"annotation":{"article_id":"gpu-tokenization-agentic-llm-serving-graphtok-2026","available_at":"2026-10-13T01:40:25.761594Z","geography_revision":"ef6ee0010dda5fe7b240fd514749b43e214c57da73222d2e40ec83b276dd0977","id":"f0313c359dc695e7c13f04814effcff2134b7608a877b9ee3c17e888b03d9f97","input_sha256":"752ddd38f81259a5a70d0dd89375a5751bd48779c681a4c9e94f57f5280dfaf5","limitations":["Rule-derived subjects are candidates, not verified editorial labels or calibrated model probabilities.","Coverage is distinct from event location. Unknown country/city/coordinates remain absent; no automatic geocoding.","Article geometries keep their stated precision and methodology; reproducing them is not independent coordinate verification."],"method":"controlled-rules-v1","places":[{"admin1":null,"city":null,"coordinate_note":null,"coordinate_reference_system":"OGC:CRS84","coordinate_source":null,"country":"SE","evidence":[{"field":"country","language":"en","note":"Explicit preserved editorial country/region; no city or national centroid inferred","source_urls":["https://arxiv.org/abs/2610.13325"]}],"geometry":null,"id":"iso3166-1:SE","kind":"country-or-area","labels":{"ar":"السويد","en":"Sweden","es":"Suecia","fr":"Suède"},"match_status":"candidate","precision":"regional","role":"coverage","uncertainty_m":null}],"rules_sha256":"fb5ad3c27c91cb840f7a0a05f7f64a5cb26ad0374409ff9457f277c7092e22b4","schema_version":1,"subjects":[{"concept_id":"medtop:20001298","confidence":null,"confidence_method":null,"evidence":[{"field":"en.title+en.summary","language":"en","note":"Subject suggested from the title or summary: AI","source_urls":["https://arxiv.org/abs/2610.13325"]}],"method":"controlled-rules-v1","rule_id":"artificial-intelligence","status":"candidate"}],"taxonomy_revision":"1f2db31d811fca07e99b16d402c10e2cc2965c9df1c7b41d9be348b40a24e7fb","taxonomy_version":"IPTC Media Topics 2026-Q2"},"api_url":"https://newsverum.com/api/v1/articles/gpu-tokenization-agentic-llm-serving-graphtok-2026?lang=en","category":"technology","content":{"key_points":[],"paragraphs":["AI agents resend earlier conversation and tool outputs with each new request. Prefix caching can skip model computation for previously processed text, but the full request still has to be converted into token IDs before the cache can be checked. GraphTok performs this work on a GPU using a CUDA graph and routes short inputs to a CPU.","In a Dynamo frontend experiment, it sustained 518 requests per second for 121,000-token traffic within a two-second P99 time-to-first-token target; gigatoken sustained 296 and the default tokenizer 18.5 requests per second. GPU workers were simulated, and the authors verified correctness only for tested tokenizer configurations, so the results are not a general serving-capacity estimate."],"sections":[]},"corrections":[],"country":"SE","data":{"blocks":[{"columns":[{"key":"context_tokens","label":{"ar":"عدد رموز السياق","de":"Anzahl der Kontexttokens","en":"Context token count","es":"Número de tokens del contexto","fr":"Nombre de jetons du contexte","pt":"Número de tokens de contexto","tr":"Bağlam belirteç sayısı"},"type":"number","unit":"count"},{"key":"load","label":{"ar":"نطاق الحمل المستدام","de":"Umfang der nachhaltigen Last","en":"Sustained-load scope","es":"Alcance de la carga sostenida","fr":"Portée de la charge soutenue","pt":"Âmbito da carga sustentada","tr":"Sürdürülen yük kapsamı"},"language":"en","type":"text"}],"description":{"ar":"يستخدم اختبار الحمل عمال GPU محاكين جزئياً.","de":"Der Lasttest verwendet teilweise simulierte GPU-Worker.","en":"The load test uses partly simulated GPU workers.","es":"La prueba de carga usa trabajadores GPU parcialmente simulados.","fr":"Le test de charge utilise des travailleurs GPU en partie simulés.","pt":"O teste de carga usa trabalhadores de GPU parcialmente simulados.","tr":"Yük testi kısmen benzetilmiş GPU çalışanları kullanır."},"id":"frontend-benchmark","methodology":{"ar":"في تجربة واجهة Dynamo بسياق مخبأ من 121 ألف رمز وهدف زمن أول رمز عند المئين 99 قدره ثانيتان، نفذت H100 NVL حقيقية الترميز بينما حوكيت عمال GPU.","de":"In einem Dynamo-Frontend-Test mit einem zwischengespeicherten Kontext von 121.000 Tokens und einem P99-Ziel von 2 Sekunden für das erste Token tokenisierte eine echte H100 NVL; die GPU-Worker wurden simuliert.","en":"In a Dynamo frontend experiment with a cached 121k-token context and a 2-second P99 time-to-first-token target, a real H100 NVL performed tokenization while GPU workers were simulated.","es":"En una prueba del frontend Dynamo con un contexto en caché de 121 000 tokens y un objetivo P99 de 2 segundos hasta el primer token, una H100 NVL real tokenizó mientras se simulaban los trabajadores GPU.","fr":"Dans un test du frontal Dynamo avec un contexte mis en cache de 121 000 jetons et un objectif de délai P99 de 2 secondes, un H100 NVL réel effectuait la tokenisation tandis que les travailleurs GPU étaient simulés.","pt":"Num teste do frontend Dynamo com contexto em cache de 121 mil tokens e meta P99 de 2 segundos até ao primeiro token, uma H100 NVL real fez a tokenização, enquanto os trabalhadores GPU foram simulados.","tr":"Dynamo ön ucunda, önbelleğe alınmış 121 bin tokenlık bağlam ve 2 saniyelik P99 ilk token süresi hedefiyle yapılan deneyde gerçek H100 NVL belirteçleme yaptı; GPU çalışanları benzetildi."},"rowTranslations":[{"load":{"ar":"518 طلباً في الثانية ضمن هدف P99 قدره ثانيتان؛ عمال GPU محاكَون","de":"518 Anfragen/s beim P99-Ziel von 2 Sekunden; GPU-Worker simuliert","en":"518 requests/s within 2-second P99 target; GPU workers simulated","es":"518 solicitudes/s con un objetivo P99 de 2 segundos; trabajadores GPU simulados","fr":"518 requêtes/s dans un objectif P99 de 2 secondes; travailleurs GPU simulés","pt":"518 pedidos/s dentro da meta P99 de 2 segundos; trabalhadores GPU simulados","tr":"2 saniyelik P99 hedefinde saniyede 518 istek; GPU çalışanları benzetilmiş"}}],"rows":[{"context_tokens":121000,"load":"518 requests/s within 2-second P99 target; GPU workers simulated"}],"sourceUrls":["https://arxiv.org/abs/2610.13325"],"title":{"ar":"مقارنة أداء الواجهة الأمامية","de":"Leistungsvergleich des Frontends","en":"Frontend performance comparison","es":"Comparación del rendimiento del frontend","fr":"Comparaison des performances du frontal","pt":"Comparação do desempenho do frontend","tr":"Ön uç performans karşılaştırması"},"type":"table","updatedAt":"2026-10-13T01:31:44Z"}],"version":1},"data_downloads":[{"formats":{"csv":"https://newsverum.com/api/v1/articles/gpu-tokenization-agentic-llm-serving-graphtok-2026/data/frontend-benchmark?lang=en&format=csv","json":"https://newsverum.com/api/v1/articles/gpu-tokenization-agentic-llm-serving-graphtok-2026/data/frontend-benchmark?lang=en&format=json"},"id":"frontend-benchmark"}],"data_types":["table"],"editorial_illustration":{"alt":"Conceptual illustration of technology, processors and connected information.","asset_id":"newsverum-v1-technology","avif_variants":[{"height":320,"url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-480.avif","width":480},{"height":480,"url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-720.avif","width":720},{"height":640,"url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-960.avif","width":960},{"height":800,"url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-1200.avif","width":1200},{"height":1024,"url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-1536.avif","width":1536}],"credit":"Newsverum","documentary":false,"height":1024,"kind":"symbolic-editorial-illustration","language":"en","scope":"topic","url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-1536.webp","variants":[{"height":320,"url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-480.webp","width":480},{"height":640,"url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-960.webp","width":960},{"height":1024,"url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-1536.webp","width":1536}],"width":1536},"event_id":"event-38db37c5-be8c-5dee-ba6f-4b6d6e21e130","event_period":"2026-10-09","event_period_basis":"source-review","id":"gpu-tokenization-agentic-llm-serving-graphtok-2026","image":null,"importance":0.6,"language":"en","modified_at":"2026-10-13T01:40:12+00:00","publication_basis":"publication-record","published_at":"2026-10-13T01:40:12+00:00","reading_minutes":1,"region":"world","section_illustration":{"alt":"Conceptual illustration of technology, processors and connected information.","asset_id":"newsverum-v1-technology","avif_variants":[{"height":320,"url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-480.avif","width":480},{"height":480,"url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-720.avif","width":720},{"height":640,"url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-960.avif","width":960},{"height":800,"url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-1200.avif","width":1200},{"height":1024,"url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-1536.avif","width":1536}],"credit":"Newsverum","documentary":false,"height":1024,"kind":"symbolic-editorial-illustration","language":"en","scope":"section","url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-1536.webp","variants":[{"height":320,"url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-480.webp","width":480},{"height":640,"url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-960.webp","width":960},{"height":1024,"url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-1536.webp","width":1536}],"width":1536},"slug":"gpu-tokenization-agentic-llm-serving-graphtok-2026","sources":[{"checkedAt":"2026-10-13","name":"arXiv","publishedAt":"2026-10-09","url":"https://arxiv.org/abs/2610.13325"}],"summary":"Researchers at KTH Royal Institute of Technology introduced GraphTok, a system that tokenizes large language model requests on a GPU. In a Dynamo frontend experiment, a real H100 NVL performed tokenization while backend GPU workers were simulated.","text":"Researchers at KTH Royal Institute of Technology introduced GraphTok, a system that tokenizes large language model requests on a GPU. In a Dynamo frontend experiment, a real H100 NVL performed tokenization while backend GPU workers were simulated.\n\nAI agents resend earlier conversation and tool outputs with each new request. Prefix caching can skip model computation for previously processed text, but the full request still has to be converted into token IDs before the cache can be checked. GraphTok performs this work on a GPU using a CUDA graph and routes short inputs to a CPU.\n\nIn a Dynamo frontend experiment, it sustained 518 requests per second for 121,000-token traffic within a two-second P99 time-to-first-token target; gigatoken sustained 296 and the default tokenizer 18.5 requests per second. GPU workers were simulated, and the authors verified correctness only for tested tokenizer configurations, so the results are not a general serving-capacity estimate.","title":"GraphTok moves long AI prompt tokenization onto the GPU","topics":[],"translations":[{"api_url":"https://newsverum.com/api/v1/articles/gpu-tokenization-agentic-llm-serving-graphtok-2026?lang=tr","language":"tr","url":"https://newsverum.com/tr/news/gpu-tokenization-agentic-llm-serving-graphtok-2026"},{"api_url":"https://newsverum.com/api/v1/articles/gpu-tokenization-agentic-llm-serving-graphtok-2026?lang=en","language":"en","url":"https://newsverum.com/en/news/gpu-tokenization-agentic-llm-serving-graphtok-2026"},{"api_url":"https://newsverum.com/api/v1/articles/gpu-tokenization-agentic-llm-serving-graphtok-2026?lang=ar","language":"ar","url":"https://newsverum.com/ar/news/gpu-tokenization-agentic-llm-serving-graphtok-2026"},{"api_url":"https://newsverum.com/api/v1/articles/gpu-tokenization-agentic-llm-serving-graphtok-2026?lang=fr","language":"fr","url":"https://newsverum.com/fr/news/gpu-tokenization-agentic-llm-serving-graphtok-2026"},{"api_url":"https://newsverum.com/api/v1/articles/gpu-tokenization-agentic-llm-serving-graphtok-2026?lang=es","language":"es","url":"https://newsverum.com/es/news/gpu-tokenization-agentic-llm-serving-graphtok-2026"},{"api_url":"https://newsverum.com/api/v1/articles/gpu-tokenization-agentic-llm-serving-graphtok-2026?lang=de","language":"de","url":"https://newsverum.com/de/news/gpu-tokenization-agentic-llm-serving-graphtok-2026"},{"api_url":"https://newsverum.com/api/v1/articles/gpu-tokenization-agentic-llm-serving-graphtok-2026?lang=pt","language":"pt","url":"https://newsverum.com/pt/news/gpu-tokenization-agentic-llm-serving-graphtok-2026"}],"url":"https://newsverum.com/en/news/gpu-tokenization-agentic-llm-serving-graphtok-2026"},"version":"1"}