{"article":{"annotation":{"article_id":"hugging-face-tokenizers-proposes-bert-normalization-speedups-2026","available_at":"2026-10-15T18:13:59.101303Z","geography_revision":"ef6ee0010dda5fe7b240fd514749b43e214c57da73222d2e40ec83b276dd0977","id":"8e4560ac6eb908d1cde96a5241c6685eddbd1b288bffe20273200fb83c015b56","input_sha256":"869d3f75de31680a9f87fbade215a718671e1cf331f92c27783d915947a56acb","limitations":["Rule-derived subjects are candidates, not verified editorial labels or calibrated model probabilities.","Coverage is distinct from event location. Unknown country/city/coordinates remain absent; no automatic geocoding.","Article geometries keep their stated precision and methodology; reproducing them is not independent coordinate verification."],"method":"controlled-rules-v1","places":[{"admin1":null,"city":null,"coordinate_note":null,"coordinate_reference_system":"OGC:CRS84","coordinate_source":null,"country":"US","evidence":[{"field":"country","language":"en","note":"Explicit preserved editorial country/region; no city or national centroid inferred","source_urls":["https://huggingface.co/blog/huggingface/tokenmaxxing-bert"]}],"geometry":null,"id":"iso3166-1:US","kind":"country-or-area","labels":{"ar":"الولايات المتحدة الأمريكية","en":"United States of America","es":"Estados Unidos de América","fr":"États-Unis d’Amérique"},"match_status":"candidate","precision":"regional","role":"coverage","uncertainty_m":null}],"rules_sha256":"fb5ad3c27c91cb840f7a0a05f7f64a5cb26ad0374409ff9457f277c7092e22b4","schema_version":1,"subjects":[{"concept_id":"medtop:13000000","confidence":null,"confidence_method":null,"evidence":[{"field":"category+topics","language":"en","note":"Preserved editorial category/topic; broad candidate only","source_urls":["https://huggingface.co/blog/huggingface/tokenmaxxing-bert"]}],"method":"controlled-rules-v1","rule_id":"technology-broad","status":"candidate"}],"taxonomy_revision":"1f2db31d811fca07e99b16d402c10e2cc2965c9df1c7b41d9be348b40a24e7fb","taxonomy_version":"IPTC Media Topics 2026-Q2"},"api_url":"https://newsverum.com/api/v1/articles/hugging-face-tokenizers-proposes-bert-normalization-speedups-2026?lang=de","category":"technology","content":{"key_points":[],"paragraphs":["Ein Beitrag des Hugging-Face-Tokenizers-Teams vom 15. Oktober beschrieb vorgeschlagene Änderungen zur Beschleunigung der BERT-Textnormalisierung. Der Ansatz überspringt Normalisierung bei ASCII-Abschnitten, speichert Unicode-Entscheidungen für Seiten mit 256 Codepoints zwischen und vermeidet eine neue Zeichenkette, wenn der Text unverändert bleibt. Laut Profil entfielen 88 % der warmen englischen Kodierzeit auf die Normalisierung und 8 % auf die WordPiece-Tokenauswahl.","Der Beitrag berichtet, dass ein Entwicklungszweig bert-base-uncased bei getesteten Eingaben bis zu etwa 50-mal schneller als Tokenizers 0.23 kodierte; die Token-IDs blieben in den genannten Korpora gleich. Die stärksten Zugewinne gab es bei Englisch und Code; Arabisch, Griechisch und Russisch erreichten das 2,2- bis 3,2-Fache, Chinesisch kaum oder gar keinen Vorteil. Kleine Eingaben konnten langsamer sein. Da die Änderungen noch als Pull Request geprüft wurden, ist der Faktor keine allgemeine Produktionsgarantie."],"sections":[]},"corrections":[],"country":"US","data":{"blocks":[{"description":{"ar":"تختلف المكاسب حسب اللغة وحجم الإدخال، وكانت التحسينات ما تزال قيد المراجعة.","de":"Die Zugewinne hängen von Sprache und Eingabelänge ab; die Änderung wurde noch geprüft.","en":"The reported gains vary by language and input size, and the optimization was still under review.","es":"Las mejoras varían según el idioma y el tamaño de entrada; los cambios seguían en revisión.","fr":"Les gains rapportés varient selon la langue et la taille des entrées; l’optimisation restait en examen.","pt":"Os ganhos variam conforme o idioma e o tamanho da entrada; as alterações ainda estavam em análise.","tr":"Ölçülen kazanımlar dile ve girdi uzunluğuna göre değişti; iyileştirme hâlâ inceleme aşamasındaydı."},"id":"announcement-record","items":[{"date":"2026-10-15","description":{"ar":"تختلف المكاسب حسب اللغة وحجم الإدخال، وكانت التحسينات ما تزال قيد المراجعة.","de":"Die Zugewinne hängen von Sprache und Eingabelänge ab; die Änderung wurde noch geprüft.","en":"The reported gains vary by language and input size, and the optimization was still under review.","es":"Las mejoras varían según el idioma y el tamaño de entrada; los cambios seguían en revisión.","fr":"Les gains rapportés varient selon la langue et la taille des entrées; l’optimisation restait en examen.","pt":"Os ganhos variam conforme o idioma e o tamanho da entrada; as alterações ainda estavam em análise.","tr":"Ölçülen kazanımlar dile ve girdi uzunluğuna göre değişti; iyileştirme hâlâ inceleme aşamasındaydı."},"label":{"ar":"فريق Hugging Face يعلن تسريعاً مقترحاً لتقطيع نصوص BERT","de":"Hugging-Face-Team meldet vorgeschlagene Beschleunigung der BERT-Tokenisierung","en":"Hugging Face team reports faster BERT tokenization in proposed changes","es":"El equipo de Hugging Face propone acelerar la tokenización BERT","fr":"L’équipe de Hugging Face propose d’accélérer la tokenisation BERT","pt":"Equipa da Hugging Face propõe acelerar a tokenização BERT","tr":"Hugging Face ekibi önerilen değişikliklerle BERT tokenizasyonunun hızlandığını bildirdi"},"status":"occurred"}],"methodology":{"ar":"يلخص سجل الحدث إعلان المصدر ولا يضيف تحققاً مستقلاً أو تعميماً يتجاوز ما ورد فيه.","de":"Dieser Ereigniseintrag fasst die Quelle zusammen und ergänzt keine unabhängige Prüfung oder Verallgemeinerung.","en":"This event record summarizes the source announcement and adds no independent verification or generalization beyond it.","es":"Este registro resume el anuncio de la fuente sin añadir verificación independiente ni generalizaciones.","fr":"Cette fiche résume l’annonce de la source sans ajouter de vérification indépendante ni généralisation.","pt":"Este registo resume o anúncio da fonte sem acrescentar verificação independente ou generalização.","tr":"Bu olay kaydı kaynağın duyurusunu özetler; kaynakta belirtilmeyen bağımsız doğrulama veya genelleme eklemez."},"sourceUrls":["https://huggingface.co/blog/huggingface/tokenmaxxing-bert"],"title":{"ar":"ملخص الإعلان","de":"Zusammenfassung der Ankündigung","en":"Announcement summary","es":"Resumen del anuncio","fr":"Résumé de l’annonce","pt":"Resumo do anúncio","tr":"Duyuru özeti"},"type":"events","updatedAt":"2026-10-15T18:03:17Z"}],"version":1},"data_downloads":[{"formats":{"csv":"https://newsverum.com/api/v1/articles/hugging-face-tokenizers-proposes-bert-normalization-speedups-2026/data/announcement-record?lang=de&format=csv","json":"https://newsverum.com/api/v1/articles/hugging-face-tokenizers-proposes-bert-normalization-speedups-2026/data/announcement-record?lang=de&format=json"},"id":"announcement-record"}],"data_types":["events"],"editorial_illustration":{"alt":"Konzeptionelle Darstellung von Technologie, Prozessoren und verbundenen Informationen.","asset_id":"newsverum-v1-technology","avif_variants":[{"height":320,"url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-480.avif","width":480},{"height":480,"url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-720.avif","width":720},{"height":640,"url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-960.avif","width":960},{"height":800,"url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-1200.avif","width":1200},{"height":1024,"url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-1536.avif","width":1536}],"credit":"Newsverum","documentary":false,"height":1024,"kind":"symbolic-editorial-illustration","language":"de","scope":"topic","url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-1536.webp","variants":[{"height":320,"url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-480.webp","width":480},{"height":640,"url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-960.webp","width":960},{"height":1024,"url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-1536.webp","width":1536}],"width":1536},"event_id":"event-0ca089fe-d902-585a-87ae-23fb0a631aed","event_period":"2026-10-15","event_period_basis":"event-or-reference-period","id":"hugging-face-tokenizers-proposes-bert-normalization-speedups-2026","image":null,"importance":0.6,"language":"de","modified_at":"2026-10-15T18:13:43+00:00","publication_basis":"publication-record","published_at":"2026-10-15T18:13:43+00:00","reading_minutes":1,"region":"world","relations":[],"section_illustration":{"alt":"Konzeptionelle Darstellung von Technologie, Prozessoren und verbundenen Informationen.","asset_id":"newsverum-v1-technology","avif_variants":[{"height":320,"url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-480.avif","width":480},{"height":480,"url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-720.avif","width":720},{"height":640,"url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-960.avif","width":960},{"height":800,"url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-1200.avif","width":1200},{"height":1024,"url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-1536.avif","width":1536}],"credit":"Newsverum","documentary":false,"height":1024,"kind":"symbolic-editorial-illustration","language":"de","scope":"section","url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-1536.webp","variants":[{"height":320,"url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-480.webp","width":480},{"height":640,"url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-960.webp","width":960},{"height":1024,"url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-1536.webp","width":1536}],"width":1536},"slug":"hugging-face-tokenizers-proposes-bert-normalization-speedups-2026","sources":[{"checkedAt":"2026-10-15","name":"Hugging Face Tokenizers team","publishedAt":"2026-10-15","url":"https://huggingface.co/blog/huggingface/tokenmaxxing-bert"}],"summary":"Die Zugewinne hängen von Sprache und Eingabelänge ab; die Änderung wurde noch geprüft.","text":"Die Zugewinne hängen von Sprache und Eingabelänge ab; die Änderung wurde noch geprüft.\n\nEin Beitrag des Hugging-Face-Tokenizers-Teams vom 15. Oktober beschrieb vorgeschlagene Änderungen zur Beschleunigung der BERT-Textnormalisierung. Der Ansatz überspringt Normalisierung bei ASCII-Abschnitten, speichert Unicode-Entscheidungen für Seiten mit 256 Codepoints zwischen und vermeidet eine neue Zeichenkette, wenn der Text unverändert bleibt. Laut Profil entfielen 88 % der warmen englischen Kodierzeit auf die Normalisierung und 8 % auf die WordPiece-Tokenauswahl.\n\nDer Beitrag berichtet, dass ein Entwicklungszweig bert-base-uncased bei getesteten Eingaben bis zu etwa 50-mal schneller als Tokenizers 0.23 kodierte; die Token-IDs blieben in den genannten Korpora gleich. Die stärksten Zugewinne gab es bei Englisch und Code; Arabisch, Griechisch und Russisch erreichten das 2,2- bis 3,2-Fache, Chinesisch kaum oder gar keinen Vorteil. Kleine Eingaben konnten langsamer sein. Da die Änderungen noch als Pull Request geprüft wurden, ist der Faktor keine allgemeine Produktionsgarantie.","title":"Hugging-Face-Team meldet vorgeschlagene Beschleunigung der BERT-Tokenisierung","topics":[],"translations":[{"api_url":"https://newsverum.com/api/v1/articles/hugging-face-tokenizers-proposes-bert-normalization-speedups-2026?lang=tr","language":"tr","url":"https://newsverum.com/tr/news/hugging-face-tokenizers-proposes-bert-normalization-speedups-2026"},{"api_url":"https://newsverum.com/api/v1/articles/hugging-face-tokenizers-proposes-bert-normalization-speedups-2026?lang=en","language":"en","url":"https://newsverum.com/en/news/hugging-face-tokenizers-proposes-bert-normalization-speedups-2026"},{"api_url":"https://newsverum.com/api/v1/articles/hugging-face-tokenizers-proposes-bert-normalization-speedups-2026?lang=ar","language":"ar","url":"https://newsverum.com/ar/news/hugging-face-tokenizers-proposes-bert-normalization-speedups-2026"},{"api_url":"https://newsverum.com/api/v1/articles/hugging-face-tokenizers-proposes-bert-normalization-speedups-2026?lang=fr","language":"fr","url":"https://newsverum.com/fr/news/hugging-face-tokenizers-proposes-bert-normalization-speedups-2026"},{"api_url":"https://newsverum.com/api/v1/articles/hugging-face-tokenizers-proposes-bert-normalization-speedups-2026?lang=es","language":"es","url":"https://newsverum.com/es/news/hugging-face-tokenizers-proposes-bert-normalization-speedups-2026"},{"api_url":"https://newsverum.com/api/v1/articles/hugging-face-tokenizers-proposes-bert-normalization-speedups-2026?lang=de","language":"de","url":"https://newsverum.com/de/news/hugging-face-tokenizers-proposes-bert-normalization-speedups-2026"},{"api_url":"https://newsverum.com/api/v1/articles/hugging-face-tokenizers-proposes-bert-normalization-speedups-2026?lang=pt","language":"pt","url":"https://newsverum.com/pt/news/hugging-face-tokenizers-proposes-bert-normalization-speedups-2026"}],"url":"https://newsverum.com/de/news/hugging-face-tokenizers-proposes-bert-normalization-speedups-2026"},"version":"1"}