{"article":{"annotation":{"article_id":"hugging-face-tokenizers-proposes-bert-normalization-speedups-2026","available_at":"2026-10-15T18:13:59.101303Z","geography_revision":"ef6ee0010dda5fe7b240fd514749b43e214c57da73222d2e40ec83b276dd0977","id":"8e4560ac6eb908d1cde96a5241c6685eddbd1b288bffe20273200fb83c015b56","input_sha256":"869d3f75de31680a9f87fbade215a718671e1cf331f92c27783d915947a56acb","limitations":["Rule-derived subjects are candidates, not verified editorial labels or calibrated model probabilities.","Coverage is distinct from event location. Unknown country/city/coordinates remain absent; no automatic geocoding.","Article geometries keep their stated precision and methodology; reproducing them is not independent coordinate verification."],"method":"controlled-rules-v1","places":[{"admin1":null,"city":null,"coordinate_note":null,"coordinate_reference_system":"OGC:CRS84","coordinate_source":null,"country":"US","evidence":[{"field":"country","language":"en","note":"Explicit preserved editorial country/region; no city or national centroid inferred","source_urls":["https://huggingface.co/blog/huggingface/tokenmaxxing-bert"]}],"geometry":null,"id":"iso3166-1:US","kind":"country-or-area","labels":{"ar":"الولايات المتحدة الأمريكية","en":"United States of America","es":"Estados Unidos de América","fr":"États-Unis d’Amérique"},"match_status":"candidate","precision":"regional","role":"coverage","uncertainty_m":null}],"rules_sha256":"fb5ad3c27c91cb840f7a0a05f7f64a5cb26ad0374409ff9457f277c7092e22b4","schema_version":1,"subjects":[{"concept_id":"medtop:13000000","confidence":null,"confidence_method":null,"evidence":[{"field":"category+topics","language":"en","note":"Preserved editorial category/topic; broad candidate only","source_urls":["https://huggingface.co/blog/huggingface/tokenmaxxing-bert"]}],"method":"controlled-rules-v1","rule_id":"technology-broad","status":"candidate"}],"taxonomy_revision":"1f2db31d811fca07e99b16d402c10e2cc2965c9df1c7b41d9be348b40a24e7fb","taxonomy_version":"IPTC Media Topics 2026-Q2"},"api_url":"https://newsverum.com/api/v1/articles/hugging-face-tokenizers-proposes-bert-normalization-speedups-2026?lang=fr","category":"technology","content":{"key_points":[],"paragraphs":["Dans un billet du 15 octobre, l’équipe Tokenizers de Hugging Face a présenté des changements proposés pour accélérer la normalisation des textes de type BERT. La méthode évite ce travail sur les séquences ASCII, mémorise les décisions Unicode par pages de 256 points de code et ne crée pas de nouvelle chaîne si le texte ne change pas. Le profil de l’équipe attribue 88 % du temps d’encodage anglais à chaud à la normalisation, contre 8 % à la sélection de tokens par WordPiece.","Le billet indique qu’une branche de développement a encodé bert-base-uncased jusqu’à environ 50 fois plus vite que Tokenizers 0.23 sur les entrées testées, avec des identifiants de tokens identiques dans les corpus rapportés. Les gains sont les plus élevés pour l’anglais et le code; l’arabe, le grec et le russe progressent de 2,2 à 3,2 fois, tandis que le gain est faible ou nul pour le chinois. Les petites entrées peuvent ralentir, et les changements étaient encore au stade de la demande de fusion : le multiplicateur ne garantit donc pas les performances en production."],"sections":[]},"corrections":[],"country":"US","data":{"blocks":[{"description":{"ar":"تختلف المكاسب حسب اللغة وحجم الإدخال، وكانت التحسينات ما تزال قيد المراجعة.","de":"Die Zugewinne hängen von Sprache und Eingabelänge ab; die Änderung wurde noch geprüft.","en":"The reported gains vary by language and input size, and the optimization was still under review.","es":"Las mejoras varían según el idioma y el tamaño de entrada; los cambios seguían en revisión.","fr":"Les gains rapportés varient selon la langue et la taille des entrées; l’optimisation restait en examen.","pt":"Os ganhos variam conforme o idioma e o tamanho da entrada; as alterações ainda estavam em análise.","tr":"Ölçülen kazanımlar dile ve girdi uzunluğuna göre değişti; iyileştirme hâlâ inceleme aşamasındaydı."},"id":"announcement-record","items":[{"date":"2026-10-15","description":{"ar":"تختلف المكاسب حسب اللغة وحجم الإدخال، وكانت التحسينات ما تزال قيد المراجعة.","de":"Die Zugewinne hängen von Sprache und Eingabelänge ab; die Änderung wurde noch geprüft.","en":"The reported gains vary by language and input size, and the optimization was still under review.","es":"Las mejoras varían según el idioma y el tamaño de entrada; los cambios seguían en revisión.","fr":"Les gains rapportés varient selon la langue et la taille des entrées; l’optimisation restait en examen.","pt":"Os ganhos variam conforme o idioma e o tamanho da entrada; as alterações ainda estavam em análise.","tr":"Ölçülen kazanımlar dile ve girdi uzunluğuna göre değişti; iyileştirme hâlâ inceleme aşamasındaydı."},"label":{"ar":"فريق Hugging Face يعلن تسريعاً مقترحاً لتقطيع نصوص BERT","de":"Hugging-Face-Team meldet vorgeschlagene Beschleunigung der BERT-Tokenisierung","en":"Hugging Face team reports faster BERT tokenization in proposed changes","es":"El equipo de Hugging Face propone acelerar la tokenización BERT","fr":"L’équipe de Hugging Face propose d’accélérer la tokenisation BERT","pt":"Equipa da Hugging Face propõe acelerar a tokenização BERT","tr":"Hugging Face ekibi önerilen değişikliklerle BERT tokenizasyonunun hızlandığını bildirdi"},"status":"occurred"}],"methodology":{"ar":"يلخص سجل الحدث إعلان المصدر ولا يضيف تحققاً مستقلاً أو تعميماً يتجاوز ما ورد فيه.","de":"Dieser Ereigniseintrag fasst die Quelle zusammen und ergänzt keine unabhängige Prüfung oder Verallgemeinerung.","en":"This event record summarizes the source announcement and adds no independent verification or generalization beyond it.","es":"Este registro resume el anuncio de la fuente sin añadir verificación independiente ni generalizaciones.","fr":"Cette fiche résume l’annonce de la source sans ajouter de vérification indépendante ni généralisation.","pt":"Este registo resume o anúncio da fonte sem acrescentar verificação independente ou generalização.","tr":"Bu olay kaydı kaynağın duyurusunu özetler; kaynakta belirtilmeyen bağımsız doğrulama veya genelleme eklemez."},"sourceUrls":["https://huggingface.co/blog/huggingface/tokenmaxxing-bert"],"title":{"ar":"ملخص الإعلان","de":"Zusammenfassung der Ankündigung","en":"Announcement summary","es":"Resumen del anuncio","fr":"Résumé de l’annonce","pt":"Resumo do anúncio","tr":"Duyuru özeti"},"type":"events","updatedAt":"2026-10-15T18:03:17Z"}],"version":1},"data_downloads":[{"formats":{"csv":"https://newsverum.com/api/v1/articles/hugging-face-tokenizers-proposes-bert-normalization-speedups-2026/data/announcement-record?lang=fr&format=csv","json":"https://newsverum.com/api/v1/articles/hugging-face-tokenizers-proposes-bert-normalization-speedups-2026/data/announcement-record?lang=fr&format=json"},"id":"announcement-record"}],"data_types":["events"],"editorial_illustration":{"alt":"Illustration conceptuelle de la technologie, des processeurs et des informations connectées.","asset_id":"newsverum-v1-technology","avif_variants":[{"height":320,"url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-480.avif","width":480},{"height":480,"url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-720.avif","width":720},{"height":640,"url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-960.avif","width":960},{"height":800,"url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-1200.avif","width":1200},{"height":1024,"url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-1536.avif","width":1536}],"credit":"Newsverum","documentary":false,"height":1024,"kind":"symbolic-editorial-illustration","language":"fr","scope":"topic","url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-1536.webp","variants":[{"height":320,"url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-480.webp","width":480},{"height":640,"url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-960.webp","width":960},{"height":1024,"url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-1536.webp","width":1536}],"width":1536},"event_id":"event-0ca089fe-d902-585a-87ae-23fb0a631aed","event_period":"2026-10-15","event_period_basis":"event-or-reference-period","id":"hugging-face-tokenizers-proposes-bert-normalization-speedups-2026","image":null,"importance":0.6,"language":"fr","modified_at":"2026-10-15T18:13:43+00:00","publication_basis":"publication-record","published_at":"2026-10-15T18:13:43+00:00","reading_minutes":1,"region":"world","relations":[],"section_illustration":{"alt":"Illustration conceptuelle de la technologie, des processeurs et des informations connectées.","asset_id":"newsverum-v1-technology","avif_variants":[{"height":320,"url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-480.avif","width":480},{"height":480,"url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-720.avif","width":720},{"height":640,"url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-960.avif","width":960},{"height":800,"url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-1200.avif","width":1200},{"height":1024,"url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-1536.avif","width":1536}],"credit":"Newsverum","documentary":false,"height":1024,"kind":"symbolic-editorial-illustration","language":"fr","scope":"section","url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-1536.webp","variants":[{"height":320,"url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-480.webp","width":480},{"height":640,"url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-960.webp","width":960},{"height":1024,"url":"https://newsverum.com/assets/newsverum/illustrations/v1/technology-1536.webp","width":1536}],"width":1536},"slug":"hugging-face-tokenizers-proposes-bert-normalization-speedups-2026","sources":[{"checkedAt":"2026-10-15","name":"Hugging Face Tokenizers team","publishedAt":"2026-10-15","url":"https://huggingface.co/blog/huggingface/tokenmaxxing-bert"}],"summary":"Les gains rapportés varient selon la langue et la taille des entrées; l’optimisation restait en examen.","text":"Les gains rapportés varient selon la langue et la taille des entrées; l’optimisation restait en examen.\n\nDans un billet du 15 octobre, l’équipe Tokenizers de Hugging Face a présenté des changements proposés pour accélérer la normalisation des textes de type BERT. La méthode évite ce travail sur les séquences ASCII, mémorise les décisions Unicode par pages de 256 points de code et ne crée pas de nouvelle chaîne si le texte ne change pas. Le profil de l’équipe attribue 88 % du temps d’encodage anglais à chaud à la normalisation, contre 8 % à la sélection de tokens par WordPiece.\n\nLe billet indique qu’une branche de développement a encodé bert-base-uncased jusqu’à environ 50 fois plus vite que Tokenizers 0.23 sur les entrées testées, avec des identifiants de tokens identiques dans les corpus rapportés. Les gains sont les plus élevés pour l’anglais et le code; l’arabe, le grec et le russe progressent de 2,2 à 3,2 fois, tandis que le gain est faible ou nul pour le chinois. Les petites entrées peuvent ralentir, et les changements étaient encore au stade de la demande de fusion : le multiplicateur ne garantit donc pas les performances en production.","title":"L’équipe de Hugging Face propose d’accélérer la tokenisation BERT","topics":[],"translations":[{"api_url":"https://newsverum.com/api/v1/articles/hugging-face-tokenizers-proposes-bert-normalization-speedups-2026?lang=tr","language":"tr","url":"https://newsverum.com/tr/news/hugging-face-tokenizers-proposes-bert-normalization-speedups-2026"},{"api_url":"https://newsverum.com/api/v1/articles/hugging-face-tokenizers-proposes-bert-normalization-speedups-2026?lang=en","language":"en","url":"https://newsverum.com/en/news/hugging-face-tokenizers-proposes-bert-normalization-speedups-2026"},{"api_url":"https://newsverum.com/api/v1/articles/hugging-face-tokenizers-proposes-bert-normalization-speedups-2026?lang=ar","language":"ar","url":"https://newsverum.com/ar/news/hugging-face-tokenizers-proposes-bert-normalization-speedups-2026"},{"api_url":"https://newsverum.com/api/v1/articles/hugging-face-tokenizers-proposes-bert-normalization-speedups-2026?lang=fr","language":"fr","url":"https://newsverum.com/fr/news/hugging-face-tokenizers-proposes-bert-normalization-speedups-2026"},{"api_url":"https://newsverum.com/api/v1/articles/hugging-face-tokenizers-proposes-bert-normalization-speedups-2026?lang=es","language":"es","url":"https://newsverum.com/es/news/hugging-face-tokenizers-proposes-bert-normalization-speedups-2026"},{"api_url":"https://newsverum.com/api/v1/articles/hugging-face-tokenizers-proposes-bert-normalization-speedups-2026?lang=de","language":"de","url":"https://newsverum.com/de/news/hugging-face-tokenizers-proposes-bert-normalization-speedups-2026"},{"api_url":"https://newsverum.com/api/v1/articles/hugging-face-tokenizers-proposes-bert-normalization-speedups-2026?lang=pt","language":"pt","url":"https://newsverum.com/pt/news/hugging-face-tokenizers-proposes-bert-normalization-speedups-2026"}],"url":"https://newsverum.com/fr/news/hugging-face-tokenizers-proposes-bert-normalization-speedups-2026"},"version":"1"}