# Hugging Face team reports faster BERT tokenization in proposed changes

> The reported gains vary by language and input size, and the optimization was still under review.

- Canonical URL: https://newsverum.com/en/news/hugging-face-tokenizers-proposes-bert-normalization-speedups-2026
- Publisher: Newsverum
- Section: Technology
- Country: United States
- Published: 2026-10-15T18:13:43+00:00
- Updated: 2026-10-15T18:13:43+00:00
- Event / data period: 2026-10-15
- Other editions: [Türkçe](https://newsverum.com/tr/news/hugging-face-tokenizers-proposes-bert-normalization-speedups-2026.md), [العربية](https://newsverum.com/ar/news/hugging-face-tokenizers-proposes-bert-normalization-speedups-2026.md), [Français](https://newsverum.com/fr/news/hugging-face-tokenizers-proposes-bert-normalization-speedups-2026.md), [Español](https://newsverum.com/es/news/hugging-face-tokenizers-proposes-bert-normalization-speedups-2026.md), [Deutsch](https://newsverum.com/de/news/hugging-face-tokenizers-proposes-bert-normalization-speedups-2026.md), [Português](https://newsverum.com/pt/news/hugging-face-tokenizers-proposes-bert-normalization-speedups-2026.md)

A Hugging Face Tokenizers team post on October 15 described proposed changes aimed at speeding up BERT-style text normalization. The approach skips normalization work on ASCII stretches, caches Unicode decisions in 256-code-point pages and avoids allocating a new string when text is unchanged. The team’s profile attributed 88% of warm English encoding time to normalization, compared with 8% for WordPiece token selection.

The post reports that a development branch encoded bert-base-uncased up to about 50 times faster than Tokenizers 0.23 on tested inputs, with identical token IDs in the reported corpora. Results were strongest for English and code; Arabic, Greek and Russian showed 2.2–3.2× gains, while Chinese gained little or none. Small inputs could run slower, and the changes were still at pull-request stage, so the multiplier is not a general production guarantee.

## Data

### Announcement summary

Methodology: This event record summarizes the source announcement and adds no independent verification or generalization beyond it.

| date | label | description | status |
|---|---|---|---|
| 2026-10-15 | Hugging Face team reports faster BERT tokenization in proposed changes | The reported gains vary by language and input size, and the optimization was still under review. | occurred |

Sources: <https://huggingface.co/blog/huggingface/tokenmaxxing-bert> · Download data: [JSON](https://newsverum.com/api/v1/articles/hugging-face-tokenizers-proposes-bert-normalization-speedups-2026/data/announcement-record?lang=en&format=json) · [CSV](https://newsverum.com/api/v1/articles/hugging-face-tokenizers-proposes-bert-normalization-speedups-2026/data/announcement-record?lang=en&format=csv)

## Sources

- [Hugging Face Tokenizers team](<https://huggingface.co/blog/huggingface/tokenmaxxing-bert>) (Source checked: 2026-10-15; Published: 2026-10-15)

---

Cite the canonical URL above. [Editorial standards and corrections](https://newsverum.com/en/editorial)