Hugging Face team reports faster BERT tokenization in proposed changes

A Hugging Face Tokenizers team post on October 15 described proposed changes aimed at speeding up BERT-style text normalization. The approach skips normalization work on ASCII stretches, caches Unicode decisions in 256-code-point pages and avoids allocating a new string when text is unchanged. The team’s profile attributed 88% of warm English encoding time to normalization, compared with 8% for WordPiece token selection.

The post reports that a development branch encoded bert-base-uncased up to about 50 times faster than Tokenizers 0.23 on tested inputs, with identical token IDs in the reported corpora. Results were strongest for English and code; Arabic, Greek and Russian showed 2.2–3.2× gains, while Chinese gained little or none. Small inputs could run slower, and the changes were still at pull-request stage, so the multiplier is not a general production guarantee.