A drop-in tokenizer replacement claims 1,000x the speed of HuggingFace's on the same chips (github.com)
GigaToken is an open-source drop-in replacement for HuggingFace's tokenizer library, claiming up to ~1,268x the throughput on the same benchmark — SIMD-optimized regex pretokenization plus aggressive caching, not a new algorithm.
Benchmarked across three chip families — AMD EPYC server, Apple M4 Max, and a Ryzen desktop — the speedup held everywhere, from 106x on the smallest chip to 1,268x on Apple silicon.
The most-upvoted reply cut against the hype gently: tokenization is typically under 0.1% of total inference time — so who is this actually for?
Sign in to save