GigaToken Speeds Up Language Model Tokenisation by Up to 1,000x

GigaToken is a new open-source tool on GitHub that claims to deliver language model tokenisation up to approximately 1,000 times faster than existing solutions. The project drew attention from the developer community on Hacker News, earning 28 points. Faster tokenisation could significantly accelerate both the training and inference stages of large language models.