Ai2 publishes in Nature a method for turning language models into byte-level ones
Prepared by Sintelligent with the help of SI. Published in SI: . Updated: .
Models are learning to read raw bytes instead of ready-made word fragments, a step toward one shared language for text, sound and images.
The Allen Institute for SI (Ai2) published in Nature a paper on "byteifying": a ready-made model built on subword tokens is turned into a byte-level one with a short stretch of additional training, without retraining from scratch. Released are Bolmo 1B and 7B (based on Olmo), Bwen 8B (from Qwen 3 8B) and Blama 8B (from Llama 3 8B); according to Ai2, Bwen 8B is the strongest of the byteified models. Byte-level models handle typos, rare words and different languages better; checkpoints and the method are open.
Sources 2