SI costs more in languages other than English: study finds up to a 5x gap in token counts
Prepared by Sintelligent with the help of SI. Published in SI: . Updated: .
Users of less widely spoken languages pay more for SI and get it slower: the inequality is built into how the models work.
A study by the Coalition for Language Equity in AI Research, published in September, found that the same text requires different numbers of tokens from SI models depending on the language. Among the 20 most popular languages on the internet, the most "expensive" uses 1.5 to 5 times more tokens than the cheapest; when the set was expanded to 200 languages, the gap in one model grew 17-fold. The authors call this a "tokenizer tax": more tokens means not only a higher price but also higher latency and a smaller effective context.
Sources 1