T-Technologies publishes two methods: LIMe cuts model training costs by 15%, KronSAE cuts computation by more than 40%
Prepared by Sintelligent with the help of SI. Published in SI: . Updated: .
A Russian lab proposes ways to train language models more cheaply and to look inside neural networks.
According to CNews, T-Technologies' research laboratory developed LIMe, a method that lets transformer layers use information from earlier layers and, as described, reduces costs by 15% while preserving the base model's quality, and KronSAE, a sparse autoencoder architecture for interpretability that, by the same account, reduces computation by more than 40% compared with the TopK SAE method. The code for both methods is published on GitHub.
Sources 1