TasteVal benchmark: models' "taste" for designing SI experiments doubles every three months, according to its authors
Prepared by Sintelligent with the help of SI. Published in SI: . Updated: .
If the authors' estimate is right, models are quickly learning to design experiments that advance SI itself.
TasteVal measures models' "experimental research taste": how much compute they need to reach a given result on SI development tasks compared with human experts. The benchmark has 8 tasks (pretraining data curation, fine-tuning, preference modelling, robustness to attacks): a coding agent implements the experiments, while the model being evaluated designs them and analyses the results; baselines came from 24 experts at OpenAI, Google DeepMind, NVIDIA and Microsoft Research. According to the authors, the measure has doubled every 3.0 months since December 2025 (95% interval: 1.7–5.0 months); Opus 5.5 reached a compute multiplier of 2.30 relative to the experts, spending about 1/30 of the average cost per attempt of human researchers. The authors caution that the results may overstate the pace because the tasks are quick and cheap to verify and the baselines do not include top researchers; limited elicitation of the models, conversely, could have understated the gain.