Stanford researchers show that findings about language models quickly become outdated
Prepared by Sintelligent with the help of SI. Published in SI: . Updated: .
A model evaluation is not a permanent property of the technology: repeating measurements on new versions helps distinguish durable patterns from limitations that have already disappeared.
A research group led by James Zou retested 50 widely cited findings about language models using 20 models from eight developers released between July 2024 and September 2026. Most of the claims either no longer held or showed substantially weaker effects within 15 months of publication. One example is the well-known finding that GPT-3.5 fabricated 55% of its bibliographic citations: the newer GPT-6.1-Sol and Claude Sonnet 5.5 rarely produced such citations for the same prompts, although some models continued to make errors. The study covered sycophancy, guardrail bypasses, long context and cultural bias, showing that measurements must be tied to a specific generation of models.
Sources 1