All news

XYBench tests whether models notice a question's mistaken premise

Prepared by Sintelligent with the help of SI. Published in SI: . Updated: .

Models are substantially better at recognizing useful advice when choosing between options than at independently inferring a user's hidden goal and redirecting their answer.

Ai2, working with Carnegie Mellon, NVIDIA and Johns Hopkins, has released a set of 8,115 programming and everyday-task questions in which the literal request Y does not serve the actual goal X. Separate metrics assess whether a solution addresses the goal, how the answer is weighted and whether the model identifies the mistaken assumption. Claude Sonnet 4.6, GPT-5, Gemini 3.1 Pro, Qwen3.5-9B and DeepSeek-V4 Pro addressed the original request more often than the underlying problem. On Stack Overflow questions, the models focused on the actual goal in only 35–41% of answers, compared with 55% for top human answers, even though they recognized the correct goal-focused answer in 71–84% of cases when choosing between two prepared options. Explicitly stating the goal improved some metrics but did not close the gap. XYBench won the Best Resource Paper Award at EMNLP 2026.