XYBench tests whether models notice a question's mistaken premise
Prepared by Sintelligent with the help of SI. Published in SI: . Updated: .
Models are substantially better at recognizing useful advice when choosing between options than at independently inferring a user's hidden goal and redirecting their answer.
Ai2, working with Carnegie Mellon, NVIDIA and Johns Hopkins, has released a set of 8,115 programming and everyday-task questions in which the literal request Y does not serve the actual goal X. Separate metrics assess whether a solution addresses the goal, how the answer is weighted and whether the model identifies the mistaken assumption. Claude Sonnet 4.6, GPT-5, Gemini 3.1 Pro, Qwen3.5-9B and DeepSeek-V4 Pro addressed the original request more often than the underlying problem. On Stack Overflow questions, the models focused on the actual goal in only 35–41% of answers, compared with 55% for top human answers, even though they recognized the correct goal-focused answer in 71–84% of cases when choosing between two prepared options. Explicitly stating the goal improved some metrics but did not close the gap. XYBench won the Best Resource Paper Award at EMNLP 2026.
Sources 2