All news

METR puts frontier models' task horizon at 27 hours on the TH2 set

Prepared by Sintelligent with the help of SI. Published in SI: . Updated: .

Models already solve multi-hour tasks, but agent swarms speed things up only where results can be checked quickly.

METR released a Q4 2026 update with evaluations of 15 public models, up to Opus 5.5 and GPT-6 Astra. On a verified subset of new TH2 tasks, the estimated time horizon of frontier models reaches 27 hours. On MirrorCode, Opus 5.5 solves 77% of medium and large programs, while GPT-6 Astra scores 98% on FrontierMath Tier 4. In 15 Al Zimmermann contests, the strongest models' median placement is the 96th–98th percentile of humans, but the absolute human leaders are still ahead. On Budget NanoGPT, the best agent run produced a 15.7% speedup in 48 hours, versus 41% for the strongest human entry. The authors conclude that public evidence is insufficient to rule out a sharp acceleration in SI R&D over the next couple of years. New metrics were introduced — the METR Capability Index and Lifecycle Cost (a proxy for the cost of an RL environment). Agent swarms help only on verifiable tasks with fast feedback.

Sources 1

History

External auditors at SI labs

11 events
    1. METR puts frontier models' task horizon at 27 hours on the TH2 setOpen now
    1. OpenAI evaluators got access to only some models; critics call it weaker than promised
    1. METR estimates Anthropic's model development has roughly doubled in pace with help from SI
    1. METR and Redwood Research spent three weeks testing OpenAI's internal agent monitoring system: by their estimate, 75–94% of traffic was eligible for monitoring; OpenAI says the vulnerabilities have been fixed
    1. METR reviews the section of Anthropic's report on misalignment risk: the risk is "low", but by its assessment because of the models' limited capabilities, not safeguards
    1. Anthropic gives six METR staff access to its internal systems close to that of its own employees; they will be able to publish findings without approval
    1. METR publishes five pieces on independently assessing labs' risk of losing control of SI
    1. Independent SI model evaluators METR, Apollo Research and Transluce are at the center of the safety debate, but how to fund them is unresolved
    1. OpenAI will let outside reviewers access models during training, but without the "desks and badges" Altman promised
    1. More than 100 researchers, including Hinton, name five conditions without which "external reviewers" inside SI labs are useless
    2. Anthropic lets outside auditors from Accenture inside the company, with the same access as employees