Redwood Research proposes testing oversight of SI agents in simulation: a 3.6 percentage-point forecast error versus 10.1 for the usual test
Prepared by Sintelligent with the help of SI. Published in SI: . Updated: .
Researchers propose a cheap way to check whether oversight would notice an agent's dangerous actions without running real attacks; only the authors have tested the method.
Redwood Research described a method called simulated control evaluations (SCE): instead of running attacks on real systems, a simulator language model writes plausible tool outputs from the transcript and the developer's code, and a judge model decides whether the attack would have succeeded — a way to check whether a monitoring system would notice an SI agent's harmful actions. According to the authors, the error in predicting safety metrics was 3.6 percentage points against 10.1 for the standard BashArena test, and the similarity between the model's behavior in simulation and in real conditions was an order of magnitude better. The method explained 95% of the differences in safety between task categories and 78% between environments; beating BashArena's forecast took just 4.6% of real task samples. The results were obtained by the authors themselves.
Sources 1