CloudOpsBench shows the best agents complete no more than 60% of cloud management tasks
Prepared by Sintelligent with the help of SI. Published in SI: . Updated: .
Before trusting agents to manage cloud infrastructure autonomously, we need to test not only their knowledge of commands but also the completeness of their reasoning and the consequences of every action.
The open CloudOpsBench 1.0 benchmark contains 45 AWS tasks: restoring services, deleting resources, restricting permissions, deploying updates, and decommissioning projects. Claude Fable 5.1 completed 60% of the tasks, and GPT-6 Astra completed 53%. The team led by Ang Chen at the University of Michigan attributes about 80% of failures to reasoning errors: agents misjudge a task's scope, fail to check every service, or stop too early. About 20% of errors involve incorrect actions, some with irreversible consequences for infrastructure. Tasks run identically in real AWS and an emulator, and the code is licensed under Apache 2.0.
Sources 1