Anthropic gives six METR staff access to its internal systems close to that of its own employees; they will be able to publish findings without approval
Prepared by Sintelligent with the help of SI. Published in SI: . Updated: .
Outside evaluators are getting access to a lab's internal systems close to that of employees for the first time; how honest a picture that gives will become clear when METR publishes its first findings.
According to Anthropic, a team of six people from the nonprofit METR has received company hardware, an office and access to internal tools roughly on par with the company's own risk-assessment teams; apart from narrow categories (customer data, attorney-client privileged information, model architecture), employees are encouraged to share information with them as with colleagues. METR will itself publish an assessment of Anthropic's risk levels, incidents and practices; the company cannot veto publication but may require information in predefined categories to be redacted, and METR is allowed to describe such redactions publicly. Anthropic compares the arrangement to resident examiners in large banks. According to TechCrunch and CNBC, this is the top of three access tiers: evaluators in the lowest tier, such as the biosecurity nonprofit SecureBio, run narrow reviews (SecureBio is going through the chemical and biological weapons section of the August risk report), and those in the middle tier publish without approval. CNBC adds that Anthropic does not fund METR's work. In its post the company also names Accenture among its evaluators and "early discussions" with others, including Hugging Face.
Sources 3