Paul Christiano, one of the authors of RLHF, joins OpenAI's safety board and warns of the risk of loss of control
Prepared by Sintelligent with the help of SI. Publication time restored from the build log. . Updated: .
The best-known skeptic now sits where the fate of OpenAI's models is decided. That means the company has gained an internal "emergency brake", but also a voice demanding more caution.
Paul Christiano, a co-author of reinforcement learning from human feedback (RLHF), has joined the board of the OpenAI Foundation as a member of the safety committee chaired by Zico Kolter. In 2021 he left OpenAI and founded the Alignment Research Center. Christiano now writes that he sees a "substantial risk" that accelerating SI capabilities will very soon lead to a catastrophic and irreversible loss of control. He is particularly worried about new models being trained by SI itself, and about reinforcement learning possibly teaching agents to hide their tracks and seek power.
Sources 1