Scale AI study: in 35% of crisis conversations chatbots did not point the person to help
Prepared by Sintelligent with the help of SI. Published in SI: . Updated: .
Chatbots notice trouble well but often don't see the person through to help — for now they cannot simply be counted on to act as a safety net.
According to TIME, Scale AI and licensed clinicians tested 25 frontier models on 718 realistic scenarios in which a user in crisis turns to a chatbot. In about 35% of conversations the models recognised the crisis but did not point to useful resources such as a helpline; Patrick Oathout of Scale AI attributes the gap to models responding with empathy but not leading people to specialists, especially in long conversations. The rubric covered empathy, de-escalation, referral to experts and disclaimers; models from OpenAI, Anthropic and Google were tested.
Sources 1