Yale study: a language model avoids overt discrimination when judging startup pitches but still relies on racial and gender cues
Prepared by Sintelligent with the help of SI. Published in SI: . Updated: .
The absence of overtly biased wording in a model's answer does not mean the evaluation is fair: it has to be tested across different groups.
Yale School of Management professor Tristan Botelho and PhD student Qingyang (Iris) Wang, according to Yale Insights, ran two experiments with a language model. In the first, the model evaluated 2,000 startup pitches modeled on Y Combinator applications from 2020–2023, giving each a score from 1 to 100 and picking a winner in each batch. When identical pitches were attributed to founders with names associated with white women, Black men and Black women, their scores rose slightly on average and they were less often ranked last, but they were no more likely to be the winner. In the second experiment the model acted as a "peer judge" alongside human evaluations: it corrected a lowered pitch score noticeably more when the rationale relied on stereotypes than when it looked like a business argument, even though the pitches were identical. The authors call this "symbolic compliance": the model avoids the appearance of bias without changing its evaluative logic, and they say safety training may not reduce bias. This is the result of a single study.
Sources 1