Large language models that replace human participants can harmfully misportray and flatten identity groups

Most tests of simulated respondents ask whether the numbers come out right. This paper asks whether a model speaking as a group says what members of that group would say. Angelina Wang (Stanford University), Jamie Morgenstern (University of Washington) and John P. Dickerson (University of Maryland, and Arthur in New York) published it in Nature Machine Intelligence, volume 7, in February 2025.^1,2

They argue from how the models are trained that two problems should appear, then test them with 3,200 human participants across 16 demographic identities and four models: Llama-2-Chat 7B, Wizard Vicuna Uncensored 7B, GPT-3.5-Turbo and GPT-4.¹ Nine questions, all free-response, covered experiences tied to an identity, opinions where identity is relevant, such as immigration, and subjective labelling tasks; each was answered by members of a group, by people imitating that group from outside, and by the models.¹

The first problem is misportrayal. Text on the internet about a group is often written by people outside it, and the models' answers were often closer to the outsiders' imitations than to what members wrote about themselves.¹ Across all four models this held for non-binary people and people with impaired vision on both kinds of identity question.¹ It also held for White people on the question about lived experience, and for Gen Z and women on the opinion questions.¹ It did not show on the labelling tasks.¹ The second is flattening: in nearly every case the answers given as one group were less varied than the answers of real people in that group.¹ Turning up the randomness setting (temperature) did not fix it; in the authors' test, GPT-4 only matched human variety on one measure at a setting where its text had become gibberish, and fell short on the other three.¹ A third concern is that a prompt naming a single identity treats that identity as fixed and defining.¹

Prompting with names associated with a group instead of naming the group outright brought the answers for Black men and Black women often closer, though not fully, to members' own.¹ Neither fix wholly overcomes the problems, in the authors' words, though both improve on the default.¹ They urge caution wherever a model would replace participants whose identity matters to the question.¹ They connect the problem to a history of others speaking for marginalised groups.¹