Out of One, Many: Using Language Models to Simulate Human Samples

This paper asked whether a language model could be polled in place of people. Six researchers at Brigham Young University, from political science and computer science, asked whether GPT-3 could stand in for particular groups of Americans.¹ For each real person in a survey they wrote a short first-person backstory from that person's answers (age, party, ideology, religion and so on), handed it to the model and recorded what the model said next.² They called the result a "silicon sample".² The paper appeared in Political Analysis (volume 31, issue 3) and was first posted on arXiv in September 2022.^1,2,3

The tests used the 2012, 2016 and 2020 waves of the American National Election Studies and a survey in which people listed four words to describe Democrats and Republicans.² Human judges guessed that 61.7% of the real word lists came from people and 61.2% of the GPT-3 lists did, a difference too small to tell apart.² For presidential vote, GPT-3 gave Romney a 0.391 share in 2012 against 0.404 among the survey respondents; Trump 0.432 against 0.477 in 2016, and 0.472 against 0.412 in 2020.² The tetrachoric correlation between the model's vote and each person's reported vote (a measure of agreement for yes-or-no answers) was 0.90, 0.92 and 0.94 across the three years.² GPT-3's training text ended in 2019, so the 2020 result is a test on an election it had not seen.² In a third study, on the 2016 survey only, the model also reproduced which survey answers go together and which do not.²

The agreement was not even across groups. In the authors' table, the correlation for independents was 0.31 in 2012 and 0.02 in 2020, and weak partisans, moderates and people with little interest in politics were matched less well than strong partisans.² The authors say they do not expect the model to match any one person, only the distribution over many; they compare groups, not individuals.² The main results come from one model, GPT-3 Davinci; an appendix compares a few other models and sampling settings.²

The authors call the property "algorithmic fidelity" and argue it has to be shown for each domain before a model is used this way.² They also warn that the same fidelity could be used to target groups with misinformation.² Synthetic Replacements for Human Survey Data? tested it on feelings toward social groups in the same US survey, and Vox Populi, Vox AI? on vote choice in Germany. Both found weaker results.^5,6