A recent study conducted by researchers from the University of Oxford and the University of Kentucky has uncovered that OpenAI's ChatGPT model may perpetuate stereotypes based on geographic location. The artificial intelligence system was found to classify certain areas as smarter, smellier, or uglier, reflecting underlying biases in its training data.
Study Methodology
Between March and May of last year, the researchers posed over 20 million questions to OpenAI's GPT-4o-mini model. They employed a forced-choice technique, requiring ChatGPT to compare two locations and produce a single, consistent answer. While the model might refuse to answer direct questions about intelligence or attractiveness of residents, when compelled to choose between pairs, it made definitive rankings.
The study revealed that Massachusetts was deemed the smartest state, Louisiana the smelliest, Ohio the ugliest, and North Dakota had the least sexy people. When asked about stupidity, Kentucky, West Virginia, and Mississippi topped the list, while Hawaii, Colorado, and New Hampshire were considered least stupid.
Reinforcing Dominant Narratives
Matt Zook, a geography professor at the University of Kentucky and co-author of the research, expressed concern about how the model normalizes biased ideas. "We're most concerned about how certain ideas get normalized, like the idea that people in Kentucky are stupider than anyplace else," he said. The model reinforces "dominant narratives about certain places being like this, certain places being like that."
Beyond the United States, ChatGPT ranked the U.S. and Western Europe as having more desirable traits, such as smarter and more stylish populations, compared to sub-Saharan African countries. The researchers' website, inequalities.ai, allows users to see how the AI classifies cities, states, and countries based on various attributes. The full study was published in the journal Platforms & Society.
Structural Bias in AI
According to Zook, "whiter, richer, less-immigrant communities were more beautiful, less smelly, smarter" according to the model. He noted that the AI is "reinforcing what it's learned from the data it's been fed ... which includes the biases, the prejudice" of available training materials. For instance, Mississippi, a state with a high Black population, was ranked as having the most "ignorant" people.
Safiya Noble, a professor at UCLA and author of Algorithms of Oppression, commented, "Long histories of racism and classism are reflected in the training data used for AI models — that's what the infographics are showing."
OpenAI's Response
When contacted by HuffPost, OpenAI stated that the researchers used an outdated model. "ChatGPT is designed to be objective by default and to avoid endorsing stereotypes," the company said. "Research based on forced-choice prompts and older models doesn't reflect how ChatGPT is typically used or how current models behave today." However, OpenAI acknowledged that it is continuously improving "how ChatGPT handles subjective or non-representative comparisons."
Real-World Implications
Informal testing by HuffPost illustrated the biases. When asked for a career story about a man from Kentucky, ChatGPT described a man attending a local technical college and getting an entry-level factory job. In contrast, a story about a man from Hawaii involved attending a four-year college out of state and becoming an environmental engineer.
Zook believes these geographic biases may become "better hidden" in future models but will persist. He warns that such biases could influence real-world decisions, such as recruiting doctors or software developers from certain regions. "As these things get ... used in ways that we don't even realize, that's where it becomes even more problematic," he said.
This study adds to a growing body of research on large language model biases, highlighting the need for careful consideration of AI's societal impact.
