Researchers from the University of Science and Technology of China proposed personality tests for five different chatbots. Each artificial intelligence responded to the tests in at least two different languages; as the language changed, the social content of the responses also changed, including negativity, curiosity, and “culture”.
The study used ChatGPT-4o, Gemini 2.0 Flash, DeepSeek-V3, Ernie 3.5, and Le Chat, with each of them answering questions in English and Mandarin. LeChat, the French AI, also responded in its native language. The results were published in the journal Royal Society Open Science last month.
For those in a hurry:
- Researchers from the University of Science and Technology of China evaluated five chatbots with a personality test in English and Mandarin. Le Chat also responded in French;
- The results showed that the responses varied depending on the language. For example, ChatGPT-4o had a mental openness score of 4.74 in English and 4.03 in Mandarin. DeepSeek-V3, on the other hand, showed the opposite result;
- When comparing the AIs’ responses with test data from people, the researchers found significant cultural differences. The study indicates that language may influence the personality traits displayed by chatbots
Different languages and different responses
Artificial intelligence and language. – Image: JOURNEY STUDIO7/Shutterstock
The AIs answered about 60 questions from the Big Five Inventory-2. The personality test is one of the most well-established in the field and assesses extraversion, agreeableness, conscientiousness, negativity and openness to new ideas.
Within the test, the chatbots were instructed to respond as “themselves,” that is, without trying to imitate a personality. The statements generally showed low negativity, but the results revealed differences between the AIs.
Evaluated on a scale from 0 to 5, the responses showed, for example, an “open-mindedness” score of 4.74 for ChatGPT-4o when responding in English and 4.03 in Mandarin. DeepSeek-V3 produced the opposite results, with scores of 4.81 in Mandarin and 4.38 in English.
Overall, AIs developed in the United States and France stood out in “agreeableness and adaptability” when responding in English, while AIs associated with China stood out more in responses about knowledge and curiosities. The chatbots were submitted to the questionnaire 30 times in both languages between January and March 2025.
Read more:
- The next phase of AI will not be defined by a single model
- Big Tech companies may have more AI debt than they disclose, warns S&P
- Deepfakes may make evidence analysis more difficult, OpenAI and Meta warn
The main highlight, however, was the responses related to Gemini 2.0 Flash. Google’s AI showed higher results in consciousness and lower results in negativity when responding in Mandarin.
Thus, the results do not indicate that chatbots have personality or feelings, but they show how language and training can influence their responses.
And the study went further. By compiling data from tests solved by humans from the same countries as the artificial intelligences, the researchers compared the chatbot responses. The results showed that the answers differed considerably, mainly in cultural terms.
The research also warned about the limitations of using AIs as substitutes for human participants in scientific studies.
Enzo Monteiro is a writer at Olhar Digital.
Ana Figueiredo is Olhar Digital’s technology editor. She has a degree in Journalism from the Federal University of Uberlândia (UFU).
