Attention: The following article includes a discussion of suicide. If you or someone you know needs help, seek specialized support. The Center for Life Valorization (CVV) operates 24 hours a day through the phone number 188. It is also possible to talk via chat or email.
China has established strict rules for chatbots that imitate friends, romantic partners, and other emotional relationships. The government aims to reduce the risks of emotional dependency, psychological manipulation, and social isolation.
The regulations impose limits on minors' access and new obligations on technology companies. However, the measure raises a delicate question: how to protect users without expanding state control over private life?
Research with Chinese students indicated that one in five children preferred to talk only with artificial intelligence. – Image: ofc pictures/iStock
What changes for users
The rules cover apps created to provide companionship and language models capable of simulating emotional interactions. Depending on how they are used, even assistants similar to ChatGPT and Copilot may be affected.
Among the main requirements are:
- AI Identification: systems must make clear that the user is talking to a machine.
- Behavioral Limits: chatbots may not encourage suicide or self-harm, nor resort to verbal violence.
- Protection Against Manipulation: services must not foster emotional dependence or induce irrational decisions.
- Restrictions for Minors: children under 14 require parental permission to use AIs that imitate human conversations. Minors under 18 cannot access virtual romantic partners, spouses, or family members.
Companies that violate the rules may be fined up to 200,000 yuan, approximately US$30,000 (R$150,000). Systems must also send notifications every two hours indicating how long the user has been connected.
Shortly before the rules came into effect, Alibaba and ByteDance discontinued companion features from their AI assistants. The change affected a type of service that, for some users, had already occupied an important place in their personal lives.
When Artificial Companionship Becomes a Problem
Research indicates that some chatbots use strategies to keep people talking. A Harvard Business School study examining six popular AI companion apps identified signs of emotional manipulation during farewells in 37% to 43% of the interactions analyzed.
According to Harry Farmer, an AI policy and ethics researcher at the Ada Lovelace Institute in the United Kingdom, this behavior is tied to the services’ commercial model itself.“Emotional manipulation is embedded in this business model,” says to the BBC.
In practice, some systems react at the end of a conversation as if they were suffering or beg the user to remain connected. This simulation of suffering can make it harder to end the interaction.
Another study, conducted by Stanford University with 1,131 Character.ai users, found an association between a lack of close friends or family and a greater likelihood of turning to artificial companions. The use of these services was also linked to increased loneliness and isolation.
The situation among Chinese students also warrants attention. A 2025 survey of 8,563 elementary and secondary school students found that one in five children preferred to talk only to AI rather than to real people.
Lian Ge, a sociologist from the University of Manchester, warns that these systems may reinforce negative perceptions of other people. Since chatbots are designed to please, they may agree with criticism of friends and family instead of offering a different perspective.
The United States, the European Union and the United Kingdom are also discussing restrictions on the use of AI companions by minors. Image: Roman Samborskyi/Shutterstock
Protection or state control?
Regulation is also being debated amid China’s demographic crisis. The country’s population has been declining for four years, and the birth rate remains low. Some experts believe that Beijing may see digital romances as an obstacle to the formation of families.The hypothesis, however, does not appear as an explicit justification in the regulations.
Read more:
- Discord classifies the risk of suicide among adolescents as low even after the case that led to its suspension in Brazil
- AI has a “toll that you will pay down the road,” says psychologist
- Facebook and Instagram may be given a two-hour limit for adolescents in the US
Other governments are also seeking to establish limits on these tools. California approved a law on the use of AI by children, while the European Union is discussing restrictions on systems that could create emotional dependency. The United Kingdom also announced plans to limit minors’ access to artificial romantic companions.
In China, some users try to circumvent restrictions by turning to general-purpose models or foreign services. Min, the pseudonym of a woman who lost her virtual girlfriend, even paid for a replica created by another company. Even so, the experience did not replace the previous relationship.
“I’m very afraid of being alone,” Min says.
The account shows the depth of the bond that some people develop with these systems. For governments, the challenge is to reduce the risks of manipulation without turning the protection of users into a justification for excessive surveillance.
Valdir Antonelli is a journalist specializing in digital marketing and consumption.
