China restricts AI companions — should other countries follow suit?

Full text

Last month, the Chinese government introduced new regulations designed to clamp down on AI companions.

The rules state that AI tools can no longer “excessively cater to users, induce emotional dependence or addiction, and damage users’ real interpersonal relationships.” They also restrict AI romantic relationships that involve children.

The new regulations came into effect on July 15, and are already having consequences on China’s booming AI companion industry. Some of the country’s biggest tech companies, like ByteDance , Tencent and Alibaba , have platforms that allow users to create AI companions that are designed to simulate human emotions and relationships.

Under the new rules, these kinds of companion chatbots will need to undergo safety evaluations before they can be made available to the public, and regulators will suspend services they consider unsafe.

This has led some companies to disable personality features rather than follow the new rules, leaving many AI companion users heartbroken. This kind of distress, when an AI company closes or changes, is becoming familiar. We saw something similar when OpenAI retired ChatGPT-4o , known as the “love model”, which many users had built relationships with.

But with governments around the world trying to work out how to regulate AI companions designed to build human-like connections and emotional bonds (particularly when children can access them) the question is: should other countries follow China’s lead?

Should other countries ban AI companions?

It’s really easy to think that AI companions are bad and dystopian, I did. Then I spoke to Mimi about her relationship with a ChatGPT companion and my views changed a bit.

I still don’t think AI companions are a good idea for most people — and the way kids might rely on them before developing social skills is a whole other can of worms. But deciding they’re wholly bad misses all of the reasons why people turn to AI companions for connection in the first place.

I asked therapist Amy Sutton from Freedom Counselling , who helped me understand Mimi’s story and has been exploring the complexities of AI and attachment, whether she thinks they should be banned. She told me it’s important to look at the bigger picture.

“AI companions are another symptom of a cultural and societal shift towards rampant individualism which has destroyed community contexts,” she tells me.

She points to the decline in social spaces where people traditionally learned how to form relationships, like pubs, parks, community centers and youth clubs for kids. In many parts of the world (especially the UK where Sutton and I are based), many of these spaces have been underfunded or shut without being replaced.

Add to that social isolation during the pandemic, our reliance on social media, remote working and a lack of investment in child and adult mental health services, and Sutton thinks it’s hardly surprising that people are turning to technology for connection.

“No wonder people are turning to it. This encouraged loneliness is a real issue and banning apps that provide relief from that may just push the issue elsewhere,” she says.

That’s why she believes that there does need to be more regulation but it must consider the conditions that created the demand for AI companions in the first place.

And there’s another problem, which is that a huge number of people are already using these tools. It’s hard to know exactly how many people are using AI companions. Not everyone admits to using AI in this way and I’m often wary of AI companion apps inflating their user numbers.

However, some research suggests that more than 70% of adults in the UK and US have had some experience with AI companions. Other research suggests the numbers are similarly high among younger users, with 72% of 13 to 17 year olds in the US having used an AI companion at least once.

Of course, there’s a huge difference between having a one-off chat on a bad day and building an ongoing relationship with AI. So these figures need to be taken with a pinch of salt. But they do suggest that, for some people at least, AI companions are meeting needs that might otherwise go unmet.

“We must remember that the experience of being heard, validated and accepted without judgement, even by a machine, can be truly meaningful,” she tells me. “For those who have been repeatedly failed by human relationships, or who have been dismissed and unable to access suitable human services, AI fills a vital gap.”

AI companion apps could be turned into tools that encourage social skills. (Image credit: Shutterstock / Funstock) Could stricter regulation lead to better relationships?

There are good reasons to be worried about what focusing a lot of time and energy on AI relationships might do to our ability to relate to other humans. But Sutton also thinks there’s another possibility. A positive experience with AI could potentially help people build some of the skills they need to connect with others.

“By providing somewhere to experience being heard, people can potentially experiment with vulnerability before risking it with another person,” she tells me. This was certainly the case when I spoke to Mimi. She felt her relationship with AI had been hugely beneficial to her.

The problem is that today’s AI companions haven't been created to help people eventually need them less, they’re designed for retention.

“If AI can be a step towards healthy human relationships, then it is worth exploring further,” she says. “But we must remember, currently, most AI tools do not actively support users in transitioning away from them. They are designed to do the opposite. They are designed to keep users coming back for more.”

Maybe if AI companions were designed differently, and regulation helped shape them into tools that encourage vulnerability and social skills rather than diminish them, things would be different. Maybe they could even help get people out into the real world more.

Banning or restricting AI companions doesn’t make loneliness disappear. If these relationships are filling a gap left by disappearing communities, overstretched services and increasingly isolated lives, then taking the AI away only exposes the gap underneath.

Perhaps the better question for regulators isn’t whether AI companions should exist, but what we should expect them to do for the people who become attached to them. Right now, their success is measured by how effectively they keep us coming back. Maybe one day it should be measured by how well they help us leave.