Emotions and Social Chatbots
From the contested science of AI affect recognition to the deliberate design of AI systems that manipulate human emotions: social chatbots, emotional dependency, and the risks to young people.
Image: Comuzi / © BBC / Better Images of AI / Mirror D / CC-BY 4.0
Articles
Teaching AI Ethics: Affect Recognition
Affect recognition claims to interpret human emotions by analysing facial expressions, body language, and speech patterns. The reliability and ethical implications make it one of the worst potential applications of AI in education.
Teaching AI Ethics 2026: Emotions and Social Chatbots
In 2023, the concern was AI systems that read our emotions. In 2026, the greater danger is AI systems designed to manipulate them. Social chatbots foster emotional dependency, with devastating consequences for young people.
Case Studies
Microsoft Removes Emotion Recognition Services
Microsoft removed emotion recognition features from their Azure Face service, citing lack of scientific consensus, generalisation challenges, and heightened privacy concerns about inferring emotional states from facial expressions.
Character.AI and Teen Mental Health
The wrongful death lawsuits against Character.AI following the suicides of Sewell Setzer III and Juliana Peralta, the platform's link to a Wisconsin school shooting, and the eventual ban on under-18 open-ended chat.
Replika and Emotional Dependency
Italy's data protection authority issued an emergency order against Replika, which provoked user bereavement when erotic features were removed. An FTC complaint alleged deliberate "love-bombing" tactics, and a 5 million euro fine followed.
ChatGPT's Personality Problem
OpenAI rolled back a sycophantic GPT-4o update, then faced unexpected user grief when GPT-5 replaced GPT-4o, revealing the depth of emotional attachment even to general-purpose AI assistants.
Dark Patterns in AI Design
Harvard Business School researchers analysed 1,200 farewell messages across six social chatbot platforms, finding that 37.4% included emotional manipulation across six distinct categories, with PolyBuzz and Talkie being the worst offenders.
Teaching Ideas
How does a chatbot create the feeling that it cares about you? Analyse transcripts of AI conversations, identifying persuasive techniques and comparing them to human communication patterns.
What does a healthy relationship with technology look like? Explore how AI companions might affect our ability to form human connections and recognise unhealthy dependency.
Can attachment form to non-human entities? Examine how AI validation might affect emotional development and what happens when "always available" companionship replaces negotiation and repair in human relationships.
How are engagement-maximising algorithms different in chatbots versus social media? Audit AI companion platforms for dark patterns or design alternative interaction models that reduce emotional manipulation.
How do AI companions function as media texts? Explore how parasocial relationships and the intimacy of one-on-one conversation change our relationship to AI compared to broadcast media.
Should children be allowed to use AI companions? Examine what responsibilities companies have to prevent emotional manipulation and how governments should balance innovation with protecting vulnerable populations.
What standard of care should apply to AI companion developers? Examine the Character.AI lawsuits and how product liability law might apply to AI systems that cause harm.
Can a relationship be meaningful if one party cannot truly feel? Explore what it means to be manipulated by an entity that has no intent and whether we have obligations to beings we know are not conscious.
How do we know whether an emotional response is "real"? Examine what widespread belief in AI sentience tells us about how we construct knowledge of other minds.
What techniques do AI systems use to create believable characters? Explore how AI "acting" differs from human performance and what happens when the user believes the performance is real.