The Illusion of Privacy

One of the most significant shifts in the GenAI era is how these tools create an illusion of privacy. When users type into a chatbot interface, the experience can feel like a private conversation: intimate, confidential, even therapeutic. But this perception is far from the truth.

In August 2024, OpenAI disclosed that it scans ChatGPT conversations for harmful content and, in cases deemed threatening, shares them with law enforcement. The announcement contradicted CEO Sam Altman's earlier statements suggesting ChatGPT interactions should have privacy protections similar to conversations with "a therapist or a lawyer."

Google's Gemini warns users not to share confidential information because conversations may be reviewed by human reviewers and retained for up to three years, even after users delete their activity. Unless you're running a local model on your own machine, whatever you tell a GenAI chatbot is never truly private.

Sharing Isn't Caring

In June 2024, Meta AI users discovered their "private" conversations were being shared with other users. What Meta called a "Discover" feed became what TechCrunch dubbed "a privacy disaster."

Users had been asking Meta AI for help with deeply personal matters, including medical questions, legal advice, and job disputes. Many believed they were having private conversations. Instead, a confusingly labeled "Share" button was broadcasting these conversations to a public feed visible to anyone on the platform.

A teacher shared an email thread about arbitration for an unjust job termination, complete with identifying details. Other users discussed medical symptoms, personal confessions, and financial information.

Privacy as a Product Feature

By October 2024, Meta announced that conversations with Meta AI — including through Ray-Ban smart glasses — would be used for ad targeting. There is no opt-out; users must simply avoid the service entirely.

University of Washington linguist Emily Bender told Fortune that Meta is "capitalizing on what she calls the 'illusion of privacy.' People often confide in chatbots about things they'd never post publicly, lulled into a sense the AI is a neutral listener."

Cross-Border Data and National Security

When DeepSeek's R1 model briefly became the most-downloaded app in the United States, regulators immediately raised alarms. DeepSeek stores all user data on servers in China, where national security laws require companies to hand over data to the government upon request.

Cybersecurity researchers discovered hidden code in DeepSeek's web application that directly connects to CMPassport.com, the online registry for China Mobile, a state-owned telecommunications company banned from US operations in 2019 due to national security concerns.

European regulators responded swiftly. Italy blocked the app outright. Ireland, Belgium, the Netherlands, and France launched formal inquiries. Australia's Department of Home Affairs issued a mandatory security notice forbidding government employees from using DeepSeek.

Bystander Privacy and Ambient AI

Meta's Ray-Ban smart glasses, equipped with cameras and AI capabilities, capture photos and videos that are sent to Meta's cloud for processing. The wearer may consent to this data collection, but what about everyone else in the frame?

In April 2025, Meta updated its privacy policy to remove the option to prevent voice recordings from being stored. Voice data is now retained for up to a year for AI training.

Harvard students demonstrated how easily the glasses could be modified to perform real-time facial recognition, matching faces to names and personal information scraped from public databases.

Retroactive Consent and Training Data

In May 2024, Meta announced plans to train its AI models on public posts from Facebook and Instagram users in the EU. Rather than seeking opt-in consent, Meta relied on GDPR's "legitimate interest" provision. Privacy advocacy group NOYB filed complaints with 11 European data protection authorities.

Case Study: Italy Fines OpenAI for GDPR Violations

On December 20, 2024, Italy's data protection authority imposed a €15 million fine on OpenAI for multiple violations of GDPR, including: failing to notify authorities of a data breach; processing personal data without legal basis; and failing to provide adequate age verification.

This case demonstrates key principles: companies must clearly explain how they collect and use personal data; simply having a business interest isn't sufficient legal basis; and AI systems accessible to children require additional safeguards.

Teaching AI Ethics

  • Legal Studies: How does GDPR's approach to AI training data compare with the emerging US framework?
  • Computer Science: What technical measures can developers implement to protect privacy in AI systems?
  • Philosophy: When does the societal benefit of AI outweigh individual privacy rights?
  • Business: How do privacy concerns affect the AI market?
  • Media Studies: How do user interface design choices affect privacy?
  • Psychology: What is the "illusion of privacy" and why are users vulnerable to it?
  • International Relations: How do different nations approach AI and data sovereignty?