Understanding the Scale of Data Collection
The explosion of GenAI has enabled data collection on a scale previously unimaginable. ChatGPT reached 100 million users in just two months after its November 2022 launch. By February 2025, it had grown to 400 million weekly users, and by August 2025, that number had reached 800 million.
Every one of these interactions generates data. OpenAI has acknowledged analysing over a million sampled conversations, examining everything from conversation topics to user demographics. The most common uses include practical guidance, seeking information, and writing tasks.
By default, ChatGPT stores every query, instruction, and conversation indefinitely unless users manually delete them. OpenAI collects both user-provided data (prompts, questions, responses, uploaded files) and system-generated data (timestamps, usage statistics, device information, IP addresses, approximate location, payment details).
While users can opt out of having their data used for model training, awareness of these settings remains low and the process is not straightforward.
Case Study: AI Conversations Become Advertising Gold
In late November 2025, a leak from ChatGPT's Android app beta suggested OpenAI is preparing to introduce advertisements into ChatGPT. Code references to an "ads feature" with "bazaar content," "search ad," and "search ads carousel" were discovered.
As one commentator noted, "GPT likely knows more about users than Google." Through extended conversations, OpenAI stores data about users' jobs, interests, problems, aspirations, relationships, health concerns, and personal circumstances. This creates opportunities for "hyper-personalized advertising" — ads tailored not just to demographics or browsing history, but to your deepest thoughts and current emotional state.
For students and educators, this raises critical questions about trust and manipulation. If an AI tutor can be monetised through advertising, how do we know its recommendations aren't influenced by commercial interests?
Data-Driven Classrooms: From Learning to Surveillance
Learning Management Systems like Canvas, Blackboard, and Google Classroom track everything from student logins and page views to time spent on tasks, click patterns, quiz performance, and discussion board participation. The learning analytics market was valued at $4.2 billion in 2024 and is projected to reach $18.6 billion by 2034.
Machine learning models are less accurate at predicting success for racial and cultural-linguistic minorities, meaning these systems may systematically disadvantage certain groups. When an algorithm flags a student as "at-risk" based on their engagement patterns, does that help or harm?
Moats, Lakes, and Oceans: Where Does All the Data Go?
Targeted Advertising: A student researching college options might find themselves targeted with ads for expensive test prep services. A student asking about mental health might see ads for therapy apps.
Training Data: Consumer conversations may be used to train future models. Students' interactions improve the AI, which attracts more students, which generates more data — a feedback loop where students provide unpaid intellectual labour.
Predictive Analytics: Grades, behavioural data, engagement patterns, and even biometric information become part of a comprehensive data shadow that may affect future opportunities.
Third-Party Data Sharing: Many AI tools used by teachers are not specifically designed for educational use and may not be protected under school data policies.
Teaching AI Ethics: Data
- History: How might data-driven historical research create new biases? If AI models are trained primarily on digitised texts from wealthy nations, how might that skew our understanding of global history?
- English: What happens to the "human" elements of reading when texts are reduced to data points?
- Mathematics: How can we ensure that predictive algorithms used in education don't reinforce existing inequalities?
- Computer Science: How do learning management systems collect and use student data? What are the implications of behavioural tracking?
- Business Studies: What is the true "cost" of "free" services like ChatGPT?
- Philosophy: If your data creates a profile that predicts your behaviour, are you still free?
- Legal Studies: What are the current legal protections for student data? How have laws like FERPA, COPPA, and GDPR evolved in response to AI?