What Does Truth Mean to a Large Language Model?
By design, large language models are incapable of truth. We have seen this in action since the release of ChatGPT, which initially had no internet connection, no way to upload or attach files, and a tendency to "hallucinate".
That word, "hallucination", has become the de facto term for what language models do when they make stuff up. But it is not an error, or a quirk: It is a feature of the way these things are built. Large language models rely on statistical matching and probability. The training data is only used to create the rules governing the languages the model can produce. There is no epistemic grounding, and no "truth" left in the system once trained.
Since 2023, most models have relied on a combination of internet access, file upload, retrieval augmented generation (RAG), and reinforcement learning from human feedback to mitigate rates of hallucinations, but none of these methods have fully guarded against the large language model producing factually incorrect information. Hallucinations are a feature, not a bug.
Even more problematically, although AI can now cite sources from the internet or use processes like RAG to access internal documents, this only increases the perceived credibility, making it harder to spot misinformation. The automation bias, a tendency of humans to believe machine-driven responses, means that people tend to believe what computers tell them.
Unfortunately, discourse on social media often compounds the problem, with AI influencers suggesting that hallucination rates are dropping when, out in the wild, the opposite seems to be true.
For example, Damien Charlotin has been compiling a fascinating database of legal cases where citations appear to have been hallucinated by artificial intelligence.
Just last week at the time of writing, the US government's "Make America Healthy Again" report clearly contained content generated by OpenAI's ChatGPT. The original release of the report carried telltale markup of ChatGPT-generated citations and many of the citations were inaccurate, misleading, or pointed to the wrong papers.
Even in medicine, a field where truth and accuracy are surely paramount, there have been issues with AI. For example, a highly successful paper in Nature detailed ways that language model-based technologies can be used to predict the function of unknown enzymes. However, as Rachel Thomas pointed out, "The Transformer model in the Nature paper made hundreds of 'novel' predictions that are almost certainly erroneous." A counter-paper pointing out the inaccuracies received much less attention than the 22,000 views-and-counting of the Nature article.
Whatever the AI pundits tell you, it is not about whether more advanced LLMs or reasoning models reduce hallucinations; it is the way in which these technologies are actually used by people in high-stakes situations like law, politics, medicine, and education.
Deepfakes and Synthetic Media
In 2023, the creation of synthetic multimodal texts was still fairly nascent and not anywhere near as convincing as text generation. But in the two years since the original Teaching AI Ethics series, that has changed. The truthfulness of audiovisual content online can no longer be taken for granted.
The creation of incredibly convincing AI-generated images is now trivial and possible through ubiquitous open-source technologies with very few restrictions or guardrails. Regulations and laws have been slow to keep up, though in Australia, at least, there have been changes regarding the creation of non-consensual, explicit deepfakes.
Deepfakes are particularly troubling when discussing artificial intelligence and truth, since it is now trivial to create convincing and entirely untrue videos of people saying and doing things they never did or said. 99% of deepfakes circulated online are nonconsensual and explicit, and 98% of those images are of women: this is gender-based abuse. These deepfakes are generally images but increasingly include video. While in 2023 this was already becoming problematic for celebrities, advances in these technologies mean that everyone is potentially vulnerable.
Deepfakes can also be used in political misinformation and deliberate reputational damage. We have already seen the deliberate sharing of deepfakes by politicians and world leaders on platforms like X, where moderation has all but disappeared. AI-generated websites, social media accounts, and bots are ubiquitous on most major platforms, generating and spreading misinformation and disinformation.
Because of the rise of these technologies, major developers have made efforts to secure their AI-generated content with watermarking and provenance standards like C2PA. These approaches identify synthetic media, often applying a content credential or metadata which articulates what platform was used to create the image, video, or audio.
While this legitimises content created on those platforms, allowing users to transparently label their content as AI-generated, it will not stop others from generating harmful deepfakes and synthetic media because they can simply use open-source equivalents which lack those credentials. Authenticity signals are part of the equation, but strong regulation is needed with heavy penalties for people creating and sharing harmful and untruthful content.
Post-Plagiarism and Moving Beyond Cheating
All of this contributes to thoughts on why we need to move discussions in education beyond cheating or catching students using artificial intelligence. Some discussions of academic integrity, for example, exclude artificial intelligence entirely from educational contexts, which makes it very difficult for students and educators to understand the full implications of these technologies in society.
Focusing on whether students using AI technologies are "cheating tools" also obfuscates a more important question of whether we can help students to learn to use artificial intelligence in ways which are truthful, accurate, and reliable.
It is not enough anymore to teach students to try to detect AI-generated fake news: it is already impossible to do so with 100% certainty. Similarly, there is no totally viable way to guarantee that anything produced by a student in the context of digital texts is their own work.
Sarah Elaine Eaton's post-plagiarism framework has six tenets:
- Hybrid Human-AI Writing Will Become Normal — Hybrid writing, co-created by human and artificial intelligence together, is becoming prevalent. Trying to determine where the human ends and where the artificial intelligence begins is pointless and futile.
- Human Creativity is Enhanced — Human creativity is enhanced, not threatened by artificial intelligence. Humans can be inspired and inspire others.
- Language Barriers Disappear — One's first language will begin to matter less and less as tools become available for humans to understand each other in countless languages.
- Humans can Relinquish Control, but not Responsibility — Humans can retain control over what they write, but they can also relinquish control to artificial intelligence tools if they choose. Although humans can relinquish control, they do not relinquish responsibility for what is written.
- Attribution Remains Important — It always has been, and always will be, appropriate and desirable to appreciate, admire, and respect our teachers, mentors, and guides.
- Historical Definitions of Plagiarism No Longer Apply — Historical definitions of plagiarism will not be rewritten because of artificial intelligence; they will be transcended.
Teaching Truth and AI Across the Curriculum
- English: Students analyse how language constructs meaning, truth, and credibility. With AI tools increasingly producing persuasive texts and opinion pieces, this is an ideal space to explore how generative tools shape narratives.
- Humanities (History / Civics / Geography): Humanities subjects centre on source reliability, perspective, and representation, areas increasingly complicated by synthetic media. Students might investigate what happens when AI rewrites history, or how deepfakes influence public trust in political institutions.
- Science (General Science / Psychology / Environmental Science): Science subjects teach students how to evaluate evidence and peer-reviewed claims. Students could analyse AI-generated scientific texts, compare summaries to source articles, or test citation credibility.
- Digital Technologies / Computer Science: Students can study the architecture of systems that generate, amplify, or suppress information, including building simple classifiers, auditing language model outputs, or exploring watermarking and provenance tools.
- Mathematics: Mathematics offers a lens for analysing the claims made by and about AI, especially through data and statistics. Students could ask what biases in training data affect how AI models "learn" truth.
- The Arts (Visual Arts / Media / Performing Arts): With AI now able to generate hyperrealistic images and synthetic voices, this is a space to ask how AI blurs the line between real and fake, and what responsibilities artists have when using AI to depict "truth".
- Languages (Other than English): In language learning, students examine meaning, nuance, and cultural context, areas where AI translations and outputs can distort or misrepresent truth.