A generative model learns patterns in its training examples. At runtime, it uses those learned patterns to predict and sample an output that matches the input prompt. A large language model repeatedly predicts a likely next token given the tokens already in context. It can generate useful language without having a built-in guarantee that its statements are true.
How a large language model generates text
- Tokenization: the input is split into tokens, which may be words, pieces of words, punctuation, or other symbols.
- Representation: tokens are converted into numerical vectors and combined with information about their positions.
- Context processing: Transformer layers use attention and other operations to build context-sensitive representations.
- Prediction: the model assigns probabilities to possible next tokens.
- Sampling: the system selects a token according to its decoding settings, adds it to the context, and repeats.
What does attention do?
Attention lets the model calculate how strongly different parts of the context should influence one another. In the sentence “The robot picked up the glass because it was fragile,” attention can help connect “it” with “glass.” Multiple attention heads can model different relationships in parallel. Attention is powerful pattern processing; it is not a truth detector or a perfect explanation of the model’s reasoning.
How are image generators different?
Many image generators use diffusion: training teaches a model to reverse a process that adds noise to images. During generation, the system starts from noise and repeatedly removes predicted noise while conditioning on the prompt. Language and image systems differ in their mechanics, but both learn distributions from examples and sample new outputs.
What is retrieval-augmented generation (RAG)?
RAG is an application pattern that searches a selected collection for relevant passages and places them in the model’s context before it answers. Retrieval can make answers more current and grounded, but it does not guarantee correctness. Search can retrieve the wrong passage, the source can be unreliable, or the model can misread it. The surrounding application — not the language model alone — should preserve sources and attach accurate citations.
Why do AI hallucinations happen?
A hallucination is generated content that sounds plausible but is unsupported or false. It happens because the training objective rewards likely output, not verified truth in every case. Hallucinations become more likely when the prompt asks for obscure facts, exact citations, events outside the system’s available knowledge, or an answer where the evidence is missing.
Use generative AI to draft, explore, transform, and compare. For facts that matter, check the original source. Never treat a confident tone as evidence.
Common questions
Is generative AI just copying?
It generally synthesizes outputs from learned statistical patterns rather than pasting one stored example. But models may sometimes reproduce memorized material, and outputs can still raise copyright, attribution, or privacy concerns.
What is a prompt?
A prompt is the input or instruction given to a generative model. It may include a task, context, examples, constraints, and a requested output format.
Can RAG eliminate hallucinations?
No. RAG can reduce unsupported answers when retrieval and sources are strong, but it adds its own failure points and still requires evaluation.
Primary and institutional sources
Research papers explain mechanisms; institutional resources provide guidance. Manufacturer pages describe their own products and are not independent evaluations.
- Attention Is All You Need (2017) — Vaswani et al.
- On the Opportunities and Risks of Foundation Models (2021) — Bommasani et al. · Stanford
- Retrieval-Augmented Generation (2020) — Lewis et al.
- Denoising Diffusion Probabilistic Models (2020) — Ho et al.
- AI Risk Management Framework — NIST
Continue with the companion in ChatGPT
Explore museum themes, ask questions and test your understanding with quizzes in ChatGPT. The companion and the game keep separate progress.
Discover the ChatGPT companionContent reviewed: 6 September 2026. Navigation and resources updated: 1 October 2026. Project method and limits.