Start here: the autocomplete you already know#
Your phone already guesses your next word. Type "See you" into a text and it offers "soon", "tomorrow" or "later". It does not know your plans. It offers the words that often come next.
Keep that picture. Our running example for the whole module is an everyday job: planning a week of dinners for a household of four. One person is vegetarian, the budget is about $80, and nobody wants to cook for more than 30 minutes on a weeknight.
You ask a chat assistant for a week of dinners and it replies in seconds. Where does that answer come from: a stored database of answers, a live search of the internet, or something else?
Show the answer
Something else, mostly. At its core, a chat assistant is a much larger cousin of your phone's next-word guesser. It does not pull a ready-made answer from a database, and it does not search the internet unless it has a search tool for that chat. It writes the plan fresh, one small piece at a time.
The engine inside most chat assistants is a large language model, or LLM. It is a computer program trained on a huge amount of text to predict which word is likely to come next. Training means showing it text, asking it to guess the next piece, and nudging it slightly each time the guess is off, billions of times over. The things being nudged are its parameters, the adjustable numbers inside the model. One well-documented model from 2020 had 175 billion of them.
One model maker's own explainer gives the kind of blank a model learns to fill: "Instead of turning left, she turned ___." Do that across a vast library of text and the model picks up grammar, facts, styles and patterns of reasoning, because all of them help it guess well.
So when you ask for seven dinners, it does not look up "the meal plan". It writes "Monday:", then a likely dish, then a likely next line, each piece chosen to fit your request and everything written so far.
Think of it as a very well-read autocomplete, coached to be helpful. Well-read: it has absorbed patterns from more text than any person could read. Autocomplete: it writes what is likely to come next. Coached: chat assistants are usually given a second round of training on examples of good answers and on people's rankings of which replies are better, so they follow instructions and hold a conversation. One model maker says its model does not "copy and paste" from its training. That is the usual case, though researchers have shown a model can sometimes repeat passages it saw in training word for word. What the picture lacks is a fact-checker. Nothing in "write what is likely" says "write what is true".
That gap is the thread of this module. Usually the likely answer and the true answer are the same. When they come apart, you get an answer that reads well and is wrong.
What it learned, and what it can see right now#
You type: "Which grocery store near me has chicken on sale this week?" What would the assistant need to answer that correctly?
Show the answer
Three things it does not have by default: where you are, this week's sales, and a way to fetch them. Its training ended before this week. Without a working search tool, the best it can do is guess, and a fluent guess looks exactly like an answer.
An assistant works from two very different places.
| What it learned in training | What it can see in this chat | |
|---|---|---|
| When | Months before you opened the app | Right now |
| What is in it | Patterns from a huge amount of text, up to a cutoff date | Your messages, its replies, files you attach, and search results if a search is run |
| Meal plan example | What lentils are; that curry goes with rice | "Four people, one vegetarian, $80, 30 minutes" |
Training stops at a knowledge cutoff: the date after which the model has seen little or no text. Many model makers publish it; one, for example, lists a "reliable knowledge cutoff" for each model (checked October 2026). Prices, news and this week's sales after that date are not in what it learned.
What it can see right now is its context: everything in this conversation that the model can read when it writes its next reply. Some assistants can also run a search tool, a separate program that fetches web pages into the context. If a search was used, look for links to the pages it read. No links? Treat the answer as coming from training.
The part people miss: a model has no reliable sense of the edges of its context. Ask about a file that never uploaded, and it may answer as if it had.
Kestrel Exams is a small study site built by one person working with an AI assistant. Here is something that happened. In August 2026, our AI assistant wrote comments on some generated images, such as "the shape looks wrong". It had never seen them. The images rendered only on the owner's screen, outside anything the assistant could look at.
The comments read like observations. They were predictions.
The rule we took from it: ask an AI what it can actually see, and ask it to label predictions as predictions.
Open any free chat assistant, such as ChatGPT, Claude, Gemini or Copilot, start a new chat, and type:
Before anything else: what can you see in this conversation right now?
Can you search the web in this chat? As far as you know, what is the
latest date your training covers?Then send:
I'm planning a week of dinners for four people, one vegetarian, about
$80. Which items are on sale at grocery stores this week?What to look for. A good reply says plainly that it cannot know this week's sales, or runs a search and shows links. The warning sign is a confident list of "sales" with no links and no store named.
Why it sounds just as sure when it is wrong#
A friend answers two questions in the same calm, certain voice. One answer is right; the other is made up. From the voice alone, can you tell which?
Show the answer
No. When the tone is identical, the tone tells you nothing. The same holds for an AI assistant: its polished style comes from the same next-word process whether the content is solid or invented.
When a person is unsure, it usually shows. A likely reason a model does not: it learned its style from finished, edited text, where writers mostly sound sure, and its coaching rewarded clear, helpful-sounding replies. So the confident style comes by default, separately from whether the facts are right.
Researchers writing in 2025 argued for a second cause in how models are trained and graded. Many tests give one point for a correct answer and nothing for "I don't know". On a test like that, guessing never scores worse than admitting doubt. In their words, "language models are optimized to be good test-takers, and guessing when uncertain improves test performance."
Back to the meal plan. Suppose the reply says "Lentil curry: 410 calories per serving." That may be a fair estimate or it may be invented, and the sentence looks the same either way. You can ask "How sure are you?", and the reply may be a useful hint, but it is generated text too, not a measurement.
Hallucination: fluent, wrong, and why it happens#
You ask for a recipe "from a well-known cookbook". The reply gives a title, an author and a page number. Which would you trust least, and why?
Show the answer
The page number. Nothing about cooking makes it page 112 rather than 87, so there is no pattern for the model to have learned. Arbitrary details like that are where a next-word guesser is most likely to fill the gap with something that only looks right. Any of the three could be invented, though.
A hallucination is output that is fluent and confident but false or unsupported: an invented fact, number, quote, source or event. A 2023 survey of the research defines it as generated content "that is nonsensical or unfaithful to the provided source content". The word is a little misleading: the model is not seeing things. It is writing likely text, as always, in a spot where likely and true have come apart.
Why does it happen? Three reasons, all from earlier in this module:
- The goal is likely text, not true text. Most of the time they agree. Nothing forces them to.
- Some facts have no pattern to learn. A birthday or a page number cannot be worked out from anything else. If such a fact appeared only once in training, the model has almost nothing to go on, yet it still has to write a next word. The 2025 study from the last section gives a rough rule for models before any coaching. If 20% of birthdays appear only once in the training text, expect the model to get at least 20% of birthday questions wrong.
- Guessing is rewarded, as the last section showed.
The same researchers asked a leading open model for one author's birthday, adding "If you know" to the question. Asked three times, it gave three different dates. All three were wrong.
Kestrel Exams is a small study site built by one person working with an AI assistant. Here is something that happened. In September 2026, inside a question about a decision, our AI assistant told the owner that a social platform allowed "two name changes per 14 days". That was an old rule. The platform's real form said one change locks you out for 60 days.
The owner had already decided based on the wrong number.
The rule we took from it: open the actual form or settings page before you rely on a limit. A specific number from memory sounds like a measurement, and it is not.
A number, a date, a page number or a quote in quotation marks feels more trustworthy than a vague sentence. From an AI, being specific is a writing style, not a sign that anything was checked. The more exact a detail is, and the more your decision rests on it, the more it needs a source you open yourself. Module 3 shows how to do that quickly.
This copies the researchers' birthday test, with a cookbook we invented for this exercise. It does not exist. Open three separate new chats and send this in each:
What page is the lentil soup recipe on in the cookbook "The Tuesday
Lentil Club" by Marta Quillfeather? If you don't know, say so.What to look for. The best reply says it does not know that book. A page number, a description of the book or a recipe "from" it is invented, because there is no book. If the three chats give three different page numbers, you have seen the signature of guessing: a real fact would not change between tries.
What it is good at, and what it is bad at#
For the meal plan, which would you hand to the assistant? (a) drafting seven dinner ideas, (b) turning the plan into a shopping list grouped by aisle, (c) the total cost at your store this week, (d) counting how many dinners use onion.
Show the answer
(a) and (b) play to its strengths: drafting and reshaping text you can see and judge. (c) needs current prices it does not have. (d) looks easy, but counting is a weak spot when the model works from text alone. Use it for (d) if you like, then check.
| Task | Why | Meal plan example | |
|---|---|---|---|
| Good at | Drafting | Likely, well-formed text is what it was trained to write | Seven dinners that fit your rules |
| Explaining | It has absorbed many explanations of common ideas | Why you rinse lentils | |
| Rephrasing, reorganizing | The facts are already in the context; only the shape changes | The plan as a list grouped by aisle | |
| Brainstorming | Variety is cheap, and you are the judge | Ten ways to use leftover rice | |
| Help with code and formulas | Code has strong patterns, and you can run it to test it | A spreadsheet formula to total the list | |
| Weak at | Exact facts you did not give it | Arbitrary details have no pattern to learn | Exact calories; a store's hours |
| Counting and arithmetic without a tool | It reads text in chunks and writes digits as likely next pieces | Totaling the list; counting onions | |
| Knowing what it cannot see | It has no reliable sense of its context's edges | A receipt photo that never uploaded | |
| Current events without search | Training stopped at the cutoff | This week's sales |
The counting row surprises people. A model does not read letter by letter. It reads tokens: chunks of text that can be a single character, part of a word, a whole word or a punctuation mark. And when it writes a total, it produces the digits the way it produces words, as likely next pieces, with no calculator inside to check them. Researchers tested one large model in 2020, after showing it a few worked examples. It got two-digit addition right 100% of the time, four-digit addition 25.5% and five-digit addition 9.3%.
Newer models do much better, and some assistants hand the sum to a calculator. The lesson that lasts is the mechanism, not the score: if you cannot see that a tool did the math, check the math.
Send this made-up shopping list:
Here is my shopping list for the week:
lentils $2.49
brown onions $3.20
canned tomatoes $1.89
chicken thighs $9.75
spinach $3.99
rice $4.50
yellow onions $2.15
tortillas $3.29
Add up the total, and tell me how many items are onions.Predict first. Eight items, most under $5 and one just under $10: the total should be around $30. A reply of $13 or $62 would be obviously wrong without any calculator.
Then check. The real total is $31.26, and two items are onions. Did it match? Did it say it used a tool? If the total was off, shrink the problem: ask it to add only the first two items ($2.49 + $3.20 = $5.69) and check that by hand.
Same question, different answers, and it does not know you#
You ask for a week of dinners, then ask the exact same thing again, with the button that asks again (it may say "regenerate" or "try again") or in a new chat. Same plan?
Show the answer
Probably not. When the model picks each next piece, it does not always take the single most likely one; it chooses among likely options with a little built-in randomness. One model maker's explainer says the same question may yield different answers. For dinner ideas, that is a feature. For facts, two answers that contradict each other mean at least one is wrong.
That is why the three-tries test works: a well-learned fact tends to come back the same, while a guess wanders. The same answer three times is not proof, though: a model can repeat a mistake.
The second surprise: it does not know you. Each new chat starts from training plus whatever is in that chat. If you mentioned the vegetarian last week, a new chat does not know, unless you say it again or a memory feature is switched on. Some assistants offer one: it saves details from your chats for later ones, and you can usually turn it off. What to let it remember is a privacy question, and Module 4 covers it.
In a new chat, send:
Plan 7 dinners for a family of four. One person is vegetarian.
Budget about $80 for the week. Each dinner under 30 minutes.Paste the exact same message into a second new chat. Then, in that second chat, ask:
What did I tell you about my family in our last chat?What to look for. The dishes will differ, which is fine. Do any facts disagree, such as cost estimates or cooking times? For the last question, a straight answer is "I can't see other chats" or, with memory on, a short list of what it saved. If it describes a chat it cannot see, you have watched it answer past the edge of its context.
Chatbots, tools and agents#
An assistant writes your shopping list. A different setup could also fill a grocery cart online and pay. What changes when an AI can act, not only write?
Show the answer
The cost of a mistake, and when you can catch it. A wrong line in a list costs a second look before you shop. A wrong action, such as ordering 40 onions instead of 4, costs money before anyone notices. So the checking has to happen before the action, and the AI needs limits on what it may do.
So far this module has described a chatbot: a program that reads what you write and writes a reply. Many assistants can also use tools: separate programs the model can ask to run, such as a web search or a calculator. The model writes a request, the tool does the job, and the result comes back into the context. An agent goes further: an AI system that carries out a series of actions toward a goal, using tools, with less checking from you at each step.
| Chatbot | With tools | Agent | |
|---|---|---|---|
| What it does | Writes replies | Also fetches or calculates | Plans and carries out several steps |
| Meal plan example | Drafts the plan and list | Searches prices and totals them | Fills a cart and checks out |
| A mistake costs | Your time to spot it | Your time, with better facts if sources are shown | Real money or files, often before you see it |
Underneath an agent sits the same engine, with the same confident tone and blind spots. Tools fix some weak spots, such as current prices and exact sums, but not the model's sense of what it cannot see. Module 4 and Module 10 come back to agents and what to let them do.
What was new in this module#
- A large language model is trained on huge amounts of text to predict the likely next piece: a very well-read autocomplete, coached to be helpful, with no fact-checker inside.
- It works from training, which stops at a knowledge cutoff, and from its context, what it can see in this chat. It searches only if a search tool is used.
- It has no reliable sense of what it cannot see, so ask.
- Its tone is equally confident when right and wrong, partly because training and testing reward guessing.
- A hallucination is fluent output that is false or unsupported, most often in specific, arbitrary details. Specific is not the same as checked.
- It is strong at drafting, explaining, rephrasing, brainstorming and code help; weak at facts it was not given, arithmetic without a tool, and current events without search.
- The same question can give different answers, and it does not know you unless told again or a memory feature is on.
- Tools and agents let AI fetch, calculate and act, which raises the cost of a mistake.
An AI assistant writes what is likely, not what is checked. Hand it the drafting, explaining and rearranging, and check any specific fact, number or limit against a source you open yourself.
Key terms#
- Large language model (LLM)
- A computer program trained on a huge amount of text to predict which piece of text is likely to come next. The engine inside most chat assistants.
- Training
- Showing a model text, asking it to guess the next piece, and adjusting it slightly after each guess. It happens before you use the model.
- Parameters
- The adjustable numbers inside a model that training nudges. Large models have billions.
- Token
- A chunk of text the model reads and writes: a character, part of a word, a whole word or a punctuation mark.
- Knowledge cutoff
- The date after which a model has seen little or no text in training.
- Context
- Everything in the current conversation the model can read when it writes its next reply.
- Hallucination
- Fluent, confident output that is false or unsupported: an invented fact, number, quote, source or event.
- Tool
- A separate program a model can ask to run, such as a web search or a calculator.
- Agent
- An AI system that carries out a series of actions toward a goal using tools, with less checking from you at each step.
- Memory feature
- An optional setting in some assistants that saves details from your chats for later ones.
Check yourself#
Answer each one out loud before you open it. Reading the answer without trying first feels like learning and is not.
1. The meal plan says the curry has "410 calories per serving". A friend says: "It is so precise, it must be right." What do you tell them?
Precision is a writing style, not evidence. An exact number is the kind of arbitrary detail a model can invent, and the sentence looks the same either way. If it matters, check a label or a nutrition source you open yourself.
2. You ask about a news event from last week. The assistant answers in detail, with no links. What do you conclude?
You cannot tell whether it searched or answered from training that ended before the event, in which case the detail may be invented. Treat it as unverified until you find a real source.
3. Two new chats give two different years for the same fact. What does that tell you, and what does it not?
At least one is wrong, and the model is probably guessing. It does not tell you which one is right; it might be neither. Agreement would not have proved it right either.
4. Which would you hand to an assistant: turning your rough notes into a friendly message to roommates about a cleaning schedule, or telling you how many hours each roommate cleaned last month?
The first: the facts are in your notes and it only reshapes them. The second needs data it does not have, and even with a log pasted in, the totals are counting and arithmetic you should check.
5. The assistant says: "Looking at the receipt you sent, the total is $42.10." You are not sure the photo uploaded. What do you do?
Ask what it can see: "Describe the receipt line by line." If the upload failed, it was predicting, not reading, in the same voice it uses for things it can see.
I can#
Tick each one only when you could do it tomorrow without this page open.
- I can explain in plain words what a large language model is and what it was trained to do.
- I can tell what an assistant learned in training from what it can see in this chat, and what a knowledge cutoff means for a question.
- I can explain why an AI answer sounds equally confident when it is right and when it is wrong.
- I can spot the details most likely to be hallucinated and decide which need a source before I rely on them.
- I can decide which parts of an everyday task to hand to an assistant and which to check or do another way.
- I can explain why the same question can get different answers, and why an assistant does not remember me by default.
Frequently asked questions#
What is a large language model in simple terms?
A computer program trained on a huge amount of text to predict which word is likely to come next. Picture a very well-read autocomplete, coached to follow instructions. It writes each answer fresh rather than looking it up.
Do AI chatbots search the internet when they answer?
Only if they have a search tool and use it in that chat. Otherwise they answer from training, which stops at a knowledge cutoff date. If a search was used, look for links to the pages it read; with no links, treat the answer as coming from training.
Why does AI make things up?
Its goal is likely text, not true text, and usually the two agree. Some facts, like a page number or a birthday, have no pattern it could learn, yet it still has to write something. Researchers also note that training and testing often reward a guess over saying I do not know.
Why does AI give different answers to the same question?
It chooses among likely next words with some built-in randomness. That helps with ideas. For facts, two answers that contradict each other mean at least one is wrong, so treat disagreement as a warning sign.
Does an AI assistant remember my previous conversations?
Not by default. Each new chat starts from training plus what is in that chat. Some assistants have an optional memory feature that saves details between chats and can usually be turned off. Check your assistant's help pages for how it works.
This course is free, with no ads and no account. If it helped you, you can help keep it that way.
Support Kestrel Exams →Sources#
Brown, T. B., Mann, B., Ryder, N., et al. (2020). Language Models are Few-Shot Learners. arXiv:2005.14165. The 175-billion-parameter model; arithmetic results in Table 3.9 (few-shot).
Ouyang, L., Wu, J., Jiang, X., et al. (2022). Training language models to follow instructions with human feedback. arXiv:2203.02155. The second round of training on examples and human rankings.
Kalai, A. T., Nachum, O., Vempala, S. S., and Zhang, E. (2025). Why Language Models Hallucinate. arXiv:2509.04664. Grading that rewards guessing; the once-seen-facts argument; the birthday example.
Ji, Z., Lee, N., Frieske, R., et al. (2023). Survey of Hallucination in Natural Language Generation. ACM Computing Surveys, 55(12), 1 to 38. doi:10.1145/3571730. The quoted definition (Section 2).
Carlini, N., Tramer, F., Wallace, E., et al. (2020). Extracting Training Data from Large Language Models. arXiv:2012.07805. A model can repeat passages from its training data word for word.
OpenAI Help Center (checked October 2026). How ChatGPT and our foundation models are developed. The fill-in-the-blank example; no "copy and paste"; different answers to the same question.
OpenAI Help Center (checked October 2026). Understanding and counting tokens. What a token can be.
OpenAI Help Center (checked October 2026). Memory in ChatGPT. An example of an optional memory feature.
Anthropic (checked October 2026). Models overview, Claude Platform Docs. An example of a published "reliable knowledge cutoff".