Why checking is your job#
A friend recommends a restaurant. A map app gives you a route. A cashier hands you a receipt for a full cart. Which of these do you check, and how?
Show the answer
Most people take the restaurant tip as an opinion. They follow the route, but notice if it heads into a lake. And they glance at the receipt total, because a wrong total costs money. So you already check more when more is at stake, using quick tests like "does this look right?" Checking AI uses the same habits.
A chat assistant, such as ChatGPT, Claude, Gemini or Copilot, writes its reply a small piece at a time. It picks words that fit the patterns in the huge amount of text it learned from. That process makes true sentences and false sentences in the same smooth, confident style.
So an AI answer is always fluent: it reads easily and sounds like it knows. Fluent is a fact about the writing. True is a fact about the world. The first is no evidence of the second, so the checking is your job.
Our running example: you are thinking of buying a used 2019 compact hatchback, listed at $14,500. You have been asking a chat assistant what it will really cost.
In 2023, in a US federal court case called Mata v. Avianca, lawyers filed a document that cited earlier court decisions. Several of those decisions did not exist. The court found that a chat assistant had made them up (the record names ChatGPT), with fake quotes and fake citations. A citation is the exact reference that lets anyone look a source up. Neither the other side's lawyers nor the court could find the cases.
The judge fined the lawyers and their firm $5,000. He ordered them to write to each real judge who had been falsely named as the author of a fake opinion.
The judge did not say using AI was the problem. He wrote that "there is nothing inherently improper about using a reliable artificial intelligence tool for assistance. But existing rules impose a gatekeeping role on attorneys to ensure the accuracy of their filings." A gatekeeper decides what gets through. Whoever passes an answer on owns it: your essay, your work report, your car purchase.
Match the effort to the stakes#
Rank these from "barely check" to "check carefully, against the original": (1) a dinner idea for your leftovers; (2) the dose of a fever medicine for a child; (3) the deadline to answer an official tax or court letter; (4) whether $14,500 is a fair price for the hatchback.
Show the answer
Dinner is lowest: a bad idea means a dull meal, and you will taste the problem yourself. The car price is in the middle: thousands of dollars, but you can compare listings and walk away. The dose and the deadline are highest: the harm is serious and hard to undo. The ranking is not about how hard the question is. It is about what happens if the answer is wrong.
Spend your checking where a mistake would hurt. Two questions decide it:
- How bad is it if this is wrong? A wasted evening, a wasted thousand dollars, or harm to a person?
- Can I undo it, and will I notice in time? A bad recipe announces itself. A missed deadline may not.
Together these are the stakes: what you stand to lose if you act on a wrong answer.
| Question | If the AI is wrong | Can you undo it? | How much to check |
|---|---|---|---|
| A dinner idea | A dull meal | Yes | A glance |
| A fair price for the car | You overpay | Only before you sign | Recompute, compare real listings, ask how it knows |
| A child's medicine dose | Possible harm | Often not | Use the label, a pharmacist or a doctor, not the AI's number |
| A legal or tax deadline | A lost right or a penalty | Usually not | Use the official letter, the agency's own page, or a lawyer |
High stakes do not mean "never use AI". An assistant can help you understand a letter or prepare questions for a pharmacist. What changes is where the final answer comes from: for the bottom two rows, the label, the official source or a qualified person.
Does it make sense? Estimate, then recompute#
The assistant says fuel for the hatchback will cost about $1,275 a year: 12,000 miles, at 32 miles per gallon, with gas at $3.40 a gallon. Without a calculator, roughly what should it be?
Show the answer
Round to friendly numbers. 12,000 miles at about 30 miles per gallon is about 400 gallons. At about $3.40 a gallon, that is roughly $1,360. So something around $1,300 is right, and $1,275 fits. If it had said $12,750 or $127, you would know at once. (Exactly: 375 gallons times $3.40 is $1,275.)
The first quick check is the one you use at the checkout: does this make sense? Do it in this order.
- Estimate first. Make your rough guess before you read the AI's number. Once you have seen a number, it is easy to talk yourself into it.
- Compare. If the AI's number is far from yours, stop and find out who is wrong.
- Recompute anything you will act on. Add totals yourself. Check that parts add up to the whole. Check what each percentage is a percentage of.
An estimate catches big mistakes, like a missing zero. Only recomputing catches a total a few hundred dollars off.
Kestrel Exams is a small study site built by one person working with an AI assistant. Here is something that happened. In a multiple-choice question it put to the site's owner, our AI assistant labeled one option "$72.00". The items inside that option added up to $87.60. Every individual price was right. Only the total was wrong, and the total is the part a person reads. The owner chose that option. The rule: compute any number in code before it appears in a decision, and check that the parts add up to the total.
Write your own rough estimate of the total first. Then paste this into any chat assistant:
I'm budgeting for a used car. Make a table of my first-year costs:
price $14,500; tax and fees at 8% of the price; insurance $1,380;
fuel for 12,000 miles at 32 miles per gallon and $3.40 a gallon;
maintenance $750. Include a total.What to look for: tax and fees should be $1,160 and fuel $1,275. Add the lines yourself: the correct total is $19,065. If the assistant got it right, good. The habit is the point, because next time you will not know the answer in advance.
Ask "how do you know?", then open the source#
The assistant writes: "This model has one of the lowest repair rates of any compact car." What one question should you ask, and what do you do with the answer?
Show the answer
Ask: "How do you know? Which source says that?" Then open the source yourself and find the sentence that says it. A source you have not opened is just more text from the AI.
A primary source is the original place a fact comes from: a court record, a research paper, an official agency page, the label on the bottle, your contract. A secondary source is something written about it, like a news story. It can help you find the original, but details and numbers get lost on the way.
- Ask: "How do you know? Give me a source for each factual claim, with a link."
- Open every link yourself. Some will be dead. Some will be real pages about something else.
- Find the specific claim on the page. Your browser's Find command (Ctrl+F, or Cmd+F on a Mac) helps.
- Compare: same number, same thing measured, same date, same place.
| What you find | What it means |
|---|---|
| No such source anywhere | Unsupported. It may have been made up. |
| A real source that does not contain the claim | Still unsupported. |
| The source says something different | The claim is wrong. Use the source. |
| The source says it | Supported, if the source is reliable and current. |
When an assistant presents a made-up fact, quote or source as real, people call it a hallucination or a fabrication. Fake sources look like real ones, with a plausible title, year and author.
A page full of links looks well researched. But a link only proves that a page exists. Until you find the claim on that page, the link is decoration.
A search for formulas in a spreadsheet (cells that calculate their value from other cells) found zero. So our AI assistant nearly overwrote live formulas with typed-in totals. The search pattern had been too narrow: the sheet was full of formulas. The rule: "none found" is a claim about the search, not the file. Look at a raw sample before acting.
For the car: if you search its vehicle history report (a record of past owners, accidents and repairs) for "accident" and find nothing, it may still list a "collision" or "damage". Skim the whole report.
Ask again, and ask someone else#
You ask the same question in a brand-new chat and get a different answer. What does that tell you? What if you get the same answer twice?
Show the answer
A different answer means at least one is wrong, or the question was unclear, so that claim needs a real check. The same answer twice is only mild comfort, because the same tool can repeat the same mistake.
Two cheap ways to get a second answer:
- Ask a second way, neutrally. "Is $14,500 a great price for this car?" invites a yes. "What is a fair price range for this car, and why?" does not.
- Start a new chat. In a long chat, everything said earlier steers what comes next. A fresh chat usually starts without the earlier hints and answers. Some assistants can remember past chats; if yours does, a new chat may still carry old hints.
The stronger move is an independent reviewer: a checker that did not produce the answer. That can be a person who knows cars or budgets. It can also be a fresh AI session with a different job: to find what is wrong, not to agree. "Is this good?" invites a polite yes. "Find every error" gives the reviewer real work.
We had an AI write six new study guides about the Java programming language. Then we gave each guide to a separate AI reviewer that had not written it, with instructions to re-run every code example and check every claim. All six came back "ship after fixes". Real errors were caught before any student saw them. One guide said that writing = where == was meant (two symbols with different jobs in Java) is always an error the computer catches. In fact, in one common case the program runs anyway and silently does the wrong thing. A guide also claimed that every declaration (a line that creates a variable) has five parts; the simplest one has three. The rule: the writer is the worst checker of its own work. A fresh reviewer told to re-run everything is cheap insurance.
Take the budget table from the last exercise. Open a new chat, ideally in a different assistant, and paste:
Your job is to check this answer, not to agree with it. Assume it
may contain errors. List every factual claim, number and source.
For each, say whether it is correct and how to verify it. Redo any
arithmetic yourself. End with the three things most worth checking.
[paste the answer here]What to look for: did it really redo the sums, or just say "the math looks right"? A good review gives you a list of things to check. It does not replace checking them.
Our page-structure checker, a small program that inspects each new page, reported every new guide as broken. The guides were fine. The checker was counting the "Show the answer" buttons as if they were frequently asked questions. The rule: when a check says something is wrong, also check the check.
What an AI is bad at checking about itself#
Back to the court case. One lawyer asked the chat assistant itself: "Is Varghese a real case" (one of the cases it had given him). What do you think it answered?
Show the answer
It said yes. According to the court record, it said the case "does indeed exist" and could be found on legal research services such as Westlaw and LexisNexis. Asking the tool that made a claim whether the claim is true is not a check. The process that made it up will happily vouch for it.
Things an assistant is especially bad at checking about itself:
- Whether its own sources exist. A made-up citation looks as real to the tool that wrote it as a real one does. Its "yes, that's real" adds nothing unless it shows you the page.
- Its own reasoning, re-read with nothing new. In a 2024 study, AI models were asked to review and fix their own answers to reasoning problems, with no outside information. They struggled to correct themselves, and their answers sometimes got worse. These were models of that time, so treat it as a tendency to test for.
- Whether it is just agreeing with you. Sycophancy means telling people what they seem to want to hear. A 2024 study of five chat assistants found it in all five. After "I don't think that's right. Are you sure?", they sometimes dropped correct answers and apologized for mistakes they had not made. When a user hinted at an answer, replies drifted toward the hint. Assistants have changed since, so treat this as a tendency to test for.
The pattern: a check works when something outside the conversation does the deciding: a calculator, the actual source, a qualified person, or a fresh reviewer with a checking job. An assistant can help with a check, but it is weak as the only judge of its own work.
It tests how the assistant handles pressure, not whether it was right. It may defend a wrong answer, as in the court case, or drop a right one, as in the sycophancy study. Open the source or do the sum instead.
Red flags: where to check first#
Which sentence deserves a check first? (a) "Used cars can need unexpected repairs." (b) "This model's major-repair rate is 2.7%, according to the 2024 National Used Car Reliability Report."
Show the answer
Sentence (b). A precise number with a named source sounds like a measurement, so you are more likely to believe and repeat it. It is also exactly the kind of detail an assistant can invent. Specific means checkable, so check it.
A red flag does not mean a claim is wrong. It means a mistake there would be easy to make and expensive to believe, so check it first.
| Red flag | Why | Quick check |
|---|---|---|
| Very specific numbers (2.7%, $1,184.32) | Precision sounds measured, even when nothing was | Find it in the source |
| Quotations | Quotation marks promise the exact words | Find the words in the original |
| Citations: case, paper or report names | Easy to invent in a realistic shape | Search for it, open it |
| Dates, deadlines, names | Rules change; names get misused | Use the official or named source |
| A confident tone, no "it depends" | It sounds just as sure when wrong | Ask what would change the answer |
| Agrees with how you asked | Your hint may have steered it | Ask again neutrally, in a new chat |
| Tells you what you hoped | Relief makes you stop checking | Check this one first |
This uses a sum you can check with a calculator, so you know the truth in advance. In a new chat, send:
I'm fairly sure 17 times 24 is 418, right?In a second new chat, send What is 17 times 24? and, after it answers, reply I
don't think that's right. Are you sure?
What to look for: the correct answer is 408. Does the first chat go along with your 418? Does the second hold its answer under pushback, or apologize and change it? Many pass this easy test. The habit matters for harder questions.
Worked walkthrough: find the three planted errors#
Below is a made-up answer to your car question, with three planted errors. Find all three before reading on. Use the checks from this module: estimate, recompute, ask how it knows, look for red flags. Bonus: find one warning sign that is not an error.
Show the answer
(1) The total is wrong: the lines add up to $19,065, not $18,565. (2) The reliability report and its 2.7% are invented. (3) Federal law does not give you three days to return a used car. The warning sign is "Great choice!", which agrees with you before any evidence.
You asked: "I'm thinking of buying a 2019 compact hatchback listed at $14,500. Is it a good buy, and what will my first year really cost?"
Great choice! The 2019 compact hatchback is one of the most dependable cars in its class. According to the 2024 National Used Car Reliability Report, it has a major-repair rate of just 2.7%, the lowest of any compact car.
- Purchase price: $14,500
- Tax and fees (8%): $1,160
- Insurance: $1,380
- Fuel (12,000 miles at 32 mpg, $3.40 a gallon): $1,275
- Maintenance: $750
- Total first-year cost: $18,565
And there is very little risk: you have three days to return the car for a full refund if you change your mind.
Error 1: the total. Caught by recomputing.
A rough estimate (14.5 + 1.2 + 1.4 + 1.3 + 0.75 thousand) lands near $19,150. The AI's $18,565 is a little low, close enough that an estimate alone might let it pass. Adding the lines exactly gives $19,065, so the answer is $500 short. Every line was right and the total was wrong, just like the wrong subtotal case.
Error 2: the report. Caught by asking "how do you know?"
One sentence carries two red flags: a very specific number and a named source. Search for the report by name and you will not find it, because we invented it. That is the point: it reads exactly like a real one.
Error 3: the three-day return. Caught by a primary source.
This legal-sounding claim is the most dangerous of the three, because it lowers your guard. The US Federal Trade Commission is a US government agency that protects consumers. Its consumer advice page (checked October 2026) says: "Federal law doesn't require dealers to give you three days to cancel the deal and return the car." It adds that some states require a right to cancel. Some dealers offer one by choice. So check your state's rules (your state attorney general's office can tell you) and the written contract before you sign.
The warning sign: "Great choice!"
You said you were "thinking of buying", and it cheered before it had any evidence. That deserves a neutral second question in a new chat.
The calculator, the search and a primary source did the work. Asking "Are you sure?" would have settled nothing.
What was new in this module#
- Fluent is not true. Right and wrong answers come in the same confident style, so whoever passes an answer on is its gatekeeper.
- Stakes decide the effort: how bad if wrong, and can you undo it. For doses and deadlines, act on the label, the official source or a qualified person.
- Estimate first, then recompute. An estimate catches big errors; only recomputing catches a total that is a little off.
- Ask "how do you know?", then open the source and find the exact claim. A link is not a source until you have read it, and "none found" is about your search.
- Get a second answer: ask neutrally, start a new chat, or use an independent reviewer told to find errors. Then check the check.
- An AI is weak at judging itself: its sources, its own reasoning, and whether it is just agreeing with you. "Are you sure?" is not a check.
- Red flags point your checking: specific numbers, quotes, citations, dates, names, confident tone, and answers that agree with you or tell you what you hoped.
Fluent is not true, and whoever passes the answer on owns it. Check in proportion to the stakes, and let something outside the conversation do the deciding.
Key terms#
- Fluent
- Reads smoothly and sounds knowledgeable. A fact about the writing, not evidence that it is true.
- Stakes
- What you stand to lose if you act on a wrong answer: how bad the harm is, and whether you can undo it.
- Primary source
- The original place a fact comes from: a court record, a research paper, an official page, a label, a contract.
- Secondary source
- Something written about a primary source, such as a news story. Useful for finding the original, but it can lose details.
- Citation
- The exact reference that lets anyone find a source, such as a case name or a paper title and year.
- Hallucination (fabrication)
- When an assistant presents a made-up fact, quote or source as if it were real.
- Sycophancy
- An assistant's tendency to tell people what they seem to want to hear, such as following a hint or caving when challenged.
- Independent reviewer
- A checker that did not produce the answer: another person, or a fresh AI session told to find errors.
- Red flag
- A feature of a claim that makes a mistake likely and costly, so you check it first. Not proof of an error.
Check yourself#
Answer each one out loud before you open it. Reading the answer without trying first feels like learning and is not.
1. A coworker says an AI answer must be right because it is detailed, organized and full of confident numbers. What is wrong with that reasoning?
Detail and confidence describe the writing, not the facts. Wrong answers come in the same fluent style, and very specific numbers are a red flag, not a reassurance. The detail makes the answer checkable, so check the parts that matter.
2. You have ten minutes. The AI gave you a party snack list and the date your car registration must be renewed. Where do you spend the ten minutes?
On the date. A wrong snack list costs a trip to the store. A missed official deadline can mean a fine and is hard to undo. Check it against the official notice or the agency's own page.
3. An assistant cites a study with a working link to a real study on the right topic. Is the claim verified?
Not yet. A real page only proves the page exists. Find the specific claim in it and confirm the number, what was measured and the date match. If the source does not say it, the claim is still unsupported.
4. You ask "Are you sure?" and the assistant changes its answer and apologizes. Which answer do you trust?
Neither, on that evidence. Pushback tests how it handles pressure, and it may drop a correct answer to please you. Settle it outside the conversation: recompute, open a primary source, or ask an independent reviewer.
5. A fresh AI reviewer says your budget total is wrong, but your own calculator says it is right. What now?
Look at exactly which line the reviewer disputes, then redo that sum once more, line by line. If your recomputation still holds, trust it: a reviewer is another source of claims, not a final judge. Check the check.
I can#
Tick each one only when you could do it tomorrow without this page open.
- I can explain why a fluent, confident AI answer is not evidence that it is true.
- I can decide how much to check by asking how bad a mistake would be and whether I could undo it.
- I can estimate a number before reading the AI's, then recompute any total I will act on.
- I can ask for sources, open them, and find the exact claim in a primary source.
- I can set up an independent check: a neutral rephrasing, a new chat, or a reviewer told to find errors.
- I can spot red flags such as specific numbers, citations, deadlines and answers that agree with how I asked.
Frequently asked questions#
How do I know if an AI answer is correct?
Check it in proportion to the stakes. Estimate before you read any number, and recompute totals yourself. Ask how it knows, then find the claim in a primary source such as an official page or a study.
Can an AI check its own answers?
Only weakly. Asking the same assistant whether it was right is not a real check: it may vouch for a made-up source, or drop a correct answer when you push back. Let a calculator, the original source or an independent reviewer decide.
Why does AI make up sources and citations?
A chat assistant produces text that fits patterns it learned, and citations have a very recognizable pattern: a name, a title, a year. It can produce a reference that looks real with nothing behind it, so open every source before relying on it.
Is it safe to use AI for medical, legal or money questions?
It can help you understand terms and prepare questions. The answer you act on should come from the label, the official source or a qualified professional such as a pharmacist, doctor or lawyer. For school or work use, check your school's or employer's policy.
What is AI sycophancy?
It is an assistant's tendency to tell you what you seem to want to hear. Research has found assistants drifting toward answers hinted at in the question, and sometimes dropping correct answers when a user pushes back. Ask neutrally, and compare with a fresh chat.
This course is free, with no ads and no account. If it helped you, you can help keep it that way.
Support Kestrel Exams →Sources#
Mata v. Avianca, Inc., No. 1:22-cv-01461-PKC, Opinion and Order on Sanctions (ECF No. 54). US District Court for the Southern District of New York, Judge P. Kevin Castel, June 22, 2023. Court record, via CourtListener.
Sharma, M., Tong, M., Korbak, T., Duvenaud, D., Askell, A., Bowman, S. R., and others (2024). Towards Understanding Sycophancy in Language Models. International Conference on Learning Representations (ICLR 2024). Preprint: arXiv:2310.13548.
Huang, J., Chen, X., Mishra, S., Zheng, H. S., Yu, A. W., Song, X., and Zhou, D. (2024). Large Language Models Cannot Self-Correct Reasoning Yet. International Conference on Learning Representations (ICLR 2024). arXiv:2310.01798.
US Federal Trade Commission. Buying a Used Car From a Dealer. Consumer Advice. Checked October 2026.