ChatGPT, 
Claude, 
Gemini, 
Grok — all four for 
Every large language model occasionally states something false with total conviction, a fabricated citation, a statistic that doesn't exist, a plausible-sounding fact that simply isn't true. This is usually called a hallucination, and it's not a glitch you can patch out. It's a direct consequence of how these models generate text in the first place. Understanding the mechanism behind it will tell you a lot more about when to trust an answer than any general rule of thumb will.
Why This Happens: There's No Fact-Checking Layer
A language model generates text by predicting the next token given everything that came before it, over and over, based on probability distributions learned from its training data. There's no separate process that cross-references a claim against a source of truth before the model commits to it. If "confident-sounding but wrong" is statistically close to "confident-sounding and correct" in the model's learned patterns, the model has no internal signal that distinguishes the two. It isn't retrieving a fact and second-guessing it; it's producing the next most probable sequence of tokens, and sometimes that sequence just happens to be false.
This is also why hallucinations tend to show up most on specific, checkable details rather than broad explanations. A model has seen thousands of general descriptions of, say, how photosynthesis works, so its probability distribution for that topic is dense and well-formed. It's seen far fewer instances of one particular obscure statistic or a niche citation, so its output there is a lot closer to an educated guess dressed up in the same confident tone as everything else it says.
Factors That Move the Needle
| Factor | Effect |
|---|---|
| Model scale | Not a clean "bigger is safer" relationship. Larger models generally score better on factuality benchmarks and make fewer obviously wrong claims, but their fluency can make the hallucinations they do produce harder to catch, since a mistake wrapped in polished, well-structured prose reads as more credible than one from a smaller model that stumbles more visibly. |
| Sampling temperature | Temperature controls how much the model deviates from its highest-probability token at each step. Higher temperature increases lexical diversity and creativity, but also increases the odds of straying from a well-supported answer. Lower temperature produces more deterministic output, though at temperature 0 the model will still confidently reproduce whatever incorrect pattern is most common in its training data, just consistently. |
| Quantization | Relevant if you're self-hosting: compressing a model to a lower bit-width to save memory measurably increases hallucination rates, and the effect compounds on smaller models. A 2025 academic study on package-name hallucination in quantized LLMs found that 4-bit quantization pushed hallucination rates dramatically higher than full precision, an effect most severe on smaller models, while 8-bit quantization introduced a much smaller, often tolerable increase. |
| Context and grounding | A model answering from parametric memory (whatever it learned during training) is far less reliable than one answering with the relevant source material sitting directly in its context window. This is the entire premise behind retrieval-augmented generation: give the model the actual text to work from instead of asking it to recall a fact from memory. |
Implications for a Self-Hosted Setup
If you're running DeepSeek or another model through Ollama, two decisions covered in our model sizes article compound directly with hallucination risk: the parameter count you choose, and the quantization level it ships with by default. A 7B model at 4-bit quantization is going to hallucinate noticeably more on specific factual questions than a 32B model at 8-bit, even before you factor in the size gap on its own.
When you pull a model without specifying a tag, Ollama commonly selects a 4-bit quantization (often labeled Q4_K_M) as that model's default, though the exact default can vary by model and has changed over time. Running ollama show <model> after pulling will tell you the actual quantization level in use, and the Ollama model library lists every quantization tag available for a given model if you'd rather choose one explicitly.
The other factor worth revisiting is the context window. Once earlier details in a long conversation have aged out of the window, a model doesn't reliably flag that it no longer has that information; models aren't trained to say "I've lost that" as a distinct behavior from actually answering. In practice it tends to reconstruct something plausible instead, and the result is functionally indistinguishable from a hallucination, even though the trigger here is lost context rather than an obscure fact the model never had a strong signal for in the first place.
Reducing the Risk
- Ground the model with actual source material. Paste in the document, dataset, or article the answer should come from rather than relying on recall. This is the single highest-leverage change available.
- Lower the temperature for factual or high-stakes tasks. Reserve higher temperature settings for creative work where variance is a feature, not a liability.
- Weigh quantization against RAM if accuracy matters. An 8-bit quantized model uses more RAM than the same model at 4-bit, but the accuracy difference is meaningful enough to justify it on tasks where a wrong answer actually costs something.
- Verify anything checkable, especially citations. Asking a model for its source doesn't confirm the source exists. Fabricated citations are one of the most well-documented hallucination patterns, and the fabricated ones are frequently formatted identically to real ones.
- Treat specificity as a risk signal, not a trust signal. A vague answer to a vague question is usually fine. A suspiciously precise number, date, or quote attached to an obscure claim is exactly where hallucinations concentrate.
Summary
Hallucinations aren't a bug to be patched; they're an inherent consequence of generating text through next-token prediction with no built-in verification step. Model scale, sampling temperature, quantization level, and how much relevant context the model actually has in front of it all shift the odds, but none of them eliminate the underlying mechanism. The most reliable mitigation isn't picking the "right" model, it's grounding answers in real source material and treating specific, checkable claims with proportional skepticism.