Understanding Model Quantization

If you've browsed models in Ollama, you've probably noticed tags like q4_K_M, q5_K_M, or q8_0 attached to model names. These refer to quantization, a compression technique used across the AI industry, not something specific to Ollama or any single tool. Every major AI provider quantizes its models to some degree; you just don't normally get to choose the level yourself. Ollama is where that choice actually becomes visible and adjustable, since it's the engine running behind most of the locally-hosted models on Bluehost's AI apps. Picking the right level here can be the difference between a model that runs smoothly on your server and one that barely fits at all.

What Is Quantization?

Quantization is a way of compressing a model by reducing the precision of the numbers that make up its internal parameters. A freshly trained model typically stores each parameter as a 16-bit number (a format called FP16), which is precise but takes up a lot of memory. Quantization rounds those numbers down to a lower precision, like 8-bit or 4-bit, so the model takes up less space in RAM and runs faster, at the cost of some accuracy.

A reasonable comparison is compressing a photo into a JPEG: you lose some fine detail, but the file becomes dramatically smaller and easier to work with. Most of the time, the difference is barely noticeable. Push the compression too far, though, and the quality loss starts to show.

Common Quantization Levels

Quantization levels are usually labeled by how many bits are used per parameter. Here's roughly what to expect from the levels you'll most commonly see in Ollama's model library, using a 7B model as a reference point:

Quantization levels compared, using a 7B model as a reference point
Level	Precision	Approximate Size (7B Model)	Quality vs. Full Precision
FP16	16-bit (full precision)	~14GB	100% (the original, uncompressed model)
Q8_0	8-bit	~7-8GB	Nearly identical to full precision
Q5_K_M	5-bit (mixed precision)	~5-6GB	Very close to full precision, small tradeoff
Q4_K_M	4-bit (mixed precision)	~4-5GB	Good for most everyday use, a more noticeable but still moderate tradeoff

These figures are approximate and will vary somewhat by model, but the pattern holds: each step down in precision meaningfully shrinks the file size and RAM needed, while giving up a little more accuracy.

Tip: Not all 4-bit quantization is the same. The "K" variants (like Q4_K_M) use mixed precision internally, keeping the most important parts of the model at slightly higher precision, which is why they generally perform better than older, simpler formats like Q4_0 at the same bit width.

Where Ollama Fits In

Quantization exists independently of any one tool, but Ollama is where it matters most for a self-hosted setup, since it's the underlying engine behind most of the locally-run models across Bluehost's AI apps: DeepSeek, and any of the other agentic apps pointed at a local model instead of a cloud provider, all run through it. When you install a model through Ollama without specifying a tag, it defaults to Q4_K_M for most models. This is a deliberate, well-supported choice, not a corner cut for convenience: it's widely considered a strong balance between capability and resource usage, and it's why the RAM figures in our model sizes article are noticeably lower than a model's full, uncompressed size would require.

That said, "default" doesn't mean "only option." If you have RAM to spare and want the best possible accuracy, you can specify a higher-precision tag (like :q8_0) when pulling a model. If you're tight on RAM, some models offer even more aggressive compression below Q4, though quality drops become more noticeable at that point.

The Connection to Accuracy and Hallucinations

Quantization isn't just a size-and-speed tradeoff, it has a measurable effect on how often a model gets things wrong. As covered in our hallucinations article, more aggressive quantization measurably increases hallucination rates, and that effect compounds on smaller models. A 7B model at Q4 is going to make more factual errors than the same model at Q8, even before accounting for its smaller parameter count on its own.

If accuracy genuinely matters for what you're using the model for, weighing a higher quantization level against the extra RAM it needs is a reasonable tradeoff to make deliberately, rather than defaulting to the smallest option without thinking about it.

Choosing a Level for Your Plan

The right quantization level comes down to how much RAM your Self-Managed VPS or VDS plan has, and how much of that RAM you can dedicate to a model. A few practical guidelines:

Start with Q4_K_M if you're not sure. It's Ollama's default for a reason, and it comfortably fits on smaller plans while remaining genuinely usable.
Move up to Q5_K_M or Q8_0 if your plan has RAM to spare and you want noticeably better accuracy, especially for tasks like coding help or anything where a wrong answer is costly.
Leave headroom for everything else. The model itself isn't the only thing using RAM. As covered in our RAM and CPU usage article, your operating system, the context window, and any other apps running on the same server all need their own share.

Summary

Quantization compresses a model by reducing the precision of its internal numbers, trading some accuracy for a meaningfully smaller memory footprint and faster performance. Ollama defaults to Q4_K_M for most models, which is a well-supported starting point for most customers, but higher levels like Q5_K_M or Q8_0 are worth considering if your plan has the RAM and your use case benefits from the extra accuracy. Since quantization level directly affects hallucination rates, it's worth choosing deliberately rather than just going with whatever a model defaults to.