Understanding Context Windows in AI

If you've had a long conversation with an AI agent and noticed it suddenly seems to forget something you mentioned earlier, that's not a bug. It's a limit called the context window, and understanding it will help you get more consistent results out of any AI tool.

What is a Context Window?

A context window is the amount of conversation an AI model can actually look at and consider at one time. It's measured in tokens, small chunks of text, roughly a few letters or a short word each.

Think of it like a desk with limited space. An AI can only keep so many papers spread out in front of it at once. As you add more to the conversation, older pages have to get pushed off the desk to make room, even if they were important.

Why This Looks Like "Forgetting"

Once a conversation or task grows beyond the model's context window, the oldest parts of it are no longer being considered at all. The AI isn't choosing to ignore what you said earlier, it genuinely can't see it anymore, the same way you can't read a page that's no longer on your desk.

This is why long conversations, very long documents, or agents working through a big multi-step task can start giving answers that seem to contradict something said earlier, or ask for information you already provided. It's not a sign the AI is broken; it's a sign the conversation has outgrown what it can hold onto at once.

How Big Is the "Desk," Really?

This is where it gets a little more interesting. A model might officially support a very large context window, but the actual limit you experience often depends on how it's configured, not just what the model is capable of.

DeepSeek's R1 models are a good example. They're officially capable of a 128,000-token context window, which is enormous, roughly the length of a short novel. But when run through Ollama, the default setting is often configured much smaller, sometimes just 2,000 to 4,000 tokens, to keep RAM usage manageable. Unless that setting is increased, you'll hit the "forgetting" point far sooner than the model's full capability would suggest.

The Tradeoff: A Bigger Window Needs More RAM

Increasing a model's context window isn't free. The more conversation a model has to hold in memory at once, the more RAM it needs, on top of whatever the model's own size already requires. This is the same kind of resource tradeoff involved in choosing a larger model size in the first place.

If your server is already close to its limit, increasing the context window can slow things down or cause instability rather than fixing the "forgetting" problem. It's worth checking your available RAM before increasing this setting, the same way you would before choosing a bigger model.

Context Window vs. an Agent's Long-Term Memory

It's worth separating two things that sound similar but aren't the same. A context window is short-term: it only covers what's in the current conversation or task. Some AI agents, though, are specifically built to keep useful information around even after a conversation ends.

Hermes Agent is a good example: it's designed to remember past tasks so it can run recurring jobs later, which is a workaround for the context window's short memory, not a bigger context window itself. This kind of long-term memory is saved separately and brought back into the conversation only when it's relevant, rather than trying to keep everything in view at once.

What You Can Do About It

• Start a new conversation for a new topic. Carrying unrelated history forward just uses up space that could go toward the task at hand.
• Summarize before continuing. If a conversation has gotten long, asking for a short summary and starting fresh from that summary keeps the important points without the full history weighing things down.
• Increase the context window only if you have RAM to spare. This helps with long documents or tasks, but it isn't free, so size it to what your server can actually support.
• Use an app built for ongoing memory if you need continuity. If you need an agent to remember things across sessions, a tool designed for that, like Hermes Agent, is a better fit than just expanding the context window.

Summary

An AI "forgetting" things mid-conversation almost always comes down to its context window, the limited amount of conversation it can consider at once, measured in tokens. Once a conversation grows past that limit, older parts genuinely drop out of view. The actual limit you experience often depends on configuration, not just the model's maximum capability, and a bigger window always costs more RAM. For real long-term memory across sessions, an agent built specifically for that, rather than a larger context window, is usually the better solution.