.svg)
.svg)
.svg)
.svg)
.svg)
.svg)
.svg)
The first agent I wired into Ollama followed its instructions for exactly three turns, then started ignoring the system prompt. No error, nothing in the logs, the model just got politely stupid. The cause was the context window: Ollama defaults to 4096 tokens, my conversation had outgrown it and everything past the limit was being silently dropped. This guide covers what the context window really does, the four ways to change it and the memory bill that arrives when you do.
What the context window does
The context window (num_ctx in Ollama's vocabulary) is the total number of tokens the model can see at once: system prompt, conversation history, your documents and the reply it's writing, all sharing one budget. Nothing outside the window exists as far as the model is concerned. The nasty part is how overflow behaves. There's no warning when input exceeds the window; the oldest content is truncated and the model answers from what's left. So the symptoms are behavioral rather than technical: a chatbot forgetting the start of a long conversation, a RAG setup ignoring half the retrieved documents, an agent losing its tool instructions mid-task.
The default is 4096 tokens
As of the v0.32 line (August 2026), Ollama's default context is 4096 tokens, roughly 3000 English words across the whole exchange. That's deliberate, since context memory is expensive and most casual chats never hit the limit. But 4096 evaporates fast in real use. A decent agent system prompt eats 1000+ tokens before the user says anything, and a single pasted source file blows the budget alone. If long inputs are your normal case, the default is wrong for you and you should change it globally.
Set OLLAMA_CONTEXT_LENGTH system-wide
The OLLAMA_CONTEXT_LENGTH environment variable sets the server-wide default, and on a Linux service install it belongs in a systemd override:
sudo systemctl edit ollamaAdd this under the comments in the editor that opens:
[Service]
Environment="OLLAMA_CONTEXT_LENGTH=16384"Then apply it:
sudo systemctl daemon-reload
sudo systemctl restart ollamaEvery model now loads with a 16K window unless a request overrides it. This is the right lever when a whole team or a fleet of clients talks to one server, because nobody has to remember per-request settings. Exporting the variable in your shell does nothing for the service, by the way. The systemd unit has its own environment, which catches almost everyone once.
Set num_ctx in an interactive session
Inside an ollama run session, change it for the current conversation only:
/set parameter num_ctx 16384The setting lasts until you exit, and /save mymodel preserves the session if you want it back later. This is the quick-experiment lever: raise the window, paste your long document, see if the answers improve. The rest of the session commands are in the Ollama CLI cheat sheet.
Set num_ctx per API request
API calls override the default through the options object:
curl http://localhost:11434/api/chat -d '{
"model": "llama3.1",
"messages": [{ "role": "user", "content": "Summarize this contract..." }],
"options": { "num_ctx": 16384 },
"stream": false
}'Per-request control fits mixed traffic: short queries stay cheap at the default while the document pipeline requests 32K only when it needs it. One caution: changing num_ctx between requests forces a model reload, so alternating values on every call adds seconds of latency each time. Pick a small set of values and stick to them. Payload details for both endpoints are in the Ollama API guide.
Bake num_ctx into a Modelfile
For a permanent per-model setting, create a variant with the parameter built in. Write a file named Modelfile:
FROM llama3.1
PARAMETER num_ctx 16384Build and run it:
ollama create llama3.1-16k -f ./Modelfile
ollama run llama3.1-16kNow every client that requests llama3.1-16k gets the bigger window with zero configuration on their side, which is exactly what you want when the client is a third-party tool that doesn't expose num_ctx at all. The full instruction set (SYSTEM prompts, sampling parameters, templates) is in the official Modelfile reference.
Which value to pick
Powers of two are the convention: 8192, 16384, 32768. For plain chat, 8K covers almost everyone. RAG pipelines want 16K as a floor, because a system prompt plus four or five retrieved chunks of 500 tokens each plus the answer already crowds 8K. Agents are the hungry ones, and 32K is a reasonable starting point there. Resist the urge to set the maximum "to be safe": the memory cost below is real, and an oversized window on undersized hardware hurts more than truncation does, because it fails loudly at load time or slows every request.
One gap to know about. The OpenAI-compatible /v1 endpoints follow OpenAI's request schema, and that schema has no num_ctx field, so tools speaking OpenAI to your server can't raise the window per request. For those clients, set the server-wide default or serve them a Modelfile variant with the window baked in.
Model maximums: check before you raise
Each model has a trained maximum context, listed on its library page, and ollama show llama3.1 prints it locally as the context length field. Setting num_ctx beyond the maximum buys nothing good: at best wasted memory, at worst degraded output or a failed load. Small models tend to have small maximums too, so check before assuming the 128K number from one model card applies to the 3B model you deployed.
The memory cost of big contexts
Here's the bill. The KV cache that backs the context grows with every configured token, scaled by model size, and a 32K window can add multiple gigabytes on top of the weights for a mid-size model. Measure it on your own hardware instead of trusting anyone's table: load the model at the default, check ollama ps, then reload with the bigger window and check again. The delta is your per-context cost.
ollama run llama3.1 "hi" && ollama ps
OLLAMA_CONTEXT_LENGTH=32768 ollama serve # or via the systemd override
ollama psOn a GPU this cache competes with the weights for VRAM, and overflow spills layers to the CPU, so a big context can quietly convert a fast GPU setup into a slow hybrid one. Size the window and the hardware together; the Ollama hardware requirements guide has the numbers per model tier.
Context settings in Open WebUI
Open WebUI keeps its own per-model parameters, including context length, in its model settings (Admin Panel, then Models). Those apply on top of whatever the server default says, which confuses people who set OLLAMA_CONTEXT_LENGTH and see different behavior in the browser. Set the window where the requests originate. If Open WebUI is the main client, configure it there; the Open WebUI quick start and our Open WebUI setup guide cover the install and the settings screens. On a LumaDock server the Ollama VPS template deploys both pieces together in one click, so the only thing left to tune is this parameter.
When agents hit the wall
Agent frameworks are the worst-case client for the 4096 default because they stack a system prompt, tool schemas, memory and multi-turn history into every single request. The failure looks like a flaky model rather than a config problem, which is why it burns so much debugging time. We hit precisely this with the Hermes agent, and the Hermes Ollama context window fix is a worked example of diagnosing truncation from the symptoms backward. If your agent forgets instructions after a few tool calls, check the context before you touch the prompts.
Validation and troubleshooting
Confirm a setting took effect from the server's perspective: run a request, then journalctl -e -u ollama and look for the context size in the model load line, or compare ollama ps memory before and after. If memory jumped, the new window is live. If behavior still looks truncated at 16K, count your actual tokens honestly (a long system prompt plus retrieved chunks reaches 16K faster than you'd think) and remember the reply shares the same budget. And when an out-of-memory kill shows up in the logs after a context increase, that's the KV cache. Halve the window or move up a hardware tier, per the official FAQ's guidance on the context length variable.
