.svg)
.svg)
.svg)
.svg)
.svg)
.svg)
.svg)
Running Ollama in Docker gives you a clean, disposable install that sits next to the rest of your containers and updates with a single image pull. This guide covers the CPU-only container, the NVIDIA and AMD GPU paths, a compose file that pairs Ollama with Open WebUI and the update routine that doesn't delete your downloaded models. Commands are for Ubuntu 24.04 with Docker Engine installed, though any Linux host with Docker works the same.
Docker or native install
Worth deciding before you type anything. If Ollama is the only service on the box, the native install from our Ollama Ubuntu install guide is simpler: one script, a systemd service, no container layer to reason about. I'd pick Docker in three of four real deployments anyway, for two reasons. First, most servers running Ollama also run other things (a web UI, an automation stack, a database) and compose keeps that whole group declared in one file you can rebuild anywhere. Second, rollback: if a new Ollama release misbehaves, pinning the previous image tag takes seconds. The overhead is negligible on CPU, and with the container toolkit configured the GPU passes straight through. The one case where I'd insist on native is a single-purpose inference box you want as boring as possible.
Run Ollama on CPU
The standard command from the official Docker documentation:
docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollamaTwo flags do the real work here. -v ollama:/root/.ollama creates a named volume called ollama and mounts it where the container stores models and keys. Without it, every model you pull dies with the container. -p 11434:11434 publishes the API port to the host. Verify it's up:
curl http://127.0.0.1:11434You get Ollama is running back. One adjustment I'd make on any internet-facing server: publish to localhost only with -p 127.0.0.1:11434:11434. Docker's port publishing bypasses ufw rules, a detail that has burned plenty of people, so an 11434 published on all interfaces is reachable from the internet even when your firewall says otherwise. Keep it local and let a reverse proxy or VPN handle remote access; our guide to hosting Ollama on a VPS walks through those setups.
Run Ollama on an NVIDIA GPU
Step 1: Install the NVIDIA Container Toolkit
Docker can't see the GPU until the toolkit bridges it. With NVIDIA drivers already working on the host (nvidia-smi prints your card), add the repository and install:
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
| sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
| sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
| sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkitStep 2: Configure the Docker runtime
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart dockerThe restart matters. Skipping it is the most common reason the next command fails with a runtime error.
Step 3: Start the container with GPU access
docker run -d --gpus=all -v ollama:/root/.ollama -p 127.0.0.1:11434:11434 --name ollama ollama/ollamaStep 4: Verify the GPU is used
Three checks, from quickest to most conclusive. Load a model and ask Ollama where it landed:
docker exec -it ollama ollama run llama3.2 "say hi"
docker exec -it ollama ollama psThe PROCESSOR column should read 100% GPU. Then docker logs ollama shows the detected card during startup, and nvidia-smi on the host shows VRAM allocated to the ollama process while a model is loaded. If everything says CPU instead, the toolkit step went wrong; rerun the configure command and restart Docker before anything else. Full passthrough cards on a GPU VPS behave exactly like local hardware here, T4 through RTX PRO 6000 Blackwell, since the container sees a real device rather than a slice.
Run Ollama on an AMD GPU
AMD cards use the ROCm image and device mappings instead of the toolkit:
docker run -d --device /dev/kfd --device /dev/dri -v ollama:/root/.ollama \
-p 127.0.0.1:11434:11434 --name ollama ollama/ollama:rocm/dev/kfd is the ROCm compute interface and /dev/dri covers the render nodes. Permission errors on these devices usually mean your user (or the container runtime) lacks membership in the video and render groups on the host. There's also a Vulkan backend, enabled by default whenever a GPU is accessible, which covers cards outside the CUDA and ROCm lists. OLLAMA_VULKAN=0 disables it and GGML_VK_VISIBLE_DEVICES selects specific devices when you have several.
Pull and manage models in the container
Every CLI command works through docker exec:
docker exec -it ollama ollama pull qwen3
docker exec -it ollama ollama list
docker exec -it ollama ollama run llama3.2The API on port 11434 works identically to a native install, so scripts and apps don't care that Ollama lives in a container. Typing the docker exec -it ollama prefix forty times a day gets old; I alias it to dol in my shell and move on. Model tags and sizes are on the Ollama library.
Docker compose with Ollama and Open WebUI
This is where Docker pulls ahead of the native install. One file declares the API and a full chat UI in front of it. Save as compose.yaml:
services:
ollama:
image: ollama/ollama
container_name: ollama
volumes:
- ollama:/root/.ollama
restart: unless-stopped
open-webui:
image: ghcr.io/open-webui/open-webui:main
container_name: open-webui
ports:
- "127.0.0.1:3000:8080"
environment:
- OLLAMA_BASE_URL=http://ollama:11434
volumes:
- open-webui:/app/backend/data
depends_on:
- ollama
restart: unless-stopped
volumes:
ollama:
open-webui:Note what's absent: the ollama service publishes no ports at all. Open WebUI reaches it as http://ollama:11434 over the compose network, so the raw unauthenticated API never touches a host interface. That's the pattern I'd ship. For NVIDIA GPU access, add this block to the ollama service:
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]Bring it up with docker compose up -d and open http://localhost:3000. Setup of the UI itself, accounts and remote connections included, is covered in our Open WebUI setup guide. If you later add services like a proxy or intrusion prevention to the same file, the structure holds; our CrowdSec Docker compose guide shows that pattern on a bigger stack.
Update the container without losing models
Models live in the named volume, and volumes outlive containers. That's the whole trick. For a docker run setup:
docker pull ollama/ollama
docker stop ollama
docker rm ollama
docker run -d --gpus=all -v ollama:/root/.ollama -p 127.0.0.1:11434:11434 --name ollama ollama/ollamaReuse whatever flags you started with. The new container mounts the same ollama volume and finds every model where it left them. Compose users get the short version:
docker compose pull
docker compose up -dOnly containers with a new image get recreated. The thing that does destroy models is docker compose down -v or docker volume rm ollama; the -v flag deletes volumes and there's no undo, so keep it out of your muscle memory.
Rolling back after a bad update is the mirror image: point the run command or compose file at the previous version tag from Docker Hub and recreate the container the same way. Confirm what's running with docker exec -it ollama ollama --version, worth a glance after every update because a failed pull quietly leaves the old image in place.
Troubleshooting Docker GPU and container issues
When --gpus=all errors with could not select device driver, the container toolkit isn't registered with Docker: rerun sudo nvidia-ctk runtime configure --runtime=docker and restart the Docker daemon. When the container runs but inference sits on CPU, compare CUDA driver versions; docker logs ollama prints why a GPU was skipped. Slow model pulls inside the container follow the same proxy rule as native installs, only HTTPS_PROXY counts, passed with -e HTTPS_PROXY=... at run time. And if port 11434 is already bound on the host, you have a native Ollama service running alongside Docker; stop one of them, or publish the container on a different host port like -p 127.0.0.1:11435:11434. For image internals and available tags, check the ollama/ollama page on Docker Hub.
