Ollama: Powerful Language Models on Your Own Machine

Installing Ollama takes about ninety seconds. Picking a model that actually runs well on your hardware takes longer, and it is the only decision in this whole exercise that changes the outcome. Everything else is a Docker file you copy once and never look at again.

So skip the part where anyone tells you local AI is hard. It is not hard. It is just constrained. You have a fixed amount of VRAM, models come in fixed sizes, and the good experience lives entirely in the gap between those two numbers.

What Ollama Actually Is

Ollama is a wrapper around llama.cpp with three useful additions: a model registry you pull from like Docker images, a scheduler that loads and unloads models from VRAM on demand, and an HTTP API on port 11434 that speaks both its own format and OpenAI’s.

That last part is why it caught on. Point any OpenAI-compatible client at http://localhost:11434/v1 and it works. Open WebUI, Continue, LibreChat, half the Python scripts on GitHub.

What Ollama is not is a serving engine. It handles one or two concurrent users on a home lab box fine. Put ten people on it and you will watch requests queue up behind each other while your GPU sits at 40% utilization. That is what vLLM exists for, and it is a different article.

Current release as of August 2026 is v0.33.2, and the project ships updates roughly weekly.

The Case for Running Local, Minus the Sales Pitch

Privacy is the real one. Your prompts never leave the machine. If you are pasting logs, client code, or anything covered by a contract into a chat box, this is the whole reason to bother.

Offline works. No network, no problem. Useful more often than you would expect.

Cost savings are conditional, and mostly false for light users. A 300W GPU pulling for an hour a day at 20 cents per kWh runs about $22 a year in electricity, which sounds great until you remember the card cost $600. If you are asking a model thirty questions a week, an API subscription is cheaper and will stay cheaper. Local wins on volume, on privacy, or when you already own the GPU for other reasons. Buying hardware specifically to save money on inference is a bad trade unless you are running it hard.

Fine-tuning is real but oversold in most write-ups. Ollama can build custom models from a Modelfile, which mostly means baking in a system prompt and sampling parameters. Actual weight training with LoRA happens elsewhere and then gets imported.

Install It

The native install is a shell script from ollama.com, and on Linux it drops in a systemd unit that starts on boot. That is the path of least resistance and what most people should use.

By default the server binds to 127.0.0.1:11434, which means loopback only. Nothing on your LAN can reach it. Keep it that way until you have read the section on exposure below.

A Compose File That Actually Works

The GPU reservation block below is the part people copy. Note the ports line, because it is the part people forget:

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    ports:
      - "127.0.0.1:11434:11434"
    volumes:
      - ./data:/root/.ollama
    environment:
      - OLLAMA_KEEP_ALIVE=10m
      - OLLAMA_CONTEXT_LENGTH=8192
    restart: unless-stopped
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

Four things worth understanding, and you can ignore the rest of the keys:

The port mapping is not optional. The official image sets OLLAMA_HOST=0.0.0.0:11434 in its Dockerfile and declares EXPOSE 11434, so the server listens on every interface inside the container. That does nothing for you without a published port. A compose file with no ports: block gives you a running Ollama you can only reach with docker exec, which defeats the point of the API.

Bind to 127.0.0.1 on the host side. Writing "11434:11434" publishes the API to your entire LAN, and Ollama has no authentication in front of it. Anyone who can route to that port can pull models onto your disk, unload the model you are using, and burn your GPU. On a flat home network that includes the TV. The 127.0.0.1: prefix keeps it on the box.

The volume is where your models live, and they are large. Point ./data at storage you actually have. Three or four mid-size models will pass 40GB without much effort.

OLLAMA_KEEP_ALIVE decides your latency. Default is five minutes, after which the model unloads and the next prompt pays the full reload cost from disk. Raise it if you use the thing all day and have VRAM to spare. Lower it to 0 if you want VRAM released the moment a request finishes, which matters if you share the GPU with anything else.

Drop the whole deploy: block if you have no NVIDIA card. Ollama runs on CPU. It is slow, and on a small model it is usable.

Picking a Model That Fits

The rule of thumb: the download size on the model page is close to the VRAM the weights need, and you should budget another 1 to 2GB for context on top. If the total exceeds your card, Ollama will split the model between GPU and system RAM and the speed falls off a cliff. Not a little. Roughly an order of magnitude.

Real numbers from the Ollama library as of August 2026:

Gemma 4 comes in e2b, e4b, 12b, 26b and 31b variants. Qwen3.6 ships 27b at 18GB and 35b at 23GB. qwen3-coder is the one to grab for code work: 30b lands at 19GB, and the 480b is 290GB, so no. gpt-oss is OpenAI’s open-weight release, 14GB for the 20b and 65GB for the 120b.

The -qat tags are quantization-aware trained, which means the quantized version was tuned to survive quantization rather than being squeezed after the fact. They hold up better than a plain Q4 at the same size. Take them when they exist.

To pull and run:

docker exec ollama ollama pull gemma4:12b
docker exec -it ollama ollama run gemma4:12b

The -it matters on the second command. ollama run with no prompt argument opens an interactive session, and without a TTY attached it has nothing to talk to. This trips people up constantly, and the error you get does not point at the missing flag.

For a one-shot question, skip interactive mode entirely:

docker exec ollama ollama run gemma4:12b "explain the difference between a Docker volume and a bind mount"

Talking to It From Code

The CLI is for poking at models. Everything else goes through the API, and there are two of them on the same port.

Ollama’s native endpoint is /api/generate for one-shot completions and /api/chat for conversations:

curl http://localhost:11434/api/generate -d '{
  "model": "gemma4:12b",
  "prompt": "name three ZFS tuning knobs",
  "stream": false
}'

Leave stream out and you get a token-by-token event stream instead of one JSON object, which is what you want in a UI and a nuisance in a shell.

The OpenAI-compatible endpoint lives at /v1 and is the one to reach for in almost every case, because every client library already speaks it:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")

resp = client.chat.completions.create(
    model="gemma4:12b",
    messages=[{"role": "user", "content": "name three ZFS tuning knobs"}],
)
print(resp.choices[0].message.content)

The api_key is required by the client library and ignored by the server. Any string works. That is worth sitting with for a second, because it is the whole security model: there is none.

Two useful management endpoints while you are in here. GET /api/tags lists what you have pulled, and GET /api/ps is the API version of ollama ps, which is how a monitoring script checks whether the GPU is actually doing the work:

curl -s localhost:11434/api/ps | jq -r '.models[] | "\(.name) \(.size_vram)"'

If size_vram comes back well under size, part of the model is sitting in system RAM and that is your slowdown.

When to Move Off Ollama

Ollama is the right default and it has a ceiling. Three signs you have hit it.

Concurrency. OLLAMA_NUM_PARALLEL lets one model serve several requests at once, but the throughput does not scale the way a real serving stack does. vLLM’s continuous batching keeps the GPU saturated across many simultaneous requests, and past roughly four concurrent users on one card the gap stops being academic. If you are putting a team behind this, plan the move.

Per-flag control. Ollama decides how many layers to offload, how to size the KV cache, and how to batch. Usually it decides correctly. When it does not, you are editing a Modelfile and hoping, where llama.cpp would give you --n-gpu-layers and --cache-type-k directly. Squeezing a model that almost fits is the classic case, and it is the point at which llama.cpp stops being harder and starts being simpler.

Model formats it does not carry. The registry is a curated subset. Anything on Hugging Face in a format Ollama has not packaged means importing GGUF by hand through a Modelfile, and if the thing you want is a safetensors release with no GGUF conversion, Ollama is not the tool.

None of that applies to one person on one GPU asking questions, which is most home labs, which is why Ollama won.

The Gotchas Nobody Warns You About

latest is not the biggest model. ollama run gemma4 pulls whatever the maintainers tagged latest, which for Gemma 4 is 9.6GB and is the e4b variant, not the 31b. Always name the tag you want.

Model names in tutorials rot fast. This post originally recommended gemma3 and llama3.3. Both still pull, and both are a generation behind. Check ollama.com/library before copying a pull command out of anything written more than six months ago, including this.

Check what is loaded before blaming the model. ollama ps shows resident models and, critically, a PROCESSOR column telling you whether inference is running on GPU or CPU. If it says 100% CPU on a machine with a GPU, your container never got the device and the deploy: block is the first thing to check.

Context length costs VRAM. OLLAMA_CONTEXT_LENGTH defaults conservatively for a reason. Push it to 32k on a model that barely fits and you will spill into system RAM. The KV cache grows with context, and it grows fast.

Two models at once needs a decision. OLLAMA_MAX_LOADED_MODELS controls how many stay resident. On a single consumer card, leave it alone. Loading a second model usually means evicting the first anyway.

Common Questions

How much VRAM do I need to run a useful local LLM?

Eight gigabytes is the practical floor. That runs a 4GB quantized model like gemma4:e2b-it-qat with room for context, and the output quality is fine for summarizing, drafting, and simple code. Twelve gigabytes opens up gemma4:12b, which is the point where most people stop feeling like they are compromising.

Can Ollama run without a GPU?

Yes. Ollama falls back to CPU inference automatically and needs no configuration change. Expect roughly 3 to 8 tokens per second on a modern desktop CPU with a small model, against 40 or more on a mid-range GPU. It is usable for background tasks and batch work, and painful for interactive chat.

Is it safe to expose Ollama’s API to my network?

No, not directly. Ollama ships no authentication, so any host that can reach port 11434 can pull models, evict the loaded model, and use your GPU. Bind it to 127.0.0.1 and put a reverse proxy with auth in front if you need remote access, or reach it over a WireGuard or Tailscale tunnel.

What is the difference between Ollama and llama.cpp?

Ollama wraps llama.cpp and adds a model registry, automatic memory management, and an OpenAI-compatible HTTP API. llama.cpp is the inference engine doing the actual work underneath. Use llama.cpp directly when you want per-flag control over batching, offload layers, and sampling. Use Ollama when you want it to work.

Why is my model suddenly slow after it worked fine?

Almost always because it unloaded. Ollama frees a model from VRAM after OLLAMA_KEEP_ALIVE expires, defaulting to five minutes, and the next prompt pays the reload cost from disk. Run ollama ps to confirm nothing is resident. Raise the keep-alive value if the pause bothers you more than the held VRAM does.

Related Reading

• Running Gemma 4 Locally with Ollama
• Ollama Beyond the Basics: Model Management, Custom Models, and Optimization
• GPU Memory Math: Will This Model Actually Fit?
• LLM Backends: vLLM vs llama.cpp vs Ollama
• API vs Self-Hosted LLMs: The Real Cost