If all you want is a chat model on your own hardware, use Ollama and stop reading. It is simpler, the model naming is saner, and the community writes more tutorials for it.
LocalAI earns its keep when you need the rest of the OpenAI API surface from one endpoint. Chat completions, embeddings, image generation, speech to text, text to speech, reranking, all on /v1/*, all from a single container. Point an application that expects OpenAI at it and the application does not need to know anything changed. That breadth is the reason to run it, and it is the only reason.
Two things have changed enough since 2024 that any older guide will actively mislead you, including the previous version of this post. Both are below.
The Compose File Everyone Copied Is Pointing at a Graveyard
Search for a LocalAI compose file and you will find one carrying an environment block like this:
environment: - 'PRELOAD_MODELS=[{"url": "github:go-skynet/model-gallery/gpt4all-j.yaml", "name": "gpt-3.5-turbo"},{"url": "github:go-skynet/model-gallery/mixtral-Q3.yaml", "name": "mixtral-Q3"}]'The go-skynet/model-gallery repository was archived on 2024-06-22. It receives no updates and never will.
The trap is that this does not fail loudly. GitHub keeps serving raw files from archived repos, so PRELOAD_MODELS still resolves, still downloads, and still boots. You get a working LocalAI serving GPT4All-J and a Q3 quant of Mixtral, models from early 2024, under an alias called gpt-3.5-turbo. Everything reports healthy. You just quietly installed the state of the art from two and a half years ago.
PRELOAD_MODELS itself is fine and still supported, now also spelled LOCALAI_PRELOAD_MODELS. The URLs inside it are the dead part.
The current gallery is built into LocalAI and defaults to github:mudler/LocalAI/gallery/index.yaml@master. You do not need to configure it.
Authentication Now Denies by Default
LocalAI 4.9.0, released 2026-08-20, changed every HTTP route to require credentials unless it appears on an explicit public list. That release credits an external report of an authentication bypass where unprefixed aliases including /models, /moderations, /backends and /mcp/chat/completions fell outside the old protected-route matching.
Read that in the context of the 2024 compose file, which published 8080:8080 to the whole LAN with no API key set. Anyone who could route to that port could enumerate and install models. If you have had a LocalAI container running on a flat home network since 2024, that is worth a minute of your attention today.
Set an API key. The variable is LOCALAI_API_KEY, and it takes a list:
LOCALAI_API_KEY=pick-something-long-and-randomLOCALAI_MODELS_PATH=/modelsLOCALAI_CONTEXT_SIZE=4096LOCALAI_WATCHDOG_IDLE=trueLOCALAI_WATCHDOG_IDLE_TIMEOUT=10mNote the LOCALAI_ prefix on all of those. LocalAI moved to prefixed variable names and kept the bare forms as aliases, so MODELS_PATH still works and LOCALAI_MODELS_PATH is what current docs use. Match the docs.
A Compose File for Now
services: localai: image: localai/localai:latest-gpu-nvidia-cuda-12 container_name: localai restart: unless-stopped ports: - "127.0.0.1:8080:8080" env_file: - .env volumes: - ./models:/models - ./backends:/backends - ./images:/tmp/generated/images healthcheck: test: ["CMD", "curl", "-f", "http://localhost:8080/readyz"] interval: 30s timeout: 10s retries: 5 start_period: 10m deploy: resources: reservations: devices: - driver: nvidia count: 1 capabilities: [gpu]Three corrections against the version this post used to carry:
The healthcheck was wrong. It set interval: 1m with timeout: 20m. A timeout longer than the interval means Docker starts a second check while the first is still running, and a curl against a local port does not need twenty minutes to answer anyway. The knob for “this container takes a long time to become ready” is start_period, which suppresses failures during startup without distorting the steady-state check. Model downloads on first boot are exactly what it is for.
Bind to loopback. "8080:8080" publishes to every interface. With the API key set you are no longer wide open, but there is no reason to offer the login page to your entire network.
Mount /backends. More on that next.
Pick your image tag to match your hardware. All of these are live: latest for CPU, latest-gpu-nvidia-cuda-12 and latest-gpu-nvidia-cuda-13 for NVIDIA, latest-gpu-hipblas for AMD, latest-gpu-intel for Intel, latest-gpu-vulkan as a vendor-neutral fallback. The latest-aio-* variants ship preconfigured models for chat, embeddings, images and audio, which is the fastest way to see what LocalAI does before you commit to curating models yourself.
Backends Are Separate Downloads Now
This is the other structural change since 2024. LocalAI used to build every inference engine into the container image, which made the image enormous. Engines are now downloadable artifacts managed like models:
local-ai backends listlocal-ai backends install llama-cpplocal-ai backends upgradeThey land in LOCALAI_BACKENDS_PATH, which is why the compose file above mounts ./backends. Skip that volume and you re-download engines on every container recreate.
Installing Models
The web UI at http://localhost:8080/browse is the easy path. From the command line:
local-ai models listlocal-ai models install gemma-4-12b-it-qat-q4_0Real entries in the current gallery, so you can skip the guessing:
| Gallery name | Roughly what it wants |
|---|---|
gemma-4-e2b-it-qat-q4_0 | 8GB VRAM |
gemma-4-12b-it-qat-q4_0 | 12GB VRAM |
qwen3.8-9b-q4 | 12GB VRAM |
qwen3.8-27b-q4 | 24GB VRAM |
gemma-4-31b-it-qat-q4_0 | 24GB VRAM |
The gallery moves faster than Ollama’s library and carries a much longer tail, including community fine-tunes and abliterated variants that Ollama will never index. That is a real advantage if you know what you are looking for, and a source of confusion if you do not, because the naming carries quantization and variant suffixes that nobody explains up front.
local-ai models install auto-selects a variant when an entry declares several. It drops builds your hardware cannot run or cannot fit, prefers the engine your hardware likes, and picks by size among what remains. Pass --variant when you want to override that.
The Model YAML Is Where the Control Actually Lives
Gallery installs are convenient and they hide the thing that makes LocalAI worth the extra setup. Every model is a YAML file in your models directory, and you can write or edit one by hand. Ollama’s Modelfile is a system prompt and some sampling parameters. This is the engine configuration.
A minimal one looks like this:
name: mymodelbackend: llama-cppparameters: model: gemma-4-12b-it-qat-Q4_0.gguf temperature: 0.3context_size: 8192gpu_layers: 999f16: truetemplate: chat: chat completion: completionThe name is what clients ask for in the model field of an API call. It has nothing to do with the filename, which is why aliasing a local model as gpt-4o works.
gpu_layers defaults to 0, which means CPU only. That deserves bold type because it is the single most common reason a LocalAI install feels broken. Gallery entries set it for you. A YAML you wrote yourself does not, so the model loads fine, answers correctly, and runs at CPU speed on a machine with a perfectly good GPU sitting idle. Set it to a number higher than the model’s layer count and the whole thing offloads.
context_size accepts -1, which reads the model’s full trained context out of the GGUF metadata. Convenient, and a way to fill your VRAM by accident, because it takes the raw maximum with no capping against what your card has. LocalAI logs a warning and loads it anyway. Pick a real number.
The memory knobs are mmap (on by default, keep it), mmlock to pin the model and stop the kernel swapping it out, low_vram for cards that are tight, and no_kv_offloading when the KV cache is what is spilling. These are llama.cpp flags with a config file in front of them, which is exactly the layer Ollama does not give you.
Text cleanup lives here too. stopwords halts generation on a phrase, cutstrings strips patterns out of responses, trimspace tidies the edges. Useful for models that will not stop emitting a chat template token.
You can also skip the gallery entirely and point at a URL, with a checksum so you know what you got:
parameters: model: my-model.gguf
download_files: - filename: my-model.gguf uri: https://example.com/model.gguf sha256: abc123...That is the answer for anything on Hugging Face the gallery has not indexed, and it is the reason people who have outgrown Ollama’s registry end up here.
One more field worth knowing: known_usecases. LocalAI infers what a model is for from its backend and config, and gets it wrong often enough that there is an override. If your embedding model shows up as a chat model in /v1/models/capabilities, that is the field to set.
One Endpoint, Every Modality
This is the payoff, and it is worth seeing concretely because it is what you cannot get from Ollama. Every call below hits the same container on the same port with the same API key.
Chat, in the shape any OpenAI client already speaks:
curl http://localhost:8080/v1/chat/completions \ -H "Authorization: Bearer $LOCALAI_API_KEY" \ -H "Content-Type: application/json" \ -d '{"model":"gemma-4-12b-it-qat-q4_0","messages":[{"role":"user","content":"one sentence on ZFS ARC"}]}'Embeddings, which is what you need for any retrieval or semantic search setup:
curl http://localhost:8080/v1/embeddings \ -H "Authorization: Bearer $LOCALAI_API_KEY" \ -H "Content-Type: application/json" \ -d '{"model":"bert-embeddings","input":"the quick brown fox"}'Image generation, speech to text, and text to speech, all on the standard paths:
curl http://localhost:8080/v1/images/generations \ -H "Authorization: Bearer $LOCALAI_API_KEY" \ -d '{"model":"stablediffusion","prompt":"a whale wearing a wrench","size":"512x512"}'
curl http://localhost:8080/v1/audio/transcriptions \ -H "Authorization: Bearer $LOCALAI_API_KEY" \
curl http://localhost:8080/v1/audio/speech \ -H "Authorization: Bearer $LOCALAI_API_KEY" \ -d '{"model":"tts-1","input":"the backup finished"}' --output out.wavCurrent builds also serve /v1/moderations, /v1/images/inpainting, /v1/images/upscale, /v1/audio/diarization, and a /v1/realtime WebRTC surface. Stack that against running Ollama plus a Whisper container plus a Stable Diffusion container plus something for embeddings, each with its own port, its own model directory, and its own idea of how to talk to a GPU. That consolidation is the argument for LocalAI in one sentence.
There is also a handy non-OpenAI route at /api/models/vram-estimate that tells you whether a model will fit before you spend an hour downloading it.
To point the official Python client at the whole thing:
from openai import OpenAI
client = OpenAI( base_url="http://localhost:8080/v1", api_key="pick-something-long-and-random",)
print(client.chat.completions.create( model="gemma-4-12b-it-qat-q4_0", messages=[{"role": "user", "content": "explain ZFS ARC in one sentence"}],).choices[0].message.content)No shim, no adapter library. That is the entire integration.
Gotchas
First boot is slow and looks broken. LocalAI downloads a backend, then a model, then loads it. On a slow link this is tens of minutes with an unhelpful log. That is what start_period: 10m is protecting you from.
Model aliases are a loaded gun. You can name any local model gpt-4o so that unmodified client code hits it. Useful for drop-in replacement, and a good way to spend an afternoon confused about why quality dropped after a colleague pointed a script at your box.
The watchdog exists because backends hang. LOCALAI_WATCHDOG_IDLE with a timeout reclaims VRAM from a backend nobody is using. Turn it on for a single-GPU machine, along with LOCALAI_SINGLE_ACTIVE_BACKEND=true if you keep multiple models configured.
CUDA image tag has to match your driver. A cuda-13 image on a driver that only supports CUDA 12 fails at backend load, not at container start, so the container sits there reporting healthy while nothing works.
Common Questions
Should I use LocalAI or Ollama?
Use Ollama for chat and code models on one machine. Use LocalAI when you need embeddings, image generation, speech to text and chat behind a single OpenAI-compatible endpoint, or when an application you cannot modify expects the full OpenAI API. LocalAI covers more surface, and it costs more setup time to get there.
Does LocalAI actually work with the OpenAI Python library?
Yes. Set base_url to http://localhost:8080/v1 and api_key to whatever you put in LOCALAI_API_KEY. Chat completions, embeddings, and the images endpoint all respond in OpenAI’s schema. Newer or niche parameters can be ignored rather than honoured, so check the response when a specific flag matters.
Can LocalAI run without a GPU?
Yes. The plain localai/localai:latest image is the CPU build, and no configuration change is needed beyond dropping the deploy: block from your compose file. Stick to models around 4GB or smaller. Expect single-digit tokens per second, which is fine for embeddings and batch jobs.
Is my old LocalAI container a security problem?
Possibly. Setups from before 4.9.0 with a published port and no LOCALAI_API_KEY exposed an unauthenticated API, and 4.9.0 exists partly because of a reported bypass affecting routes that were meant to be protected. Update the image, set an API key, and bind the port to 127.0.0.1.
Where did the model gallery go?
Into LocalAI itself. The old go-skynet/model-gallery repository was archived in June 2024, though its raw files still resolve, which is why stale compose files keep installing 2024-era models without error. The current default gallery is github:mudler/LocalAI/gallery/index.yaml@master and needs no configuration.
