Multi-GPU LLM Inference on Consumer Cards

You downloaded a 27B model, loaded it, and watched the loader fall over with an out-of-memory error. The model is 28.6 GB at Q8_0. Your card has 24 GB. You also have a second card in the closet, an older 12 GB one, and you are wondering if two wrongs make a right.

They can. Two cards add up to one pool of VRAM, and both llama.cpp and vLLM can use it. They split the work in very different ways, though. llama.cpp layer split tolerates weak PCIe slots and mismatched cards. vLLM tensor parallelism wants matching cards and a fast link, and rewards you with real concurrency. Pick by your hardware. GitHub stars tell you nothing about your PCIe slots.

Full example: Clone the working files at github.com/KingPin/sumguy-examples/llm/multi-gpu-inference

The VRAM Math That Starts This Whole Mess

Our example model is Qwen3.6-27B. Hugging Face reports 27.78 billion parameters in the safetensors metadata. The unsloth/Qwen3.6-27B-GGUF repo lists these files:

Q4_K_M fits on a single 24 GB card with room for context, so you do not need a second GPU for it. Q8_0 does not fit. It needs 28.6 GB for weights plus the KV cache, which grows with context length. That puts you in the 32 to 36 GB range, so a 24 GB + 12 GB pair (36 GB total) works and two 24 GB cards (48 GB) works with room to spare.

A second card also makes sense when you want a longer context, or want to run two models at once. It does not make sense when a smaller quant fits on one card. Always try one card first.

What Two Cards Do Not Do

Say it once so you stop hoping: a layer split does not make a single chat faster. Generating one token means running the model’s layers in order. GPU 0 runs its layers, hands a small activation vector to GPU 1, and GPU 1 runs the rest. While GPU 1 works, GPU 0 sits idle. You bought capacity, not speed.

Tensor parallelism can speed up a single stream, because every card works on every layer at the same time. The price is that the cards must exchange partial results many times per token, so the link between them becomes the bottleneck. Everything below follows from that trade.

PCIe Lane Reality

Consumer motherboards are stingy. The top slot is usually x16 wired to the CPU. The second physical x16 slot is often x4 or x8 (the second one sometimes hangs off the chipset). Riser cables and mining-style x1 adapters make it worse.

Per-direction bandwidth, from the PCIe spec numbers (encoding overhead included):

Check what you actually got before you plan anything. The query fields below exist in nvidia-smi, and check-gpus.sh in the example repo runs them together with the topology matrix:

nvidia-smi --query-gpu=index,name,pcie.link.gen.current,pcie.link.width.current,memory.total --format=csv
nvidia-smi topo -m
nvidia-smi topo -p2p r

GPUs drop their link speed at idle to save power, so read the numbers while a load runs. In the topo -m matrix, NV# means NVLink. PIX, PXB, PHB and NODE mean PCIe paths of increasing length (PHB goes through the CPU’s host bridge). SYS means the traffic also crosses the link between CPU sockets or NUMA nodes.

NVLink, P2P, and the NCCL Trap

Among recent GeForce cards, only the RTX 3090 and RTX 3090 Ti support an NVLink bridge. The 4090 and later dropped it. On those cards GPU-to-GPU traffic goes over PCIe, and peer-to-peer (P2P) access over PCIe is not reliably available on consumer boards and drivers. The third command above, nvidia-smi topo -p2p r, tells you what your box supports.

This matters because NCCL, the library vLLM uses for tensor parallel traffic, tries to use P2P. NCCL documents NCCL_P2P_DISABLE=1 as the switch that turns direct GPU-to-GPU transport off. vLLM’s troubleshooting page suggests it as a temporary workaround when a multi-GPU test script hangs, and warns that it can hurt performance.

So here is the trap. Your vLLM container hangs at startup on a consumer board. You set NCCL_P2P_DISABLE=1, it starts, and you move on. Now every tensor parallel all-reduce takes a detour through host memory. It works, and it may be slower than a plain llama.cpp layer split on the same cards. Treat the variable as a diagnostic first and a permanent setting only after you have compared both.

llama.cpp: Layer Split Is the Forgiving One

llama.cpp’s server README documents --split-mode with these values: none, layer (the default), row, and tensor. The layer mode splits layers and the KV cache across GPUs in a pipelined way. Between GPUs it only moves one activation vector per token at each split boundary. That is a tiny amount of data, so an x4 slot or a Gen3 link barely matters during generation. The one place the link can hurt is prompt processing with big batches, because the activations scale with the batch.

Here is the full compose file from the example repo:

services:
  llama:
    image: ghcr.io/ggml-org/llama.cpp:server-cuda
    restart: unless-stopped
    ports:
      - "127.0.0.1:8080:8080"
    volumes:
      - ${MODELS_DIR:-/srv/models}:/models:ro
    command: >
      -m /models/${GGUF_FILE:-Qwen3.6-27B-Q8_0.gguf}
      --host 0.0.0.0 --port 8080
      -c ${CTX_SIZE:-16384}
      -ngl 99
      --split-mode layer
      --tensor-split ${TENSOR_SPLIT:-2,1}
      --main-gpu 0
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

The flags that matter:

• --split-mode layer: split whole layers across cards. This is already the default; spelling it out documents your intent.
• --tensor-split 2,1: the README describes it as the fraction of the model for each GPU, as a list of proportions (its own example is 3,1). With a 24 GB and a 12 GB card, 2,1 puts two thirds on the big one. For our 28.6 GB file that is about 19 GB and 9.5 GB, which leaves headroom for KV cache and compute buffers on both. Do not split by raw VRAM size alone, because the context buffers need room too.
• --main-gpu 0: per the README, the GPU for intermediate results and KV when you use row mode, or the single GPU with none. In layer mode you rarely need it.
• -ngl 99: offload all layers to the GPUs. Recent llama.cpp also has --fit, on by default, which adjusts unset arguments to fit in device memory. Explicit flags win, and I prefer being explicit.

Device order is the other thing that bites. --tensor-split follows llama.cpp’s device order, which may not match the order nvidia-smi shows. Look at the startup log for each device’s index and VRAM, then confirm the split with nvidia-smi while the model is loaded.

What about row and tensor modes?

row splits each weight matrix by rows and runs the GPUs in parallel, so they talk to each other at every layer. The README labels it “parallelized”. On cards without a fast link that extra traffic is usually the cost. The newer tensor mode is labeled EXPERIMENTAL in the same README. Try both if you have an NVLink pair, benchmark with your own prompts, and keep layer as the fallback.

vLLM: Tensor Parallelism Wants a Matched Pair

vLLM’s parallelism docs say that if the model fits on one node but not one GPU, you use tensor parallelism with --tensor-parallel-size set to the GPU count. Every layer is sliced across all cards, and the cards sync up many times per token.

Two hard rules follow:

1. The tensor parallel size must divide the model’s attention head count. vLLM’s model config raises “Total number of attention heads must be divisible by tensor parallel size” otherwise. Qwen3.6-27B has 24 attention heads, 4 KV heads and 16 linear-attention key heads, so 2 or 4 work and 3 does not.
2. Cards should match. Each card holds an equal slice, so a 24 GB + 12 GB pair behaves like two 12 GB cards. You lose 12 GB.

The compose file:

services:
  vllm:
    image: vllm/vllm-openai:latest
    restart: unless-stopped
    ipc: host
    ports:
      - "127.0.0.1:8000:8000"
    volumes:
      - ${HF_CACHE_DIR:-/srv/hf-cache}:/root/.cache/huggingface
    environment:
      HF_TOKEN: ${HF_TOKEN:-}
      # Uncomment only if the launch hangs on a board without working P2P:
      # NCCL_P2P_DISABLE: "1"
    command:
      - --model
      - ${VLLM_MODEL:-Qwen/Qwen3.6-27B-FP8}
      - --tensor-parallel-size
      - ${TP_SIZE:-2}
      - --max-model-len
      - "16384"
      - --gpu-memory-utilization
      - "0.90"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

Notes on the file:

• ipc: host matches the --ipc=host flag in vLLM’s Docker docs. vLLM’s Kubernetes guide gives the reason: vLLM needs the host’s shared memory for tensor parallel inference, and Docker’s default /dev/shm is tiny. Compose’s shm_size is the alternative if you do not want to share the host IPC namespace.
• The image is vllm/vllm-openai. Pin a tag once it works. latest moves.
• The model is Qwen/Qwen3.6-27B-FP8, which exists on Hugging Face. I did not load it. Ampere cards like the 3090 have no FP8 hardware, so vLLM runs FP8 checkpoints there as weight-only FP8 through its Marlin kernel. If it refuses to start, pick a 4-bit AWQ or GPTQ repo that fits your VRAM.
• The commented-out NCCL_P2P_DISABLE is the diagnostic switch from earlier.

Pipeline parallelism, the PCIe-friendly option

vLLM also has --pipeline-parallel-size. Its docs describe it as splitting the model along layers with support for uneven splits, and recommend it over tensor parallelism when the GPUs have no NVLink (they name the L40S). In that sentence “uneven” refers to a layer count that does not divide evenly by the GPU count. Cards with different VRAM are a separate problem. By default vLLM spreads layers evenly across stages, so a 24 GB + 12 GB pair also needs VLLM_PP_LAYER_PARTITION (comma-separated layer counts per stage that sum to the model’s 64 layers, for example 42,22) to put more layers on the big card. That is vLLM’s version of what llama.cpp does by default. If your cards are PCIe-only and you want vLLM’s batching, set --tensor-parallel-size 1 and --pipeline-parallel-size 2. The docs describe the setting for this case, but I have not run it on consumer cards.

What About Ollama?

Ollama’s FAQ states its rule: if the model fits entirely on one GPU, Ollama loads it there, because that reduces PCI bus traffic. If it does not fit, it spreads the model across all available GPUs. You do nothing.

Two knobs matter. CUDA_VISIBLE_DEVICES takes a comma-separated list of GPUs and limits Ollama to that subset. OLLAMA_SCHED_SPREAD is listed in Ollama’s source as “Always schedule model across all GPUs”. Setting it spreads even models that would fit on one card, which usually costs you speed unless you are trying to leave room for other models.

Power, PSU, and the 2 AM Reboot

Two 3090s are rated at 350 W each. Add a CPU and drives and a 750 W supply is not enough. Transient spikes trip the protection on weak units, and the symptom is a hard reboot in the middle of a long generation. Your 2 AM self will not enjoy debugging that.

Power limiting is cheap insurance. LLM decoding is mostly memory-bound, so a lower cap often costs little speed (measure yours, I have no number to quote):

sudo nvidia-smi -pm 1
sudo nvidia-smi -i 0 -pl 250
sudo nvidia-smi -i 1 -pl 250

The limit resets on reboot, so put it in a systemd unit or a startup script. Also check the airflow. Stacking two three-slot cards with no gap cooks the top one.

The Verdict

1. Mismatched cards, or the second slot is x4: use llama.cpp with --split-mode layer and a --tensor-split ratio matched to VRAM. It is the simplest option here that uses all your memory.
2. Two matched cards, several users or agents hitting the API: use vLLM with --tensor-parallel-size 2. Its continuous batching is the reason to run it. Measure NCCL_P2P_DISABLE before you accept it as permanent.
3. Two matched cards, one user, PCIe only: use llama.cpp. Layer split gives no single-stream speedup. Tensor parallelism over PCIe has to beat it on your slots to justify the extra setup, so benchmark both if speed matters. llama.cpp is easier to run.
4. NVLink-bridged 3090 pair: this is the one case where vLLM tensor parallelism plays to its strengths. Try it, and compare with llama.cpp row mode.
5. You just want it to work: Ollama, and accept that you give up control of the split.

A forklift and a pickup both haul pallets. The pickup is slower on a warehouse floor, but it fits through the front door of your garage. For most home labs that door is the PCIe slot.

Common Questions

Can I mix NVIDIA and AMD GPUs for one model?

Yes, with llama.cpp. Its Vulkan backend drives NVIDIA and AMD cards, and its build docs show CUDA and Vulkan compiled into one binary, so a layer split can span both vendors. Pick cards with --device and list them with --list-devices. The compose files here are NVIDIA-only, and vLLM tensor parallelism needs one vendor.

Does NVLink matter for llama.cpp?

Not for the default layer split. Layer split moves one small activation vector per token between cards, which any PCIe slot handles. NVLink helps modes that exchange data at every layer, such as llama.cpp row mode and vLLM tensor parallelism. Among recent GeForce cards, only the RTX 3090 and 3090 Ti support a bridge.

How big a power supply do I need for two GPUs?

Add up both cards’ rated board power, add about 200 W for the CPU and drives, and add margin for spikes. For two 350 W cards that points at 1000 W or more. Power limiting with nvidia-smi -pl lets you run smaller, but verify with a load test.

Do both GPUs run at full speed at the same time?

In a llama.cpp layer split, no. Only one GPU works on a given token at a time, so you see one card busy and the other idle in nvidia-smi. Tensor parallelism keeps both busy. Running several requests at once can also fill the idle gaps.

Related Reading

• GPU Memory Math: Will This Model Actually Fit?
• Used GPU Buying Guide for Home Lab LLMs
• LLM Backends: vLLM vs llama.cpp vs Ollama
• Batch Inference: One Model, 100 Users
• CUDA vs ROCm vs CPU: Running AI on Whatever GPU You’ve Got