My 125B Local Run Ran Out of Weekend

45 Tokens a Second Is Plenty, Right?

I started an agent job on Friday with a local 125B model and a tokens-per-second number I was proud of. About 45 t/s. Faster than I read. I figured the weekend would be enough.

On Sunday I stopped it, because the GPUs had weekday jobs to go back to. It had done a large part of the work and was still making progress. A free cloud model, given the same prompt from scratch, finished the whole thing in about 7 hours.

The local model was not too dumb. It was too slow for the hours I had. The 45 t/s number was true, and it told me almost nothing about how long an agent job takes. This post is about why.

One caveat, said once: this is one run of one task on one box. It is not a benchmark. Treat it as a field report.

The Job

The task was exploratory. I wanted to pull apart an older Windows game (a few years old) and see what an agent could recover from it. No hard goal. I wanted to know if it was doable, what the result would look like, and what it would yield.

Both runs used the same setup:

• Harness: DeepSeek Harness (dsh), web UI, no extra plugins. It is an open-source agent harness from DeepSeek, MIT licensed, and the repo calls itself a developer preview with compatibility-breaking changes expected (checked on its GitHub page, November 2026). Pin your version if you care about repeating this.
• Prompt: identical for both runs.
• Starting point: from scratch each time. The cloud run did not inherit anything from the local one.

The Two Contenders

Both models sit in the same size class: roughly 120B total parameters, mixture of experts, a small slice active per token. If “total vs active” is fuzzy, the parameter-count explainer covers it, and the MoE post covers why a 125B model can run on a desktop at all.

Local: Qwen3.8-Flash-Next. The Hugging Face model card lists 125B language-model parameters with 6B active per token, 512 experts, and 10 routed plus 1 shared expert per token. I ran Unsloth’s UD-IQ3_XXS GGUF, about 3 bits per weight. I picked that quant to fit RAM, and quality came second: the experts take about 43 GB of system RAM at that size. The larger IQ3_S build (about 50 GB) left too little headroom on a 64 GB host. For more on how quant choices trade size against quality, see the quantization post.

Server: Strata. Strata is an MIT-licensed inference engine for large MoE models on consumer GPUs. Per its README it serves OpenAI- and Anthropic-compatible APIs on localhost. I run it under Docker Compose. I did not check whether the project ships an official image, so I will not claim one.

Cloud: Nemotron 3.5 Super VL. NVIDIA’s announcement lists 120B total and 12B active parameters, mixture of experts. I used OpenRouter’s free route, model id nvidia/nemotron-3.5-super-vl:free. OpenRouter’s models endpoint (checked November 2026) shows it with a 1,000,000-token context window, zero prompt and completion price, and NVIDIA as the provider. It lists the endpoint’s quantization as “unknown”, so I will not tell you what precision it ran at.

So the cloud model activates twice as many parameters per token, and someone else’s datacenter hardware serves it. Fair enough. That might explain quality. It does not explain 72 hours against 7.

The Hardware

The local box is not a rack:

• RTX 4060 8 GB plus RTX 3060 12 GB (20 GB VRAM total)
• Both cards on PCIe 3.0 x8 links
• 64 GB of system RAM
• Intel i7-9700, 8 cores, 8 threads

That is a normal secondhand-parts desktop. It also puts most of the model’s weights in system RAM, which matters in a minute.

The Scoreboard

The local run did not fail and did not finish. I pulled the plug when I needed the cards back.

The Math That Bothered Me

Here is a small calculator. It turns tokens and hours into overall tokens per second, and asks what share of the wall clock 45 t/s generation could even explain.

def rate(tokens, hours):
    return tokens / (hours * 3600)

def gen_share(tokens, hours, gen_tps, output_fraction):
    """Share of wall time spent generating, if output_fraction of tokens are output."""
    return tokens * output_fraction / gen_tps / (hours * 3600)

runs = {"local (Strata)": (5_000_000, 72), "cloud (free route)": (7_000_000, 7)}
for name, (tok, hrs) in runs.items():
    print(f"{name}: {rate(tok, hrs):.0f} tokens/s overall")

tok, hrs = runs["local (Strata)"]
print("\nLocal run, if generation runs at 45 t/s:")
for frac in (1.0, 0.2, 0.05):
    print(f"  {frac:>4.0%} of tokens are output -> generation is {gen_share(tok, hrs, 45, frac):.0%} of wall time")

Output:

local (Strata): 19 tokens/s overall
cloud (free route): 278 tokens/s overall

Local run, if generation runs at 45 t/s:
  100% of tokens are output -> generation is 43% of wall time
   20% of tokens are output -> generation is 9% of wall time
    5% of tokens are output -> generation is 2% of wall time

Three things fall out of that.

1. Overall throughput was 19 t/s locally against 278 t/s in the cloud. That is a 14x gap in tokens per second, and an even bigger one in wall time. The cloud run used more tokens (7M against 5M) and finished first.
2. Even in the absurd case where every token was generated output, 45 t/s explains only 43% of the 72 hours. Agent runs are not all output. They are mostly input.
3. The local run used fewer tokens in more time. So “the squeezed model just thinks longer” is not what these numbers support. If it had been looping through extra reasoning, the token count would be higher, not lower.

I did not log a prompt versus completion split, so I cannot give you the real output fraction. The table above is a sensitivity range, not a measurement.

Where the Time Probably Went

An agent harness does not send one prompt and read one answer. Every turn, it resends the conversation so far: the task, the tool definitions, every file it read, every command output. The context only grows. After a few dozen turns, each new request carries tens of thousands of tokens of history plus one small new step.

That means the bulk of your token count is input, and input is processed by prefill, also called prompt processing. Generation (decode) is the 45 t/s number. Prefill is a separate speed, and on a setup like mine it is likely the slow one.

Why prefill hurts here: the model is a mixture of experts, and on my box most expert weights live in system RAM. Per one detailed Strata write-up, Strata keeps the most-used experts in VRAM and computes misses on the CPU. A big batch of prompt tokens touches many experts, so more of them miss. Your CPU (8 cores, no hyperthreading, in my case) and the PCIe link do the heavy lifting.

Some published numbers, with attribution and their hardware, which is not mine:

• Per that write-up, an RTX 5090 prefilled about 100K tokens in 16 to 20 seconds. The same article quotes the README’s rule of thumb of about 60 seconds per 30K tokens, which is about 500 t/s.
• Strata’s README lists prompt speed of about 110 t/s for a 4K prompt on an RTX 5070 (12 GB) with 64 GB of RAM. That figure was measured with a DeepSeek model, not Qwen3.8-Flash-Next.
• The write-up measured the 5090’s PCIe link at about 29 GB/s on a PCIe 5.0 x8 link. PCIe 3.0 x8 tops out near 8 GB/s on paper (8 GT/s per lane, 8 lanes, 128b/130b encoding). That is my arithmetic, not a Strata figure.

Now plug in the cheap end of those numbers. A hypothetical turn with 60K tokens of context at 110 t/s of prefill (the README figure, from a different model) is over 9 minutes of waiting before the first output token. At 500 t/s it is 2 minutes. Run a few hundred turns and the weekend is gone, even with decode at 45 t/s the whole time.

I want to be straight about this: it is the likely explanation, not a measured one. I did not log prefill time per request, and I cannot tell you whether Strata reuses cached context between turns in this setup. If I rerun this, I log prompt and completion token counts and timings per request first, so the next post can replace this section with data.

The 3-bit quant is a possible quality factor, and it could have cost extra turns. But the evidence here points at speed first: the model was making real progress the whole time.

The 45 t/s number is the top speed on a forklift’s spec sheet. It is true, and it says nothing about how long the job takes when most of the work is loading and unloading pallets.

The Stall

About 2 hours were lost when the model server stopped processing requests. I noticed it, restarted DeepSeek Harness, and it worked again. I did not trace the cause. Treat it as an observation: a harness restart cleared it, and that is all I know.

For an unattended run, you do not want to find that out on Saturday at 2 AM. A watchdog is cheap. This one sends a tiny real completion request, and after N consecutive failures it restarts a named Compose service:

#!/usr/bin/env bash
# Restart a Compose service when the model server stops answering real requests.
URL="${URL:-http://localhost:8080/v1/chat/completions}"   # your server's OpenAI-compatible endpoint
SERVICE="${SERVICE:-harness}"                             # Compose service to restart
MAX_FAILS="${MAX_FAILS:-3}"
INTERVAL="${INTERVAL:-60}"
BODY='{"model":"local","messages":[{"role":"user","content":"ping"}],"max_tokens":1}'
fails=0
while true; do
  if curl -fsS --max-time 60 -o /dev/null -H 'Content-Type: application/json' -d "$BODY" "$URL"; then
    fails=0
  else
    fails=$((fails + 1))
    echo "$(date -Is) probe failed ($fails/$MAX_FAILS)"
    if [ "$fails" -ge "$MAX_FAILS" ]; then
      echo "$(date -Is) restarting $SERVICE"
      docker compose restart "$SERVICE"
      fails=0
    fi
  fi
  sleep "$INTERVAL"
done

Three gotchas:

1. Probe a real completion, not /v1/models. A model listing can answer while generation is stuck. I did not test that against Strata, so take it as a design choice that I did not verify.
2. Set --max-time above your longest prefill. The probe queues behind the agent’s own request. If the server is busy chewing a long prompt, a short timeout restarts a healthy server. With the numbers above, that means minutes, so raise --max-time or INTERVAL to match your box.
3. Point URL at your own server. I used placeholders. Strata’s README gives no port number I could verify, so check your own Compose file. A restart also kills the in-flight request, so make sure your harness can resume.

I ran bash -n on the script. I did not run it against a stalled server, so test the restart path before you trust it overnight.

The Cloud Run

Same prompt, same harness, fresh start. The cloud call shape, for reference, is the standard OpenRouter chat completions request:

curl https://openrouter.ai/api/v1/chat/completions \
  -H "Authorization: Bearer $OPENROUTER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "nvidia/nemotron-3.5-super-vl:free",
    "messages": [{"role": "user", "content": "ping"}]
  }'

In practice I pointed DeepSeek Harness at the OpenRouter endpoint instead of calling it by hand. If you want a gateway in front for routing and fallbacks, OpenRouter vs LiteLLM covers the choice.

It finished in about 7 hours and 7 million tokens, cost $0, and gave me everything I wanted from the experiment. The cloud endpoint pushes about 278 tokens through every second of wall clock, averaged over the run. My box managed 19.

Two limits on the free route (as of November 2026, from OpenRouter’s rate-limit docs):

• Free models allow 20 requests per minute.
• With under $10 of credits purchased, the daily cap is 50 requests. With at least $10, it is 1,000 requests a day.

An agent run makes a lot of requests. The 50-a-day tier would stop a multi-hour job early, so check which tier your account is on before you start.

The Privacy Trade

The free cloud route is free for a reason, and you should read the reason before pointing it at anything you care about. As of November 2026, the free NVIDIA endpoint’s page on OpenRouter says:

• “For the free endpoint, please do not upload any confidential information or personal data.”
• “Your use is logged for security purposes and to improve NVIDIA products and services.”
• The logged session data “is not linked to your identity or any persistent identifier.”

OpenRouter’s own privacy docs say account settings control whether requests can be routed to providers that may train on your data, and that there are separate settings for paid and free models. Providers follow their own policies.

That fits my task, which involved no secrets and no personal data. It does not fit your company’s source tree. The local run kept every byte on my box. If your material is private, that alone can decide it, speed or not.

What I’d Do Next Time

1. Log prompt tokens, completion tokens, and time per request. Then I can replace a hypothesis with a measurement.
2. Match the job to the hardware hours. Interactive chat and short tasks suit a big local MoE just fine. For a multi-day agent job on consumer hardware with only weekend hours, a free or cheap cloud route finished in an afternoon what my cards could not in a weekend.
3. Run the local model when the material is private. Give it a box that is not borrowed, so the job does not end when the weekend does.
4. Keep the watchdog running. Two hours of silent idle is a lot of weekend.

Related: if you are weighing cheap models for coding agents generally, the cheap workhorse post and local coding agents and context discipline cover the other half of this: keeping the context small is the cheapest prefill fix there is.

Common Questions

Can I run Qwen3.8-Flash-Next on 64 GB of RAM?

Yes, with the right quant. On my 64 GB host, the UD-IQ3_XXS build fit, with the experts taking about 43 GB of system RAM. The larger IQ3_S build, about 50 GB, left too little headroom. Pair that RAM with 20 GB of VRAM across two cards.

Is the OpenRouter free tier private?

No. As of November 2026, the free NVIDIA endpoint’s page says use is logged for security and to improve NVIDIA products, and asks you not to upload confidential or personal data. OpenRouter also has separate data-policy settings for paid and free models. Keep private material on a local server.

Would a faster GPU or PCIe link fix the slow agent run?

A faster card and link should raise prefill speed, but I did not test it. Published Strata figures show a gap: an RTX 5090 prefilled about 100K tokens in 16 to 20 seconds, per one write-up, while the README lists about 110 t/s for a 4K prompt on an RTX 5070 (measured with a different model).

Is llama.cpp’s --n-cpu-moe an alternative to Strata?

Yes, for MoE models that llama.cpp supports. The flag --n-cpu-moe N keeps the expert weights of N layers in system RAM while the rest stays on the GPU, and --cpu-moe does it for all layers. A community guide on Hugging Face notes that CPU plus GPU MoE inference is very sensitive to prompt-processing batch size.

Related Reading

• What the B in 12B Actually Means
• Mixture of Experts (MoE) for Self-Hosters, Demystified
• API vs Self-Hosted LLMs: The Real Cost
• The Free AI Stack: What $0 Gets You
• OpenRouter vs LiteLLM