The Update That Broke Confidence

Two days ago, Jev vs Von vs SemIf: Real Numbers found something that should not happen: a frozen, off-the-shelf model beat a purpose-built specialist at its own job, on every task, and free heuristics beat the specialist too. That article is closed. The numbers stand as written.

This one asks what happens next. What happened is that the specialist shipped an update, and two new systems showed up wanting a turn on the bench. The update did not touch the model’s weights. It touched something else, and the result made the model worse at the one job that matters most for a small self-hosted classifier: knowing when it doesn’t know.

What changed since round one

Same corpus pipeline, build_corpus.py over the Wyrmhole git history, but not the same rows. The repo grew from 2,441 commits to 2,692 in the meantime, so this round drew a fresh sample with SEED=20260922. Jev got rerun from scratch on that same fresh draw, so every number below compares apples to apples across systems. Round one’s Von numbers and this round’s Von numbers come from different draws and should not be compared directly, with one exception: the nonsense-abstention probes are fixed strings, not corpus rows, so those compare cleanly across rounds.

Two systems joined the field:

• reflex, a self-hosted server wrapping stock Qwen3.5-4B with a built-in dual-order calibration step. It runs each choice question forward and reversed, then combines the two answers.
• Laya (convaiinnovations/laya), a routing-agent library rather than a server. You import it and call Router(preload=True).predict(state, questions) directly in your own process, so there is no network hop between you and the answer.

I rented a GPU pod for this round and tore it down when I finished. Nothing else about the setup changed, moving on.

Von’s 2026-09-22 release is a re-skin, not a retrain

The file that actually holds the model, option_marker.pt, is byte-identical by sha256 (52202f17...) between the 09-21 and 09-22 releases. What changed is marker_calibration.json, a per-item temperature-scaling formula. Temperature scaling can rescale a model’s confidence. It cannot change which option wins, because it never touches the argmax. So the 09-22 release is the same model wearing a new confidence dial, not a new model wearing a new version number.

Two pieces of evidence back that up. First, the hash match. Second, this round’s t1 accuracy landed at 25.6%, against the old checkpoint’s 26.7% on the old draw, a gap small enough to be noise, not signal from a retrain.

Fine, same brain, new dashboard. You’d hope the new dashboard is at least an improvement. It is not.

On four content-free nonsense probes, a single period, “lorem ipsum,” “what’s the weather in Lisbon,” and ten random digits, run against two question sizes (a 6-option commit-type question and a 12-option file-routing question):

On the 09-21 checkpoint, every nonsense probe landed below the 0.85 confidence gate, on both question sizes. That was the one real fix in that release. On 09-22, two of four six-option probes and three of four twelve-option probes now clear the gate: confidently wrong on meaningless input. The twelve-option column is the sharp part. It went from near-zero confidence (0.003 to 0.075) to 0.90 through 1.00 on three of four nonsense inputs.

The lesson: a calibration-only update can make a model worse at admitting it doesn’t know, even with the weights frozen solid. Temperature scaling is monotonic, so it can’t flip which answer wins, but nothing about it guarantees nonsense gets pushed toward low confidence. This release pushed it the wrong way.

Von’s numbers on the fresh corpus, for context before the full comparison: t1 accuracy 25.6% with the confidence gate never firing at all (all 90 rows land under 0.5 confidence), t2 accuracy 17.5%, t3 AUC 0.714, and a t4 Spearman correlation of -0.046, meaning that on this run, a higher predicted change-breadth weakly tracks with a smaller true change. A negative correlation on a scoring task isn’t a rounding error, it’s the model pointing the wrong direction.

The six-system scoreboard

The full field, side by side, with the free heuristics from the original article included as a floor. SemIf, the frozen general model from round one, got rerun end to end on this exact fresh corpus draw rather than carried over from its older numbers, so it belongs in this table on the same footing as everything else.

Jev wins every row, same as round one, no surprise there. The surprise is the order of everything else. reflex, which is just stock Qwen3.5-4B with a calibration trick bolted on, beats the purpose-built specialist and the fast in-process router on every task. SemIf, the model round one already flagged as surprisingly strong, holds up on a brand new corpus draw: it beats reflex on file routing and on is-a-feature, and trails reflex only narrowly on commit type and change breadth. Von loses to a keyword-matching heuristic on file routing (17.5% versus 48.8%) and loses to plain word count on change breadth (-0.046 versus 0.222). Two free heuristics, no model weights involved, both beat the specialist.

t1: commit type (6-option choice, n=90)

Latency isn’t a fair fight here. Jev, Von, and reflex all go over a network to a served model. SemIf runs CPU-only by design, no GPU offload at all, which is most of why its p50 sits at 27.6 seconds. Laya runs in-process on local GPU with no network hop, hence the 81ms.

Break it down by class and the pattern gets funnier:

Jev leads three of the six classes outright (docs, fix, test). reflex, Laya, and Von each own exactly one (feat, refactor, chore). SemIf never leads a class outright, its best showing on test ties reflex and still trails Jev, which matches a model that is competitive everywhere and sharp nowhere. Laya’s flat 0.0% on fix matches the collapse pattern Von showed in the original article on refactor: a model that quietly picked a favorite answer and rides it regardless of input.

t2: file routing (12-option choice, n=80)

SemIf beats reflex outright on this task, one of two places in this round (the other is is-a-feature) where the frozen general model outscores the calibrated one head to head. Von’s confidence gate is inverted on this task, the same defect the original article found. 63.7% of its rows clear 0.85 confidence, but accuracy on that “confident” slice (19.6%) barely beats its unconditional accuracy (17.5%). High confidence tells you almost nothing about whether Von got the answer right on this task.

Flip the candidate order on the same items and you get a second measurement: how much of a model’s answer is really about the content, versus where the options happen to sit in the list.

Laya drops 13.75 accuracy points from a pure order flip, no content changed, the largest order sensitivity measured in this round. Jev stays the steadiest by a wide margin, matching what the original article found.

t3: is-a-feature (probability, n=180)

SemIf posts the second-best AUC of the whole field here, ahead of reflex and well clear of Von and Laya. This is where Laya’s problems stop being subtle. Its own library prints a warning at import time, quoted here exactly as it appears:

RuntimeWarning: laya: this checkpoint ships invalid temperatures or values outside [0.5, 5]; using choice:11+=0.10058280825614929 -> 0.5. Treat confidence from the affected entries as uncalibrated.

That’s the vendor’s own code telling you not to trust the number it just handed you. The table shows why: Laya’s AUC (0.614) says the ranking underneath has real signal, it can sort features from non-features better than a coin flip if you go by score. But its accuracy at the 0.5 threshold (25.0%) is worse than a coin flip, because the raw output is shifted or inverted. The actual best threshold turns out to be 0.97, nowhere near 0.5. Use Laya’s number as a plain probability and you’ll be wrong more often than guessing. Hunt for your own threshold first.

t4: change breadth (3-level score, n=90)

Jev, reflex, and SemIf all track the true score well, within 0.09 Spearman of each other. Laya barely correlates. Von is negatively correlated: use Von’s prediction to guess which commits touched the most files and you’d do slightly better flipping the guess upside down.

reflex and SemIf split the accuracy wins; only reflex abstains cleanly

reflex and SemIf trade the top local spot depending on the task: reflex leads commit type and change breadth, SemIf leads file routing and is-a-feature. Both clear Von and Laya on all four tasks. Where reflex pulls ahead is abstention: every nonsense probe on both question sizes lands under the 0.85 gate, and the answers underneath vary more than SemIf’s or Laya’s, though not perfectly.

reflex lands all four of four probes under the 0.85 gate, on both question sizes. That beats Von at its best checkpoint, and its answers vary more than either self-hosted rival: two different guesses split evenly on the six-option set, and one genuine outlier among four on the twelve-option set (three land on scripts/update_imports.py, one on old_generate_town_images.py). Not a full collapse, but not perfectly clean either.

SemIf also lands all four of four probes under the gate by raw confidence, but the six-option set already leans collapsed: three of four nonsense inputs land on chore, the fourth on docs. The twelve-option set finishes the job. Every nonsense input routes to the same predicted file, scripts/update_imports.py, regardless of what was typed. Low confidence on paper, a collapsed default underneath on both question sizes.

Laya also lands under the gate on all four of four, and on paper that looks identical. It isn’t. Look at the 12-option column: every single nonsense probe routes to the exact same file, tests/testbot/interactions.py, no matter what you typed. That isn’t uncertainty. A model has collapsed onto one default answer and happens to say it with a low number attached. The confidence looks honest. The behavior underneath matches the collapse pattern Von showed in the original article when it defaulted to “refactor” on everything, just wearing a disguise that passes the gate check.

Two of the three local systems tested on the twelve-option set, SemIf and Laya, land on one default file every single time under nonsense input. reflex’s twelve-option answers lean toward that same failure mode, one file three times out of four, but it’s the only one of the three that ever breaks from a single default at all.

Where this leaves you

Jev wins every task, again. If you need the best answer and can tolerate a network hop and someone else’s API, that’s still the call.

If you need something self-hosted, reflex is the one to reach for now. reflex beats the purpose-built specialist and the fast router on every axis measured, and its nonsense-input answers vary more than either self-hosted rival, the closest thing to telling you the truth when it’s confused instead of confidently making something up, even if it isn’t perfectly clean either. SemIf, the plain unmodified Qwen3.5-4B from round one, is close behind and even ahead of reflex on file routing and is-a-feature, worth a look if you already have spare CPU and no spare GPU, but both its six- and twelve-option abstention settle on one default answer more than reflex’s does.

Laya is fast enough that it barely counts as waiting: sub-100ms, in-process, zero network. Bank that speed for the day its calibration gets fixed. Ship Laya today and you inherit a defect the vendor’s own warning message already told you about.

Von’s 2026-09-22 release doesn’t behave like an upgrade. Same model, same weights, a confidence dial that now lies more often than the one it replaced. If you’re running the older checkpoint, don’t update just because a new tag showed up. Check what the changelog actually touched first.

Does this actually measure judgment, or does it measure mimicry?

The commit-type labels above come from what each committer typed on the prefix, not from an independent judgment of whether that label was correct. A model that agrees with the prefix might be reading the diff. It might also just be predicting how this committer tends to write commit messages. Those two things produce the same accuracy number and look identical in a table.

There’s one way to tell them apart: pull every commit where the model disagreed with the prefix, and check by hand which side got it right.

I did that for Jev, the best model in this comparison and the one most likely to be pattern-matching rather than guessing. Jev disagreed with the prefix on 23 of the 90 t1 commits. I read all 23 diffs and judged each against what the commit actually changed, not against either label.

• 11 of 23: the human prefix was right, Jev was wrong. Usually a shallow pattern match: a regenerated commands.json export called docs because the diff is JSON text, a one-line HTML title tweak called docs when the commit message itself says “test cloudflare deploy trigger,” a multi-thousand-line vendor file copy called feat because the diff was large.
• 4 of 23: Jev was right, the prefix was the sloppy one. One commit titled “let Gemini and Gemma answer instead of narrating their own reasoning” patches broken output, it doesn’t ship a feature. I typed feat when I wrote that commit. Jev said fix. The diff backs Jev.
• 8 of 23: arguable either way, settled only by house convention a model can’t know. Internal skill markdown filed as docs by habit, guessed chore by Jev. Neither answer is wrong.

Under half the disagreement, 11 of 23, is Jev failing on surface pattern-matching, not on judgment. About a fifth, 4 of 23, is Jev catching a real mislabel. The rest is convention only the committer could call. So the 74.2% headline number is closer to a mimicry score than a correctness score, but the gap between the two is smaller than the raw accuracy spread between models would suggest.

Common Questions

Is Von’s 2026-09-22 update worth installing?

No. Von’s 2026-09-22 release keeps the exact same model weights, verified by matching sha256 hashes, and only changes a calibration file, so accuracy does not improve. Nonsense-abstention gets worse: three of four twelve-option nonsense probes now clear the 0.85 confidence gate, versus zero before. Skip this release until Von ships a real retrain.

Does reflex need a GPU to run?

Yes. reflex serves stock Qwen3.5-4B, a 4-billion-parameter model, and this benchmark ran reflex on a rented GPU pod to reach the measured latency, about 1,210ms median on the six-option task. Running a model this size on CPU alone will be much slower. Budget for GPU hosting before self-hosting reflex.

Is Laya safe to use in production right now?

Not for its confidence scores. Laya’s own library prints a runtime warning admitting its checkpoint ships invalid temperature values, and Laya’s accuracy at the default 0.5 threshold, 25.0%, is worse than chance. Laya’s ranking, AUC 0.614, has real signal, but production use needs a custom threshold, 0.97 in this benchmark, before trusting Laya’s output.

What changed since the original Jev vs Von vs SemIf article?

Two new systems joined the comparison: reflex, a calibrated Qwen3.5-4B server, and Laya, an in-process routing library. Von also shipped a 2026-09-22 release, and SemIf got rerun end to end on this round’s fresh corpus rather than carried over from its original numbers. The corpus is a fresh draw from the same Wyrmhole repo, now 2,692 commits instead of 2,441, so raw accuracy numbers between rounds are not directly comparable except for the nonsense probes.

Should I trust confidence scores from small self-hosted models?

Not by default. Von’s confidence gate is inverted on file routing, where high-confidence rows score barely above Von’s overall accuracy, and Laya’s raw probability output needs a custom threshold to be usable at all. Test any small model’s confidence with content-free nonsense probes before trusting its number in production.

Related Reading

• Jev vs Von vs SemIf: Real Numbers
• API vs Self-Hosted LLMs: The Real Cost
• Function Calling in Local LLMs
• LiteLLM & vLLM: One API to Rule All Your Models
• Mixture of Experts (MoE) for Self-Hosters, Demystified