670 Tags, 452 Used Once, One Cheap Model
Measure the free baseline before you pay for anything. Then, if the paid thing clears that bar, treat it as a shortlist generator and never let it make the actual call. I only half believed that until I watched a model quietly try to strip tags off 92 of my own posts, and 18 of those tries were wrong. The 26 cents in the title barely registers. Reading the 20% it got wrong costs the real time.
My blog has 840 published posts with a tags array in the frontmatter, and the vocabulary drifted to 670 unique tags, 452 of them sitting on exactly one post. A tag used once is a dead-end page nobody browses and Google barely indexes. I wanted it cleaned up without reading 840 posts by hand, so I tried Jev on the job, benchmarked it against the free alternatives first, and kept notes on where it broke.
What Jev Actually Does
Jev is TypeSafe’s flagship model, what they call a “System One” model: built for fast structured decisions, not for writing paragraphs. You send it one state, the thing being judged, and a batch of questions evaluated against that state in parallel, in one call. You get typed answers back, not prose to parse.
Three question types cover everything I needed:
choicepicks one option from a fixed set you define, describing when each applies in a criteria object. You get back the chosen option plus a probability per option plus a confidence.noulis a yes-or-no question, answered as one probability from 0 to 1 and nothing else. There is no separate confidence field on a noul; 0.6 means “probably yes,” not “unsure.”scorerates something against ordered levels you list in acriteriaarray. You get backscore,confidence, alegendmapping each level number to its description, andprobabilitieskeyed by level number. The score itself is each level number times its probability, summed: three levels at probabilities 0.0, 0.57, and 0.43 give (0 times 0.0) plus (1 times 0.57) plus (2 times 0.43), which is 1.43. Sort by it freely, but round it before you use it to look up a level label or you will grab the wrong one.
The shape of a call, one noul question against a support ticket:
curl -s https://api.typesafe.ai/v1/systemone \ -H "Authorization: Bearer $TYPESAFE_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "jev-latest", "state": "Subject: cant login\nBody: tried resetting my password three times, still says invalid token. been locked out since yesterday.", "questions": { "needs_human": { "type": "noul", "instructions": "This ticket needs a human agent, not an automated password reset reply" } } }'{ "model": "jev-1.13.0", "answers": { "needs_human": { "type": "noul", "noul": 0.83 } }, "usage": { "input_tokens": 360, "output_tokens": 39 }}That usage block is where the number you get charged for actually lives. Pricing is $0.042 per million input tokens, and output tokens are free, so the bill is entirely a function of input_tokens in that block, as of September 2026. Rate limits sit at 250,000 tokens per second and 1,200 requests per minute, either one returns HTTP 429 if you cross it, and the docs are upfront that both numbers move without notice while they scale. Input is text only, no images, no audio, so anything else gets pre-processed into text first.
Jev is on OpenRouter as typesafe/jev-1.13, provided by TypeSafe itself, and its listed prompt price of 0.000000042 per token with completion at 0 matches the $0.042 per million TypeSafe’s own docs quote. Two independent sources, same rate, as of September 2026. It stays out of OpenRouter’s main catalog though: I fetched https://openrouter.ai/api/v1/models, got 446 models back, and none was Jev. The architecture field is the reason, text->decisions with output_modalities: ["decisions"]. That catalog covers text-output chat models, so a decisions model lives at its own endpoint (/api/v1/models/typesafe/jev-1.13/endpoints) behind an alpha decisions path rather than chat completions. “Output modality: decisions” is the honest one-line description: a decision engine that speaks JSON. On context, OpenRouter reports 32,000 tokens with a 28,800-token completion cap, while TypeSafe’s docs split it into 64k for state plus all questions, 32k for state against the longest single question. Design around 32k per question, 64k once you batch.
There is a cheap way in, and it is more generous than it first looks. TypeSafe’s billing panel labels the grant a monthly credit: $5.00 granted 20 September 2026, expiring 20 October, no rollover. I have only seen one grant, so the recurrence is the panel’s word rather than something I have watched repeat. At $0.042 per million input tokens, $5 buys about 119 million input tokens. My largest batch averaged 725 tokens per judgment, which puts the credit at roughly 164,000 judgments a month before you owe anything. A full day of experimenting here, every tagging batch and both benchmarks, came to $0.69 of it, and my purchase history is still empty. The same panel restates the rate as $0.042/MTok input, free output, which is the third place I have now seen that number.
You can spend that credit through OpenRouter instead of calling TypeSafe directly. Make a key on their site, add it to OpenRouter as a bring-your-own-key provider, and the TypeSafe credit drains before your OpenRouter balance does. OpenRouter takes 5% of list price on BYOK traffic out of your OpenRouter credits, and the free allowance is $25,000 of monthly list-price inference on pay-as-you-go, $200,000 on Enterprise. My whole 840-post run listed at $0.26, so nothing here comes close to paying that fee. None of the credit terms appear on typesafe.ai or in the docs, so read your own billing panel rather than planning around mine.
Running It Against 840 Posts
For each candidate (post, tag) pair I asked one noul question: does this tag fit this post. State was the post title plus body. One question type, nothing clever.
The biggest single batch was 8,402 judgments: 6,092,758 input tokens, $0.2559, 191.3 seconds wall clock, 266ms median latency per call, zero failures. Earlier passes the same day scaled about how you’d expect: 8,587 items in 161.8 seconds for $0.2617, 3,799 items in 71.6 seconds for $0.1156, 1,408 items in 27.2 seconds for $0.0227, 750 items in 26.6 seconds for $0.0149. Zero rows failed outright, all day.
A first pass across a broader question set produced 13,794 judgments and surfaced 321 candidates worth a human look. A second, tighter pass at confidence 0.90 and above narrowed that to 79. I applied 64 tag additions across 63 posts and rejected 15 of the 79 after reading them, roughly 81% right on its own high-confidence shortlist, in line with the benchmark numbers below.
The most useful catch was accidental: it exposed a duplicate tag, shell scripting and scripting existing as two separate tags across 5 posts. My normalizer catches punctuation and case differences. It cannot catch two spellings that differ by a whole word, because that is a judgment call, not a string comparison. I merged them and put a 301 redirect on the retired tag page.
It also suggested 92 tag removals. I declined every single one. Reading a sample of them, I estimate about 18 of the 92 would have been wrong outright. The next section explains why that number changed how I treat this model’s output.
The Asymmetry That Matters More Than the Accuracy Number
Jev reads a tag the way a dictionary reads a word. It has no way to know what that word means specifically on my blog unless my question text spells it out, and I only disambiguated four terms going in: homelab, self-hosting, llm, and ai, all folded into the instructions text. Everything I left undisambiguated produced consistent, predictable category errors, not noise.
routing means network routing on my site. The model kept reading it as request routing in application code, the more common meaning in whatever it trained on. open-source on my blog means “a post about the open-source ecosystem,” not “the software mentioned happens to carry an open-source license,” which is true of nearly everything I write about and would tag half my archive if I let it decide. automation, security, and homelab share the same shape of problem: a site-specific meaning narrower than the dictionary one, and the model defaults to the dictionary.
A noul also takes an optional criteria object with true and false descriptions, and that is the mechanism I should have reached for on all four terms instead of burying the disambiguation in prose: {"true": "homelab means a personal self-hosted server setup", "false": "any other sense"} per question, not one paragraph trying to cover every edge case at once. I’d use it from the start next time.
A wrong tag addition is loud. It shows up on the page, a reader sees a tag that makes no sense, you notice it or they tell you, and you delete it in ten seconds. A wrong tag removal is silent. The page just stops appearing in a tag listing, and nothing anywhere tells you it happened. Same model, same accuracy on both directions, completely different blast radius. That is why I took the 64 additions and threw away all 92 removals without further review: not because the removal judgments were worse, but because a wrong removal has no smoke alarm and a wrong addition does. When you gate a model’s output, gate it on how the failure announces itself, not only on the confidence number attached to it.
Benchmark It Before You Trust It
I did not take the ~80% figure on faith. I scored Jev against free baselines first, on a 247-commit Python repo whose git history is free ground truth: a conventional-commit prefix already labels “what kind of change is this,” and the file a commit touched already labels “which file handles this.” Nobody hand-annotates anything. Any repo with conventional commits gives you a labelled corpus in an afternoon, worth more than a vendor’s own numbers.
Task A, commit-type classification, n=74, balanced across 6 types, state was the diff stat plus the first 5k characters of the patch:
- Jev: 81.1%
- Path-regex baseline: 48.6%
- Per class: feat 94.4%, docs 94.4%, chore 100%, fix 66.7%, test 62.5%, refactor 57.1%
The main confusion is fix read as feat, a boundary humans argue about too. I now treat fix/feat/refactor as one bucket unless splitting them earns its keep.
Task B, file routing, n=65, 12 candidates per question (11 random src/*.py files plus the true one, each shown as its module docstring’s first line), state was the commit subject with the type(scope): prefix stripped so the question did not leak the answer:
- Jev: 78.5%
- Keyword-overlap baseline: 27.7%
- Random guessing: 8.3%
Sixty-five routing questions in 2.5 seconds for $0.002, and the strongest result in either benchmark, because “which file handles this” is otherwise a question you answer by reading files. Both benchmarks combined: 147 calls, 0 errors, 280ms median, 350ms p95, 188,000 input tokens, about $0.008.
I re-ran Task B five hours later on the same seed. The alias still resolved to the same build, and 78.5% became 76.9%, a one-item swing. That is variance. Pin the resolved model string rather than the alias and re-run whenever it moves, or you cannot tell noise from a change under you.
Confidence is the part I’d gate on, pooled across both tasks:
| reported confidence | actually correct |
|---|---|
| >= 0.95 | ~94% |
| 0.85 - 0.95 | ~94% |
| 0.70 - 0.85 | 60-75% |
| 0.50 - 0.70 | ~60% |
| < 0.50 | ~44% |
It is monotonic, so it works as a gate: accept at 0.85 and above, review 0.5 to 0.85, discard below 0.5. At that cutoff, coverage across my runs sat at 40-50% of rows. It filters the work down without removing the need to read any of it.
One trap: a noul has no confidence field, because the probability is the answer. Folding its distance from 0.5 into the same gate flagged 100% of rows for review, a gate that does nothing. Set a noul threshold on what a wrong answer costs you.
Three Question Types, One Call
A generic example so you can run this against your own data instead of my blog tags, triaging an inbound support message with all three primitives at once:
{ "category": { "type": "choice", "instructions": "What kind of issue is this support message describing?", "criteria": { "billing": "A charge, invoice, refund, or subscription question.", "bug": "Something in the product is broken or behaving unexpectedly.", "how_to": "The customer wants to know how to do something the product already supports.", "other": "None of the above applies." } }, "urgency": { "type": "score", "instructions": "How urgent is this message for the support queue?", "criteria": [ "Can wait days, no impact on the customer's work", "Should be handled today, minor impact", "Blocking the customer's work right now" ] }, "needs_human": { "type": "noul", "instructions": "This message needs a human reply rather than a template or automated response" }}One call against one support message answers all three, in parallel, and none of them can see the others’ answers. That’s deliberate. A second call is only warranted when one answer decides what to fetch next.
A minimal loop over a JSONL file, one request per row through a thread pool, with the confidence gate written explicitly as an escalate flag:
import json, sys, os, requestsfrom concurrent.futures import ThreadPoolExecutor
QUESTIONS = json.load(open("questions.json"))KEY = os.environ["TYPESAFE_API_KEY"]MIN_CONF = 0.85
def judge(line): row = json.loads(line) resp = requests.post( "https://api.typesafe.ai/v1/systemone", headers={"Authorization": f"Bearer {KEY}"}, json={"model": "jev-latest", "state": row["text"], "questions": QUESTIONS}, timeout=30, ).json() answers = resp["answers"] confs = [a["confidence"] for a in answers.values() if "confidence" in a] row["answers"] = answers row["escalate"] = bool(confs) and min(confs) < MIN_CONF return row
with ThreadPoolExecutor(max_workers=8) as pool: for row in pool.map(judge, sys.stdin): print(json.dumps(row))And splitting the output into trusted rows versus ones that need a human look:
jq -c 'select(.escalate | not)' out.jsonl > trusted.jsonljq -c 'select(.escalate)' out.jsonl > review.jsonlFree Baseline, Jev, or a General LLM
Three tiers, and here is what I actually measured versus argued.
The free baseline, measured. Path regex got 48.6% on commit types. Keyword overlap got 27.7% on file routing. Write the dumb heuristic first, always, and score it before spending anything. Sometimes the free thing is good enough, and you cannot know until you check.
Jev, measured. 81.1% on commit types, 78.5% on file routing, 266 to 280ms per judgment, $0.042 per million input tokens, and a probability attached to every answer that you can actually threshold in code.
A general LLM, local or hosted, not measured by me here. No accuracy, latency, or cost number for this tier on these tasks, so I’m arguing it structurally instead of making one up. A general LLM hands back prose you parse or JSON you validate, not a typed answer, and gives no calibrated probability to threshold on: “I’m fairly confident” is not a number you can put in an if statement. You pay for output tokens on top of input, where Jev’s output is close to free because there is barely any of it. Throughput is the real gap for a batch job like mine: 8,402 judgments landing in 191 seconds is a different order of wall-clock time than a single local GPU serving one model sequentially through the same volume.
To be fair to the other side: a general LLM can explain its reasoning, handle a task you cannot reduce to a fixed question set ahead of time, and needs no separate vendor account. If you need the “why” behind an answer, Jev is the wrong tool for that job, a real limit, not a footnote to wave away.
Two Open-Weight Clones Landed The Same Week
There is a fourth tier now, and it arrived while I was writing this. Von (wfzyx/von-1.0) is a 395M-parameter ModernBERT checkpoint, about 1.5GB on disk, Apache 2.0, claiming protocol parity with /v1/systemone so an existing client can point at it unchanged. Laya (convaiinnovations/laya) is a 421M ModernBERT-large checkpoint plus a 322M multilingual sibling, also Apache 2.0, shipping the same three primitives under the same names, installed with pip install laya.
The pitch is identical for both: roughly 18ms local inference for Von and 32.8ms for Laya, against Jev’s 266ms median over the network. No per-token bill, no vendor account, and the text never leaves your machine. For a batch like mine, 26 cents was never the deciding factor anyway; the privacy and the latency are the parts you would actually be buying.
Read their benchmark tables the way you would read any vendor’s. Both publish comparisons against Jev, and neither measured Jev themselves. Laya states that outright on its own model card, which earns it some credit. Von does not, and Von’s own numbers argue against its headline: the card claims 91.23% accuracy “surpassing published commercial alternatives” while the table directly below it shows Jev at 96.6% macro against Von’s 71.5% across a 49-task suite. Laya’s table puts Jev ahead on Banking77 by 0.870 to 0.425, a gap wide enough to decide any high-cardinality job, and Laya’s own “where Jev leads” section names that case explicitly: label spaces past about 20 options, where Jev goes up to 255. The two also disagree about each other. Von’s table ranks Laya last among the open models at 58.3% macro. Adoption tells you little so far either: Von sits at 1,223 downloads against 25 likes, Laya at roughly 1,200 likes against zero downloads, a ratio worth one raised eyebrow.
One independent data point turned up after I drafted this. A Reddit user going by look generated a synthetic suite and ran four models through it: Jev 97.4%, Von 92.3%, GLiNER2 79.5%, Laya 61.5%. Work out the denominator before reading anything into that. All four figures are exact thirty-ninths, so the suite is 39 items or a small multiple of 39. One item swings a score by 2.6 points, which puts Jev and Von two items apart and makes that gap useless for choosing between them. The Laya gap is 14 items wide and points the same direction as Laya’s own Banking77 row. A synthetic suite someone assembled in an afternoon is still the only number on this page that no vendor wrote, which is worth something and worth very little at the same time.
One line in Laya’s card is worth more than every headline score in both, and it cuts straight against the gate I recommended two sections ago. Its English checkpoint scores 0.000 accuracy on Khmer while reporting 0.952 confidence. Confidently wrong, across an entire language, with the confidence number insisting everything is fine. A calibration curve holds only inside the distribution it was measured on. Score these on your own labelled data before you wire a threshold around them, exactly the way I scored Jev against a regex.
Measure the free baseline on every new task, gate the paid model’s output on confidence, and split that gate by how a wrong answer surfaces rather than by the confidence number alone. A loud mistake you can auto-apply and fix later. A silent one you cannot.
Common Questions
How much does Jev cost for a batch of thousands of judgments?
My largest run, 8,402 judgments over 840 blog posts, cost $0.2559 and finished in 191.3 seconds. Cost scales with input tokens only, at $0.042 per million as of September 2026, since output tokens are free. A $5 monthly credit covers roughly 119 million input tokens, so a job this size costs nothing in practice. There is no pricing page on typesafe.ai; the rate comes from the docs’ Models page.
Can you self-host Jev instead of calling the API?
No. Jev is a hosted API only, reached at api.typesafe.ai or through OpenRouter’s alpha decisions route, and there is no self-hosted or downloadable version. It also accepts text input only, no images, audio, or video, so anything else needs pre-processing before it reaches the endpoint.
Does Jev replace a general-purpose LLM for classification tasks?
No. Jev returns typed answers and probabilities, not explanations, and works only within a fixed question set you define ahead of time. Use it for high-volume, well-defined judgments like tagging or routing. Use a general LLM when you need reasoning, an explanation, or a task you cannot reduce to fixed questions.
Are there open-source alternatives to Jev?
Yes. Von (395M) and Laya (421M) are Apache 2.0 typed-decision models on ModernBERT backbones, both released in September 2026, both running locally under 35ms. Von claims parity with the same API protocol. Benchmark either one on your own labelled data first, because their published comparisons against Jev are self-reported.
How big a state can I send to Jev in one call?
Up to 64,000 tokens covering the state plus all questions combined, or 32,000 tokens for the state plus your single longest question. State and questions share that budget, so a long document leaves less room for elaborate question instructions in the same call.
