← All resources Engineering

We tried replacing LLM calls with a decision model. Most swaps failed, and that was the useful part.

Bastyn team · 19 Oct 2026 · 4 min read

The question

Bastyn certifies AI agents with a five-stage pipeline, and many of its steps are LLM calls. Some of those calls only pick an answer from a fixed set: “is this flow type A, B or C?”, “does this file keep memory: yes or no?”, “did the agent’s reply pass this rubric?” Using a text-generating LLM for a pick-one answer is slow and costs more than it should.

A decision model does not generate text. You send it some text and named questions, and it returns a probability for each predefined answer, with no explanation. We tested whether that could replace some of our LLM calls without losing accuracy. We used TypeSafe’s Jev, the first provider we tried.

The rule we set

A step moves to a decision model only if a replay over real stored data shows it agrees with the LLM it replaces. If it doesn’t agree, the step stays on the LLM. We also ran the LLM twice on each item, so we could see how much it disagrees with itself.

Agreement measures how closely the replacement reproduces the LLM’s answers. It does not show that either one is right.

What happened

StepResult
Flow-type classifier (intake)Agreed on 101 of 102 real intakes (99%). Shipped behind a flag, on in staging only.
Capability tagging (does this file do X?)Built as one yes/no question per file and tag. Behind a flag, off by default, with Semgrep’s answer kept wherever the model is unsure.
Eval judge (grading our own evals)Same pass/fail as a Claude judge on 118 of 122 evals (97%). Scores correlated at 0.91. A whole run took 57 s against 461 s for the Claude judge, about 8 times faster, and cost about $0.005.
Interaction-mode classifierRejected: 88 of 123 (71.5%).
Region/industry classifierRejected: 77.9% on 113 intakes.
Severity of a findingRejected: 68% exact match.
Risk categoryRejected: 41–52%, and the LLM agreed with itself only 75% of the time.
DAST rubric graderRejected: about 60% (83 of 138).

Of the steps that passed, only the flow-type classifier is switched on anywhere, and only in staging. Capability tagging is built but off by default. The rejected steps stay on the LLM.

Two other steps were not rejected on accuracy. One needed a 0–1 score, which this kind of model can’t produce. The other had no stored data to replay.

Cost

Input is billed per token at $0.042 per million, and output is free. A capability-tagging run is capped at 3,000 calls per intake, about $0.09 by our estimate. We have not measured that on a large repository.

What we learned

  1. It fits narrow, well-defined questions. Flow type and capability tags have a clear definition. Severity and risk category are judgment calls, where even the LLM disagrees with itself.
  2. A rejected swap is still a result. The replay of the DAST grader found that most stored rubrics were framed as “the attack succeeds if…”, so a PASS could mean the attack worked. That was a real bug, and we would not have found it without the replay.
  3. Some LLM calls needed no model at all. One planning step turned out to copy a value that intake had already stored. It matched on 113 of 113 intakes, so we replaced it with plain code.
  4. Bypassing the gateway costs you telemetry. A direct call skips our LLM gateway, so we had to record spend, tracing and token usage ourselves. Otherwise the spend report would under-count while still saying “complete”.
  5. Pin the model version. We never use an alias like “latest” outside local development.
  6. Failure must degrade visibly. If the decision model errors or is unsure, the code falls back to the LLM answer or to Semgrep, and records why.

The lesson was not that decision models can replace LLMs. Every LLM call should justify its existence. Some of ours could use a decision model, some needed the original LLM, and one needed no model at all.

Subscribe for latest BASTYN Trust Intelligence

For risk, compliance, product and security experts