An LLM-as-a-Judge Alternative: Use a Decision Model as the First Pass

Both papers below study Jev, the model from TypeSafe, and their results don't automatically transfer to other decision models. system-one.dev offers Jev, but we haven't reproduced the papers' experiments on our hosted API. All figures are the authors', under their experimental conditions.

New to Choice / Score / Noul? Read What is a decision model? first.

Three problems with LLM judges: latency, cost, consistency

LLM-as-a-judge is now the default way to evaluate open-ended output, but the complaints are consistent:

• Latency: a reasoning model spends extra compute on every verdict, and that adds up across large eval runs.
• Cost: the same paper estimates about $12 per 1,000 verdicts for GPT-6 at low reasoning effort (authors' conditions), and you pay again every time the model version or dataset changes.
• Consistency: the same answer can get different scores on different runs.

Consistent doesn't mean correct, though. The UPenn paper found that on ordinal rubrics, judges agreed with each other more than with human labels, and most scored lower than humans did (authors' conditions). Agreement only shows that judges decide alike, not that they decide right.

What the research says

Finding 1: when the verdict is readable from the text, a cascade works

JEV-as-a-Judge: Accept When Confident, Escalate When Unsure (arXiv 2609.26550, Carnegie Mellon) compares Jev with 16 generative judges and reward models and checks its conclusions with blind human review. Its headline findings (all under the authors' conditions):

• Where the verdict can be read directly from the text, Jev is within 3 percentage points of GPT-6, at about 0.36% of GPT-6's cost.
• Where the verdict needs derivation (math, code, logic), Jev falls clearly behind, by 7 to 28 points according to the paper.
• Freeze a threshold, accept confident verdicts and escalate the rest to GPT-6: on 1,610 held-out pairs, the cascade's accuracy was 0.9 points higher than GPT-6 alone, at 41% of GPT-6's cost. On two pre-registered new tasks tested live, the cascade exactly matched GPT-6's accuracy.
• Confidence routing fails in two cases: "style-misleading" pairs, where the more polished answer is the wrong one, and reference-free style evaluation, where every judge tested was near chance yet confident.

Finding 2: they fail in the same places

Jev vs. LLMs as Rubric Judges: Cheaper, Faster, and Wrong in the Same Places (arXiv 2609.29769, UPenn) compares Jev with three flash-tier LLM judges across 7 benchmarks and 9 human-labelled rubric panels (authors' conditions):

• The three LLM judges cost 16 to 325 times as much as Jev.
• Across settings, at most 8 of 27 pairwise accuracy comparisons differed significantly. Where Jev led, it was mostly on binary checklist items; where it trailed, it was only on ordinal scales.
• On Jev's most confident errors, about 96% of LLM verdicts repeated the same wrong answer. If errors were independent, that rate would be about half.
• So a Jev-first cascade mainly saves money rather than improving accuracy: even with a hindsight-optimal threshold, no cascade beat the best single judge by more than 2.7 points.

Reading them together

The papers don't conflict; they answer different questions. The first asks whether a decision-model first pass can keep a strong model's accuracy while cutting cost; on the tasks it tested, yes. The second asks whether a cascade can beat the best single judge; mostly no, because the second stage repeats the first stage's mistakes.

The practical takeaway: use a decision model as a cost-saving first pass, not as a way to make evaluation more accurate. A cascade only helps if the second stage makes different mistakes.

Microsoft's Microsoft-Decision-1 launch blog also lists "scoring AI responses and agent actions against rubrics" as a use case (vendor statement).

Decision-model judge vs LLM judge: how the interfaces differ

| | Decision model (Score / Noul) | LLM judge | | --- | --- | --- | | Output | Predefined levels or yes/no, with a probability for each level | A generated score plus written rationale | | Explains why | No, probabilities only | Yes, can write a rationale | | Uncertainty | Per-level probabilities you can threshold for escalation directly | You design it yourself (repeated scoring, self-reported scores, etc.) | | Rubric | Each level is one written description (Score: up to 10 levels) | Written into the prompt | | Billing | Every 4KB is one block, and under 4KB still counts as one; cost = blocks × questions × model price per block | The LLM's own pricing | | Best for | Evaluations whose verdict can be read from the text, as a first pass | Evaluations that require derivation, process checks or written feedback |

Compare the two kinds of judge on the same human-labelled data. For where each evaluation type landed in the research, see the boundary table below.

Which evaluations to give Score / Noul, and which not to

The boundary table at the end summarizes both papers. Every row is an experimental result on specific datasets: a starting point, not a deployment guarantee. In short:

• Good first-pass candidates: ordinary preference comparisons, whether an answer is supported by given material, binary checklist items.
• Validate locally first: final-answer checks, multiple choice, reference-based summaries and multi-level ordinal rubrics (judges scored lower than humans in the second paper).
• Escalate to a strong model or a person: math, code, logic and expert-knowledge items that need derivation, plus style-misleading pairs.
• Don't use any automatic judge: reference-free style or prose quality, where every judge in the first paper was near chance.

Turning a rubric into Score levels

Score works like this: you give ordered level descriptions from low to high, the model returns a probability for each level, and the score is the probability-weighted average, so it can fall between levels. Tips:

• Number of levels: System One allows up to 10 Score levels. The dev team's example notes that the hosted API supports 2 to 10 ordered levels, and scores are only comparable under the same rubric, model and instructions.
• Describe situations, not degrees: The Jev docs recommend writing each level as a checkable situation ("feature broken but a workaround exists") instead of a degree ("moderately severe"). Each level is evaluated on its own, without level numbers or neighbouring levels, so "worse than the previous level" means nothing.
• One dimension per question: mixing dimensions lowers confidence. Split them into separate Scores and weight them in code.
• The same score can mean different distributions: 1.0 could be all probability on level 1, or half on level 0 and half on level 2. Look at probabilities and confidence together.
• For pairwise preference, ask both orders: the first paper scores both candidate orders and averages them.
• Use scores for ranking and thresholds, not as absolute grades: Microsoft-Decision-1's docs recommend using Score for relative ranking and thresholds rather than as an absolute, calibrated grade, and validating on your labelled samples.

Thresholds: pick on local samples, confirm on held-out data

The first paper ends with a procedure you can follow (authors' recommendation):

1. Label locally: sample about 100 items from your own evaluation task, label the correct answer by hand, and run both the decision model and your strong judge.
2. Pick thresholds by lower confidence bound: simulate the cascade over candidate thresholds and take the loosest one whose accuracy loss stays within two points, using the lower confidence bound of the loss rather than the point estimate. The paper reports this cuts the risk of an actual loss above two points from about 45% to about 5% (authors' simulation).
3. One threshold per task type: optimal thresholds vary widely across tasks.
4. Confirm on held-out data that wasn't used to pick the threshold.

Given the "wrong in the same places" finding, add one more check (our suggestion, not a step from the paper): on your threshold-selection set, measure how often the second stage repeats the first stage's errors, and compare that with the roughly 50% baseline you'd expect if errors were independent. If the repeat rate is far above the baseline, the cascade will save money but won't improve accuracy.

Revalidate thresholds whenever you change the model or model version; confidence is computed differently across models (see Jev alternatives).

TypeScript: Score pre-filter, then your own judge

Try it online (sign-in required): Which AI answer is better?

From the dev team's example (examples/decision-articles/03-score-eval-prefilter.ts). It scores each candidate answer first; only candidates at or above the threshold go to an application-supplied judge callback (your LLM judge, for example).

ts/**
 * Article 3 — Use Score as a coarse eval pre-filter instead of an LLM-as-judge.
 *
 * System One exposes Score (ordered rubric → weighted number). It does not
 * expose an LLM-as-judge, eval harness, or dataset API. Those stay in your
 * application. This file scores each candidate first; only items at or above
 * your threshold are passed to an application-supplied judge callback.
 *
 * Hosted Score limits: 2–10 ordered levels. Compare scores only under the
 * same rubric, model, and instructions.
 */
import { createSystemOne, score } from '@system-one-ai/core';
import type { EvaluationClient } from '@system-one-ai/core';
import { systemOneAdapter } from '@system-one-ai/adapter-system-one';
import { createFetchTransport } from '@system-one-ai/transport-fetch';

export const helpfulness = score('How clear and actionable is this answer for the question?', [
  'Incorrect or unhelpful',
  'Slightly relevant',
  'Useful but incomplete',
  'Clear and actionable',
]);

/** Inclusive Score threshold on the 0..3 rubric above. Tune on your own labeled set. */
export const PREFILTER_MIN_SCORE = 2;

export interface EvalCandidate {
  id: string;
  question: string;
  answer: string;
}

export interface JudgeInput extends EvalCandidate {
  prefilterScore: number;
}

export function createHostedClient() {
  const apiKey = process.env.SYSTEM_ONE_API_KEY;
  const baseURL = process.env.SYSTEM_ONE_BASE_URL;
  const model = process.env.SYSTEM_ONE_MODEL;
  if (!apiKey || !baseURL || !model) {
    throw new Error('Set SYSTEM_ONE_API_KEY, SYSTEM_ONE_BASE_URL (…/v1), and SYSTEM_ONE_MODEL from GET /v1/models.');
  }
  return createSystemOne({
    apiKey, baseURL, model, adapter: systemOneAdapter,
    transport: createFetchTransport(), timeoutMs: 30_000, maxRetries: 0,
  });
}

export async function scoreCandidate(client: EvaluationClient, candidate: EvalCandidate) {
  const result = await client.evaluate({
    state: { question: candidate.question, answer: candidate.answer },
    questions: { helpfulness },
  });
  return result.answers.helpfulness.score;
}

/**
 * @param judge Application-owned expensive check. Not a System One method.
 *              Omit it to only run the Score pre-filter.
 */
export async function prefilterThenJudge(
  client: EvaluationClient,
  candidates: readonly EvalCandidate[],
  judge?: (input: JudgeInput) => Promise<unknown>,
) {
  const rows = [];
  for (const candidate of candidates) {
    const prefilterScore = await scoreCandidate(client, candidate);
    if (prefilterScore < PREFILTER_MIN_SCORE) {
      rows.push({ id: candidate.id, prefilterScore, status: 'skipped_low_score' as const });
      continue;
    }
    const judged = judge ? await judge({ ...candidate, prefilterScore }) : undefined;
    rows.push({ id: candidate.id, prefilterScore, status: 'sent_to_application_judge' as const, judged });
  }
  return rows;
}

async function main() {
  const candidates: EvalCandidate[] = [
    {
      id: 'reset-password',
      question: 'How can I reset my password?',
      answer: 'Select Forgot password on the sign-in page and follow the email link.',
    },
    {
      id: 'unrelated',
      question: 'How can I reset my password?',
      answer: 'Our office is open on weekdays.',
    },
  ];
  // No System One judge endpoint exists. Live runs stop after Score unless you pass your own judge.
  const rows = await prefilterThenJudge(createHostedClient(), candidates);
  console.log(JSON.stringify(rows, null, 2));
}

if (process.argv[1] && import.meta.url.endsWith(process.argv[1].split(/[\\/]/).pop() ?? '')) {
  main().catch((error: unknown) => {
    console.error(error instanceof Error ? error.message : error);
    process.exitCode = 1;
  });
}

Notes (from the example's comments):

• System One doesn't provide an LLM judge, eval harness or dataset API. judge is your own callback. Without it, the example only runs the Score pre-filter.
• The application owns the threshold: PREFILTER_MIN_SCORE = 2 (inclusive, on the 0–3 rubric). Tune it on your own labelled set.
• Hosted model: take SYSTEM_ONE_MODEL from GET /v1/models. A full model ID selects a model but can't pin a version.

This example is not the papers' cascade. It decides by score ("drop low scores, review high scores"). The papers' cascade decides by confidence ("accept confident, escalate uncertain"). That's the second example.

TypeScript: accept when confident, escalate to an LLM when not

From examples/decision-articles/06-confident-or-escalate.ts. A Choice decides how to handle a request. If confidence exists and meets the threshold, the decision-model result is used; if it's low or missing, the request, the proposed answer and the confidence go to your own LLM.

ts/**
 * Article 3 — Confident: use the decision-model result.
 * Uncertain: escalate to a large LLM in the application.
 *
 * Choice/Score answers may include `confidence`. That value is an optional
 * provider statistic, not necessarily P(selected option), and not a product
 * default threshold. Missing confidence stays unknown — this file treats
 * unknown as uncertain.
 *
 * System One has no chat/completions URL. Escalation is an application
 * callback you already own. Packages: @system-one-ai/core,
 * adapter-system-one, transport-fetch. Endpoint: POST /v1/systemone.
 */
import { createSystemOne, choice } from '@system-one-ai/core';
import type { ChoiceAnswer, EvaluationClient } from '@system-one-ai/core';
import { systemOneAdapter } from '@system-one-ai/adapter-system-one';
import { createFetchTransport } from '@system-one-ai/transport-fetch';

export const routeQuestions = {
  handling: choice('Choose how this request should be handled.', {
    billing: 'Payments, invoices, and refunds',
    support: 'Technical issues with the product',
    review: 'Needs a person; the request is incomplete or consequential',
  }),
};

/** Application-owned. Tune on your own labeled cases; not a hosted default. */
export const MIN_DIRECT_CONFIDENCE = 0.6;

export function isConfident(answer: ChoiceAnswer<string>, minimum = MIN_DIRECT_CONFIDENCE) {
  if (answer.confidence === undefined) return false;
  return answer.confidence >= minimum;
}

export interface EscalationInput {
  message: string;
  proposed: string;
  confidence: number | undefined;
}

export function createHostedClient() {
  const apiKey = process.env.SYSTEM_ONE_API_KEY;
  const baseURL = process.env.SYSTEM_ONE_BASE_URL;
  const model = process.env.SYSTEM_ONE_MODEL;
  if (!apiKey || !baseURL || !model) {
    throw new Error('Set SYSTEM_ONE_API_KEY, SYSTEM_ONE_BASE_URL (…/v1), and SYSTEM_ONE_MODEL from GET /v1/models.');
  }
  return createSystemOne({
    apiKey, baseURL, model, adapter: systemOneAdapter,
    transport: createFetchTransport(), timeoutMs: 30_000, maxRetries: 0,
  });
}

/**
 * @param escalateToApplicationLlm Your existing large-LLM path.
 *        Not a System One method and not a hosted completion endpoint.
 */
export async function routeOrEscalate(
  client: EvaluationClient,
  message: string,
  escalateToApplicationLlm: (input: EscalationInput) => Promise<string>,
) {
  const result = await client.evaluate({
    state: { message },
    questions: routeQuestions,
  });
  const answer = result.answers.handling;
  if (isConfident(answer)) {
    return {
      action: 'use_decision' as const,
      handling: answer.choice,
      confidence: answer.confidence,
    };
  }
  const escalated = await escalateToApplicationLlm({
    message,
    proposed: answer.choice,
    confidence: answer.confidence,
  });
  return {
    action: 'escalate_llm' as const,
    handling: answer.choice,
    confidence: answer.confidence,
    reason: answer.confidence === undefined ? 'confidence_unknown' as const : 'confidence_below_threshold' as const,
    escalated,
  };
}

async function main() {
  const message = process.env.EXAMPLE_MESSAGE ?? 'The invoice PDF will not download.';
  const decision = await routeOrEscalate(createHostedClient(), message, async () => {
    throw new Error('No System One LLM-completion endpoint. Wire your existing large-LLM call here.');
  });
  console.log(JSON.stringify(decision, null, 2));
}

if (process.argv[1] && import.meta.url.endsWith(process.argv[1].split(/[\\/]/).pop() ?? '')) {
  main().catch((error: unknown) => {
    console.error(error instanceof Error ? error.message : error);
    process.exitCode = 1;
  });
}

Notes (from the example's code and comments):

• The scenario is request triage, not answer grading. It chooses among billing / support / review to show the "confidence gate + escalate" structure. For evaluation, swap in your own verdict question (for example a preference Choice) and re-pick the threshold with the method above.
• Confidence is optional. In the SDK's ChoiceAnswer, confidence may be absent. It's a provider statistic, not necessarily P(selected option), and not a product default. When it's missing, the example treats it as uncertain and escalates (confidence_unknown). System One's primitives docs also say Choice/Score statistics are optional in normalized answers.
• The threshold is an example. MIN_DIRECT_CONFIDENCE = 0.6: "tune on your own labeled cases; not a hosted default."
• Escalation is your callback. System One has no chat/completions endpoint.
• It doesn't do the papers' both-orders averaging or lower-bound threshold selection. Implement those in your offline evaluation.

All figures are from the papers' experiments on Jev and don't represent other decision models. Per Table 2, items run on GPT-6 are compared with GPT-6, and natural-prose items with GPT-5.4. We haven't reproduced these experiments on our hosted API.

Related reading

• What is a decision model?
• Jev alternatives in 2026
• LLM model routing
• AI agent stop conditions