Judge an AI Answer Before You Show It

Why a typed verdict first

An LLM judge writes a rationale and a score. Two runs can disagree, and the score has no stated uncertainty, so a pipeline has to accept whatever it gets. A decision model answers the same narrow questions every time and returns a probability for each option.

That makes the verdict comparable across items and lets your code choose a threshold. It does not make the judgement correct — thresholds still come from your own labelled samples.

Question shape

Three questions cover most answer judging:

• supported (Noul): is every factual claim in the answer supported by the reference text?
• relevance (Score): how much of the user's question does the answer address?
• verdict (Choice): ship, revise, or send to a person.

Keep the instruction narrow and local: "using only the reference, assess each claim" is a different question from "is this a good answer", and it is the one you can measure.

Illustrative code, not tested. Thresholds are placeholders.

tsimport { booleanQuestion, choice, score } from '@system-one-ai/core';
import type { EvaluationClient } from '@system-one-ai/core';

export const judgeQuestions = {
  supported: booleanQuestion(
    'Is every factual claim in candidate_answer supported by reference?',
  ),
  coverage: score('How much of user_question does candidate_answer address?', {
    none: 'Does not address the requested information',
    partial: 'Addresses some of the request but omits a required part',
    full: 'Addresses the request completely',
  }),
  verdict: choice('What should happen to this answer?', {
    ship: 'Supported and complete',
    revise: 'Fixable gaps',
    escalate: 'Needs a person or a stronger model',
  }),
};

export async function judgeAnswer(
  client: EvaluationClient,
  input: { question: string; reference: string; candidate: string },
) {
  const result = await client.evaluate({
    state: input,
    questions: judgeQuestions,
  });
  return {
    supported: result.answers.supported.probability,
    coverage: result.answers.coverage.score,
    verdict: result.answers.verdict.choice,
    confidence: result.answers.verdict.confidence,
  };
}

Order the checks by cost, not by importance

Run the lower-cost decision checks first and escalate only what survives:

1. Support check on every answer. It is one yes/no question, and it catches the failure that matters most in a retrieval pipeline.
2. Coverage score on answers that pass. A fully supported answer can still miss half the question.
3. Stronger judge or a person on anything that fails either check, or where confidence is low.

See an LLM-as-a-judge alternative for the research caveats on this split.

What the verdict does not tell you

• It is not a correctness proof. A supported answer can still be wrong if the reference is wrong.
• It does not grade style. If tone or format matters, ask a separate question about it.
• It does not explain itself. Pair it with a generative model when a human needs a written reason.

Cost

Every 4KB is one block, and anything smaller still counts as one block. Cost = blocks × questions × the model rate. This page's request asks three questions on Jev (1 credit per question per block).

New accounts start with 100 credits, and credits never expire. See pricing.

Run the judging sample (sign-in required) to see the same question shape on sample data.