---
title: "Judge an AI Answer Before You Show It"
description: "Score an answer against a rubric and check whether it is supported by the source before it reaches a user. Typed verdicts, probabilities you can threshold, and an escalation path for low confidence."
canonical: "https://system-one.dev/en/use-cases/answer-judging"
language: "en"
---

# Judge an AI Answer Before You Show It

**Key points**

1. A judge needs two things a generated score does not give you: a verdict that is comparable across items, and a probability you can threshold.
2. Ask narrow questions — support, relevance, completeness — and score each against a rubric you define, rather than asking for one overall "quality" number.
3. Send low-confidence verdicts to a stronger model or a person; the decision model only carries the cases it is sure about.

## Why a typed verdict first

An LLM judge writes a rationale and a score. Two runs can disagree, and the score has no stated uncertainty, so a pipeline has to accept whatever it gets. A decision model answers the same narrow questions every time and returns a probability for each option.

That makes the verdict comparable across items and lets your code choose a threshold. It does not make the judgement correct — thresholds still come from your own labelled samples.

## Question shape

Three questions cover most answer judging:

- **supported** (Noul): is every factual claim in the answer supported by the reference text?
- **relevance** (Score): how much of the user's question does the answer address?
- **verdict** (Choice): ship, revise, or send to a person.

Keep the instruction narrow and local: "using only the reference, assess each claim" is a different question from "is this a good answer", and it is the one you can measure.

> Illustrative code, not tested. Thresholds are placeholders.

```ts
import { booleanQuestion, choice, score } from '@system-one-ai/core';
import type { EvaluationClient } from '@system-one-ai/core';

export const judgeQuestions = {
  supported: booleanQuestion(
    'Is every factual claim in candidate_answer supported by reference?',
  ),
  coverage: score('How much of user_question does candidate_answer address?', {
    none: 'Does not address the requested information',
    partial: 'Addresses some of the request but omits a required part',
    full: 'Addresses the request completely',
  }),
  verdict: choice('What should happen to this answer?', {
    ship: 'Supported and complete',
    revise: 'Fixable gaps',
    escalate: 'Needs a person or a stronger model',
  }),
};

export async function judgeAnswer(
  client: EvaluationClient,
  input: { question: string; reference: string; candidate: string },
) {
  const result = await client.evaluate({
    state: input,
    questions: judgeQuestions,
  });
  return {
    supported: result.answers.supported.probability,
    coverage: result.answers.coverage.score,
    verdict: result.answers.verdict.choice,
    confidence: result.answers.verdict.confidence,
  };
}
```

## Order the checks by cost, not by importance

Run the lower-cost decision checks first and escalate only what survives:

1. **Support check** on every answer. It is one yes/no question, and it catches the failure that matters most in a retrieval pipeline.
2. **Coverage score** on answers that pass. A fully supported answer can still miss half the question.
3. **Stronger judge or a person** on anything that fails either check, or where confidence is low.

See [an LLM-as-a-judge alternative](https://system-one.dev/en/blog/llm-as-a-judge-alternative) for the research caveats on this split.

## What the verdict does not tell you

- **It is not a correctness proof.** A supported answer can still be wrong if the reference is wrong.
- **It does not grade style.** If tone or format matters, ask a separate question about it.
- **It does not explain itself.** Pair it with a generative model when a human needs a written reason.

## Cost

Every 4KB is one block, and anything smaller still counts as one block. Cost = blocks × questions × the model rate. This page's request asks three questions on Jev (1 credit per question per block).

New accounts start with 100 credits, and credits never expire. See [pricing](https://system-one.dev/en/pricing).

[Run the judging sample (sign-in required)](https://system-one.dev/en/demo/llm-judge) to see the same question shape on sample data.


## Frequently asked questions

### Does a decision model replace an LLM judge?

For narrow, checkable questions it can carry most of the load. Derivations such as math, code and logic still need a stronger model or a person.

### How do I pick the threshold?

Label a sample yourself, pick the threshold that matches the error cost you accept, then confirm it on held-out items. Thresholds tuned on one model do not transfer to another.

### Can I use this with a retrieval pipeline?

Yes: pass the retrieved passages as reference and the generated answer as candidate, then ask the support question before the answer is shown.

### Where does the verdict live?

In your code. The model returns probabilities and a chosen option; the action — ship, revise, escalate — is yours to implement.