AI Agent Stop Conditions: Decide \"Done\" with Yes/No Probabilities

Background on Noul (yes/no) and the other primitives: What is a decision model?

Try it online (sign-in required): Code diff risk

Three kinds of stop condition

| Type | Examples | Answers | Can't answer | | --- | --- | --- | --- | | Hard limit | Max steps, runtime, token or cost budget | "Should we force a stop?" | Whether the task is actually done | | Deterministic evidence | Tests pass, exit code 0, file created, clean git status | "Does verifiable output exist?" | Requirements the tests don't cover | | Semantic judgment | "Was every requested item handled?" | "Was the goal met?" | On its own, can't prove anything happened |

Existing frameworks already cover the first two:

• Vercel AI SDK: ToolLoopAgent stops after 20 steps by default (stepCountIs(20)), which the docs say prevents runaway loops from causing excessive API calls and cost. You can combine several stopWhen conditions (any one stops the loop) or write a custom one, such as stopping when cumulative token cost exceeds a budget.
• Claude Code /goal: the docs recommend /goal for work with a verifiable end state. A community tutorial's example is "npm test exits 0 and git status is clean, or stop after 20 turns", which puts deterministic evidence and a turn limit in one condition.

Why the agent shouldn't declare itself done

Claude Code's design shows one approach. Per its docs, without /goal Claude stops when it judges the work complete. /goal adds an independent evaluator that checks your condition after every turn, so a fresh model, not the one doing the work, decides whether it's done. The evaluator reads the condition and the conversation so far, uses the configured small fast model by default (Haiku on the Claude API), and returns yes or no with one line of reasoning. On "no", Claude keeps going and uses that reason as guidance.

Same principle here: something other than the worker judges "done". The difference is that we split one big condition into several yes/no questions, get a probability for each, and let code apply thresholds.

Split "done?" into yes/no questions

Noul is a yes/no question: give a clear proposition and get P(true). The System One SDK defines it with booleanQuestion(), which HTTP adapters send as native noul; the SDK answer field is probability.

Writing acceptance criteria (from the System One primitives docs and the Jev Noul docs):

• One proposition per question. When several conditions can be true at once, define one Noul per condition rather than forcing them into a Choice. The probabilities refer to different propositions and don't need to sum to 1.
• A low probability isn't low confidence. 0.2 means the model thinks the condition is unlikely to hold, not that it's unsure. Noul has no separate confidence; values near 0.5 mean the model can't decide.
• A probability isn't a fact. P(true)=0.8 isn't a confirmed fact or an automatic boolean. "The output says tests passed" and "the tests passed" are different; read the test report for the second.
• Use data you already have. The System One docs say that if authoritative application data already settles a fact, use it rather than asking a model. Test results, exit codes and file existence belong to code.
• Questions don't see each other. Questions in one call evaluate the same state independently; one answer doesn't feed another. The docs note this independence is about the question interface; it doesn't guarantee that the model's errors on different questions are independent.

A starter checklist (example wording; adapt to your task):

| Acceptance criterion | Check | | --- | --- | | Every requirement the user listed is handled | Noul: "Does the output cover every acceptance criterion listed in the task?" | | No blocking question left for the user | Noul: "Does the output leave no question the user must answer?" | | All tests pass, and tests actually ran | Deterministic: read the test runner's result | | Runtime, steps and cost within limits | Hard limit: counters in the framework or your code |

The first two rows are the two Noul questions in the example below.

Threshold → action table

With probabilities and evidence in hand, your code decides. The Jev Noul docs give the principle: raise the threshold when a false "yes" is costly, lower it when missing a real "yes" is costly, and send the middle band to a person. Each row below maps to a branch in 05-completion-actions.ts (order explained in the next section); the last row applies only to the Claude Code Stop hook. 0.7, 0.4, 20 steps and 3 rounds are example values; there is no universal threshold.

| Situation | Action | | --- | --- | | Tests fail and the model also says not done | Run another round with the failure output | | Tests fail but the model says done | Evidence contradicts the model: hand to a person | | No tests ran | Not done: run another round | | Tests pass, both criteria ≥ 0.7 | Stop and report | | Tests pass, lowest probability between 0.4 and 0.7 | Run another round | | Tests pass, lowest probability below 0.4 | Hand to a person | | Hit the 20-step limit, or 3 rounds with no progress | Force stop, report status, hand to a person | | (Claude Code Stop hook only) already blocked once (stop_hook_active is true) and tests still fail or none ran | Stop blocking, hand to a person |

"3 rounds with no progress" is stall detection: the example builds a progress fingerprint from test results plus agent output, and three identical fingerprints in a row count as a stall. Application code implements this. Microsoft's Microsoft-Decision-1 launch blog also lists agent control as a use case: evaluating an agent's proposed next step and deciding whether to continue, stop, retry, or hand off to a model, tool or person (vendor statement).

Pick thresholds on your own human-labelled cases. For how to select on labelled samples and confirm on held-out data, see An LLM-as-a-judge alternative.

Implementing the action table

The dev team's examples/decision-articles/05-completion-actions.ts imports checkCompletion() and the thresholds from the main example (05-agent-completion-check.ts, below) and adds loop counters. decideCompletionAction() checks in this order and returns on the first match:

1. Step limit MAX_AGENT_STEPS = 20 reached: hand to a person.
2. STALL_ROUNDS = 3 identical progress fingerprints (tests + output): hand to a person.
3. No tests ran: run another round.
4. Tests failed and the lower of the two probabilities is ≥ 0.7 (model says done): evidence contradicts the model, hand to a person.
5. Tests failed and the model agrees it's not done: run another round.
6. Tests passed and the lower probability is ≥ 0.7: stop and report.
7. Tests passed and the lower probability is between MID_BAND_LOW = 0.4 and 0.7: run another round.
8. Otherwise (lower probability under 0.4): hand to a person.

Rule 3 comes before rules 4 and 5: if the result is marked failed but zero tests ran, it counts as "no tests ran" (another round), not as a failure. All numbers are examples; tune them on your own cases.

ts/**
 * Article 5 — Action table: retry, human handoff, hard cap, stall detection.
 *
 * `05-agent-completion-check.ts` only returns done true/false. This file maps
 * probabilities + test evidence + loop counters onto the article action table.
 * System One still has no `done` primitive, step counter, or stall detector.
 */
import type { EvaluationClient } from '@system-one-ai/core';
import {
  MIN_CRITERION_PROBABILITY,
  type TestEvidence,
  checkCompletion,
  createHostedClient,
} from './05-agent-completion-check.ts';

/** Inclusive lower bound of the mid band. Tune on labeled cases. */
export const MID_BAND_LOW = 0.4;
export const MAX_AGENT_STEPS = 20;
export const STALL_ROUNDS = 3;

export type CompletionAction =
  | { action: 'stop_done'; reason: 'tests_and_criteria_passed' }
  | { action: 'retry'; reason: 'tests_failed' | 'no_tests_ran' | 'criteria_mid_band' }
  | {
      action: 'handoff_human';
      reason: 'criteria_low' | 'evidence_contradicts_model' | 'hard_cap' | 'stalled';
    };

export interface AgentLoopState {
  step: number;
  recentFingerprints: readonly string[];
}

export function minCriterionProbability(probabilities: {
  acceptanceCovered: number;
  noOpenQuestions: number;
}) {
  return Math.min(probabilities.acceptanceCovered, probabilities.noOpenQuestions);
}

export function fingerprintProgress(tests: TestEvidence, output: string) {
  return JSON.stringify({
    passed: tests.passed,
    failed: tests.failed,
    ran: tests.ran,
    output,
  });
}

export function isStalled(fingerprints: readonly string[], window = STALL_ROUNDS) {
  if (fingerprints.length < window) return false;
  const recent = fingerprints.slice(-window);
  return recent.every(value => value === recent[0]);
}

/**
 * Deterministic evidence wins. Mid-band retries. Low probability or a
 * contradiction (tests failed while the model is above the done threshold)
 * hands off to a person. Hard cap and stall always stop the loop.
 */
export function decideCompletionAction(
  probabilities: { acceptanceCovered: number; noOpenQuestions: number },
  tests: TestEvidence,
  loop: AgentLoopState,
): CompletionAction {
  if (loop.step >= MAX_AGENT_STEPS) {
    return { action: 'handoff_human', reason: 'hard_cap' };
  }
  if (isStalled(loop.recentFingerprints)) {
    return { action: 'handoff_human', reason: 'stalled' };
  }

  const minimum = minCriterionProbability(probabilities);
  const modelSaysDone = minimum >= MIN_CRITERION_PROBABILITY;
  const modelMid = minimum >= MID_BAND_LOW && minimum < MIN_CRITERION_PROBABILITY;

  // Flagged failed with zero runs is "no tests ran", not a real failure.
  if (tests.ran === 0) {
    return { action: 'retry', reason: 'no_tests_ran' };
  }
  if (!tests.passed) {
    if (modelSaysDone) {
      return { action: 'handoff_human', reason: 'evidence_contradicts_model' };
    }
    return { action: 'retry', reason: 'tests_failed' };
  }
  if (modelSaysDone) {
    return { action: 'stop_done', reason: 'tests_and_criteria_passed' };
  }
  if (modelMid) {
    return { action: 'retry', reason: 'criteria_mid_band' };
  }
  return { action: 'handoff_human', reason: 'criteria_low' };
}

export async function decideCompletionAfterEvaluate(
  client: EvaluationClient,
  input: {
    task: string;
    output: string;
    acceptance: readonly string[];
    tests: TestEvidence;
    loop: AgentLoopState;
  },
) {
  const checked = await checkCompletion(client, input);
  return {
    ...decideCompletionAction(checked.probabilities, input.tests, input.loop),
    probabilities: checked.probabilities,
    model: checked.model,
  };
}

async function main() {
  const tests: TestEvidence = {
    passed: process.env.EXAMPLE_TESTS_PASSED !== '0',
    failed: process.env.EXAMPLE_FAILED_TESTS ? process.env.EXAMPLE_FAILED_TESTS.split(',') : [],
    ran: Number(process.env.EXAMPLE_TESTS_RAN ?? 3),
  };
  const output = 'Implemented decideCompletionAction with mid-band retry and stall detection.';
  const decision = await decideCompletionAfterEvaluate(createHostedClient(), {
    task: 'Stop only when tests pass and both yes/no checks are above threshold.',
    output,
    acceptance: ['Failed tests prevent done', 'Mid-band retries instead of stopping'],
    tests,
    loop: {
      step: Number(process.env.EXAMPLE_STEP ?? 1),
      recentFingerprints: [fingerprintProgress(tests, output)],
    },
  });
  console.log(JSON.stringify(decision, null, 2));
}

if (process.argv[1] && import.meta.url.endsWith(process.argv[1].split(/[\\/]/).pop() ?? '')) {
  main().catch((error: unknown) => {
    console.error(error instanceof Error ? error.message : error);
    process.exitCode = 1;
  });
}

Combining with AI SDK stopWhen

AI SDK stopWhen accepts custom conditions that see every executed step, and you can put them in an array with the built-in step-count condition; any one stops the loop. So keep the step limit as the hard limit and add a custom condition that reads test results and calls System One for the completion check.

examples/decision-articles/05-ai-sdk-stop-when.ts shows two ways: pass "step limit + async completion check" as a stopWhen array to ToolLoopAgent, or, like the AI SDK docs' manual loop, run generateText one step at a time and call decideCompletionAction() after each step. The comments note that StopCondition may return PromiseLike<boolean>, so an async System One check works.

This repo uses ai@6.0.82, where the step condition is stepCountIs(). (AI SDK 7 renames it isStepCount(); see the AI SDK 7 migration guide.)

ts/**
 * Article 5 — Combine completion checks with the Vercel AI SDK loop.
 *
 * In `ai@6` the published name is `stepCountIs` (the article draft's
 * `isStepCount` is not an export). `StopCondition` may return
 * `PromiseLike<boolean> | boolean`, so an async System One check is valid.
 * Multiple `stopWhen` entries stop when any condition is true.
 *
 * Alternative: a manual `generateText` loop (default `stopWhen` is
 * `stepCountIs(1)`) that calls `decideCompletionAction` after each step.
 */
import { ToolLoopAgent, generateText, stepCountIs } from 'ai';
import type { LanguageModel, StopCondition, ToolSet } from 'ai';
import type { EvaluationClient } from '@system-one-ai/core';
import type { TestEvidence } from './05-agent-completion-check.ts';
import {
  MAX_AGENT_STEPS,
  decideCompletionAction,
  fingerprintProgress,
  type CompletionAction,
} from './05-completion-actions.ts';
import { checkCompletion } from './05-agent-completion-check.ts';

export function hardCapStopWhen(maxSteps = MAX_AGENT_STEPS) {
  return stepCountIs(maxSteps);
}

/**
 * Custom async stop condition. Receives the AI SDK `steps` array.
 */
export function createCompletionStopWhen<TOOLS extends ToolSet>(
  inspect: (steps: Parameters<StopCondition<TOOLS>>[0]['steps']) => Promise<boolean> | boolean,
): StopCondition<TOOLS> {
  return async ({ steps }) => inspect(steps);
}

export function completionStopWhen<TOOLS extends ToolSet>(
  checkDone: (steps: Parameters<StopCondition<TOOLS>>[0]['steps']) => Promise<boolean> | boolean,
  maxSteps = MAX_AGENT_STEPS,
): Array<StopCondition<TOOLS>> {
  return [hardCapStopWhen(maxSteps), createCompletionStopWhen(checkDone)];
}

export function createCompletionAgent<TOOLS extends ToolSet>(settings: {
  model: LanguageModel;
  tools?: TOOLS;
  checkDone: (steps: Parameters<StopCondition<TOOLS>>[0]['steps']) => Promise<boolean> | boolean;
}) {
  return new ToolLoopAgent({
    model: settings.model,
    tools: settings.tools,
    stopWhen: completionStopWhen(settings.checkDone),
  });
}

export interface ManualLoopStep {
  text: string;
}

/**
 * Testable loop body. `generateStep` is your `generateText` (or any other
 * application step). System One is consulted only through `evaluateCriteria`.
 */
export async function runManualCompletionLoop(input: {
  maxSteps?: number;
  generateStep: (step: number) => Promise<ManualLoopStep>;
  readTests: () => TestEvidence;
  evaluateCriteria: (output: string) => Promise<{
    acceptanceCovered: number;
    noOpenQuestions: number;
  }>;
}) {
  const maxSteps = input.maxSteps ?? MAX_AGENT_STEPS;
  const fingerprints: string[] = [];
  let lastAction: CompletionAction = { action: 'handoff_human', reason: 'hard_cap' };
  let lastText = '';

  for (let step = 1; step <= maxSteps; step += 1) {
    const generated = await input.generateStep(step);
    lastText = generated.text;
    const tests = input.readTests();
    fingerprints.push(fingerprintProgress(tests, generated.text));
    lastAction = decideCompletionAction(
      await input.evaluateCriteria(generated.text),
      tests,
      { step, recentFingerprints: fingerprints },
    );
    if (lastAction.action !== 'retry') {
      return { ...lastAction, text: lastText, step };
    }
  }

  return { action: 'handoff_human' as const, reason: 'hard_cap' as const, text: lastText, step: maxSteps };
}

/**
 * Manual `generateText` loop from the AI SDK docs: one model step, then decide.
 */
export async function runGenerateTextCompletionLoop(input: {
  model: LanguageModel;
  prompt: string;
  tools?: ToolSet;
  client: EvaluationClient;
  task: string;
  acceptance: readonly string[];
  readTests: () => TestEvidence;
  maxSteps?: number;
}) {
  return runManualCompletionLoop({
    maxSteps: input.maxSteps,
    generateStep: async () => {
      const result = await generateText({
        model: input.model,
        prompt: input.prompt,
        tools: input.tools,
        stopWhen: stepCountIs(1),
      });
      return { text: result.text };
    },
    readTests: input.readTests,
    evaluateCriteria: async (output) => {
      const checked = await checkCompletion(input.client, {
        task: input.task,
        output,
        acceptance: input.acceptance,
        tests: input.readTests(),
      });
      return checked.probabilities;
    },
  });
}

async function main() {
  console.log(JSON.stringify({
    hardCap: 'stepCountIs(20)',
    asyncStopWhenSupported: true,
    note: 'Pass a Vercel AI SDK LanguageModel into createCompletionAgent or runGenerateTextCompletionLoop. System One is not that model.',
  }, null, 2));
}

if (process.argv[1] && import.meta.url.endsWith(process.argv[1].split(/[\\/]/).pop() ?? '')) {
  main().catch((error: unknown) => {
    console.error(error instanceof Error ? error.message : error);
    process.exitCode = 1;
  });
}

TypeScript: yes/no questions + test evidence

From examples/decision-articles/05-agent-completion-check.ts. Two booleanQuestion()s ask "are all acceptance criteria covered?" and "is nothing left blocking?", combined with the test runner's result. Failed tests or zero tests mean not done; both probabilities must meet the threshold to count as done.

ts/**
 * Article 5 — Agent completion check: yes/no questions plus hard evidence.
 *
 * Split "is the task done?" into independent Noul questions via
 * `booleanQuestion()` (wire type `noul`, SDK answer `.probability`).
 * Combine those probabilities with deterministic evidence your runtime
 * already has — here, a test-runner summary. System One does not run
 * tests, read CI, or expose a "done" primitive.
 *
 * A low probability means the statement is estimated unlikely, not that
 * the model lacks confidence. Failed tests override a high probability.
 */
import { createSystemOne, booleanQuestion } from '@system-one-ai/core';
import type { EvaluationClient } from '@system-one-ai/core';
import { systemOneAdapter } from '@system-one-ai/adapter-system-one';
import { createFetchTransport } from '@system-one-ai/transport-fetch';

export const doneCriteria = {
  acceptanceCovered: booleanQuestion(
    'Does the agent output address every acceptance criterion listed in the task?',
    { true: 'Each written criterion has a corresponding result', false: 'At least one criterion is missing' },
  ),
  noOpenQuestions: booleanQuestion(
    'Does the agent output leave no blocking question for the user?',
    { true: 'The user can stop; nothing essential is unanswered', false: 'The user must still answer a blocking question' },
  ),
};

/** Application-owned. Tune on your own cases; this is not a product default. */
export const MIN_CRITERION_PROBABILITY = 0.7;

export interface TestEvidence {
  passed: boolean;
  failed: readonly string[];
  ran: number;
}

export function createHostedClient() {
  const apiKey = process.env.SYSTEM_ONE_API_KEY;
  const baseURL = process.env.SYSTEM_ONE_BASE_URL;
  const model = process.env.SYSTEM_ONE_MODEL;
  if (!apiKey || !baseURL || !model) {
    throw new Error('Set SYSTEM_ONE_API_KEY, SYSTEM_ONE_BASE_URL (…/v1), and SYSTEM_ONE_MODEL from GET /v1/models.');
  }
  return createSystemOne({
    apiKey, baseURL, model, adapter: systemOneAdapter,
    transport: createFetchTransport(), timeoutMs: 30_000, maxRetries: 0,
  });
}

export function combineDoneSignal(
  probabilities: { acceptanceCovered: number; noOpenQuestions: number },
  tests: TestEvidence,
) {
  const modelSaysDone =
    probabilities.acceptanceCovered >= MIN_CRITERION_PROBABILITY
    && probabilities.noOpenQuestions >= MIN_CRITERION_PROBABILITY;
  if (tests.ran === 0) {
    return { done: false, reason: 'no_tests_ran' as const, modelSaysDone };
  }
  if (!tests.passed) {
    return { done: false, reason: 'tests_failed' as const, failed: tests.failed, modelSaysDone };
  }
  if (!modelSaysDone) {
    return { done: false, reason: 'criteria_below_threshold' as const, modelSaysDone };
  }
  return { done: true, reason: 'tests_and_criteria_passed' as const, modelSaysDone };
}

export async function checkCompletion(
  client: EvaluationClient,
  input: { task: string; output: string; acceptance: readonly string[]; tests: TestEvidence },
) {
  const result = await client.evaluate({
    state: {
      task: input.task,
      output: input.output,
      acceptance: [...input.acceptance],
      testsPassed: input.tests.passed,
      testsRan: input.tests.ran,
      failedTests: [...input.tests.failed],
    },
    questions: doneCriteria,
  });
  const probabilities = {
    acceptanceCovered: result.answers.acceptanceCovered.probability,
    noOpenQuestions: result.answers.noOpenQuestions.probability,
  };
  return {
    ...combineDoneSignal(probabilities, input.tests),
    probabilities,
    model: result.model,
  };
}

async function main() {
  const tests: TestEvidence = {
    passed: process.env.EXAMPLE_TESTS_PASSED !== '0',
    failed: process.env.EXAMPLE_FAILED_TESTS ? process.env.EXAMPLE_FAILED_TESTS.split(',') : [],
    ran: Number(process.env.EXAMPLE_TESTS_RAN ?? 3),
  };
  const decision = await checkCompletion(createHostedClient(), {
    task: 'Add a refund-eligibility check that refuses when tests fail.',
    output: 'Implemented combineDoneSignal; unit tests cover failed and passing suites.',
    acceptance: [
      'Failed tests prevent a done=true result',
      'Yes/no model checks are combined with AND',
    ],
    tests,
  });
  console.log(JSON.stringify(decision, null, 2));
}

if (process.argv[1] && import.meta.url.endsWith(process.argv[1].split(/[\\/]/).pop() ?? '')) {
  main().catch((error: unknown) => {
    console.error(error instanceof Error ? error.message : error);
    process.exitCode = 1;
  });
}

Notes (from the code and comments):

• System One doesn't run tests and has no done primitive. It doesn't read CI. Test results are your app's deterministic evidence, passed in as TestEvidence.
• Deterministic evidence wins. However high the probabilities, combineDoneSignal checks tests first: no_tests_ran when none ran, tests_failed when they ran and failed.
• The two checks are ANDed. Both must be ≥ MIN_CRITERION_PROBABILITY = 0.7 for modelSaysDone. The comment says this is "not a product default"; tune it on your cases.
• Hosted model: take SYSTEM_ONE_MODEL from GET /v1/models. A full model ID selects a model but can't pin a version.

For the action table (retry band, human handoff, hard limit, stall detection) see "Implementing the action table"; for AI SDK see "Combining with AI SDK stopWhen".

Claude Code Stop hook

examples/decision-articles/05-claude-code-stop-hook.ts wraps checkCompletion() as a Claude Code Stop hook. It reads the hook JSON from stdin, uses last_assistant_message as the agent output, and runs the completion check together with the test evidence. Its decisions mostly match the table, with two differences: when tests pass but the yes/no checks fall short, it always blocks for another round (no separate "below 0.4 → person" band); and it has no step limit or stall detection. It has three outputs:

• Block, run another round: when tests fail and the model also says not done, when no tests ran, or when tests pass but a check is below threshold, it prints {"decision": "block", "reason": "..."} so Claude keeps working with that reason.
• Hand to a person: when tests fail but the model says done, or when stop_hook_active is true and tests still fail or none ran, it stops blocking, prints {"systemMessage": "Tests failing but agent believes it is done — needs human review"} and writes the same line to stderr, so Claude stops and a person takes over. The hooks docs define systemMessage as a warning shown to the user. The example prints the same message for every handoff; reword it by cause when you adopt it.
• Allow: when tests pass and both checks meet the threshold, it prints nothing and Claude can stop.

stop_hook_active: true means Claude is already continuing because of an earlier block. The example still checks tests first: failing or missing tests go to a person instead of blocking again; passing tests are allowed without re-running the yes/no checks. So it blocks at most once and then hands off, rather than relying on Claude Code's own cap. The hooks docs say that after a Stop hook has continued the turn 8 times in a row, Claude Code ignores the next block and ends the turn. That's only a backstop, and a force-ended turn has nobody picking it up, so the example doesn't rely on it. The comments say the contract was checked against the Claude Code hooks docs (2026-10-11). System One has no Claude Code integration and no done primitive; this is your own hook script calling the System One API.

ts/**
 * Article 5 — Claude Code Stop hook around System One completion checks.
 *
 * Contract verified against https://code.claude.com/docs/en/hooks (2026-10-11):
 *   Input (stdin JSON), Stop-specific: hook_event_name, stop_hook_active,
 *   last_assistant_message, plus common session_id / transcript_path / cwd /
 *   permission_mode. Use last_assistant_message; do not parse transcript_path.
 *   `stop_hook_active` is true when Claude is already continuing because a
 *   previous Stop hook blocked. This file still reads test results: failed
 *   tests hand off to a person (allow stop + `systemMessage` / stderr) so the
 *   hook cannot loop; passing tests allow stop. Claude Code also applies an
 *   8-consecutive-continuation cap.
 *   Output: omit JSON to allow stop; or
 *     { "decision": "block", "reason": "..." }
 *   or, when handing off without blocking:
 *     { "systemMessage": "..." }
 *   `reason` is required when decision is "block". Exit 0 with that JSON.
 *   Exit 2 + stderr also blocks; this file uses the JSON form.
 *   hookSpecificOutput.additionalContext is the non-error continue path —
 *   this hook uses decision/reason when Claude should keep working.
 *
 * System One still has no Claude Code integration or `done` primitive. Test
 * results are application evidence. Merge 05-claude-code-stop-hook.settings.json
 * into project `.claude/settings.json`.
 */
import type { EvaluationClient } from '@system-one-ai/core';
import {
  type TestEvidence,
  checkCompletion,
  createHostedClient,
} from './05-agent-completion-check.ts';

/** Documented Stop input fields used by this hook. Extra keys are ignored. */
export interface StopHookInput {
  hook_event_name?: string;
  session_id?: string;
  transcript_path?: string;
  cwd?: string;
  permission_mode?: string;
  stop_hook_active?: boolean;
  last_assistant_message?: string;
}

/** Documented Stop block output. The only `decision` value is `"block"`. */
export interface StopHookBlockOutput {
  decision: 'block';
  reason: string;
}

/** Allow stop and show a user-visible note. No `decision`, so Claude may stop. */
export interface StopHookHumanHandoff {
  systemMessage: string;
}

export type StopHookResult = StopHookBlockOutput | StopHookHumanHandoff | undefined;

/** Exact copy for the tests-failed / model-says-done handoff. */
export const HUMAN_REVIEW_NOTE = 'Tests failing but agent believes it is done — needs human review';

export function isStopHookHumanHandoff(result: StopHookResult): result is StopHookHumanHandoff {
  return Boolean(result && 'systemMessage' in result && !('decision' in result));
}

export function parseStopHookInput(raw: unknown): StopHookInput {
  if (typeof raw !== 'object' || raw === null || Array.isArray(raw)) {
    throw new Error('Stop hook stdin must be a JSON object.');
  }
  const record = raw as Record<string, unknown>;
  const text = (key: string) => {
    const value = record[key];
    return typeof value === 'string' ? value : undefined;
  };
  return {
    hook_event_name: text('hook_event_name'),
    session_id: text('session_id'),
    transcript_path: text('transcript_path'),
    cwd: text('cwd'),
    permission_mode: text('permission_mode'),
    stop_hook_active: typeof record.stop_hook_active === 'boolean' ? record.stop_hook_active : undefined,
    last_assistant_message: text('last_assistant_message'),
  };
}

export function readTestEvidenceFromEnv(env: NodeJS.ProcessEnv = process.env): TestEvidence {
  return {
    passed: env.EXAMPLE_TESTS_PASSED !== '0',
    failed: env.EXAMPLE_FAILED_TESTS ? env.EXAMPLE_FAILED_TESTS.split(',').filter(Boolean) : [],
    ran: Number(env.EXAMPLE_TESTS_RAN ?? 0),
  };
}

export function blockReasonFromCompletion(result: {
  done: boolean;
  reason: string;
  failed?: readonly string[];
  probabilities?: { acceptanceCovered: number; noOpenQuestions: number };
}) {
  if (result.reason === 'tests_failed') {
    const failed = result.failed?.length ? result.failed.join(', ') : 'unspecified tests';
    return `Stop blocked: tests failed (${failed}). Fix the failures before finishing.`;
  }
  if (result.reason === 'no_tests_ran') {
    return 'Stop blocked: no tests ran. Run the test suite and try again.';
  }
  if (result.reason === 'criteria_below_threshold') {
    const covered = result.probabilities?.acceptanceCovered;
    const open = result.probabilities?.noOpenQuestions;
    return `Stop blocked: yes/no completion checks are below threshold (acceptanceCovered=${covered}, noOpenQuestions=${open}). Address remaining criteria or questions.`;
  }
  return `Stop blocked: ${result.reason}.`;
}

/**
 * Map tests + model signal onto the documented Stop decision.
 *
 * (a) Tests failed and the model does not say done → block, another round.
 * (b) Tests failed but the model says done → allow stop + human-review note.
 * (c) No tests ran → block, another round.
 * When `stop_hook_active` is true, failed tests or zero runs hand off (b)
 * so the hook cannot loop; passing tests allow stop.
 */
export function decideStopHook(
  input: StopHookInput,
  checked: {
    done: boolean;
    reason: string;
    modelSaysDone: boolean;
    failed?: readonly string[];
    probabilities?: { acceptanceCovered: number; noOpenQuestions: number };
  },
  tests: TestEvidence,
): StopHookResult {
  if (input.hook_event_name !== undefined && input.hook_event_name !== 'Stop') {
    return undefined;
  }

  const noTestsRan = tests.ran === 0;
  const testsFailed = tests.ran > 0 && !tests.passed;

  if (testsFailed) {
    if (input.stop_hook_active === true || checked.modelSaysDone) {
      return { systemMessage: HUMAN_REVIEW_NOTE };
    }
    return { decision: 'block', reason: blockReasonFromCompletion({ ...checked, reason: 'tests_failed' }) };
  }

  if (noTestsRan) {
    if (input.stop_hook_active === true) {
      return { systemMessage: HUMAN_REVIEW_NOTE };
    }
    return { decision: 'block', reason: blockReasonFromCompletion({ ...checked, reason: 'no_tests_ran' }) };
  }

  if (input.stop_hook_active === true || checked.done) {
    return undefined;
  }
  return { decision: 'block', reason: blockReasonFromCompletion(checked) };
}

export function stopHookOutputFromCompletion(result: {
  done: boolean;
  reason: string;
  modelSaysDone?: boolean;
  failed?: readonly string[];
  probabilities?: { acceptanceCovered: number; noOpenQuestions: number };
}): StopHookResult {
  return decideStopHook(
    { hook_event_name: 'Stop', stop_hook_active: false },
    { ...result, modelSaysDone: result.modelSaysDone ?? false },
    {
      passed: result.reason !== 'tests_failed' && result.reason !== 'no_tests_ran',
      failed: result.failed ?? [],
      ran: result.reason === 'no_tests_ran' ? 0 : 1,
    },
  );
}

export async function evaluateStopHook(
  input: StopHookInput,
  client: EvaluationClient,
  context: {
    task: string;
    acceptance: readonly string[];
    tests: TestEvidence;
  },
): Promise<StopHookResult> {
  if (input.hook_event_name !== undefined && input.hook_event_name !== 'Stop') {
    return undefined;
  }
  const checked = await checkCompletion(client, {
    task: context.task,
    output: input.last_assistant_message ?? '',
    acceptance: context.acceptance,
    tests: context.tests,
  });
  return decideStopHook(input, checked, context.tests);
}

async function readStdin() {
  const chunks: Buffer[] = [];
  for await (const chunk of process.stdin) chunks.push(Buffer.from(chunk));
  const text = Buffer.concat(chunks).toString('utf8').trim();
  return text === '' ? {} : JSON.parse(text) as unknown;
}

async function main() {
  const input = parseStopHookInput(await readStdin());
  const output = await evaluateStopHook(input, createHostedClient(), {
    task: process.env.EXAMPLE_TASK ?? 'Complete the requested work and leave no blocking questions.',
    acceptance: process.env.EXAMPLE_ACCEPTANCE
      ? process.env.EXAMPLE_ACCEPTANCE.split('|').filter(Boolean)
      : ['Acceptance criteria are covered', 'No blocking questions remain'],
    tests: readTestEvidenceFromEnv(),
  });
  if (isStopHookHumanHandoff(output)) {
    console.error(output.systemMessage);
    process.stdout.write(`${JSON.stringify(output)}\n`);
    return;
  }
  if (output) process.stdout.write(`${JSON.stringify(output)}\n`);
}

if (process.argv[1] && import.meta.url.endsWith(process.argv[1].split(/[\\/]/).pop() ?? '')) {
  main().catch((error: unknown) => {
    console.error(error instanceof Error ? error.message : error);
    process.exitCode = 1;
  });
}

Then merge this config (examples/decision-articles/05-claude-code-stop-hook.settings.json) into your project's .claude/settings.json:

json{
  "hooks": {
    "Stop": [
      {
        "hooks": [
          {
            "type": "command",
            "command": "node",
            "args": [
              "--experimental-strip-types",
              "${CLAUDE_PROJECT_DIR}/examples/decision-articles/05-claude-code-stop-hook.ts"
            ]
          }
        ]
      }
    ]
  }
}

The hook and /goal are different paths: /goal uses Claude Code's built-in evaluator; the Stop hook lets your own test evidence and System One yes/no probabilities decide when to stop.

These stack: hard limits as the backstop, deterministic checks first, semantic judgment for what tests don't cover. We haven't compared these approaches head to head, so we give no cost or accuracy figures.

Hosted billing: every 4KB of request is one block, and anything under 4KB still counts as one; cost = blocks × questions × the model's price per block. Two yes/no questions per check means two questions' worth of blocks. New accounts get 100 free credits on sign-up; see pricing.

Related reading

• What is a decision model?
• Jev alternatives in 2026
• An LLM-as-a-judge alternative
• Route agent tool calls with a Choice question
• System One primitives docs