AI Agent Guardrails: Gate Risky Tool Calls with Yes/No Probabilities

How do yes/no probabilities fit in? Start with deterministic controls: allowlists, permission checks, regex for well-defined patterns. For the fuzzy part of a policy, split it into yes/no questions such as "does this target a specific group?" or "will this tool call modify production data?", and have a decision model return P(true) for each. Add a Score question if you need severity levels. Code applies two thresholds: above the upper one, block; in between, send to a person; below the lower one, allow. If the check errors, fail closed. A probability is not authorization: high-risk actions still need permission checks and human approval.

Background on Choice, Score and Noul (yes/no probability): What is a decision model?

Try it online (sign-in required): Will this tweet get moderated?

Three checkpoints: input, output, tool

The OpenAI Agents SDK (JS) docs define three kinds of guardrail:

• Input guardrails run on the initial user input, only for the first agent that receives it.
• Output guardrails run on the agent's final output, only on the agent that produces it.
• Tool guardrails run before and after each custom function tool call. A tool input guardrail can return allow (continue), rejectContent (skip the call with a message) or throwException (trip a circuit-breaker error).

The docs describe two execution modes. Input guardrails run in parallel with the agent by default (runInParallel: true) for the lowest latency, but the model may already have spent tokens or run tools when the guardrail trips. Set it to false to finish the guardrail before calling the model. The docs also suggest running a lightweight model as the guardrail before calling a more expensive one.

Alibaba Cloud's AgentScope integration guide gives another set of hooks: pre_reply checks user input before the agent replies, post_reasoning checks final output after reasoning, and toolkit middleware checks "tool name + arguments" before execution and blocks violations.

Guardrails and human approval are different things. In the OpenAI Agents SDK, approval uses a tool's needsApproval: the run pauses, returns interruptions, and resumes from the same RunState after approve or reject. By default, tool input guardrails run after approval and before execution; with toolExecution: { preApprovalInputGuardrails: true } they also run before the approval request goes out.

These frameworks are clear about where checkpoints go. What the guardrail function uses to decide is your call.

What makes the decision

Four common options:

• Rules and regex: deterministic and cheap, good for well-defined patterns, such as the OpenAI docs example that blocks arguments containing sk- keys. They miss rephrased content. Use these first.
• Another generative LLM call: the OpenAI docs' input guardrail example runs another agent that outputs a boolean field. It understands meaning, but outputs yes/no with no tunable threshold.
• A dedicated moderation service: for example Alibaba Cloud's AI guardrails, plugged into AgentScope's hooks. The service defines categories and criteria.
• Decision model Noul: returns P(true) for yes/no questions you write. You write questions for your own policy and set thresholds by error cost.

Combine them: rules catch well-defined cases, Noul handles policies that need semantic understanding, and high-risk actions still go through human approval.

Turn a policy into Noul questions

A vague policy like "don't post offensive content" is hard to judge directly. Split it into yes/no questions:

• Does this content target a specific group?
• Does this content ask for passwords, verification codes or other credentials?
• Will this tool call modify production data?

The System One primitives docs say Noul estimates whether a specific condition holds and returns P(true) from 0 to 1. When several conditions can hold at once, define one Noul per condition; the probabilities refer to different propositions and don't need to sum to one. In the SDK, use booleanQuestion() and state what true and false mean in the criteria. Microsoft Learn's Microsoft-Decision-1 docs also recommend noul for gating and filtering, with the example question "does this reply promise the customer a refund?".

For severity levels, add a Score question with ordered levels such as "safe / borderline / violation / must remove". The System One docs say the hosted API requires 2–10 ordered levels, and the score is a probability-weighted value, not a percentage of correctness.

Gate tool calls before execution

Send the tool call as the model's input (state). The AgentScope guide formats it as Tool: {name}\nArguments: {json}, which is a reasonable template. Put only what the decision needs and is allowed to share into state; never secrets.

With the OpenAI Agents SDK, put the check in a tool input guardrail: return allow for low risk; rejectContent with an explanation to refuse the call but let the agent continue; throwException to stop the whole run. Payments, data deletion and outbound messages still get needsApproval so a person approves them.

The System One agent integration docs say a model result is not authorization: the operator's identity and permissions, action allowlists and required approvals are all checked outside the model. A missing answer or failed request is "unresolved", never an automatic yes or no. So when the guardrail call errors, treat it as not passed (fail closed), not as allow. For the allowlist approach in tool routing, see Route agent tool calls with a Choice question.

Three bands and human review

When the question asks "is this a violation?":

• P(true) above the upper threshold → block
• between the thresholds → human review
• below the lower threshold → allow

If you phrase it as "is this safe?", flip the direction. Neither threshold should be a generic percentage. The System One docs say thresholds come from your own eval cases and the consequences of mistakes; Microsoft Learn also recommends setting them by the cost of false positives and false negatives. Where a miss is expensive (payments), lower the upper threshold so more requests reach review; where a false block is expensive, narrow the review band. For picking thresholds on labelled samples and confirming on held-out data, see An LLM-as-a-judge alternative. For a Noul threshold → action table, see AI agent stop conditions.

Microsoft Learn adds: use clear, neutral wording for questions and options; add an abstain option like "cannot tell" when a choice shouldn't be forced; consider randomizing option order and testing whether order affects results; and for consequential decisions about people (credit, employment, housing, health, legal), use the model only as decision support with meaningful human review.

Evaluate misses and false blocks separately

On your own labelled set, count two things separately: should have blocked but didn't (misses) and shouldn't have blocked but did (false blocks). They have different costs, and one combined accuracy number hides that. The System One agent integration docs likewise ask you to track wrong choices and missed reviews separately and to include inputs that try to force the model to change its choice.

Include in your test set: the same meaning phrased differently, prompt injection (for example "ignore previous rules" inside arguments), and content that sounds harsh but is harmless. Microsoft Learn's known limitations include scores shifting with question and option wording and order, and, when used as a safety filter, missing subtle harmful content or flagging harmless content. Only your own data will show how big these effects are.

TypeScript example

The dev team's examples/decision-articles/08-agent-guardrails.ts defines three booleanQuestion()s: does it target a specific group, does it ask for credentials or secrets, and will this tool call modify production data or send an outbound message. The maximum of the three probabilities is the risk value, so any single rule can trigger the guardrail. Risk ≥ BLOCK_IF_RISK_AT_LEAST (0.8) blocks; ≥ REVIEW_IF_RISK_AT_LEAST (0.4) goes to your human-review callback; otherwise it's allowed. Both thresholds are examples to tune on labelled cases. The comments also note that P(true) isn't a confirmed fact or authorization, and that System One has no moderation, approval or needsApproval API.

ts/**
 * Article 8 — Agent guardrails ("Agent 护栏", /zh-CN/blog/agent-guardrails).
 *
 * Split a moderation policy into independent yes/no questions via
 * `booleanQuestion()` (wire type `noul`, SDK answer `.probability`). Those
 * probabilities do not sum to 1. This file combines them with max() — any
 * rule can trip the guardrail — then maps the combined risk onto two
 * application-owned cutoffs: block / human review / allow.
 *
 * P(true) is not a confirmed fact and is not authorization. Human review is
 * an application callback. System One has no moderation, approval, or
 * `needsApproval` API. Packages: @system-one-ai/core, adapter-system-one,
 * transport-fetch. Endpoint: POST /v1/systemone.
 */
import { createSystemOne, booleanQuestion } from '@system-one-ai/core';
import type { EvaluationClient } from '@system-one-ai/core';
import { systemOneAdapter } from '@system-one-ai/adapter-system-one';
import { createFetchTransport } from '@system-one-ai/transport-fetch';

export const guardrailQuestions = {
  targetsGroup: booleanQuestion(
    'Does this content target a specific group with abuse or harassment?',
    { true: 'The text attacks a named group', false: 'The text does not target a group' },
  ),
  requestsSecrets: booleanQuestion(
    'Does this content ask for credentials, API keys, or other secrets?',
    { true: 'The user is asked to reveal a secret', false: 'No secret is requested' },
  ),
  changesProduction: booleanQuestion(
    'Would executing this tool call change production data or send an external message?',
    { true: 'The call writes, deletes, pays, or posts', false: 'The call is read-only or local' },
  ),
};

export type GuardrailRule = keyof typeof guardrailQuestions;
export type GuardrailAction = 'block' | 'review' | 'allow';

export interface GuardrailProbabilities {
  targetsGroup: number;
  requestsSecrets: number;
  changesProduction: number;
}

/** High risk: false allow is costly. Tune on labeled cases; not a hosted default. */
export const BLOCK_IF_RISK_AT_LEAST = 0.8;
/** Mid band: send to a person. Must be below the block cutoff. */
export const REVIEW_IF_RISK_AT_LEAST = 0.4;

export interface HumanReviewInput {
  text: string;
  tool?: string;
  args?: string;
  risk: number;
  probabilities: GuardrailProbabilities;
}

export function createHostedClient() {
  const apiKey = process.env.SYSTEM_ONE_API_KEY;
  const baseURL = process.env.SYSTEM_ONE_BASE_URL;
  const model = process.env.SYSTEM_ONE_MODEL;
  if (!apiKey || !baseURL || !model) {
    throw new Error('Set SYSTEM_ONE_API_KEY, SYSTEM_ONE_BASE_URL (…/v1), and SYSTEM_ONE_MODEL from GET /v1/models.');
  }
  return createSystemOne({
    apiKey, baseURL, model, adapter: systemOneAdapter,
    transport: createFetchTransport(), timeoutMs: 30_000, maxRetries: 0,
  });
}

/** Conservative combine: the highest P(true) among independent Noul rules. */
export function combineRiskProbability(probabilities: GuardrailProbabilities) {
  return Math.max(probabilities.targetsGroup, probabilities.requestsSecrets, probabilities.changesProduction);
}

export function applyGuardrailPolicy(risk: number): GuardrailAction {
  if (risk >= BLOCK_IF_RISK_AT_LEAST) return 'block';
  if (risk >= REVIEW_IF_RISK_AT_LEAST) return 'review';
  return 'allow';
}

/**
 * @param sendToHumanReview Your approval queue. Not a System One method.
 */
export async function screenWithGuardrails(
  client: EvaluationClient,
  input: { text: string; tool?: string; args?: string },
  sendToHumanReview: (input: HumanReviewInput) => Promise<string>,
) {
  const result = await client.evaluate({
    state: {
      text: input.text,
      tool: input.tool ?? null,
      args: input.args ?? null,
    },
    questions: guardrailQuestions,
  });
  const probabilities: GuardrailProbabilities = {
    targetsGroup: result.answers.targetsGroup.probability,
    requestsSecrets: result.answers.requestsSecrets.probability,
    changesProduction: result.answers.changesProduction.probability,
  };
  const risk = combineRiskProbability(probabilities);
  const action = applyGuardrailPolicy(risk);

  if (action === 'block') {
    return { action, risk, probabilities, model: result.model };
  }
  if (action === 'review') {
    const reviewed = await sendToHumanReview({
      text: input.text,
      tool: input.tool,
      args: input.args,
      risk,
      probabilities,
    });
    return { action, risk, probabilities, model: result.model, reviewed };
  }
  return { action, risk, probabilities, model: result.model };
}

async function main() {
  const text = process.env.EXAMPLE_MESSAGE ?? 'Please post this refund to the production ledger.';
  const decision = await screenWithGuardrails(
    createHostedClient(),
    { text, tool: process.env.EXAMPLE_TOOL, args: process.env.EXAMPLE_TOOL_ARGS },
    async () => {
      throw new Error('No System One human-review API. Wire your approval queue here.');
    },
  );
  console.log(JSON.stringify(decision, null, 2));
}

if (process.argv[1] && import.meta.url.endsWith(process.argv[1].split(/[\\/]/).pop() ?? '')) {
  main().catch((error: unknown) => {
    console.error(error instanceof Error ? error.message : error);
    process.exitCode = 1;
  });
}

This example isn't wired into an OpenAI Agents SDK tool input guardrail and doesn't use Score for severity. When you integrate it, map the returned block / review / allow to your framework's rejectContent, needsApproval, allow and so on.

Hosted billing: every 4KB of request is one block, and anything under 4KB still counts as one; cost = blocks × questions × the model's price per block. Three guardrail questions means three questions' worth of blocks per check. New accounts get 100 free credits on sign-up; see pricing.

Related reading

• What is a decision model?
• An LLM-as-a-judge alternative
• Route agent tool calls with a Choice question
• AI agent stop conditions
• Docs: primitives, agent integration