决策模型和 Choice / Score / Noul 三种原语的基础知识见 决策模型是什么？；怎么在不同模型服务之间切换见 Jev 开源替代有哪些？。
为什么工具越多，Agent 越容易选错
接了几个 MCP 服务器之后，工具定义很快就会塞满上下文。Anthropic 的工具搜索文档描述了两个问题：
- 上下文膨胀：一个常见的多服务器配置（GitHub、Slack、Sentry、Grafana、Splunk）在 Claude 开始工作之前，光工具定义就可能占用约 5.5 万个 token。
- 选择准确率下降：工具数超过 30 到 50 个之后，Claude 选对工具的能力就会下降。
这是 Anthropic 对自家模型的说法，其他模型的拐点不一定相同。原因不难理解：候选越多，描述相近的工具就越多，比如几个都叫“搜索”、几个都能“发消息”，模型也更容易被无关的定义分散注意力。开发者社区的抱怨也很直接：“too many tools and no clarity on which to use”，意思是工具太多，不知道该用哪个（原帖）。
三层架构：检索 → 选择 → 执行
社区目前的解法大多集中在第一层：先检索、懒加载、改写工具描述。把它们和后面两步放在一起看，工具路由可以分成三层：
- 检索：缩小候选。 用关键词、BM25、向量检索，或者模型自己调用的“工具搜索”，把成百上千个工具缩小到少数几个。Anthropic 的工具搜索默认每次返回最多 5 个工具，并建议把最常用的 3 到 5 个工具一直加载着，其余的延迟加载。
- 选择：从候选中定一个。 这是一次有边界的选择：答案只能是候选工具之一，或者“都不合适”。这一步正好是决策模型 Choice 的用武之地。
- 执行：生成参数并调用。 由应用的权限检查和主模型（或代码）负责生成参数，再调用工具。System One 不会替你调用工具，也不生成参数。
需要注意的是，检索这一层失败了，后面两层都救不回来：正确的工具没出现在候选里，再好的选择器也选不到它。第三方的对比测试（Stacklok 博客，厂商自测）也是这样解释工具选择准确率差距的。所以决策模型不能代替检索，它只负责检索之后的那一步。
下图是开发团队提供的三层架构示意图（examples/decision-articles/04-three-tier-architecture.mmd；同目录下还有渲染好的 04-three-tier-architecture.svg）。检索由应用负责，Choice 只给出工具名，置信度分档是应用自己设的阈值，不是托管服务的默认值。
mermaid%% Article 4 — three-tier tool routing after Choice.
%% Retrieve is application-owned. Choice only names a tool.
%% Confidence bands are application thresholds, not a hosted default.
flowchart TB
subgraph retrieve [Layer 1 - Retrieve]
Message[User message] --> Catalog[Allowlisted catalog]
Catalog --> Candidates[Candidate tools]
end
subgraph select [Layer 2 - Select]
Candidates --> Choice[System One Choice]
Choice --> Band{Confidence band}
end
subgraph act [Layer 3 - Act]
Band -->|high| Execute[Propose execute tool]
Band -->|medium| Confirm[Ask user to confirm]
Band -->|low or missing| Fallback[Main application LLM]
Confirm -->|approved| Execute
Confirm -->|declined| Stop[Do not execute]
end
用 Choice 做工具选择
Choice 就是从一组具名选项里选一个。System One 的决策原语文档写明，原生 Choice 的答案包含选中的选项名（choice）、以选项名为键的概率分布（probabilities），以及独立报告的统计值 confidence。System One SDK 的写法见下文示例。
选项 = 工具名 + 一句“何时使用”
选项的键就是工具名，值写清“什么情况下选它”。System One 的接入文档建议：工具说明应解释何时选择该项，不要包含凭据，也不能用描述来授予权限。决策原语文档也要求保持各选项含义清晰，让选项彼此区分开。
留一条“都不合适”的出路
System One 决策原语文档建议，强制归类会误导时加入审核选项；接入文档也建议在适合时提供明确的 think 或 review 选项，让模型不必硬选一个工具。下文示例用的是 defer_to_llm（交回主模型），白名单中还有 ask_user（询问用户）和 human_review（人工审核）两个“非工具”选项。
选项上限 255 个，为什么还要先缩小候选
System One 文档规定，一个 Choice 问题最多 255 个选项。TypeSafe 甚至建议做分类时直接给模型完整列表，而不是缩短的候选列表，因为每多一个选项只多花几个 token。
工具路由有所不同。System One 的接入文档明确建议：从一个小型白名单开始，不要把 Agent 能访问的所有工具都交给模型，也不要让模型生成的名称、URL 或 shell 命令绕过白名单。原因有两个：工具调用会产生实际后果，候选集本身就是权限边界；而每个工具的“何时使用”往往比分类标签长得多。实际做法可以按候选数量来定：候选少时直接放进 Choice；工具多、描述长、权限各异时，先检索出候选再交给 Choice。哪种方式更准，要用你自己的评估集比较，本站没有做过实测。
置信度阈值与回退
拿到 Choice 结果后，应用按置信度决定下一步。TypeSafe 的置信度文档给了一个“三段式”的起点：
- 高置信：自动执行。工具路由里，就是交给应用的权限检查和调度器。
- 中等置信：谨慎处理，比如让用户确认、标记复核，或先收集更多信息。
- 低置信：不要执行，转给人工、请用户澄清，或回退到别的系统（比如主模型）。
几个要点：
- 置信度不等于“选对的概率”。 System One 决策原语文档写明，
confidence不一定等于选中选项的概率，两者都不是通用动作阈值。TypeSafe 文档补充说明，TypeSafe 的 Choice 置信度是根据概率分布算出来的统计量，概率全部集中在一个选项上为 1，平均分散为 0。TypeSafe 还建议同时参考最高概率、第一名与第二名的概率比。下文示例的注释也写明：置信度是一项可选的服务端统计，不一定代表“选中工具的概率”，也不是通用阈值。 - 阈值随风险变化。 只读的查询可以用较低的阈值；付款、改权限、对外发消息这类动作，应该要求更高的置信度，或者无论如何都要人工确认。System One 的 Agent 接入文档也要求，付款、访问权限变更和对外发送消息必须保留政策要求的确认。
- 缺失就是未解决。 System One 文档写道：答案缺失或请求失败表示未解决，不是自动的“是”“否”或最低分。示例在
confidence缺失时一律不执行。 - 模型结果不是授权。 模型选中了某个工具，不代表这个用户有权执行它。身份、权限和审批要在模型之外检查。
- 阈值从哪里来：System One 文档写明，阈值应该来自你自己的评估案例和后果分析，而不是一个通用的百分比。换了模型、说明、选项或描述之后，都要重新评估。
三段式的写法
开发团队提供了一个三段式示例 examples/decision-articles/04-three-tier-routing.ts。它从主示例 04-agent-tool-routing.ts 导入白名单、检索函数和 DEFER_TO_LLM，主示例见下文“TypeScript 示例”一节。置信度不低于 HIGH_EXECUTE_CONFIDENCE（0.75）时返回“执行”；介于 MEDIUM_CONFIRM_CONFIDENCE（0.45）和高阈值之间时，调用应用自己的确认回调，由用户决定；更低、缺失，或模型选了 defer_to_llm 时，交回应用的主模型。两个阈值都是示例值，注释写明要按标注案例和动作风险调。
ts/**
* Article 4 — Three-tier tool routing after Choice.
*
* TypeSafe's confidence guidance is a three-band starting point: high →
* execute, medium → confirm with the user, low → do not execute. System One
* only returns Choice + optional confidence. Thresholds, the confirm UI, and
* the main-LLM fallback are application-owned. Missing confidence stays
* unknown and is treated as "do not execute".
*
* This file does not generate tool arguments and does not call tools.
*/
import { choice } from '@system-one-ai/core';
import type { ChoiceAnswer, EvaluationClient } from '@system-one-ai/core';
import {
DEFER_TO_LLM,
TOOL_CATALOG,
createHostedClient,
retrieveCandidateTools,
} from './04-agent-tool-routing.ts';
/** Application-owned. Tune on labeled cases and action risk; not a hosted default. */
export const HIGH_EXECUTE_CONFIDENCE = 0.75;
export const MEDIUM_CONFIRM_CONFIDENCE = 0.45;
export type RouteTier = 'execute' | 'confirm' | 'fallback';
export function buildToolChoiceCriteria(
tools: readonly { id: string; whenToUse: string }[],
) {
return Object.fromEntries([
...tools.map(tool => [tool.id, tool.whenToUse]),
[DEFER_TO_LLM, 'None of the allowlisted tools fits; continue with the main application LLM'],
]);
}
export function classifyRouteTier(answer: ChoiceAnswer<string>): RouteTier {
if (answer.choice === DEFER_TO_LLM) return 'fallback';
if (answer.confidence === undefined) return 'fallback';
if (answer.confidence >= HIGH_EXECUTE_CONFIDENCE) return 'execute';
if (answer.confidence >= MEDIUM_CONFIRM_CONFIDENCE) return 'confirm';
return 'fallback';
}
export interface ConfirmToolInput {
message: string;
proposedTool: string;
confidence: number;
}
/**
* @param askUserToConfirm Your confirm UI or approval queue. Not a System One method.
* @param completeWithApplicationLlm Your agent's existing completion path.
*/
export async function routeToolThreeTier(
client: EvaluationClient,
message: string,
callbacks: {
askUserToConfirm: (input: ConfirmToolInput) => Promise<boolean>;
completeWithApplicationLlm: (prompt: string) => Promise<string>;
},
) {
const candidates = retrieveCandidateTools(message, TOOL_CATALOG);
const result = await client.evaluate({
state: { message, retrievedTools: candidates.map(tool => tool.id) },
questions: {
tool: choice('Choose the next allowed tool. Do not execute it.', buildToolChoiceCriteria(candidates)),
},
});
const answer = result.answers.tool;
const tier = classifyRouteTier(answer);
if (tier === 'execute') {
return {
action: 'execute_tool' as const,
tier,
proposedTool: answer.choice,
confidence: answer.confidence,
};
}
if (tier === 'confirm') {
const approved = await callbacks.askUserToConfirm({
message,
proposedTool: answer.choice,
confidence: answer.confidence as number,
});
return approved
? {
action: 'execute_tool' as const,
tier,
proposedTool: answer.choice,
confidence: answer.confidence,
confirmedByUser: true as const,
}
: {
action: 'user_declined' as const,
tier,
proposedTool: answer.choice,
confidence: answer.confidence,
};
}
const fallback = await callbacks.completeWithApplicationLlm(message);
return {
action: 'fallback_llm' as const,
tier,
proposedTool: answer.choice,
confidence: answer.confidence,
fallback,
};
}
async function main() {
const message = process.env.EXAMPLE_MESSAGE ?? 'Can you explain how credits are calculated?';
const decision = await routeToolThreeTier(createHostedClient(), message, {
askUserToConfirm: async () => {
throw new Error('No System One confirm UI. Wire your approval callback here.');
},
completeWithApplicationLlm: async () => {
throw new Error('No System One LLM-completion endpoint. Wire your existing model call here.');
},
});
console.log(JSON.stringify(decision, null, 2));
}
if (process.argv[1] && import.meta.url.endsWith(process.argv[1].split(/[\\/]/).pop() ?? '')) {
main().catch((error: unknown) => {
console.error(error instanceof Error ? error.message : error);
process.exitCode = 1;
});
}
度量：选择准确率和参数有效率要分开算
工具调用失败有两种不同的情况：一是选错了工具，二是选对了工具但参数错了。开发者社区也有人提出，Agent 可能选错工具，也可能选对工具却给了错误参数，所以工具选择和参数有效性应该分开度量（原帖）。在本文的三层架构里，这两种错误分别出在第 2 层和第 3 层，所以建议分开统计：
- 选择准确率：在带标注的评估集上，Choice 选出的工具与预期工具一致的比例。除了总体准确率，还要单独看“该转人工却没转”的漏判。System One 文档建议分别统计错误工具选择和遗漏审核的情况，并加入模糊请求，以及试图强制模型选择其他工具的指令。
- 检索召回率：正确工具出现在候选中的比例。它决定了选择准确率的上限。
- 参数有效率：在选对工具的前提下，参数通过校验、调用成功的比例。这属于主模型或代码的职责，与决策模型无关。
- 回退率：被交回主模型、用户或人工的比例。它直接影响成本和用户体验。
System One 文档还提醒：HTTP 请求成功、模型给出较高概率，都不能证明路由是安全的。应该先以“只提供建议”的方式运行，再根据已有政策决定哪些只读动作可以自动执行。
参数生成：选完工具再单独生成、单独校验
下面的示例（examples/decision-articles/04-generate-tool-params.ts）在三段式路由判定“执行”之后，才调用应用自己的参数生成回调，并用 paramsAreValid() 检查必填参数是否齐全、非空。Choice 只负责选出工具名，System One 不生成、不校验参数，也不调用工具。示例中的必填字段表（TOOL_PARAM_REQUIRED）只是演示。
ts/**
* Article 4 — Generate tool-call parameters after routing.
*
* Choice only names the tool. System One does not emit arguments, validate
* them, or invoke the tool. Parameter generation stays in the application
* (deterministic code or your main LLM). Measure parameter validity separately
* from routing accuracy.
*/
import type { EvaluationClient } from '@system-one-ai/core';
import { DEFER_TO_LLM, createHostedClient } from './04-agent-tool-routing.ts';
import { routeToolThreeTier } from './04-three-tier-routing.ts';
export interface GeneratedToolParams {
readonly [key: string]: string;
}
export const TOOL_PARAM_REQUIRED: Readonly<Record<string, readonly string[]>> = {
search_docs: ['query'],
ask_user: ['question'],
human_review: ['reason'],
};
export function paramsAreValid(toolId: string, params: unknown): boolean {
const required = TOOL_PARAM_REQUIRED[toolId];
if (!required) return false;
if (typeof params !== 'object' || params === null || Array.isArray(params)) return false;
const record = params as Record<string, unknown>;
return required.every(key => typeof record[key] === 'string' && record[key].trim().length > 0);
}
export interface GenerateParamsInput {
message: string;
toolId: string;
}
/**
* @param generateParams Application-owned. Not a System One method.
* The decision model has already picked `toolId` and must not invent args.
*/
export async function routeThenGenerateParams(
client: EvaluationClient,
message: string,
generateParams: (input: GenerateParamsInput) => Promise<unknown>,
callbacks: {
askUserToConfirm: (input: { message: string; proposedTool: string; confidence: number }) => Promise<boolean>;
completeWithApplicationLlm: (prompt: string) => Promise<string>;
},
) {
const routed = await routeToolThreeTier(client, message, callbacks);
if (routed.action !== 'execute_tool' || routed.proposedTool === DEFER_TO_LLM) {
return { ...routed, params: undefined, paramsValid: false as const };
}
const params = await generateParams({ message, toolId: routed.proposedTool });
return {
...routed,
params,
paramsValid: paramsAreValid(routed.proposedTool, params),
};
}
async function main() {
const message = process.env.EXAMPLE_MESSAGE ?? 'Can you explain how credits are calculated?';
const decision = await routeThenGenerateParams(
createHostedClient(),
message,
async ({ message: userMessage, toolId }) => {
if (toolId === 'search_docs') return { query: userMessage };
throw new Error(`No application parameter builder wired for ${toolId}.`);
},
{
askUserToConfirm: async () => true,
completeWithApplicationLlm: async () => {
throw new Error('No System One LLM-completion endpoint. Wire your existing model call here.');
},
},
);
console.log(JSON.stringify(decision, null, 2));
}
if (process.argv[1] && import.meta.url.endsWith(process.argv[1].split(/[\\/]/).pop() ?? '')) {
main().catch((error: unknown) => {
console.error(error instanceof Error ? error.message : error);
process.exitCode = 1;
});
}
离线评估：检索召回、选择准确率、漏判率、参数有效率
下面的示例（examples/decision-articles/04-routing-eval-harness.ts）在一组带标注的案例上同时算出四个指标：检索召回率、Choice 选择准确率、该转人工却没转的漏判率，以及“选对工具的前提下”的参数有效率。System One 没有评估数据集或裁判 API，这些都由这段应用代码完成。示例里的 4 条案例（ROUTE_EVAL_CASES）只用于演示，注释写明调阈值前要换成你自己的案例。
ts/**
* Article 4 — Offline routing-accuracy harness on a labeled set.
*
* System One has no eval-dataset or judge API. This file scores:
* 1. retrieval recall — expected tool present in the candidate set
* 2. choice accuracy — Choice equals the labeled tool
* 3. missed-review rate — labeled human_review but Choice picked something else
* Parameter validity is a separate column, computed only on correctly routed
* rows, because argument generation is not a decision-model job.
*/
import { choice } from '@system-one-ai/core';
import type { EvaluationClient } from '@system-one-ai/core';
import {
DEFER_TO_LLM,
TOOL_CATALOG,
createHostedClient,
retrieveCandidateTools,
} from './04-agent-tool-routing.ts';
import { paramsAreValid } from './04-generate-tool-params.ts';
import { buildToolChoiceCriteria } from './04-three-tier-routing.ts';
export interface LabeledRouteCase {
id: string;
message: string;
expectedTool: string;
/** When set, used only after a correct Choice to score argument generation. */
expectedParams?: Readonly<Record<string, string>>;
}
/** Small labeled set for the article. Replace with your own cases before tuning thresholds. */
export const ROUTE_EVAL_CASES: readonly LabeledRouteCase[] = [
{ id: 'credits', message: 'How are credits calculated?', expectedTool: 'search_docs' },
{ id: 'vague', message: 'It broke and does not work', expectedTool: 'ask_user' },
{ id: 'refund', message: 'I was charged twice and want a refund', expectedTool: 'human_review' },
{ id: 'unrelated', message: 'Write a haiku about rain', expectedTool: DEFER_TO_LLM },
];
export interface RouteEvalRow {
id: string;
expectedTool: string;
predictedTool: string | undefined;
retrieved: readonly string[];
retrievedExpected: boolean;
correctChoice: boolean;
missedReview: boolean;
paramsValid: boolean | undefined;
}
export function evaluateRetrieval(cases: readonly LabeledRouteCase[] = ROUTE_EVAL_CASES) {
return cases.map((labeled) => {
const retrieved = retrieveCandidateTools(labeled.message, TOOL_CATALOG).map(tool => tool.id);
const retrievedExpected =
labeled.expectedTool === DEFER_TO_LLM || retrieved.includes(labeled.expectedTool);
return { id: labeled.id, retrieved, retrievedExpected };
});
}
export async function evaluateRoutingAccuracy(
client: EvaluationClient,
cases: readonly LabeledRouteCase[] = ROUTE_EVAL_CASES,
generateParams?: (input: { message: string; toolId: string }) => Promise<unknown>,
) {
const rows: RouteEvalRow[] = [];
for (const labeled of cases) {
const candidates = retrieveCandidateTools(labeled.message, TOOL_CATALOG);
const retrieved = candidates.map(tool => tool.id);
const retrievedExpected =
labeled.expectedTool === DEFER_TO_LLM || retrieved.includes(labeled.expectedTool);
const result = await client.evaluate({
state: { message: labeled.message, retrievedTools: retrieved },
questions: {
tool: choice('Choose the next allowed tool. Do not execute it.', buildToolChoiceCriteria(candidates)),
},
});
const predictedTool = result.answers.tool.choice;
const correctChoice = predictedTool === labeled.expectedTool;
const missedReview = labeled.expectedTool === 'human_review' && predictedTool !== 'human_review';
let paramsValid: boolean | undefined;
if (correctChoice && labeled.expectedParams && generateParams) {
const params = await generateParams({ message: labeled.message, toolId: predictedTool });
paramsValid = paramsAreValid(predictedTool, params);
}
rows.push({
id: labeled.id,
expectedTool: labeled.expectedTool,
predictedTool,
retrieved,
retrievedExpected,
correctChoice,
missedReview,
paramsValid,
});
}
return rows;
}
export function summarizeRouteEval(rows: readonly RouteEvalRow[]) {
const reviewCases = rows.filter(row => row.expectedTool === 'human_review');
const correctWithParams = rows.filter(row => row.correctChoice && row.paramsValid !== undefined);
return {
n: rows.length,
retrievalRecall: rows.length === 0 ? 0 : rows.filter(row => row.retrievedExpected).length / rows.length,
choiceAccuracy: rows.length === 0 ? 0 : rows.filter(row => row.correctChoice).length / rows.length,
missedReviewRate: reviewCases.length === 0
? 0
: reviewCases.filter(row => row.missedReview).length / reviewCases.length,
paramValidityAmongCorrect: correctWithParams.length === 0
? undefined
: correctWithParams.filter(row => row.paramsValid).length / correctWithParams.length,
};
}
async function main() {
const retrieval = evaluateRetrieval();
console.log(JSON.stringify({ retrieval, retrievalOnly: true }, null, 2));
if (!process.env.SYSTEM_ONE_API_KEY) return;
const rows = await evaluateRoutingAccuracy(createHostedClient());
console.log(JSON.stringify({ rows, summary: summarizeRouteEval(rows) }, null, 2));
}
if (process.argv[1] && import.meta.url.endsWith(process.argv[1].split(/[\\/]/).pop() ?? '')) {
main().catch((error: unknown) => {
console.error(error instanceof Error ? error.message : error);
process.exitCode = 1;
});
}
工具路由和模型路由有什么区别
两者都是从预设选项中做一次选择，都可以用决策模型实现，但选项不同、指标也不同：
- 工具路由决定这一步调用哪个工具（或者不调用）。主要指标是选择准确率、漏判率和参数有效率。
- 模型路由决定把请求交给哪个模型处理。Microsoft 在 Microsoft-Decision-1 的发布博客中把它描述为“按质量、成本和延迟要求选出最合适的模型”（厂商说法），主要指标是质量、成本和延迟。
不少决策模型厂商都把“路由”列为主要用途：Microsoft 称 Microsoft-Decision-1 适用于 routing、classification 等场景，RouterPlus 的文档也说它的 /v1/decisions 可用于 routing、classification、moderation（均为厂商说法）。选哪家模型、怎么在不同服务之间切换，可以参考 Jev 开源替代有哪些？2026 决策模型选型地图与 TypeScript 方案。如果要把决策模型当成评估环节的第一道关，参见 用 Score 替代 LLM 当裁判。
按难度把请求分给小模型或大模型的完整做法，详见 模型路由：先判断难度，再分给小模型或大模型。
TypeScript 示例：检索候选 → Choice → 低置信回退
在线试一下：NPC 下一步动作（示例结果，登录后可跑真实数据）
下面这段代码来自开发团队提供的示例（examples/decision-articles/04-agent-tool-routing.ts）。它用关键词从一份静态白名单中检索候选工具，再用 Choice 选一个，并加入一个 defer_to_llm 选项。只有当置信度存在且不低于阈值时，才返回建议调用的工具；否则交回应用自己的主模型。
ts/**
* Article 4 — Agent tool routing: retrieve candidates, Choice, fall back on low confidence.
*
* System One can pick one named option from the set you send. It does not
* retrieve tools, execute them, or run a general LLM completion. Retrieval
* and the "main LLM" fallback are application functions in this file.
*
* Confidence is an optional provider statistic, not necessarily P(selected
* tool) and not a universal threshold. Missing confidence stays unknown —
* this example treats unknown as "do not execute".
*/
import { createSystemOne, choice } from '@system-one-ai/core';
import type { ChoiceAnswer, EvaluationClient } from '@system-one-ai/core';
import { systemOneAdapter } from '@system-one-ai/adapter-system-one';
import { createFetchTransport } from '@system-one-ai/transport-fetch';
export interface ToolSpec {
id: string;
whenToUse: string;
keywords: readonly string[];
}
/** Static allowlist. There is no System One tool-catalog endpoint. */
export const TOOL_CATALOG: readonly ToolSpec[] = [
{ id: 'search_docs', whenToUse: 'A product or usage question can be answered from documentation', keywords: ['credit', 'docs', 'how', 'price'] },
{ id: 'ask_user', whenToUse: 'Essential details are missing; ask before proceeding', keywords: ['it broke', 'does not work', 'help'] },
{ id: 'human_review', whenToUse: 'A payment dispute, refund, or policy exception needs a person', keywords: ['refund', 'charged', 'dispute'] },
];
export const DEFER_TO_LLM = 'defer_to_llm';
export const MIN_EXECUTE_CONFIDENCE = 0.55;
export function retrieveCandidateTools(message: string, catalog: readonly ToolSpec[] = TOOL_CATALOG) {
const text = message.toLowerCase();
const hits = catalog.filter(tool => tool.keywords.some(keyword => text.includes(keyword)));
return hits.length > 0 ? hits : catalog.filter(tool => tool.id === 'ask_user' || tool.id === 'human_review');
}
export function shouldExecuteChosenTool(answer: ChoiceAnswer<string>) {
if (answer.choice === DEFER_TO_LLM) return false;
if (answer.confidence === undefined) return false;
return answer.confidence >= MIN_EXECUTE_CONFIDENCE;
}
export function createHostedClient() {
const apiKey = process.env.SYSTEM_ONE_API_KEY;
const baseURL = process.env.SYSTEM_ONE_BASE_URL;
const model = process.env.SYSTEM_ONE_MODEL;
if (!apiKey || !baseURL || !model) {
throw new Error('Set SYSTEM_ONE_API_KEY, SYSTEM_ONE_BASE_URL (…/v1), and SYSTEM_ONE_MODEL from GET /v1/models.');
}
return createSystemOne({
apiKey, baseURL, model, adapter: systemOneAdapter,
transport: createFetchTransport(), timeoutMs: 30_000, maxRetries: 0,
});
}
/**
* @param completeWithApplicationLlm Your agent's existing completion path.
* System One has no chat/completions URL for this fallback.
*/
export async function routeTool(
client: EvaluationClient,
message: string,
completeWithApplicationLlm: (prompt: string) => Promise<string>,
) {
const candidates = retrieveCandidateTools(message);
const criteria = Object.fromEntries([
...candidates.map(tool => [tool.id, tool.whenToUse]),
[DEFER_TO_LLM, 'None of the allowlisted tools fits; continue with the main application LLM'],
]);
const result = await client.evaluate({
state: { message, retrievedTools: candidates.map(tool => tool.id) },
questions: {
tool: choice('Choose the next allowed tool. Do not execute it.', criteria),
},
});
const answer = result.answers.tool;
if (!shouldExecuteChosenTool(answer)) {
const fallback = await completeWithApplicationLlm(message);
return {
action: 'fallback_llm' as const,
proposedTool: answer.choice,
confidence: answer.confidence,
fallback,
};
}
return {
action: 'propose_tool' as const,
proposedTool: answer.choice,
confidence: answer.confidence,
};
}
async function main() {
const message = process.env.EXAMPLE_MESSAGE ?? 'Can you explain how credits are calculated?';
const decision = await routeTool(createHostedClient(), message, async () => {
throw new Error('No System One LLM-completion endpoint. Wire your existing model call here.');
});
console.log(JSON.stringify(decision, null, 2));
}
if (process.argv[1] && import.meta.url.endsWith(process.argv[1].split(/[\\/]/).pop() ?? '')) {
main().catch((error: unknown) => {
console.error(error instanceof Error ? error.message : error);
process.exitCode = 1;
});
}
需要说明几点（均出自示例代码和注释）：
- 检索和回退都是应用自己的函数。 System One 不检索工具、不执行工具，也不提供通用的 LLM 补全接口；也没有工具目录接口，
TOOL_CATALOG是代码里的静态白名单。示例中的关键词匹配只是演示用的占位实现，生产环境可以换成 BM25 或向量检索。 - 只建议，不执行。 置信度足够时，示例返回
action: 'propose_tool'，把是否执行交给你的权限检查和调度器。 - 阈值是示例值。
MIN_EXECUTE_CONFIDENCE = 0.55只是示例，需要按上文的方法在自己的评估集上重新选。置信度缺失（undefined）时，示例按“不执行”处理。 - 托管模型：
SYSTEM_ONE_MODEL要从GET /v1/models获取。system-one.dev 提供 Jev（typesafe/jev）、Clef Flash（cloudflare/clef-flash）和 Clef（cloudflare/clef）模型服务。
上文“三段式的写法”和“度量”两节中的示例补上了询问用户的中间档、参数生成和离线评估。下面两个示例演示如何接入 Vercel AI SDK 和 MCP。
接入 Vercel AI SDK
这个示例（examples/decision-articles/04-vercel-ai-sdk-tools.ts）用 AI SDK 的 tool() 定义白名单里的三个工具，先用三段式路由选出工具，再调用 generateText，通过 activeTools 和 toolChoice 把模型限定在这一个工具上，由 AI SDK 绑定的语言模型填写参数。示例注释写明：@system-one-ai/adapter-vercel 是 Vercel 的评估（Evaluation）适配器，不是 AI SDK 的工具适配器，也不是这个仓库的依赖，所以没有引入；传给 generateText 的 model 是你自己的模型服务，不是 System One。示例基于仓库使用的 ai@6.0.82。
ts/**
* Article 4 — Vercel AI SDK tools wrapping System One routing.
*
* `@system-one-ai/adapter-vercel` is a published Vercel *Evaluation* adapter
* (not an AI SDK tools adapter) and is not a dependency of this repo. This
* file uses the already-installed `ai` package: `tool()`, `generateText`,
* `activeTools`, and `toolChoice: { type: 'tool', toolName }`.
*
* Choice still only picks the tool. The language model bound to `generateText`
* fills `inputSchema`. System One does not generate arguments.
*/
import { generateText, tool } from 'ai';
import type { LanguageModel, ToolSet } from 'ai';
import { z } from 'zod';
import type { EvaluationClient } from '@system-one-ai/core';
import { createHostedClient } from './04-agent-tool-routing.ts';
import { routeToolThreeTier } from './04-three-tier-routing.ts';
export const searchDocsTool = tool({
description: 'A product or usage question can be answered from documentation',
inputSchema: z.object({
query: z.string().describe('Documentation search query'),
}),
execute: async ({ query }) => ({ source: 'application-docs-search', query }),
});
export const askUserTool = tool({
description: 'Essential details are missing; ask before proceeding',
inputSchema: z.object({
question: z.string().describe('Clarifying question for the user'),
}),
execute: async ({ question }) => ({ source: 'application-ask-user', question }),
});
export const humanReviewTool = tool({
description: 'A payment dispute, refund, or policy exception needs a person',
inputSchema: z.object({
reason: z.string().describe('Why a person must review this request'),
}),
execute: async ({ reason }) => ({ source: 'application-human-review', reason }),
});
export const allowlistedAiTools = {
search_docs: searchDocsTool,
ask_user: askUserTool,
human_review: humanReviewTool,
} satisfies ToolSet;
export type AllowlistedToolName = keyof typeof allowlistedAiTools;
export function routedGenerateTextOptions(
proposedTool: AllowlistedToolName,
message: string,
) {
return {
prompt: message,
tools: allowlistedAiTools,
activeTools: [proposedTool] as Array<AllowlistedToolName>,
toolChoice: { type: 'tool' as const, toolName: proposedTool },
};
}
/**
* Route with System One, then let the AI SDK model fill that tool's arguments.
*
* @param model A Vercel AI SDK `LanguageModel` from your existing provider
* package (for example `@ai-sdk/openai`). Not a System One model.
*/
export async function routeThenGenerateText(
client: EvaluationClient,
model: LanguageModel,
message: string,
callbacks: {
askUserToConfirm: (input: { message: string; proposedTool: string; confidence: number }) => Promise<boolean>;
completeWithApplicationLlm: (prompt: string) => Promise<string>;
},
) {
const routed = await routeToolThreeTier(client, message, callbacks);
if (routed.action !== 'execute_tool' || !(routed.proposedTool in allowlistedAiTools)) {
return { routed, generation: undefined };
}
const proposedTool = routed.proposedTool as AllowlistedToolName;
const generation = await generateText({
model,
...routedGenerateTextOptions(proposedTool, message),
});
return { routed, generation };
}
async function main() {
const message = process.env.EXAMPLE_MESSAGE ?? 'Can you explain how credits are calculated?';
const routed = await routeToolThreeTier(createHostedClient(), message, {
askUserToConfirm: async () => true,
completeWithApplicationLlm: async () => {
throw new Error('No System One LLM-completion endpoint. Wire your existing model call here.');
},
});
if (routed.action === 'execute_tool' && routed.proposedTool in allowlistedAiTools) {
console.log(JSON.stringify({
routed,
generateTextOptions: routedGenerateTextOptions(routed.proposedTool as AllowlistedToolName, message),
}, null, 2));
return;
}
console.log(JSON.stringify({ routed }, null, 2));
}
if (process.argv[1] && import.meta.url.endsWith(process.argv[1].split(/[\\/]/).pop() ?? '')) {
main().catch((error: unknown) => {
console.error(error instanceof Error ? error.message : error);
process.exitCode = 1;
});
}
接入 MCP 工具
这个示例（examples/decision-articles/04-mcp-tool-routing.ts）只用官方 @modelcontextprotocol/sdk 的客户端类型：先用 listTools() 列出已连接 MCP 服务器上的工具，把工具名和描述转成 Choice 选项（加上 defer_to_llm），置信度达到高阈值且选中的工具确实在列表里时，才返回建议调用的工具和它的 inputSchema；真正调用由应用通过 callTool() 完成，参数来自你的代码或主模型。System One 本身不支持 MCP，也不检索、不执行工具。注意，这个示例把所有列出的 MCP 工具都交给 Choice，工具很多时，仍建议先按上文做检索。
ts/**
* Article 4 — Route among tools advertised by an MCP client.
*
* Uses official `@modelcontextprotocol/sdk` client types only: `Client.listTools()`
* and `Client.callTool({ name, arguments })`. System One does not speak MCP,
* retrieve tools, or execute them. After Choice, the application calls MCP.
*/
import { choice } from '@system-one-ai/core';
import type { EvaluationClient } from '@system-one-ai/core';
import type { Client } from '@modelcontextprotocol/sdk/client';
import { DEFER_TO_LLM } from './04-agent-tool-routing.ts';
import { HIGH_EXECUTE_CONFIDENCE } from './04-three-tier-routing.ts';
export type McpToolLister = Pick<Client, 'listTools'>;
export type McpToolCaller = Pick<Client, 'callTool'>;
export type McpListedTool = Awaited<ReturnType<Client['listTools']>>['tools'][number];
export function mcpToolsToChoiceCriteria(tools: readonly McpListedTool[]) {
return Object.fromEntries([
...tools.map(tool => [
tool.name,
tool.description ?? `MCP tool ${tool.name}. Use only when this tool is the best next action.`,
]),
[DEFER_TO_LLM, 'None of the listed MCP tools fits; continue with the main application LLM'],
]);
}
export async function listMcpTools(mcp: McpToolLister) {
const listed = await mcp.listTools();
return listed.tools;
}
/**
* List MCP tools, then let Choice pick one name. Does not call the tool.
*/
export async function routeMcpTool(
client: EvaluationClient,
mcp: McpToolLister,
message: string,
) {
const tools = await listMcpTools(mcp);
if (tools.length === 0) {
return {
action: 'fallback_llm' as const,
proposedTool: DEFER_TO_LLM,
confidence: undefined,
available: [] as string[],
};
}
const result = await client.evaluate({
state: { message, retrievedTools: tools.map(tool => tool.name) },
questions: {
tool: choice('Choose the next MCP tool. Do not execute it.', mcpToolsToChoiceCriteria(tools)),
},
});
const answer = result.answers.tool;
const available = tools.map(tool => tool.name);
if (
answer.choice === DEFER_TO_LLM
|| answer.confidence === undefined
|| answer.confidence < HIGH_EXECUTE_CONFIDENCE
|| !available.includes(answer.choice)
) {
return {
action: 'fallback_llm' as const,
proposedTool: answer.choice,
confidence: answer.confidence,
available,
};
}
return {
action: 'propose_mcp_tool' as const,
proposedTool: answer.choice,
confidence: answer.confidence,
available,
inputSchema: tools.find(tool => tool.name === answer.choice)?.inputSchema,
};
}
/**
* Application-owned execution step. Arguments come from your code or main LLM,
* not from System One.
*/
export async function callRoutedMcpTool(
mcp: McpToolCaller,
name: string,
args: Record<string, unknown>,
) {
return mcp.callTool({ name, arguments: args });
}
async function main() {
throw new Error(
'This file routes tools from an MCP Client you already connected. There is no System One MCP endpoint.',
);
}
if (process.argv[1] && import.meta.url.endsWith(process.argv[1].split(/[\\/]/).pop() ?? '')) {
main().catch((error: unknown) => {
console.error(error instanceof Error ? error.message : error);
process.exitCode = 1;
});
}
更完整的鉴权、幂等和重试说明，见站内文档 Agent 接入。
表中各方案可以组合使用。Anthropic 的数据针对的是 Claude，不代表其他模型的表现。本站没有做过同类实测。