LLM Model Routing: Judge Difficulty First, Then Send to a Small or Large Model

Jev, Clef Flash and Clef on system-one.dev are all decision models. They only judge (score difficulty, answer yes/no); they don't answer the user and aren't routing targets. Your app connects its own small and large models; System One has no text-generation endpoint. Background: What is a decision model?

Try it online (sign-in required): System One demos

Model routing vs tool routing

Both pick from predefined options and both can use a decision model, but the options and metrics differ:

• Tool routing picks which tool (if any) to call next. Metrics: selection accuracy, missed escalations, argument validity.
• Model routing picks which model handles the request. Microsoft's Microsoft-Decision-1 launch blog describes it as choosing the best model for quality, cost and latency requirements (vendor statement). Metrics: quality, cost, latency.

For tool routing, see Route agent tool calls with a Choice question.

Routing strategies

The research literature groups routers into a handful of strategies. In practice you'll see four:

• Rule-based: route by length, keywords, user tier or request type. Cheap and predictable, but rules are never complete.
• Judge, then route: a judgment step estimates difficulty or type, then picks the model. This article covers this one, using a decision model as the judge.
• Cascade: let the small model answer, check the result, escalate only if it fails. Saves money only if the check is cheap and reliable.
• Managed router: a platform-hosted router model picks the target model for you, usually deployed as a single endpoint.

Real systems combine them: rules catch the obvious cases, a judge handles the rest, and a cascade is the safety net.

Who judges difficulty: four options

• Rules: no extra model call, but they only cover the cases you thought of.
• Another generative LLM call: flexible, but that call has its own cost and latency. If the judge costs as much as the answering model, routing saves little.
• A decision model: returns options, scores or probabilities, never text. Microsoft Learn lists "choosing a model, tool or agent for a request" as a Microsoft-Decision-1 use case and suggests routing or gating with a decision model before a generative model (vendor statement).
• A managed router: no routing logic to write, but you don't define the rules, and the effective context window can be limited by the smallest model in the pool. Compare against your current baseline.

Using a decision model as the router

Choice: options = model tiers

Make model tiers the options and say when to use each, for example:

• small: factual lookups, format conversion, short Q&A
• large: multi-step reasoning, long documents, synthesis across sources
• review: money, legal or medical content that needs a person or the strongest model

Wording is illustrative. Microsoft Learn recommends describing each option clearly and treating a low top probability as a signal to escalate to a person or another model. The Jev Choice docs suggest adding other or none of the above when options may not cover every case.

Score: difficulty levels, mapped by threshold

If difficulty is more of a continuum, use Score with ordered levels (say "answerable in one sentence / needs lookup / needs multi-step reasoning / needs expert judgment") and map the score to a model in code. Microsoft Learn says Score returns a probability-weighted average of level indices that can fall between levels, and recommends using it for relative ranking and thresholds, not as an absolute calibrated grade.

Add a "can't tell" option

Microsoft Learn suggests an abstain option such as cannot tell when the model shouldn't be forced to choose. In routing, "can't tell" can go straight to the default large model or human review. It also recommends clear, neutral wording, optionally randomizing option order, and testing whether order changes the result.

Confidence and thresholds

For Choice and Score, the Jev docs describe confidence as a statistic derived from the distribution, not "the probability of being right", and thresholds should scale with risk. The default stance in routing: when unsure, go to the more capable model. Sending an easy request to a large model only costs money; sending a hard one to a small model can produce a wrong answer. For picking thresholds on labelled samples and confirming on held-out data, see An LLM-as-a-judge alternative, which also has the dev team's "accept when confident, escalate to an LLM" TypeScript example.

Cascade: small model first, Noul check, escalate when needed

Alternatively, skip the difficulty judgment: let the small model answer, then check the result with yes/no questions such as:

• "Does the answer fully address the user's request?"
• "Does the answer contain a refusal, obvious uncertainty or a contradiction?"

Each Noul returns P(true), and code decides whether to keep the small model's answer or escalate. It's the same pattern as AI agent stop conditions: one proposition per question, and anything code can check directly (does the JSON parse, do tests pass) stays out of the model.

One caveat. The research cited in An LLM-as-a-judge alternative found that when the second stage repeats the first stage's mistakes, a cascade mostly saves money without improving accuracy. Routing is no different: if the check fails on the same kinds of requests as the small model, bad answers still get through. Before launch, measure "bad answers that passed the check" on labelled data.

Proving routing saves money without hurting quality

Evaluate along four dimensions:

• Quality: minimum acceptable score or win rate (accuracy, completeness, clarity, usefulness).
• Cost: a per-request cost ceiling, or a minimum saving versus baseline.
• Latency: acceptable median and tail latency (for example p90 or p95).
• Policy: which models, regions and deployment configurations your organization allows.

Also (our recommendations):

• Count end to end: include the routing judgment, cascade checks, repeated calls after escalation and human fallback, not just per-call price.
• Watch the escalation rate: the more you escalate, the less you save.
• Slice by category: if one request type behaves differently, evaluate it separately. Don't route high-risk categories on cost alone.
• Revalidate on any change: a new judge model, answering model, option wording or threshold means rerunning the evaluation.

We haven't measured before/after routing ourselves, so we don't quote a savings figure.

TypeScript: Score for difficulty, Noul to check the small model

From the dev team's examples/decision-articles/07-model-routing.ts. Three steps:

1. A decision model scores difficulty (Score, three levels 0–2). At or above the threshold, the request goes straight to your large model.
2. Otherwise your small model drafts an answer, and the same decision model answers a booleanQuestion: "Does this draft fully and correctly address the request without hedging or refusing?"
3. If P(adequate) meets the threshold, keep the small model's answer; otherwise escalate to the large model.

ts/**
 * Article 6 — Model routing: Score difficulty, then cascade LLMs.
 *
 * System One estimates request difficulty (Score) and whether a small-model
 * draft is adequate (Noul via `booleanQuestion`). It does not run chat
 * completions. Small/large generation stays in application callbacks.
 *
 * The decision-model id comes from GET {baseURL}/models (hosted: GET /v1/models).
 * When catalog entries include `pricing.creditsPerBlock`, this file defaults to
 * the cheapest name (on the current hosted catalog that is typically
 * `typesafe/jev`). It does not invent that id if the catalog omits pricing.
 *
 * Packages: @system-one-ai/core, adapter-system-one, transport-fetch.
 * Evaluate: POST {baseURL}/systemone. Catalog listing is not an SDK method.
 */
import { createSystemOne, score, booleanQuestion } from '@system-one-ai/core';
import type { EvaluationClient } from '@system-one-ai/core';
import { systemOneAdapter } from '@system-one-ai/adapter-system-one';
import { createFetchTransport } from '@system-one-ai/transport-fetch';

export const difficultyQuestion = score('How much generative capacity does this request need?', [
  'A short factual or formatting answer a small model can handle',
  'Some reasoning or multi-step wording; try a small model first',
  'Specialized, high-stakes, or long-form work that needs a larger model',
]);

export const adequacyQuestion = booleanQuestion(
  'Does this draft fully and correctly address the request without hedging or refusal?',
  {
    true: 'The user can use this answer as-is',
    false: 'The answer is incomplete, wrong, hedged, or a refusal',
  },
);

/** Inclusive Score threshold on the 0..2 rubric. Tune on labeled traffic. */
export const LARGE_IF_DIFFICULTY_AT_LEAST = 2;
/** Keep the small draft only when P(adequate) meets this application threshold. */
export const KEEP_SMALL_IF_ADEQUATE_AT_LEAST = 0.7;

export interface CatalogModel {
  name: string;
  aliases: readonly string[];
  pricing?: { creditsPerBlock: number; bytesPerBlock: number };
}

export type LlmSize = 'small' | 'large';

export interface LlmCallbacks {
  completeWithSmallLlm: (prompt: string) => Promise<string>;
  completeWithLargeLlm: (prompt: string) => Promise<string>;
}

function isRecord(value: unknown): value is Record<string, unknown> {
  return typeof value === 'object' && value !== null && !Array.isArray(value);
}

/** Parse the published `{ models: [...] }` envelope. Extra fields are ignored. */
export function parseCatalogModels(payload: unknown): CatalogModel[] {
  if (!isRecord(payload) || !Array.isArray(payload.models)) {
    throw new Error('GET /v1/models must return { models: [...] }.');
  }
  return payload.models.map((entry, index) => {
    if (!isRecord(entry) || typeof entry.name !== 'string' || entry.name.trim() === '') {
      throw new Error(`Catalog model ${index} is missing name.`);
    }
    let pricing: CatalogModel['pricing'];
    if (
      isRecord(entry.pricing)
      && typeof entry.pricing.creditsPerBlock === 'number'
      && typeof entry.pricing.bytesPerBlock === 'number'
    ) {
      pricing = {
        creditsPerBlock: entry.pricing.creditsPerBlock,
        bytesPerBlock: entry.pricing.bytesPerBlock,
      };
    }
    const aliases = Array.isArray(entry.aliases)
      ? entry.aliases.filter((alias): alias is string => typeof alias === 'string' && alias.length > 0)
      : [];
    return { name: entry.name, aliases, pricing };
  });
}

export async function fetchCatalogModels(input: {
  baseURL: string;
  apiKey: string;
  fetchImpl?: typeof fetch;
}) {
  const response = await (input.fetchImpl ?? fetch)(`${input.baseURL.replace(/\/$/, '')}/models`, {
    headers: { Authorization: `Bearer ${input.apiKey}`, Accept: 'application/json' },
  });
  if (!response.ok) {
    throw new Error(`GET /v1/models failed with HTTP ${response.status}.`);
  }
  return parseCatalogModels(await response.json());
}

/** Lowest `pricing.creditsPerBlock`, then name. Undefined when no entry is priced. */
export function pickCheapestCatalogModel(models: readonly CatalogModel[]) {
  const priced = models.filter(model => model.pricing !== undefined);
  if (priced.length === 0) return undefined;
  return priced.reduce((lowest, model) => {
    const current = model.pricing!.creditsPerBlock;
    const best = lowest.pricing!.creditsPerBlock;
    if (current < best) return model;
    if (current === best && model.name < lowest.name) return model;
    return lowest;
  });
}

export function findCatalogModel(models: readonly CatalogModel[], requested: string) {
  const name = requested.trim();
  return models.find(model => model.name === name || model.aliases.includes(name));
}

/**
 * Prefer an explicit catalog name/alias (SYSTEM_ONE_MODEL). Otherwise the
 * cheapest priced entry. Do not invent a hosted id such as typesafe/jev.
 */
export function resolveDecisionModelId(models: readonly CatalogModel[], preferred?: string) {
  if (preferred && preferred.trim() !== '') {
    const match = findCatalogModel(models, preferred);
    if (!match) {
      throw new Error(`SYSTEM_ONE_MODEL ${preferred} is not in GET /v1/models.`);
    }
    return match.name;
  }
  const cheapest = pickCheapestCatalogModel(models);
  if (!cheapest) {
    throw new Error('Set SYSTEM_ONE_MODEL to a name from GET /v1/models (catalog has no pricing to pick a cheapest default).');
  }
  return cheapest.name;
}

export function routeByDifficulty(difficultyScore: number): LlmSize {
  return difficultyScore >= LARGE_IF_DIFFICULTY_AT_LEAST ? 'large' : 'small';
}

export function keepSmallAnswer(adequacyProbability: number) {
  return adequacyProbability >= KEEP_SMALL_IF_ADEQUATE_AT_LEAST;
}

export async function createDecisionClientFromCatalog(options?: { fetchImpl?: typeof fetch }) {
  const apiKey = process.env.SYSTEM_ONE_API_KEY;
  const baseURL = process.env.SYSTEM_ONE_BASE_URL;
  if (!apiKey || !baseURL) {
    throw new Error('Set SYSTEM_ONE_API_KEY and SYSTEM_ONE_BASE_URL (…/v1). Model id is read from GET /v1/models.');
  }
  const models = await fetchCatalogModels({ apiKey, baseURL, fetchImpl: options?.fetchImpl });
  const model = resolveDecisionModelId(models, process.env.SYSTEM_ONE_MODEL);
  return {
    model,
    models,
    client: createSystemOne({
      apiKey,
      baseURL,
      model,
      adapter: systemOneAdapter,
      transport: createFetchTransport(),
      timeoutMs: 30_000,
      maxRetries: 0,
    }),
  };
}

export async function estimateDifficulty(client: EvaluationClient, prompt: string) {
  const result = await client.evaluate({
    state: { prompt },
    questions: { difficulty: difficultyQuestion },
  });
  return {
    difficulty: result.answers.difficulty.score,
    model: result.model,
  };
}

export async function checkSmallAnswer(client: EvaluationClient, prompt: string, draft: string) {
  const result = await client.evaluate({
    state: { prompt, draft },
    questions: { adequate: adequacyQuestion },
  });
  return {
    adequate: result.answers.adequate.probability,
    model: result.model,
  };
}

/**
 * @param llms Your existing small/large completion paths.
 *        System One has no chat/completions URL for either callback.
 */
export async function routeToLlm(
  client: EvaluationClient,
  prompt: string,
  llms: LlmCallbacks,
) {
  const estimated = await estimateDifficulty(client, prompt);
  const lane = routeByDifficulty(estimated.difficulty);
  if (lane === 'large') {
    const text = await llms.completeWithLargeLlm(prompt);
    return {
      action: 'used_large' as const,
      reason: 'difficulty_at_or_above_threshold' as const,
      difficulty: estimated.difficulty,
      decisionModel: estimated.model,
      text,
    };
  }

  const small = await llms.completeWithSmallLlm(prompt);
  const checked = await checkSmallAnswer(client, prompt, small);
  if (keepSmallAnswer(checked.adequate)) {
    return {
      action: 'used_small' as const,
      reason: 'small_answer_adequate' as const,
      difficulty: estimated.difficulty,
      adequate: checked.adequate,
      decisionModel: checked.model,
      text: small,
    };
  }

  const text = await llms.completeWithLargeLlm(prompt);
  return {
    action: 'escalated_large' as const,
    reason: 'small_answer_below_threshold' as const,
    difficulty: estimated.difficulty,
    adequate: checked.adequate,
    decisionModel: checked.model,
    smallDraft: small,
    text,
  };
}

async function main() {
  const prompt = process.env.EXAMPLE_MESSAGE ?? 'How many credits does a single Score question cost?';
  const { client, model } = await createDecisionClientFromCatalog();
  const decision = await routeToLlm(client, prompt, {
    completeWithSmallLlm: async () => {
      throw new Error('No System One LLM-completion endpoint. Wire your small-model call here.');
    },
    completeWithLargeLlm: async () => {
      throw new Error('No System One LLM-completion endpoint. Wire your large-model call here.');
    },
  });
  console.log(JSON.stringify({ catalogModel: model, ...decision }, null, 2));
}

if (process.argv[1] && import.meta.url.endsWith(process.argv[1].split(/[\\/]/).pop() ?? '')) {
  main().catch((error: unknown) => {
    console.error(error instanceof Error ? error.message : error);
    process.exitCode = 1;
  });
}

Notes (from the code and comments):

• No hard-coded decision model ID. The example reads the catalog from GET /v1/models. With SYSTEM_ONE_MODEL set it uses the matching entry; without it, it takes the entry with the lowest pricing.creditsPerBlock. If the catalog has no pricing, it errors instead of inventing an ID. A full model ID selects a model but can't pin a version.
• Your app connects the answering models. completeWithSmallLlm and completeWithLargeLlm are your own calls. System One has no chat/completions endpoint, and the example throws if they aren't wired.
• Thresholds are examples. LARGE_IF_DIFFICULTY_AT_LEAST = 2 (0–2, inclusive) and KEEP_SMALL_IF_ADEQUATE_AT_LEAST = 0.7; tune both on your own labelled traffic.
• Not covered: Choice over model tiers, a "can't tell" option, end-to-end cost accounting.

Billing on the hosted API: every 4KB of request is one block, and anything under 4KB still counts as one; cost = blocks × questions × the model's price per block. Per-block prices on the API page: Jev 1 credit, Clef Flash 3, Clef 6. Credits come in prepaid packs; see the pricing page. Packs are one-time purchases with no auto-renewal. New accounts get 100 free credits on sign-up. The example makes up to two decision-model calls per request (difficulty score + yes/no check), so count both.

No accuracy or latency comparison: the sources we checked only have vendor self-reported numbers, and we haven't measured any.

Related reading

• What is a decision model?
• Jev alternatives in 2026
• An LLM-as-a-judge alternative
• Route agent tool calls with a Choice question
• AI agent stop conditions
• System One primitives docs