---
title: "LLM Model Routing: Judge Difficulty First, Then Send to a Small or Large Model"
description: "Sending every request to your biggest model wastes money, and asking an LLM to route is slow. Use a decision model to score difficulty, route by threshold and escalate."
canonical: "https://system-one.dev/en/blog/llm-model-routing"
language: "en"
---

# LLM Model Routing: Judge Difficulty First, Then Send to a Small or Large Model

**Short answer:** model routing means judging a request's type, difficulty and risk *before* it reaches a model, then sending it to the right one: small models for simple requests, large or reasoning models for hard or risky ones. The hard part is deciding who judges difficulty. Rules are too coarse, and calling another LLM to judge eats the savings. With a decision model, the judgment becomes a Choice or Score question with probabilities, and your code routes by threshold. After the small model answers, a yes/no (Noul) check decides whether to keep the answer or escalate to a stronger model.

Jev, Clef Flash and Clef on system-one.dev are **all decision models**. They only judge (score difficulty, answer yes/no); they don't answer the user and aren't routing targets. Your app connects its own small and large models; System One has no text-generation endpoint. Background: [What is a decision model?](https://system-one.dev/en/blog/what-is-a-decision-model)

**Try it online (sign-in required):** [System One demos](https://system-one.dev/en/demo)

## Model routing vs tool routing

Both pick from predefined options and both can use a decision model, but the options and metrics differ:

- **Tool routing** picks which tool (if any) to call next. Metrics: selection accuracy, missed escalations, argument validity.
- **Model routing** picks which model handles the request. Microsoft's Microsoft-Decision-1 launch blog describes it as choosing the best model for quality, cost and latency requirements (vendor statement). Metrics: quality, cost, latency.

For tool routing, see [Route agent tool calls with a Choice question](https://system-one.dev/en/blog/agent-tool-routing).

## Routing strategies

The research literature groups routers into a handful of strategies. In practice you'll see four:

- **Rule-based:** route by length, keywords, user tier or request type. Cheap and predictable, but rules are never complete.
- **Judge, then route:** a judgment step estimates difficulty or type, then picks the model. This article covers this one, using a decision model as the judge.
- **Cascade:** let the small model answer, check the result, escalate only if it fails. Saves money only if the check is cheap and reliable.
- **Managed router:** a platform-hosted router model picks the target model for you, usually deployed as a single endpoint.

Real systems combine them: rules catch the obvious cases, a judge handles the rest, and a cascade is the safety net.

## Who judges difficulty: four options

- **Rules:** no extra model call, but they only cover the cases you thought of.
- **Another generative LLM call:** flexible, but that call has its own cost and latency. If the judge costs as much as the answering model, routing saves little.
- **A decision model:** returns options, scores or probabilities, never text. Microsoft Learn lists "choosing a model, tool or agent for a request" as a Microsoft-Decision-1 use case and suggests routing or gating with a decision model before a generative model (vendor statement).
- **A managed router:** no routing logic to write, but you don't define the rules, and the effective context window can be limited by the smallest model in the pool. Compare against your current baseline.

## Using a decision model as the router

### Choice: options = model tiers

Make model tiers the options and say when to use each, for example:

- `small`: factual lookups, format conversion, short Q&A
- `large`: multi-step reasoning, long documents, synthesis across sources
- `review`: money, legal or medical content that needs a person or the strongest model

Wording is illustrative. Microsoft Learn recommends describing each option clearly and treating a low top probability as a signal to escalate to a person or another model. The Jev Choice docs suggest adding `other` or `none of the above` when options may not cover every case.

### Score: difficulty levels, mapped by threshold

If difficulty is more of a continuum, use Score with ordered levels (say "answerable in one sentence / needs lookup / needs multi-step reasoning / needs expert judgment") and map the score to a model in code. Microsoft Learn says Score returns a probability-weighted average of level indices that can fall between levels, and recommends using it for relative ranking and thresholds, not as an absolute calibrated grade.

### Add a "can't tell" option

Microsoft Learn suggests an abstain option such as `cannot tell` when the model shouldn't be forced to choose. In routing, "can't tell" can go straight to the default large model or human review. It also recommends clear, neutral wording, optionally randomizing option order, and testing whether order changes the result.

### Confidence and thresholds

For Choice and Score, the Jev docs describe confidence as a statistic derived from the distribution, not "the probability of being right", and thresholds should scale with risk. The default stance in routing: **when unsure, go to the more capable model.** Sending an easy request to a large model only costs money; sending a hard one to a small model can produce a wrong answer. For picking thresholds on labelled samples and confirming on held-out data, see [An LLM-as-a-judge alternative](https://system-one.dev/en/blog/llm-as-a-judge-alternative), which also has the dev team's "accept when confident, escalate to an LLM" TypeScript example.

## Cascade: small model first, Noul check, escalate when needed

Alternatively, skip the difficulty judgment: let the small model answer, then check the result with yes/no questions such as:

- "Does the answer fully address the user's request?"
- "Does the answer contain a refusal, obvious uncertainty or a contradiction?"

Each Noul returns P(true), and code decides whether to keep the small model's answer or escalate. It's the same pattern as [AI agent stop conditions](https://system-one.dev/en/blog/agent-stop-condition): one proposition per question, and anything code can check directly (does the JSON parse, do tests pass) stays out of the model.

One caveat. The research cited in [An LLM-as-a-judge alternative](https://system-one.dev/en/blog/llm-as-a-judge-alternative) found that when the second stage repeats the first stage's mistakes, a cascade mostly saves money without improving accuracy. Routing is no different: if the check fails on the same kinds of requests as the small model, bad answers still get through. Before launch, measure "bad answers that passed the check" on labelled data.

## Proving routing saves money without hurting quality

Evaluate along four dimensions:

- **Quality:** minimum acceptable score or win rate (accuracy, completeness, clarity, usefulness).
- **Cost:** a per-request cost ceiling, or a minimum saving versus baseline.
- **Latency:** acceptable median and tail latency (for example p90 or p95).
- **Policy:** which models, regions and deployment configurations your organization allows.

Also (our recommendations):

- **Count end to end:** include the routing judgment, cascade checks, repeated calls after escalation and human fallback, not just per-call price.
- **Watch the escalation rate:** the more you escalate, the less you save.
- **Slice by category:** if one request type behaves differently, evaluate it separately. Don't route high-risk categories on cost alone.
- **Revalidate on any change:** a new judge model, answering model, option wording or threshold means rerunning the evaluation.

We haven't measured before/after routing ourselves, so we don't quote a savings figure.

## TypeScript: Score for difficulty, Noul to check the small model

From the dev team's `examples/decision-articles/07-model-routing.ts`. Three steps:

1. A decision model scores difficulty (Score, three levels 0–2). At or above the threshold, the request goes straight to your large model.
2. Otherwise your small model drafts an answer, and the same decision model answers a `booleanQuestion`: "Does this draft fully and correctly address the request without hedging or refusing?"
3. If P(adequate) meets the threshold, keep the small model's answer; otherwise escalate to the large model.

```ts
/**
 * Article 6 — Model routing: Score difficulty, then cascade LLMs.
 *
 * System One estimates request difficulty (Score) and whether a small-model
 * draft is adequate (Noul via `booleanQuestion`). It does not run chat
 * completions. Small/large generation stays in application callbacks.
 *
 * The decision-model id comes from GET {baseURL}/models (hosted: GET /v1/models).
 * When catalog entries include `pricing.creditsPerBlock`, this file defaults to
 * the cheapest name (on the current hosted catalog that is typically
 * `typesafe/jev`). It does not invent that id if the catalog omits pricing.
 *
 * Packages: @system-one-ai/core, adapter-system-one, transport-fetch.
 * Evaluate: POST {baseURL}/systemone. Catalog listing is not an SDK method.
 */
import { createSystemOne, score, booleanQuestion } from '@system-one-ai/core';
import type { EvaluationClient } from '@system-one-ai/core';
import { systemOneAdapter } from '@system-one-ai/adapter-system-one';
import { createFetchTransport } from '@system-one-ai/transport-fetch';

export const difficultyQuestion = score('How much generative capacity does this request need?', [
  'A short factual or formatting answer a small model can handle',
  'Some reasoning or multi-step wording; try a small model first',
  'Specialized, high-stakes, or long-form work that needs a larger model',
]);

export const adequacyQuestion = booleanQuestion(
  'Does this draft fully and correctly address the request without hedging or refusal?',
  {
    true: 'The user can use this answer as-is',
    false: 'The answer is incomplete, wrong, hedged, or a refusal',
  },
);

/** Inclusive Score threshold on the 0..2 rubric. Tune on labeled traffic. */
export const LARGE_IF_DIFFICULTY_AT_LEAST = 2;
/** Keep the small draft only when P(adequate) meets this application threshold. */
export const KEEP_SMALL_IF_ADEQUATE_AT_LEAST = 0.7;

export interface CatalogModel {
  name: string;
  aliases: readonly string[];
  pricing?: { creditsPerBlock: number; bytesPerBlock: number };
}

export type LlmSize = 'small' | 'large';

export interface LlmCallbacks {
  completeWithSmallLlm: (prompt: string) => Promise<string>;
  completeWithLargeLlm: (prompt: string) => Promise<string>;
}

function isRecord(value: unknown): value is Record<string, unknown> {
  return typeof value === 'object' && value !== null && !Array.isArray(value);
}

/** Parse the published `{ models: [...] }` envelope. Extra fields are ignored. */
export function parseCatalogModels(payload: unknown): CatalogModel[] {
  if (!isRecord(payload) || !Array.isArray(payload.models)) {
    throw new Error('GET /v1/models must return { models: [...] }.');
  }
  return payload.models.map((entry, index) => {
    if (!isRecord(entry) || typeof entry.name !== 'string' || entry.name.trim() === '') {
      throw new Error(`Catalog model ${index} is missing name.`);
    }
    let pricing: CatalogModel['pricing'];
    if (
      isRecord(entry.pricing)
      && typeof entry.pricing.creditsPerBlock === 'number'
      && typeof entry.pricing.bytesPerBlock === 'number'
    ) {
      pricing = {
        creditsPerBlock: entry.pricing.creditsPerBlock,
        bytesPerBlock: entry.pricing.bytesPerBlock,
      };
    }
    const aliases = Array.isArray(entry.aliases)
      ? entry.aliases.filter((alias): alias is string => typeof alias === 'string' && alias.length > 0)
      : [];
    return { name: entry.name, aliases, pricing };
  });
}

export async function fetchCatalogModels(input: {
  baseURL: string;
  apiKey: string;
  fetchImpl?: typeof fetch;
}) {
  const response = await (input.fetchImpl ?? fetch)(`${input.baseURL.replace(/\/$/, '')}/models`, {
    headers: { Authorization: `Bearer ${input.apiKey}`, Accept: 'application/json' },
  });
  if (!response.ok) {
    throw new Error(`GET /v1/models failed with HTTP ${response.status}.`);
  }
  return parseCatalogModels(await response.json());
}

/** Lowest `pricing.creditsPerBlock`, then name. Undefined when no entry is priced. */
export function pickCheapestCatalogModel(models: readonly CatalogModel[]) {
  const priced = models.filter(model => model.pricing !== undefined);
  if (priced.length === 0) return undefined;
  return priced.reduce((lowest, model) => {
    const current = model.pricing!.creditsPerBlock;
    const best = lowest.pricing!.creditsPerBlock;
    if (current < best) return model;
    if (current === best && model.name < lowest.name) return model;
    return lowest;
  });
}

export function findCatalogModel(models: readonly CatalogModel[], requested: string) {
  const name = requested.trim();
  return models.find(model => model.name === name || model.aliases.includes(name));
}

/**
 * Prefer an explicit catalog name/alias (SYSTEM_ONE_MODEL). Otherwise the
 * cheapest priced entry. Do not invent a hosted id such as typesafe/jev.
 */
export function resolveDecisionModelId(models: readonly CatalogModel[], preferred?: string) {
  if (preferred && preferred.trim() !== '') {
    const match = findCatalogModel(models, preferred);
    if (!match) {
      throw new Error(`SYSTEM_ONE_MODEL ${preferred} is not in GET /v1/models.`);
    }
    return match.name;
  }
  const cheapest = pickCheapestCatalogModel(models);
  if (!cheapest) {
    throw new Error('Set SYSTEM_ONE_MODEL to a name from GET /v1/models (catalog has no pricing to pick a cheapest default).');
  }
  return cheapest.name;
}

export function routeByDifficulty(difficultyScore: number): LlmSize {
  return difficultyScore >= LARGE_IF_DIFFICULTY_AT_LEAST ? 'large' : 'small';
}

export function keepSmallAnswer(adequacyProbability: number) {
  return adequacyProbability >= KEEP_SMALL_IF_ADEQUATE_AT_LEAST;
}

export async function createDecisionClientFromCatalog(options?: { fetchImpl?: typeof fetch }) {
  const apiKey = process.env.SYSTEM_ONE_API_KEY;
  const baseURL = process.env.SYSTEM_ONE_BASE_URL;
  if (!apiKey || !baseURL) {
    throw new Error('Set SYSTEM_ONE_API_KEY and SYSTEM_ONE_BASE_URL (…/v1). Model id is read from GET /v1/models.');
  }
  const models = await fetchCatalogModels({ apiKey, baseURL, fetchImpl: options?.fetchImpl });
  const model = resolveDecisionModelId(models, process.env.SYSTEM_ONE_MODEL);
  return {
    model,
    models,
    client: createSystemOne({
      apiKey,
      baseURL,
      model,
      adapter: systemOneAdapter,
      transport: createFetchTransport(),
      timeoutMs: 30_000,
      maxRetries: 0,
    }),
  };
}

export async function estimateDifficulty(client: EvaluationClient, prompt: string) {
  const result = await client.evaluate({
    state: { prompt },
    questions: { difficulty: difficultyQuestion },
  });
  return {
    difficulty: result.answers.difficulty.score,
    model: result.model,
  };
}

export async function checkSmallAnswer(client: EvaluationClient, prompt: string, draft: string) {
  const result = await client.evaluate({
    state: { prompt, draft },
    questions: { adequate: adequacyQuestion },
  });
  return {
    adequate: result.answers.adequate.probability,
    model: result.model,
  };
}

/**
 * @param llms Your existing small/large completion paths.
 *        System One has no chat/completions URL for either callback.
 */
export async function routeToLlm(
  client: EvaluationClient,
  prompt: string,
  llms: LlmCallbacks,
) {
  const estimated = await estimateDifficulty(client, prompt);
  const lane = routeByDifficulty(estimated.difficulty);
  if (lane === 'large') {
    const text = await llms.completeWithLargeLlm(prompt);
    return {
      action: 'used_large' as const,
      reason: 'difficulty_at_or_above_threshold' as const,
      difficulty: estimated.difficulty,
      decisionModel: estimated.model,
      text,
    };
  }

  const small = await llms.completeWithSmallLlm(prompt);
  const checked = await checkSmallAnswer(client, prompt, small);
  if (keepSmallAnswer(checked.adequate)) {
    return {
      action: 'used_small' as const,
      reason: 'small_answer_adequate' as const,
      difficulty: estimated.difficulty,
      adequate: checked.adequate,
      decisionModel: checked.model,
      text: small,
    };
  }

  const text = await llms.completeWithLargeLlm(prompt);
  return {
    action: 'escalated_large' as const,
    reason: 'small_answer_below_threshold' as const,
    difficulty: estimated.difficulty,
    adequate: checked.adequate,
    decisionModel: checked.model,
    smallDraft: small,
    text,
  };
}

async function main() {
  const prompt = process.env.EXAMPLE_MESSAGE ?? 'How many credits does a single Score question cost?';
  const { client, model } = await createDecisionClientFromCatalog();
  const decision = await routeToLlm(client, prompt, {
    completeWithSmallLlm: async () => {
      throw new Error('No System One LLM-completion endpoint. Wire your small-model call here.');
    },
    completeWithLargeLlm: async () => {
      throw new Error('No System One LLM-completion endpoint. Wire your large-model call here.');
    },
  });
  console.log(JSON.stringify({ catalogModel: model, ...decision }, null, 2));
}

if (process.argv[1] && import.meta.url.endsWith(process.argv[1].split(/[\\/]/).pop() ?? '')) {
  main().catch((error: unknown) => {
    console.error(error instanceof Error ? error.message : error);
    process.exitCode = 1;
  });
}
```

Notes (from the code and comments):

- **No hard-coded decision model ID.** The example reads the catalog from `GET /v1/models`. With `SYSTEM_ONE_MODEL` set it uses the matching entry; without it, it takes the entry with the lowest `pricing.creditsPerBlock`. If the catalog has no pricing, it errors instead of inventing an ID. A full model ID selects a model but can't pin a version.
- **Your app connects the answering models.** `completeWithSmallLlm` and `completeWithLargeLlm` are your own calls. System One has no chat/completions endpoint, and the example throws if they aren't wired.
- **Thresholds are examples.** `LARGE_IF_DIFFICULTY_AT_LEAST = 2` (0–2, inclusive) and `KEEP_SMALL_IF_ADEQUATE_AT_LEAST = 0.7`; tune both on your own labelled traffic.
- **Not covered:** Choice over model tiers, a "can't tell" option, end-to-end cost accounting.

**Billing on the hosted API:** every 4KB of request is one block, and anything under 4KB still counts as one; cost = blocks × questions × the model's price per block. Per-block prices on the API page: Jev 1 credit, Clef Flash 3, Clef 6. Credits come in prepaid packs; see the [pricing page](https://system-one.dev/en/pricing). Packs are one-time purchases with no auto-renewal. New accounts get 100 free credits on sign-up. The example makes up to two decision-model calls per request (difficulty score + yes/no check), so count both.

> No accuracy or latency comparison: the sources we checked only have vendor self-reported numbers, and we haven't measured any.

## Related reading
- [What is a decision model?](https://system-one.dev/en/blog/what-is-a-decision-model)
- [Jev alternatives in 2026](https://system-one.dev/en/blog/jev-alternatives)
- [An LLM-as-a-judge alternative](https://system-one.dev/en/blog/llm-as-a-judge-alternative)
- [Route agent tool calls with a Choice question](https://system-one.dev/en/blog/agent-tool-routing)
- [AI agent stop conditions](https://system-one.dev/en/blog/agent-stop-condition)
- [System One primitives docs](https://docs.system-one.dev/en/docs/primitives)


## Judging approaches compared (checked 2026-10-11)

| Approach | How it judges | Extra call | Control | Source |
| --- | --- | --- | --- | --- |
| Rules | Length, keywords, request type, etc. | None | You define them, but coverage is incomplete | Our summary |
| Generative LLM judge | An LLM reads the request and names a model | One generative call | Free-text output you must parse | Our summary |
| Decision model (e.g. Microsoft-Decision-1)<br><small>[Microsoft Learn](https://learn.microsoft.com/en-us/azure/foundry/foundry-models/how-to/use-foundry-models-microsoft-decision) · 2026-10-11</small> | choice over models or score levels; per-option probabilities | One decision call; Microsoft-Decision-1 lists $0.042 per 1M input tokens, output free<br><small>[Microsoft launch blog](https://commandline.microsoft.com/microsoft-decision-1-model-foundry/) · 2026-10-11</small> | You define options, thresholds and abstain rules | Official docs (vendor statement) |
| Managed router | A trained router model picks the target model per request | Handled by the platform as one deployment | You pick a mode and a model subset; context window limited by the smallest model | Our summary |
| System One hosted decision API<br><small>[System One API](https://system-one.dev/en/api) · 2026-10-11</small> | `choice()`, `score()`, `booleanQuestion()` defined in the SDK | One decision call, billed in credits | Jev, Clef Flash and Clef are all decision models; your app connects the answering models | Official page |
## Frequently asked questions

### What is LLM model routing?

Judging a request's type, difficulty and risk before it reaches a model, then sending it to the right one: small models for simple requests, large ones for complex or risky ones. Like tool routing it's a choice among predefined options, but it's judged on quality, cost and latency.

### Is it worth using one model to decide which model to use?

It depends on what the judgment costs. If the judge is as expensive as the answering model, savings are hard to get. A decision model only returns options and probabilities, so each call is small. Work out end-to-end cost on your own traffic.

### How do I route with Choice or Score?

With Choice, options are model tiers (small / large / review), each with a "when to use" line. With Score, define difficulty levels and map them to models in code. Either way, add a "can't tell" option and route uncertain requests to the stronger model.

### How do I decide whether to escalate after the small model answers?

Check the answer with yes/no questions ("fully addresses the request?", "contains a refusal or obvious uncertainty?"), each with a probability, and let code decide by threshold. Prefer code for anything verifiable (format, tests).

### Can I route high-risk requests to a small model to save money?

Don't decide on cost alone. Microsoft recommends meaningful human review for consequential decisions about people (credit, employment, housing, health, legal). Give high-risk categories their own rule that sends them to a stronger model or a person.

### Managed router or build my own?

A managed router saves you writing routing logic, but you don't define the rules and you're bound by the platform's model pool and context window. With your own router, options, thresholds and escalation live in your code and can be tuned to business risk. Either way, compare quality, cost and latency against your current baseline.

### Do I need to revalidate thresholds after changing models?

Yes. A new judge model, answering model, option wording or threshold means re-evaluating on labelled data. Confidence algorithms also differ between decision models.