Turn Screenshots into Typed JSON for Your Agent


Why agents need structured input

A screenshot is pixels. Your agent's routing logic can't run an if on it. What it needs is a product name, a timestamp and a duration, each with a known type.

That's the "Agent pre-processing" use listed on the Clairic model page: turn the key values into structured data first, then let the agent route or decide on them.

Define the fields

Describe what you want with defineFields and the field factories. This example uses textField, dateTimeField and durationField({ unit: 'minutes' }). Use instructions to say what each field means.

Field names must match [A-Za-z_][A-Za-z0-9_.-]{0,63}.

Send the image

ImageInput takes either { mediaType, base64 } or a base64 data URL. When you send images, state can be ''. Remote image URLs are rejected, so read the file and embed it.

Install the SDK:

bashnpm install @system-one-ai/core@0.8.0 @system-one-ai/adapter-system-one@0.7.0 @system-one-ai/transport-fetch@0.8.0

Save this as screenshot-to-json.mts and set SYSTEM_ONE_API_KEY in .env:

tsimport { readFile } from 'node:fs/promises';
import {
  createSystemOne, defineFields, textField, dateTimeField, durationField,
  type ImageInput,
} from '@system-one-ai/core';
import { systemOneAdapter } from '@system-one-ai/adapter-system-one';
import { createFetchTransport } from '@system-one-ai/transport-fetch';

const apiKey = process.env.SYSTEM_ONE_API_KEY;
const imagePath = process.argv[2];
if (!apiKey || !imagePath) throw new Error('Set SYSTEM_ONE_API_KEY and supply a PNG path.');

const images: readonly ImageInput[] = [{
  mediaType: 'image/png',
  base64: (await readFile(imagePath)).toString('base64'),
}];

const client = createSystemOne({
  adapter: systemOneAdapter,
  transport: createFetchTransport(),
  apiKey,
  timeoutMs: 30_000,
  maxRetries: 0,
});

const fields = defineFields({
  product: textField({ instructions: 'The product or service named in the error screenshot.' }),
  error_time: dateTimeField({ instructions: 'When the error happened, as shown in the screenshot.' }),
  impact: durationField({ unit: 'minutes', instructions: 'How long the problem has lasted.' }),
});

const result = await client.extract({
  model: 'clairic',
  state: '',
  images,
  fields,
});

console.log({
  model: result.model,
  product: result.answers.product.value,
  errorTime: result.answers.error_time.value,
  impactMinutes: result.answers.impact.value,
  impactPNull: result.answers.impact.pNull,
  truncated: result.truncated,
  usage: result.usage,
});

Run it with node --env-file=.env screenshot-to-json.mts ./error.png. Every run creates a new billable request.


Read the Clairic SDK docs

Read the result: value, confidence, pNull, span

Only value is guaranteed. Scalar fields can also carry guess, confidence and pNull when the model returns them. When pNull is above the null threshold (0.5 by default), value is null, while guess may still hold something.

Here's the shape of the output (illustrative, not a real result):

json{
  "model": "clairic",
  "product": "<product name from the screenshot>",
  "errorTime": "<datetime string>",
  "impactMinutes": null,
  "impactPNull": "<0–1, above the null threshold>",
  "truncated": false,
  "usage": { "inputTokens": "<n>", "outputTokens": "<n>", "totalTokens": "<n>" }
}

The sample screenshot never says how long the outage lasted, so impact comes back null instead of a guessed number (guess may still have one). Set your own thresholds on confidence and pNull, and send anything below them to a person. result.model may differ from the model you requested, so log both. truncated is always a boolean, and usage reports token counts.


Chain with Jev or Clef

The ticket-triage use on the model page works like this: Clairic pulls the product, error time and impact, then Jev or Clef decides which team gets it. Add this after the code above:

tsimport { choice } from '@system-one-ai/core';

const decision = await client.evaluate({
  model: 'jev',
  state: {
    product: result.answers.product.value,
    error_time: result.answers.error_time.value,
    impact_minutes: result.answers.impact.value,
  },
  questions: {
    team: choice('Choose the team that should handle this error report.', {
      billing: 'Payments and refunds',
      support: 'Technical issues',
      review: 'More information is needed',
    }),
  },
});

console.log(decision.answers.team.choice);

Don't pass questions to extract(), and don't pass fields to evaluate(). The decision step sees structured values, not the original image. More on this pattern in extract, then decide, or see Jev.

 → typed fields → Jev evaluate() → team. Designed.)

Image limits

Clairic is billed once per request based on input size, not per field. See the Clairic model page for the rate.

Get an API key

Get an API key

Last verified: 2026-10-22