---
title: "Turn Screenshots into Typed JSON for Your Agent"
description: "Turn screenshots into typed JSON fields with Clairic's extract(): text, numbers, dates, with confidence and null when a value is missing."
canonical: "https://system-one.dev/en/use-cases/screenshot-to-json"
language: "en"
---

# Turn Screenshots into Typed JSON for Your Agent

**Key points**\n\n1. Clairic's `extract()` takes up to 4 PNG, JPEG or WebP images as base64 and returns one typed value per field you define.
2. Each answer has a `value`; scalar fields can also carry `confidence` and `pNull`, and a value the image can't support comes back as `null`.
3. Feed the typed fields to Jev or Clef to decide what happens next, for example which team gets a ticket.

To turn a screenshot into JSON, call Clairic's `extract()` with the image as base64, `state` set to an empty string, and the fields you want. You get back one typed value per field, and anything the screenshot doesn't show comes back as `null`.

![Error screenshot converted into typed JSON fields by Clairic](TODO-IMG: Cover. Left: mock app error screenshot; arrow; right: three JSON lines for product / error_time / impact. Designed, mock data only.)

## Why agents need structured input

A screenshot is pixels. Your agent's routing logic can't run an `if` on it. What it needs is a product name, a timestamp and a duration, each with a known type.

That's the "Agent pre-processing" use listed on the [Clairic model page](https://system-one.dev/en/models/clairic): turn the key values into structured data first, then let the agent route or decide on them.

## Define the fields

Describe what you want with `defineFields` and the field factories. This example uses `textField`, `dateTimeField` and `durationField({ unit: 'minutes' })`. Use `instructions` to say what each field means.

Field names must match `[A-Za-z_][A-Za-z0-9_.-]{0,63}`.

| Field | Type | What it holds |
|---|---|---|
| `product` | text | The product named in the error screenshot |
| `error_time` | datetime | When the error happened |
| `impact` | duration (minutes) | How long the problem has lasted |

## Send the image

`ImageInput` takes either `{ mediaType, base64 }` or a base64 data URL. When you send images, `state` can be `''`. Remote image URLs are rejected, so read the file and embed it.

Install the SDK:

```bash
npm install @system-one-ai/core@0.8.0 @system-one-ai/adapter-system-one@0.7.0 @system-one-ai/transport-fetch@0.8.0
```

Save this as `screenshot-to-json.mts` and set `SYSTEM_ONE_API_KEY` in `.env`:

```ts
import { readFile } from 'node:fs/promises';
import {
  createSystemOne, defineFields, textField, dateTimeField, durationField,
  type ImageInput,
} from '@system-one-ai/core';
import { systemOneAdapter } from '@system-one-ai/adapter-system-one';
import { createFetchTransport } from '@system-one-ai/transport-fetch';

const apiKey = process.env.SYSTEM_ONE_API_KEY;
const imagePath = process.argv[2];
if (!apiKey || !imagePath) throw new Error('Set SYSTEM_ONE_API_KEY and supply a PNG path.');

const images: readonly ImageInput[] = [{
  mediaType: 'image/png',
  base64: (await readFile(imagePath)).toString('base64'),
}];

const client = createSystemOne({
  adapter: systemOneAdapter,
  transport: createFetchTransport(),
  apiKey,
  timeoutMs: 30_000,
  maxRetries: 0,
});

const fields = defineFields({
  product: textField({ instructions: 'The product or service named in the error screenshot.' }),
  error_time: dateTimeField({ instructions: 'When the error happened, as shown in the screenshot.' }),
  impact: durationField({ unit: 'minutes', instructions: 'How long the problem has lasted.' }),
});

const result = await client.extract({
  model: 'clairic',
  state: '',
  images,
  fields,
});

console.log({
  model: result.model,
  product: result.answers.product.value,
  errorTime: result.answers.error_time.value,
  impactMinutes: result.answers.impact.value,
  impactPNull: result.answers.impact.pNull,
  truncated: result.truncated,
  usage: result.usage,
});
```

Run it with `node --env-file=.env screenshot-to-json.mts ./error.png`. Every run creates a new billable request.

![Sample error screenshot used as Clairic image input](TODO-IMG: Mock error screenshot showing a product name and a time, but NO duration. Designed.)

[Read the Clairic SDK docs](https://docs.system-one.dev/en/docs/sdk/clairic) *(CTA button)*

## Read the result: value, confidence, pNull, span

Only `value` is guaranteed. Scalar fields can also carry `guess`, `confidence` and `pNull` when the model returns them. When `pNull` is above the null threshold (0.5 by default), `value` is `null`, while `guess` may still hold something.

Here's the shape of the output (illustrative, not a real result):

```json
{
  "model": "clairic",
  "product": "<product name from the screenshot>",
  "errorTime": "<datetime string>",
  "impactMinutes": null,
  "impactPNull": "<0–1, above the null threshold>",
  "truncated": false,
  "usage": { "inputTokens": "<n>", "outputTokens": "<n>", "totalTokens": "<n>" }
}
```

The sample screenshot never says how long the outage lasted, so `impact` comes back `null` instead of a guessed number (`guess` may still have one). Set your own thresholds on `confidence` and `pNull`, and send anything below them to a person. `result.model` may differ from the model you requested, so log both. `truncated` is always a boolean, and `usage` reports token counts.

![Terminal output of extract() showing value and pNull per field](TODO-IMG: Real terminal run of screenshot-to-json.mts, API key masked as sk-****.)

| Key | When it appears | Meaning |
|---|---|---|
| `value` | Always | The typed answer, or `null` |
| `guess` / `confidence` / `pNull` | Scalar fields, when the model returns them | Best guess, confidence, probability the input can't determine the value |
| `span` | Text fields, when there's `state` text | Start and end character positions in `state` |
| `mean` / `median` / `p10` / `p90` / `pBelow` / `pAbove` | Estimate and forecast fields only | Distribution of the estimate |

## Chain with Jev or Clef

The ticket-triage use on the model page works like this: Clairic pulls the product, error time and impact, then Jev or Clef decides which team gets it. Add this after the code above:

```ts
import { choice } from '@system-one-ai/core';

const decision = await client.evaluate({
  model: 'jev',
  state: {
    product: result.answers.product.value,
    error_time: result.answers.error_time.value,
    impact_minutes: result.answers.impact.value,
  },
  questions: {
    team: choice('Choose the team that should handle this error report.', {
      billing: 'Payments and refunds',
      support: 'Technical issues',
      review: 'More information is needed',
    }),
  },
});

console.log(decision.answers.team.choice);
```

Don't pass `questions` to `extract()`, and don't pass `fields` to `evaluate()`. The decision step sees structured values, not the original image. More on this pattern in [extract, then decide](https://system-one.dev/en/blog/clairic-vs-jev-clef), or see [Jev](https://system-one.dev/en/models/jev).

![Flow from screenshot to Clairic fields to a Jev routing decision](TODO-IMG: Flow diagram: screenshot → Clairic extract() → typed fields → Jev evaluate() → team. Designed.)

## Image limits

| Limit | Value |
|---|---|
| Images per request | Up to 4 |
| Formats | PNG, JPEG, WebP |
| Size per image | 4MB |
| Total image size | 8MB |
| Full JSON request body | Up to 13MB |
| How to send | base64 only, no remote URLs |

Clairic is billed once per request based on input size, not per field. See the [Clairic model page](https://system-one.dev/en/models/clairic) for the rate.

[Get an API key](https://system-one.dev/en/signup) *(CTA button)*

[Get an API key](https://system-one.dev/en/signup) *(CTA button, end of page)*

Last verified: 2026-10-22

## Frequently asked questions

### Which image formats does Clairic accept?

PNG, JPEG and WebP, up to 4 images per request, up to 4MB each and 8MB in total.

### Can I pass an image URL?

No. Images must be embedded as base64, either as a data URL or as `{ mediaType, base64 }`. Remote URLs are rejected.

### What comes back when a value can't be read?

The field's value is null when pNull is above the null threshold (0.5 by default). guess may still hold a value, so you can decide whether to show it or ask a human.

### How many images can I send at once?

Up to 4 in one extract() request. When images are present, state can be an empty string.

### How is this different from asking an LLM to output JSON?

Clairic doesn't generate text. It returns one typed value per field, with null for values the input can't support, and confidence and pNull on scalar fields. See [LLM extraction vs Clairic](https://system-one.dev/en/blog/clairic-vs-llm-extraction).

### Does extracting more fields cost more?

No. Clairic is billed once per request from input size, not per field. See the Clairic model page for the rate.