---
title: "Moderate a Post Before Publishing"
description: "Ask one yes/no question about whether a post breaks a specific rule, point at the rule text in state, and route on the probability instead of a single label."
canonical: "https://system-one.dev/en/use-cases/content-moderation"
language: "en"
---

# Moderate a Post Before Publishing

**Key points**

1. Moderation is not one question: a joke and an attack aimed at a group can look similar in isolation, so name the rule and send it with the post.
2. A probability per post lets you split outcomes into publish, review and block instead of a binary verdict.
3. The model applies your written rule; it does not write the rule, and appeals still need a human.

## Ask about one rule at a time

A generic "is this toxic?" question mixes harassment, spam, self-harm and copyright into one label, and gives an operator nothing to act on. Name the rule — "does this attack a protected group?" — and send the rule text with the post.

The demo on this site shows the difference: a joke about Monday meetings stays up, while abuse aimed at a group is flagged. Same length, same tone, different rule outcome.

## Question shape

```ts
import { booleanQuestion } from '@system-one-ai/core';

export const policyQuestions = {
  groupAttack: booleanQuestion(
    'Does this post attack, dehumanize, or demean a protected group?',
  ),
  spamLink: booleanQuestion(
    'Is this post primarily an unsolicited link or product pitch?',
  ),
};
```

State carries the post text and the policy paragraph the reviewer would cite. Without the rule text, the judgement drifts toward whatever the model absorbed in training.

> Illustrative code, not tested. The band values are placeholders.

```ts
export async function moderate(client, post: { text: string; policy: string; context?: string }) {
  const result = await client.evaluate({ state: post, questions: policyQuestions });
  const attack = result.answers.groupAttack.probability;
  const spam = result.answers.spamLink.probability;

  if (attack >= 0.85) return 'block';
  if (attack >= 0.45 || spam >= 0.7) return 'review';
  return 'publish';
}
```

## Bands, not labels

Each band maps to a different cost of being wrong:

- **Block** only above a high probability, because a wrong block is visible to the author and hard to appeal.
- **Review** in the middle: a queue with the probability attached, so reviewers prioritise.
- **Publish** below the floor, and keep the number for later calibration.

## What it will not do

- **Write your policy.** It applies the text you send; ambiguous rules produce ambiguous verdicts.
- **Read images or video.** Jev reads text and JSON. Moderate the transcript or caption, or use a model that reads images.
- **Handle appeals.** A person decides those, with the probability as context, not as evidence.

For the general pattern of gating actions on a probability, see [agent guardrails](https://system-one.dev/en/blog/agent-guardrails).

## Cost

Every 4KB is one block, and anything smaller still counts as one block. Cost = blocks × questions × the model rate. Two questions on a short post is two credits on Jev.

New accounts start with 100 credits, and credits never expire. See [pricing](https://system-one.dev/en/pricing).

[Run the moderation sample (sign-in required)](https://system-one.dev/en/demo/tweet-moderation) to see a flagged and a passing post.


## Frequently asked questions

### Can one question cover every policy?

No. Name one rule per question; a combined "is this bad" question produces a verdict nobody can appeal or audit.

### Should the policy text go in the question or the state?

Put the rule text in state and keep the question short, so the same question can be evaluated against different policy versions.

### Does it read images and video?

No. It reads text and JSON; pass the caption or transcript, or use an image-capable model.

### How do I set the bands?

Label a sample of real posts with the policy applied, then choose bands that match the cost of a wrong block versus a missed post in your community.