Flag possible threats for human review

Ask about one policy category

This example asks whether a message contains a direct, credible threat of physical violence. A report quoting someone else’s words needs different treatment from a threat made by the author. Define that distinction in the question. Other categories, such as fraud or harassment, need their own policies.

Run the local example

Download the ZIP above, unzip it and open a terminal in that folder. You need Python 3.10 or later; there are no packages to install. dry-run writes request payloads. demo reads the supplied, hand-written responses and runs the application logic. Use a new output folder for each run.

python jev_lab.py --mode dry-run --out requests
python jev_lab.py --mode demo --out demo-run

Create a review band

The demo maps values at or below 0.2 to allow, at or above 0.8 to flag, and the middle to review. Both flag and review enter a human queue in this starter. Nothing is deleted. The numbers are example settings; the boundary cases test what happens at exactly 0.2 and 0.8.

if p <= low:
    decision = 'allow'
elif p >= high:
    decision = 'flag'
else:
    decision = 'review'
# flag and review both enter the review queue in this starter.

Check the six local outcomes

The supplied responses produce three allow, one review and two flag outcomes. Compare results.json with fixture-responses.json. The cases cover benign wording, a quoted report, ambiguity, a threat and two artificial boundary values. Matching all six expected outcomes confirms these branches, not the model’s threat detection ability.

Choose cutoffs using labeled messages

Have reviewers apply your written policy to representative messages. Try several cutoff pairs on a validation set. Count missed threats, benign messages flagged and total review workload. Keep a separate test set for the final check. Noul has no confidence field; use its noul value in this policy.

Run with your selected cutoffs

Set TYPESAFE_API_KEY, prepare your own samples and replace the numbers below with your validated cutoffs. --calibrated is an explicit opt-in required by this starter. It does not calculate thresholds or verify your evaluation.

python jev_lab.py --mode live --samples own-policy-data.json \
  --low 0.2 --high 0.8 --calibrated --out live-review
# Replace both numbers with your validated thresholds.

Keep failed requests out of the allow path

When a request times out or fails validation, preserve the message and mark it pending according to your policy. The model request is only one part of moderation: reviewers still need a queue, the applicable policy version and a way to correct mistakes.