jevRead the exact input that failed
Start with the state actually sent, not the original message in your database. Preprocessing may have removed the order ID, quote marker or preceding sentence needed to interpret it.
Ask a person to apply the same rubric
For “Do I pay to return this?”, would the written rules select billing, returns or review? If reviewers disagree, clarify that boundary first. Moving the confidence cutoff cannot repair an undefined category policy.
Check whether code should make this decision
Use code for exact arithmetic, date comparisons and known mappings. The official model-limitations page describes weaknesses that are useful sources of test cases. Keep the model’s job focused on judgments that need it.
Change one thing, then check new examples
Try a clearer category description or a different cutoff on a validation set. Inspect false positives and false negatives separately. Once you choose a version, test it on unseen examples. A fix that only memorizes the failures you edited against may not hold up.
Request example
# Keep these with each result: # sample_id, rubric_version, model, expected, predicted, raw_answer