Classify support tickets

Decide what counts as a correct label

“The same invoice was paid twice” is billing. “The shoes do not fit” is returns. “Do I pay to return this?” crosses the boundary, so this lab expects review. Agree on these cases before judging a model. Unlike the routing lab, this article focuses on comparing labels, not choosing an application action.

Run the local example

Download the ZIP above, unzip it and open a terminal in that folder. You need Python 3.10 or later; there are no packages to install. dry-run writes request payloads. demo reads the supplied, hand-written responses and runs the application logic. Use a new output folder for each run.

python jev_lab.py --mode dry-run --out requests
python jev_lab.py --mode demo --out demo-run

Read the six example labels

samples.json contains four messages with a team label and two with review. One delivery message is in Chinese. The supplied responses let you check Unicode handling and review outcomes locally; you will need real predictions to evaluate language capability.

Open evaluation.csv in Evaluation Bench

After the demo, import demo-run/evaluation.csv into Evaluation Bench. It has id, expected and predicted columns. The bench compares labels exactly, including review. A label match therefore measures agreement with this lab’s review policy as well as its categories.

id,expected,predicted
en,shipping,shipping
short,review,review

Find a weak category

For billing, precision asks: of the messages labeled billing, how many belonged there? Recall asks: of all billing messages, how many did we find? The bench shows these values for configuration A. Also count review outcomes separately; the bench does not automatically calculate an automation coverage rate.

Compare two sets of predictions

Run the same messages with rubric A and rubric B. Join by id, putting B’s labels in predicted_b. Do not join by row position: a failed request can shift the rows. Use a separate set to choose wording and cutoffs, and keep expected labels out of the request state.

Try your own messages with the API

Copy the structure of samples.json into own-data.json and replace the examples and expected labels. Set TYPESAFE_API_KEY in your terminal, then run the command below. Use --model with an available version identifier to repeat an experiment against the same model. Check failures.json before interpreting the results.

python jev_lab.py --mode live --samples own-data.json --out live-run

Read the disagreements before choosing a version

Start with messages where A and B differ. Would the proposed change fix a common mistake or only one unusual example? Check whether either version pushes more work into review. Use new messages for the final check after editing the rubric.