Rerank retrieval results

Start with results from search

A search system has already returned four passages for “How do I track my order?” This lab only scores and sorts those candidates. It does not create a search index. Keep one query per run so passages from unrelated searches are not ranked against each other.

Run the local example

Download the ZIP above, unzip it and open a terminal in that folder. You need Python 3.10 or later; there are no packages to install. dry-run writes request payloads. demo reads the supplied, hand-written responses and runs the application logic. Use a new output folder for each run.

python jev_lab.py --mode dry-run --out requests
python jev_lab.py --mode demo --out demo-run

Pair each passage with the question

doc-a explains how to open Orders and track a shipment. doc-d describes blue hats. Both are evaluated against the same tracking question and the same rubric. Keep each candidate’s ID when sending requests so the result can be joined back to its source.

Read the four-level score

The rubric has levels 0 through 3. Score can return a fractional position between levels; it is not a 0–1 probability of relevance. The client checks the score against its distribution and legend before sorting. The classification bench accepts labels, so use the ranking metrics in manifest.json for this task.

Check the top-two result

The supplied answers give doc-a through doc-d scores of 3, 2, 1 and 0. The first two should be retained. Recall@2 is 2/3 because three passages have a positive relevance grade. NDCG@2 is 1 because the two highest-grade passages are in the right order. These are fixture calculations, not measured Jev results.

python jev_lab.py --mode demo --out ranking-demo
# results.json is sorted by score descending, then stable sample ID.
# manifest.json includes k, recall_at_k and ndcg_at_k.

Compare with the order you already had

Use human relevance grades to calculate the same metrics for the original search results. Then compare the reranked results per query. Include searches with no useful result and ones where the useful passage originally ranked below the cutoff. Report the number of queries as well as the average score.

Score your own candidates with the API

Copy the structure of samples.json into own-data.json and replace the examples and expected labels. Set TYPESAFE_API_KEY in your terminal, then run the command below. Use --model with an available version identifier to repeat an experiment against the same model. Check failures.json before interpreting the results.

python jev_lab.py --mode live --samples own-data.json --out live-run

Check the answer produced from those passages

A better ranking metric is only part of the result. Run the answer-writing model with the original and selected passages. Inspect whether an important source was dropped or an answer became unsupported. Keep the full search results so you can recover omitted evidence.