Measuring relevance with golden query sets and nDCG

"Did that ranking change help?" is usually answered one of two ways: watch click-through rate after shipping, and hope nothing else moved at the same time — or don't measure at all, and rely on someone's gut feeling that the new order "looks better." Neither tells you, before you ship, whether a change is actually an improvement.

What a golden query set is

A golden set is a fixed list of real queries, each with a graded judgment of which documents are relevant and how relevant — typically 0 (irrelevant) through 3 (exactly what the user wanted). Judgments can come from human raters, or be derived from behaviour: a purchase after a search is strong evidence of relevance, a click is weaker evidence, and AACSearch's quality lab can seed a set automatically from click and purchase events before you've labeled anything by hand.

Once you have judgments, you can score any ranking — the current production ranking, a candidate change, last month's ranking — against the same fixed yardstick.

The metrics

nDCG (normalized discounted cumulative gain) rewards relevant documents more the higher they rank, and normalizes against the best possible ranking so scores are comparable across queries with different numbers of relevant documents:

A perfect ranking scores 1.0 regardless of how many relevant documents exist for that query — which is what makes nDCG the right metric to average across a whole query set, unlike raw DCG.

MRR (mean reciprocal rank) answers a narrower question — "how far down did I have to scroll to find a good result?" — using 1/rank of the first relevant hit. It's the right metric when there's usually one correct answer (a support article, a specific product) rather than many acceptable ones.

MAP (mean average precision) and precision/recall@K round out the picture for queries where multiple documents are genuinely relevant and you care about getting most of them into the top K, not just one.

Running an evaluation

A quality set in AACSearch is a named list of queries against a collection, each with its judgments. Starting an evaluation run fires the queries — batched through multi_search — against a given configuration (a specific preset, a candidate search-rules change, or just the live config), scores every metric per query, and aggregates.

Comparing against a baseline

The number on its own is less useful than the delta against a known-good baseline. Every run can be compared against another, producing per-query score changes and a regression gate: if mean nDCG drops by more than a configurable threshold, the run is flagged as a regression before anyone promotes it to production. That comparison is what turns "I think this ranking change is better" into "nDCG@10 improved from 0.79 to 0.84 with zero regressed queries."

Offline plus online

A golden set catches whether a change is correct relative to your judgments. It won't catch a change that's technically correct but that real users respond to differently than your judgments predicted — that's what A/B experiments with confidence intervals are for. Run both: golden query sets as a pre-ship gate, experiments to confirm the win in production.