Introduction
Adopting a library metric takes a minute. What takes longer — and is the part worth doing — is setting the passing criteria, because the library supplies the measurement and you supply the standard.
Adding one
Open the library and filter by test case type — A metric built for conversational cases will not score single-turn ones.
Check its parameters against your dataset — It scores only cases carrying every parameter it declares. This is the check that prevents a metric quietly returning nothing.
Add it — You now have your own copy of the metric.
Set the passing criteria — Two levels, and this is the real work — see below.
Test it before relying on it — Run it against cases whose right answer you already know.
What you get is your copy
Adding a metric creates a metric belonging to you, carrying a reference back to the library record it came from. The practical consequences:
You can change it — criteria, thresholds, parameters — without affecting anyone else.
Two teams adopting the same library metric get two metrics, which will drift as each is tuned. That is usually right; different teams have different standards.
Your copy does not automatically inherit library updates. Worth knowing if you adopted something early and want a later improvement.
The passing criteria are yours, and they are the standard
A library metric measures something. What counts as good enough is your decision, expressed at two levels the product keeps separate:
Level | The product says | Decides |
| "Define passing criteria for a particular test case" | Whether one case passed |
| "Define passing criteria for an experiment" | How many cases must pass for the run to |
Note: The second level is where most of the judgement lives, and it is the one people skip. Requiring 100% of cases to pass makes a run fail on a single edge case. Requiring 60% means a run can pass with four in ten answers wrong. Neither is obviously right — but leaving it at whatever it defaults to means you never made the decision.
Thresholds on a scale metric
A scale metric returns a number, and a number is not a verdict until you say where the line is. Set it by looking at real scores rather than by picking a round figure: run the metric over cases you have judged yourself and put the threshold where it agrees with you.
A threshold chosen before seeing any scores is a guess, and it will either pass everything or fail everything for the first few runs.
"The library tells you how to measure. It cannot tell you what you are willing to ship."