Scores and report metrics

Interpret verifier readings, success statistics, reliability, and resource cost.

On this page

Facts, scores, and verdicts

A Verifier records facts. Its projection may reduce those facts to a numeric score, and its decision rule turns the result into pass, fail, or an indeterminate outcome. Each report snapshot carries a success policy that classifies rollouts from Verifier verdicts or scores; report statistics aggregate those classifications. Registry UI separately projects terminal results into Passed, Failed, or Unscorable.

Statistic reference

Statistic	Meaning
`pass@k`	Estimated probability that at least one of k sampled rollouts passes.
`pass^k`	Estimated probability that all k sampled rollouts pass; a reliability measure.
Pass rate	Observed share of requested rollouts that passed.
Learnability	`4 × pass_rate × (1 − pass_rate)`; highest near a 50% pass rate and zero at always-pass or always-fail.
Mean score	Mean of per-rollout mean verifier scores over rollouts that produced one.
Best / worst of k	Expected best or worst score among k sampled rollouts.
Mean duration	Mean wall-clock duration over rollouts that recorded it.
Mean total tokens	Mean input-plus-output tokens over rollouts whose agent reported usage.
Metric mean / variance	Per Task, mean or population variance of one unambiguous numeric `(Verifier, subject, metric)` series across that Task's rollouts. Multiple series within a Task are refused.
Agreement	Per Task, the modal-value share for one unambiguous metric series; omitted until at least two readings exist.

Run-level metric statistics are unweighted means across Tasks that produced the label.

Generate a CLI report

Terminal

fleet-runtime runs report run-... \
  --statistic pass@1 \
  --statistic pass@5 \
  --statistic pass^5 \
  --statistic mean_score \
  --statistic mean_total_tokens

Choose k no larger than the Run's rollouts per Task. Score-based best_of_k and worst_of_k also require at least kobserved scores or their labels are omitted.

Success policies

Policy	Meaning
`verifier_verdict.v1`	The current Runtime report default: a succeeded rollout passes only when at least one Verifier verdict was recorded and every recorded verdict passes. Indeterminate, a legacy indeterminate outcome such as `error`, and an empty verdict set are non-pass outcomes; frozen `passed`/`failed` aliases retain their determinate meaning.
`score_threshold.v1`	Pass only when no verifier verdict is indeterminate, at least one verifier score exists, and the mean recorded verifier score reaches the configured threshold.

Registry UI and Runtime reports are separate projections

Missing data is not zero

Score-, duration-, token-, and metric-based statistics are omitted when they lack the observations their estimator requires. Pass-based statistics instead use the Run's requested rollout count: unfinished, cancelled, or missing outcomes contribute no successes rather than disappearing, so zero observed successes produces zero.