Common problems
| Symptom | Check | Action |
|---|---|---|
| Hosted worker cannot resolve an image | For fleet-runtime runs submit --target, was the exact published TaskSet re-pulled? | Pull the flt://… release and submit that alias. Registry UI and flt experiments submit do not require a client-side re-pull. |
| Create Experiment is disabled | Result project, agent fields, rollout limit, and every required binding. | Resolve the inline field message; an unchanged retry will be refused again. |
| Agent credential is missing | The harness variable and candidate credential source. | Select a project credential or use --harness-var NAME=@. |
| A tag appears to have changed content | Compare root digests. | Pin a digest and publish changed content under a new release tag. |
| A succeeded rollout is not Passed | Verifier outcomes and re-verification runs. | Repair the grading input or credential; do not rerun the agent unless execution itself was wrong. |
| A succeeded rollout is Unscorable | Verifier execution state, recorded score, verdict, and required bindings. | A missing score is not zero. Fix the grading input; Re-verify when the same sealed evidence is sufficient, or create a fresh Experiment when Task content or its required bindings changed. |
| Run is Active but no new model trace appears | Rollout states and journal claims or results. | Use Runtime lifecycle records for progress. Trace Pending should be short-lived; if it persists after the Attempt seals, diagnose trace delivery or indexing. |
| Environment never becomes ready | Environment-open trace, readiness endpoint, tool discovery, bounded container-log diagnostic, and whether the Environment belongs to rollout execution or verification. | A rollout Environment readiness failure happens before the agent starts; a fresh Verifier Environment can fail later during grading. Fix the owning Environment before choosing recovery. |
| Image has no matching worker-platform manifest | The imported or built image platform versus the admitted worker platform. | Rebuild and republish for the worker platform, commonly linux/amd64. |
| Private content or a prior Run looks missing | Repository or results-project read access. | Ask the owner or administrator for access; some protected resources deliberately use the same not-found response for missing and unreadable records. |
| Only one Task slot should run again | The stable task_key#index. | Use Retry, not Re-run. |
| Historical results show unexpected content | The Run's recorded TaskSet digest and Attempt, not a current tag. | Navigate from the Run provenance and sealed evidence. |
| Registry says Runtime unavailable | Whether the run API is reachable. | Wait and refresh; the absence of a read is not a failure verdict. |
When execution succeeds but evaluation does not
Keep four facts separate: rollout execution status, Verifier execution state, numeric score, and derived verdict. A model can finish successfully while a Verifier errors, produces no numeric score, or returns an indeterminate verdict. In the grading projection, score: null accompanies an indeterminate result; it is neither zero nor pass. The unchanged v1alpha7 durable record keeps an aligned decimal placeholder for compatibility, but current reports use the verdict to exclude that placeholder from score means and threshold decisions. An explicit fail verdict is a determinate Verifier judgment. Separately, a report's stored success policy can count the rollout as unsuccessful, and a grading-substrate failure can fail execution during verification. Inspect the selected Attempt and each Verifier record before choosing Re-verify, Re-run, or a content fix.
Diagnostic order
- 1Confirm identity
Record the TaskSet digest, Run ID, rollout key, and selected Attempt ID.
- 2Read the first journal failure
Do not start with the final aggregate label.
- 3Open the failing Attempt
From the Run, open Rollouts, choose the failed row, select the latest or relevant Attempt, then use Trace → Errors. Separate Environment, harness, submission, sealing, and Verifier evidence.
- 4Choose the narrowest recovery
Retry, Re-verify, Re-run, or correct content and publish a new release.
What is usually retryable
Provider rate limits, temporary unavailability, transport failures, and explicit retryable rollout failures can justify Retry. Invalid content, missing bindings, model configuration, harness protocol errors, unsupported platform requirements, and admission refusals need correction before a fresh run. Trust the recorded retryable classification; do not infer it from a generic final label.