Skip to documentation
Fleet
DocsGet started

Build, run, and debug agent evaluations

Fleet pins benchmark inputs, runs one or more agent configurations, and records the evidence needed to compare and diagnose results.

On this page

Start here

Run your first evaluationUse existing content to run locally or create a hosted Experiment.Bring in a benchmarkImport an existing benchmark, inspect its TaskSet, and publish it to Registry.Compare agentsHold Task inputs constant across one or more explicit agent configurations.Understand results and failuresRead progress, rollout outcomes, traces, and verifier results.

Run locally

Use fleet-runtime runs submit with --follow to create a direct Run and execute its eligible Rollouts on this machine. You can start with a local TaskSet or a prompt; Registry publication and an Experiment are not required.

TaskSetExact inputs
RunOne agent
RolloutOne Task slot
AttemptOne sealed try
ResultEvidence and grading

Local execution is useful while authoring a benchmark, checking an Environment, or debugging one agent configuration. Follow the local CLI workflow or run a prompt without publishing content.

Run hosted

Use Registry UI or flt experiments submit to create an Experiment over one or more comparable agent configurations. Registry freezes the TaskSet and Task selection, then creates one member Run per agent configuration.

TaskSetExact inputs
ExperimentComparable Runs
RunOne agent
RolloutOne Task slot
AttemptOne sealed try
ResultEvidence and grading

Use this path for shared execution, durable comparison, and inspection in Registry. See the hosted Experiment workflow. If you need one hosted Run without an Experiment, use the direct Runtime CLI path.

Local and hosted Runs both pin exact inputs, record explicit agent configuration and bindings, and preserve their Rollout, Attempt, evidence, and grading history. A retry adds a new Attempt to the same Rollout. Tasks without Verifiers can still complete without producing a score or pass/fail verdict.

Choose how you work

SurfaceBest forBoundary
Registry UIBrowse content, create hosted Experiments, compare member Runs, and inspect results.Initial import of locally authored content remains a CLI workflow.
fltCompile, import, inspect, and transfer immutable content; manage Registry credentials and submit hosted Experiments.Does not execute Tasks locally.
fleet-runtimeRun Tasks directly, inspect Runs and Attempts, and verify sealed evidence.Does not resolve Registry names while executing.
Registry APIIntegrate with the canonical Registry v1 HTTP API.It is not a complete Platform API reference.
fleetAdminister mutable access policy.For platform administrators, not normal evaluation workflows.