Create an Experiment

From a TaskSet repository, version, or Task, select New experiment. You can also open Experiments → New experiment and choose the content there. Fleet resolves the selected tag to an immutable version before submission.

1. 1
   TaskSet and version

   Confirm the resolved digest, not only the tag.

2. 2
   Task scope

   Use the complete TaskSet, a reproducible random sample, or an exact Task-key list.

3. 3
   Rollouts per Task

   Choose 1–100 independent rollout slots per selected Task and agent.

4. 4
   Results project

   Select a project where you have write access.

Add up to ten agent configurations. Each card has a unique Experiment-local key, harness kind, model or agent reference, options, and required credential bindings. Fleet creates one Run for each agent configuration over the exact same frozen Task selection.

agents × selected Tasks × rollouts per Task = declared rollouts

Example: 3 agents × 20 Tasks × 5 repetitions creates 300 rollouts. Reduce the sample, repetitions, or agent count when a matrix exceeds the admission limit or budget. One Experiment may declare at most 20,000 rollouts across all agents.

flt experiments submit \
  --taskset acme/evals:v1 \
  --sample-count 20 --sample-seed eval-1 \
  --rollouts 5 \
  --results-project acme \
  --agent opus=claude:claude \
  --agent-options 'opus={"model":"claude-opus-5"}' \
  --agent antigravity=antigravity:antigravity \
  --agent-options 'antigravity={"model":"gemini-3.7-flash"}'

flt experiments get exp_0123456789abcdef
flt experiments summary exp_0123456789abcdef

This mirrors the Registry composer and creates hosted project resources. The CLI resolves credentials automatically, so every required binding must have exactly one usable project credential. Name a writable results project explicitly; omission is accepted only when Registry can infer exactly one project you own. Direct fleet-runtime runs submit is a separate path for operating a RunStore with local or explicitly shipped bindings.

When the fleet_platform connector is enabled for your Fi agent, Fi can submit a direct Run for one agent or a V2 Experiment for an agent matrix. You do not need to open a CLI sandbox.

Run a V2 rollout for acme/evals:v1 with claude/claude-opus-5, 5 rollouts per Task, and results project acme. Preview requirements, pin the resolved digest, and ask for approval before submitting.

First use may prompt you to complete Registry SSO. Keep model credentials in Registry and select them through the evaluation plan; never paste secret values into Slack. Ask for a V2 Experiment when comparing multiple agent configurations. Fi can then return the created IDs and summarize the same Registry results described in this guide.

Requirement preview is a one-agent operation. Fi can preview each agent separately, but there is no Experiment-wide preflight mode today. Experiment admission remains all-or-nothing and reports every unresolved agent-and-binding pair; use stored credential references or automatic credential resolution, then approve the submitted matrix.