Run evaluations from the CLI

Submit locally, hand work to a hosted target, and inspect durable Runs.

On this page

Run locally

Terminal

run_id=$(fleet-runtime runs submit \
  benchmark/task-key -n 1 \
  --agent claude --harness-var ANTHROPIC_API_KEY=@ --follow)

fleet-runtime runs show "$run_id" --rollouts

--follow admits the Run and executes its pending rollouts on this machine. Omit the Task suffix to run every Task in the local alias.

Configure the hosted target

~/.flt/config.toml

[targets.fleet-alpha.run_store]
kind = "http"
base_url = "https://runtime-alpha.fleetai.me"
credential_host = "registry-alpha.fleetai.me"

The target reuses the Registry credential created by flt auth login.

Submit and inspect hosted work

Terminal

run_id=$(fleet-runtime runs submit \
  benchmark-release/task-key -n 1 \
  --agent claude --harness-var ANTHROPIC_API_KEY=@ \
  --target fleet-alpha)

fleet-runtime runs status "$run_id" --target fleet-alpha
fleet-runtime runs show "$run_id" \
  --rollouts --target fleet-alpha