Import an existing benchmark

Lower Harbor and other supported ecosystem formats into a native immutable TaskSet.

On this page

Adapters enter Fleet through explicit contracts

External-format imports are flt pull workflows. Harbor imports atomically materialize one complete native TaskSet before the local alias becomes visible; Runtime then reads ordinary immutable Fleet content and keeps no Harbor dependency. Other adapters keep their explicit contracts: TaskDump bulk import can create one TaskSet per project, while OpenEnv and ORS retain their native Environment Kinds and require a compatible Runtime Driver.

Source family	Use it for
Harbor	Published packages, GitHub-backed Tasks, or one local Task directory.
TaskDump, Prime, Gym-Anything, SWE-bench, MCPMark	Import content through its supported compatibility adapter.
OpenEnv and ORS	Import protocol-backed content into native Fleet declarations.
Authored Fleet YAML	Use `file://` when the source is already a Fleet TaskSet.

Import a published Harbor package

Terminal

flt pull 'harbor://sldbench/sldbench@1' sldbench
flt pull 'harbor://sldbench/sldbench@1' sldbench-resumable --import-workspace ./sldbench-import
flt list sldbench
flt inspect sldbench

Package locators use the separately installed Harbor CLI and its authentication. Pin @REF when source selection must be reproducible; omitting it requests Harbor's mutable latest package version. Harbor authentication and Docker Registry authentication are separate.

For a large published package, --import-workspace retains acquired source and checkpoints completed Tasks. Resume with the exact same source locator and workspace; unchanged Tasks and pinned image resolutions can be reused, while edited Tasks rebuild.

Import from GitHub or a local directory

Terminal

flt pull 'harbor://github.com/harbor-framework/harbor/examples/tasks/hello-world?ref=main' harbor-live
flt pull 'harbor:///absolute/path/to/task' harbor-local

Check trust and worker platform before publishing

A local image build can follow the host platform unless the supported adapter profile pins another platform; the Harbor Compose compatibility profile requires linux/amd64. Before remote execution, confirm every image serves the admitted worker platform. Unsupported adapter declarations fail closed instead of becoming hidden Runtime behavior.

Inspect, publish, and run the native TaskSet

Terminal

flt push sldbench registry-alpha.fleetai.me/acme/sldbench:v1 --dry-run
flt push sldbench registry-alpha.fleetai.me/acme/sldbench:v1

After import, use the same validation and publication flow as any TaskSet. Registry UI and flt experiments submit use Registry authority directly. Only direct fleet-runtime runs submit --target needs a fresh pull of the published reference to record worker-reachable image locations. Interactive flt pulloutput identifies the import as local and prints the kind-correct publish command; JSON and quiet modes remain machine-clean. Continue with Preview and push.

Published Harbor package locators are pull-only and cannot be passed to flt export. Editable export is limited to local or GitHub-backed single-Task Harbor sources. Resumable import workspaces currently support only published Harbor packages; native Fleet or OCI pulls, local or GitHub single-Task Harbor locators, and the other ecosystem adapters reject --import-workspace.

A workspace is bound to its exact protocol and source locator, is locked against concurrent use, remains after success or failure until you remove it, and is never canonical Fleet content. The destination alias still appears atomically only after the complete imported TaskSet succeeds; a workspace never exposes a partial TaskSet.