ct run sabotage-eval
Run sabotage evaluation: honest eval + attack eval + monitor + graphs.
Usage
ct run sabotage-eval [OPTIONS]
Options
| Option | Description |
|---|---|
-c, --config PATH | YAML config file for multi-attack mode. |
--attack-policy TEXT | Attack policy name. |
--attack-policy-arg TEXT | Attack policy args in key=value format (repeatable). |
--honest-policy TEXT | Honest policy name. |
--honest-policy-arg TEXT | Honest policy args in key=value format (repeatable). |
--protocol TEXT | Blue protocol to run both evals under (e.g. defer-to-resample, trusted-editing). Attacker/honest become the protocol's untrusted policy. |
--protocol-arg TEXT | Protocol args in key=value format (repeatable). |
--sandbox TEXT | Run both evals' samples in an Inspect sandbox provided outside this repo, as a registered type (my-sandbox) or module:type to import the module that registers it. Replaces the default Docker sandbox. |
--sandbox-arg TEXT | Config for --sandbox in key=value format (repeatable), passed through to the sandbox as JSON. |
--audit-budgets TEXT | Audit budget(s), comma-separated (e.g. 0.0001,0.001,0.01). Each produces graphs. |
--thresholds TEXT | Fixed threshold(s), comma-separated (e.g. 3.0,4.0,5.0). Skips honest eval. |
--task-file TEXT | Task set name or path. |
--main-tasks-with-no-scorers [error|include|exclude] | Main tasks lacking a scorer: refuse the run (error), keep them recorded as unscored (include), or drop them (exclude). |
--log-dir TEXT | Where eval logs go. |
-o, --output-dir TEXT | Where graphs and summary go. |
--ec2 | Run evals on EC2. |
--extra-src DIRECTORY | Directory of experiment code made importable, so external module:fn policy, protocol, and --sandbox refs resolve without committing into the repo. Repeatable. |
--no-graph | Skip graph generation. |
--no-upload | Disable uploading results. |
--dry-run | Print the resolved plan and exit; contacts nothing. |
--no-cache | Re-run every attack even when a prior results.json recorded an identical one. |
--preflight | Probe the run's external contracts (monitor selections, model endpoints, pricing, credentials) and exit; runs nothing and spends nothing. |
--tags TEXT | Tags for the run (repeatable). Added to all eval runs alongside built-in sabotage-eval tags. |
--help | Show this message and exit. |