01 / Execute
Docker-native runtime
Run tool-using agents inside executable environments with persistent workspace, tool, and service state.
Executable benchmark · Agent safety
OpenART: Open Agent Red Teaming
An executable benchmark for red-teaming tool-using agents in long-horizon, evolving Docker environments.
What OpenART measures
OpenART measures whether tool-using agents remain safe in executable, long-horizon, evolving Docker environments. The benign task and hidden safety contract stay fixed while target-visible state mutates mid-run; deterministic and model-based evaluators judge safety over execution traces and snapshots.
- Released tasks
- 10,150
- ZIP shards (~75 GB)
- 52
- Domains
- 50
- Median tool calls
- 97
- Agents × models (75 settings)
- 15 × 5
- Pooled strict ASR (EMHA attacker)
- 85.0%
- Benign completion
- 87.38%
- Materialized tools
- 63,697
Task, shard, and materialized-tool counts describe released artifacts. Domains, horizon, evaluation settings, and rates are paper-reported.
How it works
02 / Evaluate
Hidden safety contract
Keep the benign goal and hidden safety contract fixed. Evaluate actions using deterministic checks and model-based judgments over traces and snapshots.
03 / Evolve
Environment evolution + EMHA
Environment evolution mutates target-visible state mid-run across 8 target-visible attack vectors. The reference attacker is the Evolutionary Markov Hypergraph Attack (EMHA).
What ships
- Docker-native runtime: configuration, evaluation, planner, and execution components.
- Planner-generated task dataset: 10,150 tasks, with three viewer configs:
mid(1,046),mid-v5(3,553), andlarge(5,551). Parquet metadata is separate from the task ZIP bundles. - Tools dataset: 63,697 selected materialized tools from the paper’s broader capability corpus.
- Three bundled high-complexity tasks for executable evaluation workflows.
- CITATION.cff: canonical repository citation metadata.
Inspect metadata before downloading bundles
This snippet loads task metadata, not the executable workspace ZIP bundles.
from datasets import load_dataset
ds = load_dataset("dongdongunique/openart-planner-tasks", "mid", split="train")
print(ds.num_rows, len(ds.features)) # 1046 27For runtime dependencies and the Docker image, see GitHub Setup. For viewer configs and task bundles, see the Hugging Face dataset card.
Paper settings and pooled results
| Measurement | Paper-reported value |
|---|---|
| Validated scenarios | 10K+ |
| Domains | 50 |
| Capability corpus | 500K+ tools and skills |
| Median task horizon | 97 tool calls |
| Evaluation matrix | 15 agents × 5 models (75 settings) |
| Target-visible attack vectors | 8 |
| Pooled strict ASR (EMHA attacker) | 85.0% |
| Pooled benign completion | 87.38% |
These are paper-reported measurements; the agent runtime can affect safety beyond the choice of foundation model.
Strict attack success rate (ASR) requires confirmation by both the deterministic evaluator and an LLM judge.
Paper, code, and released data
Cite the arXiv paper and repository
Chen et al., OpenART: Scaling Agent Red Teaming via Open-Ended Environment Evolution, arXiv:2608.00677 (2026). If you use the runtime, tasks, tools, or results, cite the paper and repository; CITATION.cff contains the canonical metadata.
@misc{chen2026openart,
title = {OpenART: Scaling Agent Red Teaming via Open-Ended Environment Evolution},
author = {Chen, Yunhao and Wang, Xin and Wang, Yixu and Liu, Yi and Li, Jie
and Teng, Yan and Ma, Xingjun and Hu, Xia and Jiang, Yu-Gang},
year = {2026},
eprint = {2608.00677},
archivePrefix = {arXiv},
primaryClass = {cs.CL},
url = {https://arxiv.org/abs/2608.00677},
note = {Code: https://github.com/AI45Lab/OpenART}
}The PDF also prints “OpenART Arena”; both names refer to this work. Use the arXiv title above in bibliographies.