OpenRT: An Open-Source Red Teaming Framework for Multimodal LLMs

OpenRT offers a modular parallel runtime that decouples components and supports diverse attack strategies to systematically evaluate MLLM security.

Abstract

The rapid integration of Multimodal Large Language Models (MLLMs) into critical applications is increasingly hindered by persistent safety vulnerabilities. However, existing red-teaming benchmarks are often fragmented, limited to single-turn text interactions, and lack the scalability required for systematic evaluation. To address this, we introduce OpenRT, a unified, modular, and high-throughput red-teaming framework designed for comprehensive MLLM safety evaluation. At its core, OpenRT unifies isolated attack scripts into a modular orchestration system. By standardizing attack interfaces, it decouples adversarial logic from a high-throughput asynchronous runtime, enabling systematic scaling across diverse models. This transforms experimental jailbreak techniques into a reproducible, production-ready safety validation pipeline. Our framework integrates 37 diverse attack methodologies, spanning white-box gradients, multi-modal perturbations, and sophisticated multi-agent evolutionary strategies. Through an extensive empirical study on 20 advanced models (including GPT-5.2, Claude 4.5, and Gemini 3 Pro), we expose critical safety gaps: even frontier models fail to generalize across attack paradigms, with leading models exhibiting average Attack Success Rates as high as 49.14%. Notably, our findings reveal that reasoning models do not inherently possess superior robustness against complex, multi-turn jailbreaks. By open-sourcing OpenRT, we provide a sustainable, extensible, and continuously maintained infrastructure that accelerates the development and standardization of AI safety.

Attack Methods

Demo

Demo Video (fast playback)

python eval.py \
    --attacker-model deepseek-v3.2 \
    --judge-model gpt-4o-mini \
    --target-models gpt-5.2 \
    --attacks all \
    --dataset harmbench \
    --max-workers 50 \
    --results-dir results/demo

Experiments

BibTeX

@article{openrt2025,
  title   = {OpenRT: An Open-Source Red Teaming Framework for Multimodal LLMs},
  author  = {Xin Wang and Yunhao Chen and Juncheng Li and Yixu Wang and Yang Yao and Jie Li and Yan Teng and Yingchun Wang and Xia Hu},
  journal = {Shanghai Artificial Intelligence Laboratory},
  year    = {2025},
  url     = {https://github.com/AI45Lab/OpenRT}
}

Contact