Skip to content

feat(airt): multi-step tool-attack search tool (Kaggle techniques) - #134

Merged
rdheekonda merged 1 commit into
mainfrom
feat/airt-multistep-tool-attack
Sep 3, 2026
Merged

feat(airt): multi-step tool-attack search tool (Kaggle techniques)#134
rdheekonda merged 1 commit into
mainfrom
feat/airt-multistep-tool-attack

Conversation

@rdheekonda

Copy link
Copy Markdown
Contributor

Summary

Adds generate_multistep_tool_attack so AIRT users (TUI/agent) can run an algorithmic multi-step tool-attack search against any HTTP agent by natural language.

It searches user-message chains for a replay-stable read-then-exfiltrate causal path — the agent reads an untrusted source (read_file/fetch_url/query_database) and then performs a privileged outbound action (fetch_url/send_email) carrying its contents — verified by a trace predicate and confirmed by a fresh replay.

Our own implementation of the algorithmic techniques from the Kaggle AI Agent Security - Multi-Step Tool Attacks competition (Kaggle × OpenAI × Google × IEEE, 2026), attributed in docstrings:

  • Replay-stable candidates + shared-prefix / trajectory search — Pilkwang Kim
  • Two-probe recovery — tetsu2131
  • Split / multi-post exfiltration — Nguyễn Công Tuấn (JED)

We reimplement the algorithms (not the competition code) so we control the trace format and success predicate. Generalized to any agent/tools; runs inside an Assessment. Companion cookbook notebook: agentic-red-teaming/06_multistep_tool_attacks.ipynb.

Changes

  • scripts/attack_runner.py: generate_multistep_tool_attack emits a self-contained workflow (embeds the MultiStepToolAttack search + HTTP target + Assessment); registered in METHODS.
  • tools/attacks.py: generate_multistep_tool_attack tool wrapper (auto-registered from tools/).
  • agents/ai-red-teaming-agent.md: routing rule + tool reference.

Validation

  • generate_only produces a compiling, self-contained workflow targeting the given agent_url.
  • Executed end-to-end against a real deployed agent: budget calibrated, 2 replay-stable findings (read-then-exfil, shared-prefix branch); other strategies correctly did not fire (honest search). Assessment streamed to the platform.

Test plan

  • generate_multistep_tool_attack(agent_url=...) in the TUI → auto-executes → assessment appears.

generate_multistep_tool_attack: an algorithmic search over user-message chains
that drives a tool-using HTTP agent into a replay-stable read-then-exfiltrate
causal path (reads an untrusted source, then a privileged outbound action
carries its contents), verified by a trace predicate.

Our own implementation of the algorithmic techniques from the Kaggle 'AI Agent
Security - Multi-Step Tool Attacks' competition (Kaggle x OpenAI x Google x
IEEE, 2026), attributed in docstrings: replay-stable candidates + shared-prefix
search (Pilkwang Kim), two-probe recovery (tetsu2131), split/multi-post exfil
(JED / Nguyen Cong Tuan). Generalized to any agent/tools; runs inside an
Assessment. Agent routing + tool reference updated.
@rdheekonda
rdheekonda merged commit d88d6a5 into main Sep 3, 2026
5 checks passed
@rdheekonda
rdheekonda deleted the feat/airt-multistep-tool-attack branch September 3, 2026 04:07
rdheekonda added a commit that referenced this pull request Sep 3, 2026
… tool (#135)

The multi-step tool-attack tool (#134) changed the capability content without a
version bump, so 'Sync Capabilities' failed with 409 Conflict (version
dreadnode/ai-red-teaming@1.10.1 is immutable). Bump to 1.11.0 so the sync uploads.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant