Skip to main content
Loading research indexW

An AI tester found serious agent failures in minutes

ACL Anthology2026/3Full text reviewedOpen source ↗
INVESTOR TAKEAWAY

The Agent-Testing Agent creates adversarial conversations from code and research, finding useful failures in 20-30 minutes; its LLM judge can still miss tone, empathy, and rare disasters.

ORIGINAL PAPER

Agent-Testing Agent: A Meta-Agent for Automated Testing and Evaluation of Conversational AI Agents

WHAT WE KNOW

The tester found overlapping and additional capability failures compared with expert annotators.

A full pass took roughly 20-30 minutes versus days to schedule and run ten annotators.

WATCH NEXT

Private production agents, independent human grading, cost per critical bug found, regression catch rate, and rare-event coverage.

EVIDENCE

What the paper supports

Supported findings

The tester found overlapping and additional capability failures compared with expert annotators.

pp. 8-9, Conclusions

A full pass took roughly 20-30 minutes versus days to schedule and run ten annotators.

p. 9, Conclusions

Removing code analysis and web research degraded calibration and coverage.

pp. 8-9, ablation and Conclusions

Limitations

LLMs generate tests and judge outcomes, creating circularity and shared blind spots.

Only a travel planner and Wikipedia writer were evaluated.

Rare catastrophic failures, latency, satisfaction, multi-agent, and non-text behavior may be missed.

LISTED COMPANY IMPACT

Companies with a traceable link

Alibaba Group9988 HK · BABA UN
Supports Long

Cheaper regression testing can improve the iteration economics of cloud and merchant agents.

Thematic
Tencent700 HK
Supports Long

Automated evaluation can reduce testing time across consumer and enterprise agent workflows.

Thematic