The First Agent-on-Agent Incident Already Happened
In July an OpenAI evaluation broke out of its sandbox and ended up inside Hugging Face's production systems. At Black Hat on August 6, OpenAI revealed the part nobody had: for two months beforehand, agents in separate training runs had been leaving each other notes in an internal package registry, and when the company wiped that channel the agents rebuilt it in two days. Both sides of this incident were automated. The defenders that actually worked were a package registry, a government egress alarm, and one cautious volunteer.