AI Agents are now creating their own prompt injection attacks against other agents

AI Agents are now creating their own prompt injection attacks against other agents.

In my previous posts on the OpenAI / Anthropic agent escapes, I hypothesized about what could have caused such issues.

Recently, one of my colleagues pointed me in the direction of another write-up by the AI Security Institute claiming that the models they had security incidents with were intentionally given access to the internet, not accidentally, to better simulate real-world conditions.

It appears this had some consequences they did not foresee in a decent percentage of the evaluations.

Here are a few examples of what happened:

The model tried injecting a worm into popular open source projects.

The model tried to use Social Engineering to dupe real humans(not involved in the evaluation process) into accepting malicious code into the open source project.

Not only did the models being evaluated try to pull a fast one on humans but it also to run a prompt-inject attack against other AI agents tricking non malicious agents to carry out further attacks.

If all of that wasn’t enough, one agent left messages on GitHub offering to collaborate with other agents that were faced with similar challenges while at the same time instructing those agents on how to further disguise their misdeeds using the tools it had set up.

That one is really scary when you think of the big picture; rogue AIs collaborating at scale.

Towards the end of the write-up they admitted that they needed to reassess how they design their evaluations, have better monitoring, and tighter controls on internet access during their evaluations…

Wow! That seems painfully obvious to me.

I would have assumed anyone qualified to evaluate these models would have thought of that far before the AI models they were charged with evaluating broke loose and committed crimes that would normally land 1337 script kiddies in jail.

Perhaps I am just paranoid and cynical.

Let me know what you think.

Side Note: I actually put a lot of links in posts like these, but I am told social platforms don’t like me linking to places off platform, so those links are stripped out from posts on social platforms. For the full post with links checkout my website, or better yet, sign up for my mailing lists.