In my line of work, fending off DDoS attacks and malicious crawlers you see a lot of weird traffic coming through.

Oftentimes I see calls to a .well-known directory.

These are oddly enough oftentimes legitimate. The .well-known directory is just another way for websites to pass common meta data back and forth.

It is where the new Agent 2 Agent stores its agent.json.

There are domain specific files for Nostr, OpenAI, SMTP and a whole lot more that live in that directory.

So if you see traffic coming in to the .well-known URI you should be able to map it out to some protocol or service to get an idea who is crawling you and why.

Good luck fending off malicious traffic.

PS: If you want to know about how to scrape the web at scale, come join us for this Friday’s Tech Talk on how to do just that.