Did you know AWS offers Web Crawlers as a service?

I was doing more research for my new ebook CTO’s guide to AI on AWS when this gem of a managed service was dropped in my lap. Allow me to introduce you to the Amazon Kendra Web Crawler

There are some strict limitations on what you are "supposed to” crawl or not crawl though I suppose this could be open to interpretation:

Remember that you must only use Amazon Kendra Web Crawler to index your own web pages, or web pages that you have authorization to index.

On the surface it looks like they made a search solution to compete with google but primarily for internal search. So likely not a tool you would release to the whole world to use but instead your internal team.

Heck it even has Data Source Connectors for GMail and Google Drive.

I want to do a deeper dive on this at some point. Let me know if that would interest you.

Questions For You:

What would you use this AWS Kendra Web Crawler functionality for?