The next hurdle for AI agents: getting websites to let them in

A new wave of consumer AI agents, such as Meta’s Muse and ChatGPT’s Dots, are designed to perform tasks like booking flights or ordering groceries for users. However, these agents face a significant hurdle: website access. Companies like Cloudflare, which protect sites from AI agents training on their data, are suspected of disrupting traffic from agents like Muse. This disruption is linked to a September 15 change in crawler defaults, allowing site owners to block AI training while permitting other bots, and shifting existing sites to block AI agents on ad-containing pages.

This report uniquely details how Cloudflare's September 15 crawler default change, unlike previous settings, now specifically blocks AI agents on ad-containing pages.