Stay discoverable in search while disallowing AI training

Website owners face a dilemma: allow AI training or risk search discoverability, as major organizations use mixed-use crawlers for both. Cloudflare offers new controls, including "Search setting," "Training setting," and "Agent setting," to manage this. Applebot allows opting out of training via robots.txt for "Applebot-Extended" and uses the nosnippet directive for AI Summaries. Apple has confirmed that disallowing training does not affect search ranking and is developing a URL-level inspection tool for next year.

Cloudflare's new controls offer a granular solution, unlike previous methods that forced an all-or-nothing choice between AI training and search discoverability.