Cloudflare announced a system for evaluating whether its Verified Bots continue to respect website preferences. The company distinguishes a bot's verified identity from its behavior over time.
What the system is intended to observe
- Whether a crawler follows a site's stated crawling preferences.
- Whether it responds to requests to slow down, including HTTP 429 responses.
- Whether repeated observations support an explainable trust band rather than relying only on a one-time identity check.
Cloudflare says a verified bot can still behave badly because of a bug or configuration error. Its monitoring is a Cloudflare program for listed bots, not a universal enforcement mechanism for every crawler on the web.
Site owners should publish clear preferences and monitor their own traffic. robots.txt communicates crawler preferences but is not access control; sensitive resources still need authentication and authorization.