Vercel added separate controls for AI crawlers and AI assistants in its Firewall AI Bots managed ruleset. A project can log or block these categories independently.
Different agents have different purposes
- AI crawlers collect content for indexing or model training.
- AI assistants retrieve pages in response to user actions or search queries.
- The ruleset can block one category while allowing the other, and Vercel says it updates the rules as new bots are added.
The controls operate at the firewall, not through a page's robots.txt. Decide which automated access the site permits, and test the policy before publishing it so useful services are not blocked by mistake.
Vercel says the managed ruleset is available on all plans. Review the current rules and documentation because bot classification and product behavior can change.