OpenAI announced that it plans to honor a shared signal in robots.txt that lets website operators express preferences about using their content for foundational AI training. The signal is part of an IETF working-group effort intended to reduce the need for publishers to maintain separate rules for every provider.
What changes for site owners
- OpenAI says it aims to implement the signal in the coming months and plans to share implementation details later.
- The shared signal is voluntary and will not replace OpenAI's existing crawler controls.
- Other participating providers are being encouraged to honor the same preference.
The announcement describes a planned change, not a feature already active on every site. Website operators should continue using documented crawler controls and watch for implementation guidance.
A robots.txt preference helps communicate a policy to crawlers; it does not guarantee that every automated system will follow it or affect ordinary search indexing in the same way.