IETF AI preferences: Content-Usage status
A standard way to say “search yes, training no” is close but not finished. Here is what the IETF drafts define and why you can’t rely on them yet.
The IETF’s AI Preferences (aipref) working group is drafting a standard way for site owners to say how AI systems may use their content.
What happened
There are two main documents:
- Vocabulary: defines usage categories, including
train-aiandsearch, each set to allowed, disallowed or unknown. - Attachment: defines where to put those preferences, with a new
Content-Usagerule in robots.txt (for exampleContent-Usage: train-ai=n) and an HTTP header.
The working group’s milestones targeted sending both drafts to the IESG for publication around August 2026, and revisions continued after that. As of November 2026 they are still drafts, not RFCs. Separate proposals, such as an ai.txt file, also exist.
What it means for you
- Adding a Content-Usage line won’t break anything, since crawlers ignore rules they don’t understand. But don’t count on it being honoured yet.
- User-agent rules still do the real work. Blocking
GPTBotorClaudeBotis the reliable way to opt out of training today, while allowing the search bots. - Watch for adoption. Once major crawlers announce support, one line will express “search yes, training no” for all of them.
Build per-bot rules with the AI robots.txt Generator.