1 · Choose your stance
Presets set every toggle at once; fine-tune individual bots afterwards.
| Crawler | Vendor | Declared purpose | Policy |
|---|
Training vs. retrieval: "Training" bots feed model weights — your content shapes the model but you get no traffic back. "Retrieval" / search / user-triggered bots fetch pages to answer a live query, often with a citation link. Some vendors run separate user-agents for each (OpenAI and Perplexity do); others use one token with a separate opt-out mechanism (Google-Extended and Applebot-Extended are preference tokens — they don't crawl themselves, they signal your preference to the company's other crawlers). Crawler lists drift. Vendors rename and add bots without announcement — verify this table against current vendor docs before relying on it, and remember robots.txt is a request, not a fence: enforcement needs the server-level snippets below.
2 · Your generated policy
robots.txt blocks, response headers, and server-level enforcement.
3 · Check my current policy
Enter your site URL — we fetch your live robots.txt, parse it, and diff the effective rule per crawler against the toggles you set above.