Loading page content
Build a robots.txt with per-crawler rules, sitemap declarations, and explicit directives for AI training crawlers. The group most sites have not decided about yet.
Generate a valid robots.txt file for your website. Configure rules per user-agent, add sitemaps, and block AI crawlers with one click.
Presets
User-agent: * Allow: / Disallow: /api/ Disallow: /admin/ Disallow: /_next/ Sitemap: https://example.com/sitemap.xml
The most important thing to understand about robots.txt is what it does not do. Disallow stops crawling, not indexing. A URL blocked in robots.txt can still appear in search results. Google just cannot see the page contents, so it shows the URL with no snippet. To keep something out of the index you need a noindex meta tag or header, and that requires the page to be crawlable so the directive can be read. Blocking a page in robots.txt and adding noindex is self-defeating.
AI crawler directives are a newer and optional decision. GPTBot, ClaudeBot, CCBot, PerplexityBot and others respect robots.txt. Blocking them keeps your content out of training corpora and, depending on the crawler, out of AI search citations too, which for a content site may cost more traffic than it protects.