Robots.txt Generator
Seconds. Google ignores this; Bing honours it.
One path per line.
One path per line. Use to carve an exception out of a Disallow.
One per line.
Disallow is not a privacy control
robots.txt asks well-behaved crawlers not to fetch a path. It does not stop anyone, and the file is public, listing /admin/ tells the world where your admin is. A disallowed page can also still be indexed from links elsewhere, without its content. Use authentication for anything private, and noindex for anything that must stay out of search.
The robots.txt generator builds a valid file with one group per crawler, allow and disallow rules, and sitemap lines. It is worth being clear about what the file does: it asks well-behaved crawlers not to fetch paths, and it stops nobody.
How it works
- Each group starts with a User-agent line and applies to that crawler only. A crawler obeys the most specific group matching it and ignores the rest, including the wildcard.
- Disallow with no value means nothing is disallowed. Disallow: / blocks everything, one character apart, opposite meanings.
- Allow carves an exception out of a broader Disallow. The longest matching rule wins in Google's implementation.
- Sitemap lines are independent of any group and must be absolute URLs.
- Crawl-delay is honoured by Bing and ignored by Google, which uses its own rate limiting.
The file lives at the root of the domain and nowhere else. A robots.txt in a subdirectory is simply a text file that nothing reads, and subdomains each need their own.
The most consequential misunderstanding: disallowing a page does not remove it from search. Google can still index a disallowed URL from links elsewhere, showing it with no description. To keep a page out of results you need a noindex tag, which requires the crawler to be allowed to fetch the page and see it.
Examples
A typical site
Rules
Disallow /admin/ and /cart/ for all crawlers
Result
User-agent: * Disallow: /admin/ Disallow: /cart/ Sitemap: https://example.com/sitemap.xml
One group plus a sitemap line. The blank line before Sitemap is convention rather than a requirement.
The one-character disaster
Rule
Disallow: / against Disallow:
Result
Block everything against block nothing
A trailing slash is the difference between a normal site and one that vanishes from search. It happens when a staging file reaches production.
Blocking an AI crawler
User-agent
GPTBot
Result
A separate group for that crawler
GPTBot and CCBot honour robots.txt. A specific group applies to that crawler only, which then ignores the wildcard group entirely.
Frequently asked questions
Does robots.txt stop a page appearing in Google?
No, and this trips up a lot of people. Google can index a disallowed URL discovered through links, listing it with no description because it was not allowed to read the page. To keep a page out of results, allow it to be crawled and serve a noindex tag.
Is robots.txt a security measure?
Not remotely. It is a public file that politely asks compliant crawlers to stay away, and listing /admin/ in it advertises exactly where your admin panel is. Anything that must not be accessed needs authentication.
Where does the file have to live?
At the root of the domain, https://example.com/robots.txt and nowhere else. A file in a subdirectory is ignored entirely, and every subdomain needs its own, because blog.example.com does not inherit the rules from example.com.
Can I block AI training crawlers?
The major ones honour robots.txt: GPTBot for OpenAI, ClaudeBot for Anthropic, CCBot for Common Crawl, Google-Extended for Google's AI training. Add a group for each. Crawlers that ignore robots.txt will continue to ignore it, since compliance is voluntary.
What happens if the file is missing?
Everything is allowed, which is the correct default for most sites. A missing robots.txt is not an error. What does cause problems is a file that returns a 500 error, which some crawlers treat as a signal to stop crawling entirely.