Robots.txt Generator
Build a robots.txt with per-bot allow and disallow rules, crawl-delay and sitemap lines, from presets or by hand.
Build a robots.txt
Runs in your browser. Pick a preset or build the groups by hand; the file updates as you type.
Group 1
An empty Disallow means no restriction for these bots. Disallow: / blocks the whole site.
robots.txt
User-agent: *
Disallow:
Upload the file to the root of the domain so it is served at /robots.txt, then test a URL against it with the robots.txt tester.
What the Robots.txt Generator does
robots.txt is a plain text file at the root of a domain that tells crawlers which paths they may fetch. It is read before any other page, so a typo in it can hide a whole site from Google or let a scraper into the one folder you wanted closed. The syntax is small (User-agent, Allow, Disallow, Crawl-delay, Sitemap) but the details trip people up: rules apply per bot group, the longest matching path wins, and a rule meant for one bot silently applies to none if its group name is misspelled.
This generator assembles the file for you. Pick the bots a group applies to (everyone, Googlebot, Bingbot, the AI training crawlers such as GPTBot and ClaudeBot, and others), add allow and disallow paths, set an optional crawl-delay for the bots that honour it, and list your sitemaps. The file updates as you type and can be copied or downloaded as robots.txt. Four presets cover the usual starting points: allow everything, block everything, WordPress defaults and block AI crawlers.
Everything runs in your browser. The file is not uploaded anywhere and the paths you type are not recorded. Once it is in place, check it with the robots.txt tester to confirm a specific URL is allowed or blocked for the bot you care about.
Step by step
- 01Start from a preset that is closest to what you need, or from the empty default group.
- 02For each group, tick the user agents it applies to and add Allow and Disallow rows. Paths start with / and may end in * or $.
- 03Add a crawl-delay only if a bot is hitting the server too hard; Googlebot ignores it and uses Search Console instead.
- 04Enter the full URL of each sitemap, including https:// and the domain.
- 05Copy the text or download robots.txt and upload it to the root of the domain, so it is reachable at /robots.txt.
Common questions
No. It stops Google from crawling the page, but the URL can still be indexed from links on other sites, usually without a description. To remove a page from search results, let it be crawled and add a noindex meta tag or return a 404 or 410 status.
An empty Disallow allows everything for that group; it is a legal way to say "no restrictions". Disallow: / blocks every path on the site. The generator writes the empty form for the "allow everything" preset so the intent is explicit.
The preset lists the agents that publish their names and honour robots.txt: GPTBot and ChatGPT-User (OpenAI), ClaudeBot and anthropic-ai (Anthropic), Google-Extended (Gemini training), CCBot (Common Crawl), PerplexityBot, Bytespider, Applebot-Extended, Amazonbot, cohere-ai and Meta-ExternalAgent. Blocking Google-Extended does not affect normal Google Search indexing.
Google, Bing and most large crawlers support * for any run of characters and $ to anchor the end of the URL, for example Disallow: /*.pdf$. Smaller crawlers may treat them literally, so keep the rules you care most about simple prefixes.
At the root of the host, for example https://www.example.com/robots.txt. A file in a subfolder is ignored, and each subdomain and protocol needs its own file. The sitemap lines may point to any host, which is the one place a cross-domain reference is allowed.
Keep checking
Robots.txt Tester
Fetch a site's robots.txt and test whether a path is allowed or blocked for Googlebot or any user agent, with the matching rule.
Free toolXML Sitemap Generator
Paste a list of URLs and get a valid sitemap.xml with lastmod, changefreq and priority, deduplicated and escaped.
Free toolXML Sitemap Validator
Validate an XML sitemap or index: well-formed XML, URL count, loc, lastmod, priority and changefreq, limits and live URL checks.
Free toolMeta Tag Analyzer
Check a page's title, meta description, canonical, robots, viewport, Open Graph and Twitter tags with a 0-100 score.
Want these fixes done for you?
Our team runs the audit, makes the changes and reports the result.