Robots.txt Tester
Fetch a site's robots.txt and test whether a path is allowed or blocked for Googlebot or any user agent, with the matching rule.
Test a path against robots.txt
The file is fetched from the domain you enter and the rules are applied the way Google does: most specific group, longest matching rule, Allow wins a tie.
What the Robots.txt Tester does
robots.txt is the first file a well-behaved crawler reads on a site, and the one most likely to hide a costly mistake. A single Disallow: / left over from staging removes a whole site from Google; a rule meant for one folder blocks another because of a wildcard; a group for Googlebot overrides the wildcard group in a way the author did not expect. Because the file does nothing visible to a human visitor, these mistakes can live for months.
This tester fetches robots.txt from the domain you enter and parses it the way Google does: user-agent lines start groups, the most specific matching group wins, and within that group the longest Allow or Disallow pattern that matches the path decides, with Allow winning a tie. Wildcards (*) and end anchors ($) are honoured. You pick the path and the user agent, and the result tells you allowed or blocked, which rule decided it and in which group. The Sitemap lines are listed, and the raw file is shown so you can read it alongside the verdict.
Since Google retired its own robots.txt tester from Search Console, there is no official place to try a path before deploying a change. Use this one to confirm that /admin/ is blocked and /blog/ is open, that image search is not blocked for Googlebot-Image, and that a new AI crawler rule does what you meant.
Step by step
- 01Enter the site URL (or a full page URL; its domain is used and its path is pre-filled).
- 02Type the path to test, such as /blog/my-post/ or /search?q=shoes, and pick a user agent.
- 03Click Test. The file is fetched and the matching rule is applied.
- 04Read the verdict and the rule that produced it; the group list below shows every rule parsed.
- 05Edit the file on your server, then test the same path again.
Common questions
The rule with the longest path pattern wins, measured in characters. If an Allow and a Disallow have the same length, Allow wins. So "Disallow: /folder/" with "Allow: /folder/public/" lets /folder/public/page through, because the Allow pattern is longer. This tool applies exactly that logic.
Only the Googlebot group. Google uses the single most specific group whose name is a prefix of the crawler's token and ignores the wildcard group completely for that crawler. Rules you want to apply to Googlebot as well must be repeated inside its group.
No. It stops Google from fetching the page, but the URL can still be indexed from links on other sites, showing without a description. To remove a page from results, let Google crawl it and add a noindex meta tag or return 404 or 410.
That there are no restrictions: every crawler may fetch everything. That is fine for most sites. A 5xx error is different; Google treats a server error as if the whole site were disallowed until the file can be fetched again, so a broken robots.txt endpoint can stop crawling.
Google, Bing, Yandex and most modern crawlers support them. The original 1994 standard did not, so a very old or simple bot may treat "/*.pdf$" as a literal path. For rules that matter, prefer plain folder prefixes where you can.
Keep checking
Robots.txt Generator
Build a robots.txt with per-bot allow and disallow rules, crawl-delay and sitemap lines, from presets or by hand.
Free toolXML Sitemap Validator
Validate an XML sitemap or index: well-formed XML, URL count, loc, lastmod, priority and changefreq, limits and live URL checks.
Free toolGoogle Index Checker
Build the site: and exact-URL Google searches for up to ten pages and open them in your browser to see what is indexed.
Free toolMeta Tag Analyzer
Check a page's title, meta description, canonical, robots, viewport, Open Graph and Twitter tags with a 0-100 score.
Free toolCanonical Tag Checker
Find every canonical tag and Link header on a page and check it is absolute, self-referencing and free of conflicts.
Want these fixes done for you?
Our team runs the audit, makes the changes and reports the result.