How to Edit robots.txt in WordPress

When search engines like Google or Bing crawl your WordPress site, they don’t just look at your content—they follow every file, folder, and link they can access. But what if you want to control which parts of your website are visible to them? That’s exactly where the robots.txt file comes in.

The robots.txt file acts like a traffic signal for search engine bots. It tells them which pages or directories they can crawl and which ones they should avoid. For example, you might want search engines to skip your admin area, private files, or duplicate content while focusing only on your main posts and pages.

For WordPress users, learning to edit robots.txt is an important step toward better SEO and smarter crawl management. It can improve your site’s visibility, save crawl budget, and prevent indexing errors. 

Where to Find robots.txt in WordPress

The robots.txt file in WordPress works a little differently compared to static websites. By default, WordPress automatically generates a virtual robots.txt file, which means you won’t see an actual file in your site’s root directory unless you create one manually.

To check if your site already has a robots.txt file:

1. Open your browser.
2. Type your domain name followed by /robots.txt (e.g., www.example.com/robots.txt).
3. Press Enter.

If a file loads, congratulations—you’ve found your robots.txt. Most WordPress sites will display a basic file that blocks access to the /wp-admin/ directory while allowing access to important files like admin-ajax.php.

Methods to Edit robots.txt in WordPress

There are three main methods to edit your robots.txt file in WordPress:

Method no 1: Editing robots.txt with Plugins (Yoast SEO, RankMath, AIOSEO)

Steps to Edit robots.txt with Yoast SEO:

1. Log in to your WordPress Dashboard.
2. Go to SEO in the left-hand menu, then click on Tools.
3. Click on the File Editor option under “SEO Tools”.
4. You will see your robots.txt file. Yoast will prompt you to create one if it’s not created yet.
5. Make the necessary edits:
   • Example: Disallow: /wp-admin/ to block the admin section from being crawled.
6. After editing, click Save Changes to apply the updates.

Method no 2: Editing robots.txt with Rank Math or AIOSEO

Steps to Edit robots.txt with Rank Math (in AIOSEO the editor is under Tools → Robots.txt Editor):

1. Log in to your WordPress Dashboard.
2. Navigate to RankMath in the left-hand menu and click General Settings.
3. Scroll to the robots.txt section.
4. Click on Edit robots.txt.
5. Make your changes in the editor:
   • Example: Disallow: /?s= to stop internal search result pages from being crawled.
6. After making the changes, click Save Changes to apply them.

Method no 3: Manually editing robots.txt via FTP/cPanel

Steps to Edit robots.txt Manually via FTP/cPanel:

1. Log in to your hosting account (via FTP or cPanel).
2. If using cPanel, open File Manager; if using FTP, connect using your FTP client.
3. Navigate to the public_html directory (or the root folder where WordPress is installed).
4. Look for robots.txt in the folder.
   • If it’s not present, create a new file and name it robots.txt.
5. Open robots.txt in a text editor.
6. Add or modify the rules you need. For example:
   • Disallow: /wp-admin/ to block the WordPress admin area.
   • Allow: /wp-admin/admin-ajax.php to allow specific admin files.
7. Save and upload the file back to the root directory.

Best Practices for robots.txt in WordPress

Once you’ve learned how to edit your robots.txt file, it’s important to follow best practices to ensure that it benefits your site’s SEO without causing issues. Here are some of the most effective rules for your WordPress robots.txt file.

Recommended robots.txt Rules

• Allowing Essential Files
  You should always allow bots to access important files like CSS, JavaScript, and images to ensure proper page rendering. Without these files, your site might not display correctly in search results.

• Disallowing Sensitive Areas

Block search engines from accessing areas of your site that shouldn’t be indexed, like your wp-admin directory. This prevents unnecessary crawl requests and avoids search engines from indexing admin pages.

• Preventing Duplicate Content

WordPress often creates duplicate URLs through parameters, filters and internal search. Do not rely on Disallow to fix this: a disallowed URL can still be indexed, and Google cannot see a canonical tag on a page it is not allowed to crawl. Use canonical tags for parameter and filter duplicates, and reserve Disallow for infinite spaces such as internal search results (Disallow: /?s=) and endless filter combinations. Never Disallow a URL you want removed from the index; use a noindex tag instead.

• Allowing Admin AJAX Requests

While blocking the /wp-admin/ section, it is essential to allow the admin-ajax.php file for many WordPress plugins to function properly.

Setting Crawl Rate and Crawl Delay

If you notice your site is over-crawled (which can negatively affect your site speed), you can set a crawl delay in the robots.txt file. This instructs search engines to wait a specific amount of time between requests. However, Google ignores this directive; Bing and some other crawlers respect it.

Blocking Specific Bots

Add user-agent-specific rules to block specific bots from accessing your site. For example, to block Bingbot:

Directing Search Engines to Your Sitemap

Including a Sitemap line in your robots.txt file helps search engines find your sitemap without you submitting it anywhere. WordPress core already adds Sitemap: https://yourdomain.com/wp-sitemap.xml to its virtual file; Yoast SEO and Rank Math replace this with their own sitemap index (for example /sitemap_index.xml). Open yourdomain.com/robots.txt to see which line your site outputs, and make sure it points at a sitemap that actually loads.

Testing & Validating Your robots.txt

After editing your robots.txt file, testing and validating your changes is important to ensure that everything works as expected. Incorrect configurations can prevent search engines from crawling your website properly or may result in unwanted pages being indexed.

Using Google Search Console’s robots.txt Report

Google Search Console has a robots.txt report that shows the file Google last fetched, when it fetched it and any parsing errors. The old per-URL robots.txt Tester was retired in 2023; per-page checks now live in URL Inspection. Here’s how to use the report:

Steps to Test robots.txt in Google Search Console:

1. Log in to Google Search Console.
2. Go to Settings and open the robots.txt report (the old robots.txt Tester was retired in 2023).
3. The report lists each robots.txt file Google found for your site, with its fetch status, the date it was last crawled and any errors or warnings.
4. Click a file to see the exact content Google has cached. If you have just fixed a mistake, use Request a recrawl so Google fetches the new version sooner.
5. To test a single page, use URL Inspection instead: enter the URL, click Test live URL, and check the Crawl section for “Crawl allowed?” and “Blocked by robots.txt”.
6. If you find any issues, modify your robots.txt file accordingly.

What to Look For:

• Blocked pages: Ensure that important pages are not accidentally blocked by robots.txt.
• Unblocked pages: Confirm that search engines can crawl and index critical pages on your site.

Using URL Inspection to Check Crawl Behaviour

Once your robots.txt file is live, use the URL Inspection tool to see how Googlebot fetches a specific page. (Fetch as Google was retired in 2019; URL Inspection replaced it.) This is the quickest way to confirm that a page you care about is not blocked.

Steps to Use URL Inspection:

1. Go to Google Search Console.
2. Click on “URL Inspection” in the left-hand menu.
3. Enter the URL of a page you want to test.
4. Click “Test Live URL”.
5. Google will show whether the page can be crawled and indexed properly.
6. If there are issues (e.g., blocked by robots.txt), it will display an error.

Using External Tools to Validate Your robots.txt

In addition to Google Search Console, you can use external tools to validate your robots.txt file, including:

• Bing Webmaster Tools: Bing also offers a robots.txt tester.
• SEO crawlers like Screaming Frog crawl your site and list every URL your robots.txt blocks, and free checkers such as the marketers.pk robots.txt Tester show which rule matches a given URL and user-agent.

Best Practices for Ongoing Validation

After initial testing, it’s important to check your robots.txt file as your website evolves regularly:

• Monitor site performance in Google Search Console.
  
• Revisit robots.txt when you make major site changes (e.g., adding new pages or restructuring the site).
• Test after every update to ensure that search engines can still crawl and index your important pages.

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Bytespider
Disallow: /

Conclusion

Editing your robots.txt file in WordPress is essential in controlling how search engines crawl and index your site. By following best practices, testing your changes, and avoiding common mistakes, you can enhance your site’s SEO, prevent duplicate content issues, and ensure that search engines are crawling the most important pages. Whether you choose to edit via an SEO plugin or manually through FTP, the right configuration can significantly improve how your site performs in search results. Make sure to revisit your robots.txt file regularly to optimise your site for search engines and users.