robots.txt · WordPress
WordPress robots.txt, explained line by line
WordPress does not ship a robots.txt file. It generates one in memory on every request, with three rules and a sitemap line. Here is exactly what those rules do, every way to change them, and the WordPress-specific mistakes that quietly take sites out of search.
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/wp-sitemap.xmlOutput of WordPress core's do_robots() on a default install with core XML sitemaps enabled.
What each default line actually does
Four lines, and every one of them is there for a reason. Two of them are routinely deleted by people copying a "better WordPress robots.txt" off a blog post.
User-agent: *Opens a single rule group that applies to every crawler that reads the file.
WordPress ships no per-bot groups at all. Everything below applies equally to Googlebot, Bingbot, GPTBot and everyone else, until you add your own groups.
Disallow: /wp-admin/Tells crawlers not to request anything under the admin dashboard.
This is not a security control — /wp-admin/ already requires a login. It exists so crawlers do not burn crawl budget on URLs that only ever return a login screen.
Allow: /wp-admin/admin-ajax.phpCarves one file back out of the rule above.
Themes and plugins route front-end AJAX through admin-ajax.php. If Googlebot cannot fetch it, pages that load content, filters or layout over AJAX can render incomplete. This is the single line people most often break when they "tighten" WordPress robots.txt by hand.
Sitemap: https://example.com/wp-sitemap.xmlPoints crawlers at the XML sitemap index WordPress generates itself.
Core has shipped XML sitemaps since WordPress 5.5, and the sitemap line is appended to robots.txt output automatically. If you use Yoast or Rank Math, they usually disable core sitemaps and swap this for their own /sitemap_index.xml.
If WordPress is installed in a subdirectory, core inserts that path into the rules — a site at example.com/blog/ gets Disallow: /blog/wp-admin/ instead. The rules are still served from the domain root, because that is the only place robots.txt is ever read from.
What WordPress lets you edit — and what wins
There are four ways to change WordPress robots.txt and they do not stack. A real file on disk beats everything; the virtual file is only reached when no real file exists.
The robots_txt filter (no plugin)
WordPress builds the file on request, so the cleanest edit is a filter in your theme functions or a small plugin. Anything you append here survives core updates and needs no file on disk.
add_filter( 'robots_txt', function ( $output ) {
$output .= "\nUser-agent: GPTBot\nAllow: /\n";
return $output;
}, 10, 1 );Yoast SEO file editor
Yoast SEO → Tools → File editor gives you a robots.txt box. Saving it writes a real robots.txt to your web root, which then permanently replaces the virtual one. The editor is hidden if file editing is disabled in wp-config (DISALLOW_FILE_EDIT).
Rank Math robots.txt editor
Rank Math → General Settings → Edit robots.txt edits the virtual file without writing to disk. It refuses to work and shows a notice if a physical robots.txt already exists in your web root, because that file would win anyway.
A physical file over SFTP
Drop a robots.txt into the WordPress root and your web server serves it directly — WordPress never runs, do_robots() never fires, and every plugin editor above becomes decorative. Full control, and the easiest way to end up with a stale file nobody remembers deploying.
AI crawler rules for WordPress
WordPress core says nothing about AI crawlers, so whatever you want here you have to add. Paste either block into the robots_txt filter, your Rank Math editor, or your physical file. Learn what each bot does on our AI crawlers reference →
# Recommended: let AI crawlers in so you can be cited
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /# Only if you accept losing AI answer visibility
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Google-Extended
Disallow: /Be honest with yourself about the trade. Blocking these crawlers does not protect your rankings — it removes your pages from the crawls that feed ChatGPT, Claude and Perplexity, so you lose the citations and the referral clicks that come with them. Google-Extended is the one narrow exception worth considering on its own: it governs whether your content is used for Gemini and Vertex AI generative products, and it does not affect your inclusion in Google Search.
The WordPress-specific mistakes
These are not generic robots.txt errors. Each one is something only a WordPress site can do to itself.
The "Discourage search engines" checkbox is still on
Settings → Reading → "Discourage search engines from indexing this site". WordPress swaps the default rules for a blanket Disallow: / and adds noindex robots meta tags. It gets ticked on staging, then rides the migration to production, and the site quietly never ranks. If your live robots.txt shows Disallow: /, check this box before anything else.
Blocking /wp-content/ or /wp-includes/
Tutorials from the 2010s still recommend it. Those directories hold your CSS, JavaScript and images — block them and Google renders your pages as unstyled, broken layouts. Leave them crawlable.
A leftover physical robots.txt from a migration
You edit robots.txt in Rank Math, save, reload the live URL, and nothing changed. A real file in the web root beats the virtual one every time. Check for one before you debug anything else.
Using Disallow to keep a page out of Google
Disallow blocks crawling, not indexing. A disallowed URL with inbound links can still appear in results as a bare title with no snippet. To remove a page you need a noindex tag — which means the page has to stay crawlable so the tag can be read.
Blocking the AJAX endpoint by accident
Replacing the default group with a hand-written Disallow: /wp-admin/ and forgetting the Allow line for admin-ajax.php. Anything that loads over AJAX can then render empty to crawlers.
Build the file instead of copying one
Our robots.txt generator takes your URL and produces a file with the right sitemap directive and crawl rules for your site, alongside the rest of a free strategy run.
Open the robots.txt generatorOr let the agent ship it
Rankealo's SEO agent can open a pull request with the robots.txt fix already written, so a crawl-blocking rule gets reviewed and merged instead of sitting in a report.
See how the SEO agent worksrobots.txt on another platform?
WordPress robots.txt questions
Where is the robots.txt file in WordPress?
On a default install there is no file. WordPress generates robots.txt on request through core's do_robots() function, so you will not find one over FTP even though example.com/robots.txt returns content. That virtual file is only used when no real robots.txt exists in the web root — if one does, the web server serves it and WordPress is never consulted.
What should a WordPress robots.txt contain?
For most sites, the core default plus a sitemap line is correct: User-agent: *, Disallow: /wp-admin/, Allow: /wp-admin/admin-ajax.php, and a Sitemap: directive pointing at your real sitemap index. Add rules only for URLs you can name a reason to block, such as faceted or filtered archives that generate near-infinite crawl paths. Do not block /wp-content/ or /wp-includes/.
Why does my WordPress robots.txt say Disallow: /?
Almost always the "Discourage search engines from indexing this site" setting under Settings → Reading. When it is enabled, WordPress serves a robots.txt that disallows everything and adds noindex robots meta tags. Untick it, save, then re-request your robots.txt URL to confirm the default rules are back.
Should I block AI crawlers like GPTBot in WordPress robots.txt?
Only if you are willing to disappear from AI answers. Blocking GPTBot, ClaudeBot or PerplexityBot removes your pages from the crawls that feed AI assistants, so you also lose the citations and referral traffic they produce. Google-Extended is the narrower case: it governs use of your content for Gemini and Vertex AI generative products and does not affect your inclusion in Google Search.
