Home / Free tools / Website tools / WordPress robots.txt generator
WordPress robots.txt generator
This WordPress robots.txt generator builds a clean robots.txt for your site. Keep the WordPress defaults, add your sitemap, block search pages or AI crawlers, then copy or download the file.
Loading the WordPress robots.txt generator.
How to use the WordPress robots.txt generator
- Admin area. Leave this on. It blocks /wp-admin/ and keeps admin-ajax.php open, the same as WordPress does.
- Internal search results. Turn this on to stop crawlers spending time on /?s= and /search/ pages.
- Sitemap URL. Paste the full address of your sitemap. Yoast SEO and Rank Math use /sitemap_index.xml. WordPress on its own uses /wp-sitemap.xml.
- AI crawlers. Tick each crawler you want to keep out.
- Extra paths. Add one path per line, such as /cart/ or /checkout/.
- Copy or download. Upload the file to the top folder of your site, so it opens at yoursite.com/robots.txt.
The default WordPress robots.txt
WordPress serves a virtual robots.txt when no file exists. There is nothing on the disk. WordPress writes the text each time a crawler asks for /robots.txt. It looks like this:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
A real robots.txt file in the site root replaces the virtual one. The web server sends your file and WordPress is never asked, so anything a plugin adds to the virtual file stops showing too.
Worked example
With the options as they load, the generator writes the three default lines, a blank line and the sitemap:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/sitemap_index.xml
Tick GPTBot and it adds a group for that crawler alone:
User-agent: GPTBot
Disallow: /
A crawler follows the group that names it and skips the general group. That is why each AI crawler gets its own block with a full Disallow.
AI crawlers you can block
| Name in robots.txt | Run by | What it is for |
|---|---|---|
| GPTBot | OpenAI | Collects pages to train AI models |
| ClaudeBot | Anthropic | Collects pages to train AI models |
| CCBot | Common Crawl | Open web archive, often used for AI training |
| Google-Extended | Controls use of your pages for Gemini | |
| PerplexityBot | Perplexity | Indexes pages for its AI search answers |
Google-Extended is a control name, not a separate crawler. Blocking it does not change how your site appears in Google Search.
Rules for a safe robots.txt for WordPress
- Never block /wp-content/ or /wp-includes/. Google renders your pages like a browser. It needs the CSS, JavaScript and images in those folders. Block them and Google sees a broken page, which can hurt how it is ranked. The generator refuses to write those rules.
- Robots.txt controls crawling, not indexing. A blocked URL can still appear in Google if other pages link to it. To keep a page out of search, use a noindex tag and leave the page open to crawling. The meta tag generator writes that tag.
- Google ignores crawl-delay. The generator does not add it.
- The sitemap line needs a full URL. It must start with https:// and can sit anywhere in the file.
- It is not a lock. Robots.txt is public and well-behaved crawlers choose to follow it. Do not use it to hide private pages.
- Check it after upload. Open yoursite.com/robots.txt in a browser, then look at the robots.txt report in Google Search Console.
Next, describe your pages to search engines with the schema markup generator. More tools like this are on the website tools page.
How this robots.txt generator was checked
Built by Rasel Hakkani, WordPress developer at Swift Web Dev. The output was checked line by line for the default options and with every option switched on, and the downloaded file was compared with the text shown. It has not been run through Google’s own robots.txt parser, so check the robots.txt report in Search Console after you upload it.
Sources
- Google Search Central: Introduction to robots.txt
- WordPress Developer Resources: do_robots(), the function that outputs the default virtual robots.txt
Last checked: October 2026. Found a wrong answer? Tell us and we will fix it.
WordPress robots.txt generator questions
Where is the robots.txt file in WordPress?
By default there is no file. WordPress creates a virtual robots.txt each time someone opens yoursite.com/robots.txt. If you upload a real robots.txt to the root folder of the site, that file is served and the virtual one is no longer used.
What is the default WordPress robots.txt?
It has three lines. User-agent: * comes first, then Disallow: /wp-admin/ and Allow: /wp-admin/admin-ajax.php. Since WordPress 5.5 it also adds a Sitemap line that points to wp-sitemap.xml, unless a plugin has turned the core sitemap off.
Does robots.txt stop a page from showing in Google?
No. It only stops crawling. A blocked page can still be indexed if other pages link to it. To keep a page out of search results, add a noindex robots meta tag and leave the page open to crawling so Google can see the tag.
Should I block wp-content or wp-includes?
No. Google loads your CSS, JavaScript and images from those folders to render the page the way a visitor sees it. Blocking them can make the page look broken to Google. This generator never writes those rules.
How do I block AI crawlers in robots.txt?
Add a group for each crawler with its user agent name and a Disallow line for the whole site. For GPTBot, that is User-agent: GPTBot followed by Disallow: /. Tick the crawlers in the generator and it writes the groups for you. Crawlers that ignore robots.txt need a firewall rule instead.
Does Google follow crawl-delay?
No. Googlebot ignores the crawl-delay line. Some other crawlers, such as Bingbot, do read it. This generator leaves it out.
More free website tools
Need a tool like this on your website?
Swift Web Dev builds custom calculators and tools for WordPress sites: quote forms, pricing estimators, booking logic and more.