robots.txt is a small text file that tells search engine bots which parts of your website they may crawl. It always lives at the root of your domain, for example example.com/robots.txt.
Crawlers usually read this file before visiting your other pages. One wrong line here can quietly hide your whole site from Google, so it is worth understanding.
What does robots.txt do?
Think of robots.txt as a "Staff Only" sign on certain doors of a shop. Polite visitors such as Googlebot and Bingbot read the sign and stay out. Bad bots, however, can simply ignore it.
The file uses a few simple rules. User-agent names the bot, Disallow blocks a path and Allow makes an exception.
The default WordPress robots.txt
WordPress creates a virtual robots.txt for you, so there is no real file on your server unless you upload one. On a typical site, it looks similar to this:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/wp-sitemap.xmlThis blocks the admin area but still lets bots reach a file that some themes and plugins use on the front end. To change it, you can upload your own file or use an SEO plugin. Here is our guide on how to edit robots.txt through the Yoast SEO plugin.
Mistakes to avoid in robots.txt
- Don't block your CSS and JavaScript files. Google needs them to see your pages properly.
- Don't add
Disallow: /unless you really want to block the entire site. - Don't list secret folders here. Anyone can read robots.txt, so it is not a security tool.
After every change, open yoursite.com/robots.txt in your browser to confirm that the file looks right.
robots.txt controls crawling, not indexing. A blocked page can still appear in Google if other sites link to it. To keep a page out of search results, leave it crawlable and add a noindex tag, as shown in our guide on how to noindex a page in WordPress.
A tidy robots.txt guides good bots to the right places without hiding anything important. That's all about robots.txt. If you have any questions, leave them in the comment box below.