How to Edit robots.txt through Yoast SEO Plugin?

Robots.txt tell search engines robots which pages you would like them not to visit. The bots won’t go to those restricted areas anymore. Normally, the WP-Content areas and Plugin areas in your websites are blocking the search bots to secure the files or to avoid unwanted indexing. There are several situations where you need to edit your robots.txt file for a better configuration or to Allow or Disallow the bots access from a specific area of your website. To edit the robots.txt file, you don’t need to do anything more if you have the Yoast SEO plugin installed on your WordPress website.

In this article on “How to edit robots.txt through Yoast SEO Plugin?” we are teaching you to easily edit the robots.txt file of your website using the Yoast SEO plugin. So, let’s have a look at it.

What is Robots.txt

Robots.txt is a file that webmasters can use to communicate with the search engine bots. The file is placed in the root of your website and is generally accessible from the URL – example.com/robots.txt.

The file can be used for various purposes such as allowing / disallowing crawler access to certain sections of your website, define the sitemap location, etc. For more usage details, refer to the next section.

Even though it is possible to define rules in the robots.txt file, it is completely up to the bots to decide whether to respect it or not. There are crawlers that wouldn’t even check the file before crawling your website. Such crawlers are mostly from aggressive web scrapers or other similar bad actors.

Uses of Robots.txt in WordPress

There are many uses for a robots.txt file on your WordPress website. Some of them are listed below –

• To tell search engines whether to crawl your website or not.
• To prevent search engine bots from indexing certain parts of your website.
• Preserve crawl budget by preventing search engines from crawling unnecessary pages.
• To define the location of your site map, especially if it is placed in a non-traditional location.

How to edit robots.txt through Yoast SEO Plugin?

Follow the simple steps given below to edit the robots.txt file using the Yoast SEO plugin.

1. Login to your WordPress Dashboard.
   
2. Go to SEO then Tools tab on the left sidebar.
   
3. Click on File Editor option.
   
4. Now, you will be shown a window where you can edit your robots.txt.
5. Make Necessary changes and save it.
6. Now, the robots.txt of your website is successfully edited

Note: Be careful when making changes to the robots.txt file of your website. As it can control the crawlers, the misconfigurations can harm your website. If you are not aware of what you are going to do, don’t edit it.

Example of a normal WordPress robots.txt file

Sitemap: https://example.com/sitemap_index.xml

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

If you don’t have a Robots.txt file on your website, Create one

1. Create an Empty text file named as robots.txt using your favorite text editor.
2. Upload it to your server root folder.
3. Then, you can edit the file using the Yoast SEO Plugin or Directly using an FTP software.

If you face any problems related to the file uploading, contact the hosting support team. They may help you to resolve the issue.