How to set up robots.txt in Magento 2?

Robots.txt configuration in Magento 2

Step 1

Go to Content > Design> Configuration.

Step 2

In the opened grid, find the line Global, click the Edit link, and open the Search Engine Robots tab.

Step 3

Now choose what Default Robots you need from the following:

• INDEX, FOLLOW: Search engines will regularly index your pages and check the changes.
• NOINDEX, FOLLOW: Search engine bot won’t index your website but it’ll still monitor changes.
• INDEX, NOFOLLOW: Search bot will index your store once but never came back to track the changes.
• NOINDEX, NOFOLLOW: This setting will hide your store from search engine bots.

Step 4

In the Edit custom instruction of robots.txt File line, you can write the needed custom instructions. Read on to find out more about the option.

Step 5

If you want to reset the default settings, you need to click the Reset To Default button, which will remove all your custom instructions.

Step 6

Don’t forget to Save Configuration to apply the changes.

Custom instructions for Magento 2 robots.txt

As we already mentioned, you can add custom instructions to the robot.txt configurations. Below, we provide examples that you can use in Magento robots.txt for your needs.

Allow Full Access

User-agent:*

Disallow:

Disallow Access to All Folders

User-agent:*

Disallow: /

Default Instructions

Disallow: /lib/

Disallow: /*.php$

Disallow: /pkginfo/

Disallow: /report/

Disallow: /var/

Disallow: /catalog/

Disallow: /customer/

Disallow: /sendfriend/

Disallow: /review/

Disallow: /*SID=

Restrict User Account & Checkout Pages

Disallow: /checkout/

Disallow: /onestepcheckout/

Disallow: /customer/

Disallow: /customer/account/

Disallow: /customer/account/login/

Disallow Catalog Search Pages

Disallow: /catalogsearch/

Disallow: /catalog/product_compare/

Disallow: /catalog/category/view/

Disallow: /catalog/product/view/

Disallow URL Filter Searches

Disallow: /*?dir*

Disallow: /*?dir=desc

Disallow: /*?dir=asc

Disallow: /*?limit=all

Disallow: /*?mode*

Restrict CMS Directories

Disallow: /app/

Disallow: /bin/

Disallow: /dev/

Disallow: /lib/

Disallow: /phpserver/

Disallow: /pub/

Disallow Duplicate Content

Disallow: /tag/

Disallow: /review/

To forget about glitches in Google's indexing, generate your XML website sitemaps automatically. As a result, you'll achieve faster content indexing, exclude irrelevant pages from indexing, define the frequency of updates for pages and get higher positions in SERPs.