How to Add or Modify the robots.txt File in Your Shopify Store

What is robots.txt?

The robots.txt file serves as a communication tool between websites and web crawlers. It tells search engine bots like Google, Bing, and others which pages they should or should not crawl.

Why Modify robots.txt?

There are three primary reasons for modification:

Control Crawling: Prevent search engines from indexing internal pages such as checkout, cart, or account sections
Improve SEO: Optimize crawl budgets by directing search engines toward important pages
Exclude Duplicate Content: Prevent indexing of duplicate content to improve rankings

Shopify’s robots.txt.liquid

Since 2021, Shopify has allowed merchants to customize the robots.txt file using the robots.txt.liquid template, enabling users to add or remove rules for different bots without extensive coding.

Steps to Add or Modify robots.txt

1. Access the robots.txt.liquid File

Log into Shopify Admin
Navigate to Online Store > Themes
Click Actions > Edit Code for your active theme
In Templates section, select “Add a new template” and choose robots.txt
Name the file robots.txt.liquid

2. Add Rules to Your robots.txt.liquid File

Here are several example configurations:

Blocking Specific Pages:

User-agent: *
Disallow: /cart
Disallow: /checkout

Blocking Specific Search Engine Crawlers:

User-agent: Bingbot
Disallow: /

Allowing Specific Crawlers:

User-agent: Googlebot
Allow: /

User-agent: *
Crawl-delay: 10

3. Save the File

Click Save in the top right corner to apply modifications.

4. Check Your robots.txt File

View live results by visiting https://yourstore.com/robots.txt

Common Use Cases

Blocking Internal Pages:

User-agent: *
Disallow: /cart
Disallow: /checkout
Disallow: /account

Blocking Duplicate Content:

User-agent: *
Disallow: /collections/all?sort_by=best-selling

Preventing Image Indexing:

User-agent: *
Disallow: /images/

Allowing Specific Bots:

User-agent: Googlebot
Allow: /

User-agent: *
Disallow: /

Best Practices

Use Specific Rules: Avoid excessive blocking. Focus on protecting sensitive pages like cart and checkout.
Test Changes: Utilize Google Search Console’s robots.txt Tester tool to verify modifications.
Use noindex Tags: Consider adding noindex meta tags for more granular control over indexing.

Conclusion

Modifying robots.txt allows merchants to control how search engines crawl and index your website while protecting sensitive pages from indexing through Shopify’s accessible template system.