Shopify robots.txt.liquid, Tags and Filters: How to Stop Wasting Crawl Budget

Quick answer: Shopify generates a robots.txt file automatically, and Shopify says its default works for most stores. You can customise it by adding a robots.txt.liquid template to your theme, which lets you add or remove rules while keeping Shopify’s defaults. Use it to block genuinely useless URL patterns or to manage AI crawlers. Use noindex, not robots.txt, for pages you want out of Google. Shopify treats editing this file as an unsupported customisation, so test it and have a developer review it first.

What is robots.txt?

Robots.txt is a small text file with a big job: it tells search engines and other crawlers which parts of your site they may visit. Get it right, and Googlebot spends its time on your products and collections. Get it wrong, and a single line can block your whole store from Google.

Shopify has let store owners edit this file since 2021, through a theme template called robots.txt.liquid. This guide explains what Shopify’s default file does, when you should and shouldn’t change it, and how to make the common changes safely.

robots.txt is a text file at the root of your domain (yourstore.com.au/robots.txt) that tells crawlers which URLs they’re allowed to request. It uses simple rules. This is a plain example of the format, not something to paste into Shopify:

User-agent: *
Disallow: /cart/
Sitemap: https://yourstore.com.au/sitemap.xml

• User-agent says which crawler the rules apply to (* means all).
• Disallow blocks a path. Allow makes an exception.
• Sitemap tells crawlers where your sitemap is.

Two things to remember:

1. robots.txt controls crawling, not indexing. Google says a blocked URL can still be indexed if other pages link to it. Google just can’t read the page.
2. It’s a request, not a lock. Reputable crawlers follow it. Others might not.

Rules are also case-sensitive and use * as a wildcard for “any characters”, which matters later when we get to mistakes.

What Shopify’s default robots.txt blocks

Open yourstore.com.au/robots.txt to see your current file. According to Shopify’s Help Center, the current default looks like this:

Two cautions. First, your live file may differ: Shopify says that if your theme has a robots.txt.liquid template that outputs the default rule groups, your file can include rules the current default no longer has, such as Disallow: /search and Disallow: /policies/. Second, Shopify updates its defaults over time, so read your own file rather than trusting a list in a blog post. Your file also lists your sitemap.

For most stores, the defaults are good. They cover the biggest sources of crawl waste without touching products, collections, pages or blog posts.

Do you need to edit it?

Edit robots.txt.liquid only if you have a specific, tested reason. Common good reasons:

• An app or theme creates crawlable URL patterns with no value, such as alternate views (?view=quick), app-generated parameters or duplicate pages.
• A large store where Search Console shows heavy crawling of junk URLs. Look in Settings > Crawl stats, or for many “Crawled - currently not indexed” pages from filters.
• You want to allow or block specific AI crawlers.
• You need to remove a default rule that blocks something you genuinely want crawled (rare).

Bad reasons:

• Trying to remove pages from Google (use noindex).
• Hiding duplicate product URLs (fix the internal links instead).
• “Improving SEO” without evidence of a crawl problem.

Shopify’s Help Center says editing robots.txt.liquid is an unsupported customisation and that Shopify Support can’t help with edits to it. It also warns that incorrect use can result in loss of all traffic. Make changes carefully.

robots.txt vs noindex: which to use

The classic mistake: blocking a page in robots.txt and adding noindex. Google says the page must not be blocked by robots.txt for noindex to work, so if you block it, Google never sees the noindex and the URL can stay in the index.

Noindex snippet for tag-filtered collection pages

This is a fragment, not a robots.txt.liquid file. It goes inside the <head> of your theme’s theme.liquid layout, not in robots.txt.liquid:

{%- if template.name == 'collection' and current_tags -%}
  <meta name="robots" content="noindex, follow">
{%- endif -%}

Two warnings before you use it:

• current_tags is Shopify’s object for the tags applied to a collection or blog page. Shopify’s documentation lists it for collection and blog templates, so ask your developer to confirm it works where you place it.
• Storefront filtering uses filter.* URL parameters (for example ?filter.p.tag=new). Shopify’s documentation doesn’t say current_tags is filled on those URLs, so this snippet may not catch them. Check a real filtered URL (view the page source, or use URL Inspection in Search Console) before you rely on it.

How to create robots.txt.liquid

Follow Shopify’s steps, and do the safety steps below before you save anything.

1. In your Shopify admin, open Online Store > Themes. On the theme, open the menu (…) and choose Edit code.
2. Click Add a new template, choose robots, and click Create template.
3. Edit the file Shopify creates, then save it. Shopify’s Help Center says to save the changes in your published theme.

Menu labels can change, so check Shopify’s current Help Center steps on the day. Shopify’s developer documentation also says the template can’t be a JSON template and that adding it replaces Shopify’s generated robots.txt output.

Before you publish: duplicate, review, then test

Changes to robots.txt are instant on a live store, so treat this like any other change to live theme code.

1. Duplicate your theme first (Online Store > Themes > … > Duplicate). It gives you a backup to return to.
2. Save a copy of your current /robots.txt. Shopify’s developer documentation recommends this so you can compare it with the rendered output.
3. Have a developer review the code before it goes live. Shopify Support won’t review it for you.
4. Publish at a quiet time, then compare immediately. Open /robots.txt, check it against your saved copy, and confirm only the lines you meant to change are different. If anything looks wrong, delete the template (see the FAQ) to go back to Shopify’s default.

The standard template

This is a complete robots.txt.liquid template. It’s the standard loop Shopify’s documentation builds on. Shopify generates the starting file for you, so begin with what Shopify created rather than retyping this:

{% for group in robots.default_groups %}
  {{- group.user_agent }}

  {%- for rule in group.rules -%}
    {{ rule }}
  {%- endfor -%}

  {%- if group.sitemap != blank -%}
    {{ group.sitemap }}
  {%- endif -%}
{% endfor %}

It loops through Shopify’s default groups and prints each user agent, its rules and the sitemap. Always build on this loop rather than replacing it with a hard-coded file. Shopify strongly recommends using its Liquid objects where possible, and the loop keeps the defaults in place.

Warning: pasting a fragment from this guide over the whole file wipes Shopify’s defaults. Only the snippets labelled “complete robots.txt.liquid template” replace the whole file. A “fragment” is added to your existing template, in the place the label tells you.

Code examples: add, remove and customise rules

After every change, open /robots.txt and confirm each directive sits on its own line.

Example 1: Add a rule for all crawlers

This is a complete robots.txt.liquid template. It adds one rule inside the loop, to the group for all crawlers (*). It blocks an alternate-view parameter, such as one created by a quick-view feature:

{% for group in robots.default_groups %}
  {{- group.user_agent }}

  {%- for rule in group.rules -%}
    {{ rule }}
  {%- endfor -%}

  {%- if group.user_agent.value == '*' -%}
    {{ 'Disallow: /*?view=*' }}
  {%- endif -%}

  {%- if group.sitemap != blank -%}
    {{ group.sitemap }}
  {%- endif -%}
{% endfor %}

Check first that no valuable page uses that parameter. This pattern only matches URLs where view= is the first parameter after the ?. If the parameter can appear after another one, ask your developer to add a matching rule and check the rendered file.

Example 2: Remove a default rule

Each rule has a directive (such as Disallow) and a value (the path). To skip one default rule, for example if you deliberately want /policies/ crawled, wrap the rule output in unless. This is a complete robots.txt.liquid template:

{% for group in robots.default_groups %}
  {{- group.user_agent }}

  {%- for rule in group.rules -%}
    {%- unless rule.directive == 'Disallow' and rule.value == '/policies/' -%}
      {{ rule }}
    {%- endunless -%}
  {%- endfor -%}

  {%- if group.sitemap != blank -%}
    {{ group.sitemap }}
  {%- endif -%}
{% endfor %}

Only remove defaults when you understand exactly why they’re there.

Example 3: Add a group for a specific crawler

This is a fragment. Paste it after the final {% endfor %}, at the very bottom of your existing template, outside the loop. It blocks one crawler from the whole site (replace ExampleBot with the real crawler name):

User-agent: ExampleBot
Disallow: /

A crawler follows only the most specific group that matches it, and Google says specific groups aren’t merged with the * group. So this block applies to ExampleBot instead of the * rules.

Example 4: Add an extra sitemap

This is a fragment too. Paste it after the final {% endfor %}, outside the loop. Use it if you host a separate sitemap, for example for a headless section. Shopify’s robots.txt.liquid template only supports a small set of Liquid objects, so write the full URL out by hand:

Sitemap: https://yourstore.com.au/sitemap-extra.xml

Managing AI crawlers

AI companies use different crawlers for different purposes. Blocking the wrong one can remove you from AI search answers. This table is based on each company’s own documentation (Google, OpenAI, Perplexity, Anthropic and Common Crawl), which can change.

Anthropic also runs Claude-SearchBot and Claude-User, which relate to search and user requests, so blocking those can reduce visibility in Claude’s answers. Perplexity-User is a user-initiated fetcher that Perplexity says generally ignores robots.txt.

A sensible starting point for Shopify brands that want AI visibility: allow the search-related crawlers (Googlebot, OAI-SearchBot, PerplexityBot), and treat the training crawlers (GPTBot, Google-Extended, ClaudeBot, CCBot) as a business choice.

Example: block GPTBot (training) but keep ChatGPT search. This is a fragment. Paste it after the final {% endfor %}, outside the loop:

User-agent: GPTBot
Disallow: /

No rule is needed for OAI-SearchBot. It’s allowed unless you block it. OpenAI says it can take about 24 hours after a robots.txt update for its systems to adjust.

Two more things worth knowing:

• Robots.txt rules are advisory, and Shopify says not all crawlers are guaranteed to follow them. Shopify also says it handles the network layer for stores on Shopify, and doesn’t recommend putting a proxy in front of your store. If you’ve added a bot-blocking app or proxy anyway, check it isn’t blocking AI search crawlers without you knowing.
• Shopify says that blocking AI crawlers in robots.txt doesn’t stop Shopify Catalog from sending product data to agentic storefronts you’ve activated. For the wider picture of AI visibility, see the state of AI Overviews for Shopify and our AI SEO service.

Tags, filters and crawl budget

Crawl budget is roughly how many URLs Google is willing and able to crawl on your site in a period. Google’s own crawl budget guide says it’s aimed mainly at very large sites: roughly 1 million or more unique pages, or 10,000 or more with very fast-changing content. Google calls those figures a rough estimate. So most small stores don’t need to worry about it. It’s worth a look when:

• You have a large catalogue.
• Filters and tags create very large numbers of URL combinations.
• Search Console shows Google spending most of its crawls on parameters instead of products.

How to check: in Search Console, open Settings > Crawl stats. Look at the crawl file types breakdown and click through to the example URLs. Google notes example URLs aren’t comprehensive, but if most you see are filter, sort, tag or app URLs, you may have crawl waste.

How to fix, in order:

1. Stop linking to junk URLs. Filters shouldn’t output crawlable links for every combination.
2. Noindex thin tag pages (snippet above), or turn valuable ones into real collections. See our guide to Shopify collection page SEO and why category pages are such a high-leverage SEO asset.
3. Block patterns with no value in robots.txt.liquid, after confirming they get no organic traffic (Search Console > Performance, filtered by URL). Google’s guide says that if you can’t consolidate duplicate pages, you can block unimportant ones with robots.txt.
4. Fix internal links to duplicate product URLs. Shopify’s within filter puts a product link inside a collection’s URL (/collections/sale/products/example). If your theme uses it in collection grids, ask your developer whether those links should point to the main product URL.

How to test changes

1. View the live file. Open yourstore.com.au/robots.txt and read it line by line against the copy you saved.
2. Use Search Console’s robots.txt report (under Settings) to check Google fetched the new file and to see any errors. Google refreshes its cached copy about once a day, and the report has a “Request a recrawl” option if you need it sooner.
3. Use URL Inspection on key URLs, such as your homepage, a collection and a product, to make sure they aren’t blocked.
4. Test patterns before you publish with a robots.txt testing tool, using a list of real URLs. Include some that should be blocked and, more importantly, ones that must stay crawlable. Google publishes an open-source robots.txt library developers can use.
5. Monitor for the next few weeks. Watch Crawl stats, the Page indexing report and organic traffic. Shopify notes that changes are instant but crawlers don’t always react immediately.

Mistakes that can deindex your store

1. Disallow: / under User-agent: *. Blocks your entire store from every crawler that follows the rules. One line is all it takes.
2. Replacing Shopify’s loop with a hard-coded file. You lose the default rules and any future updates to them.
3. Blocking /collections/ or /products/ to “fix duplicates”. This blocks your money pages.
4. Blocking JavaScript, CSS or image files your pages rely on. Google says it won’t render JavaScript from blocked files or on blocked pages.
5. Blocking and noindexing the same URL, so Google never sees the noindex.
6. Blocking AI search crawlers by accident, through security tools or copied rules.
7. Wildcards that match more than intended. Disallow: /*sale matches any path containing “sale”, including /collections/sale. Because paths are case-sensitive, test the exact URLs you care about.

Key takeaways

• Shopify generates robots.txt automatically, and its defaults already block cart, checkout, account, sorted collections and filtered collection pages.
• You can customise it through a robots.txt.liquid theme template. Keep Shopify’s default loop and add to it, and label every change as either a complete template or a fragment before you paste it.
• robots.txt controls crawling, not indexing. Use noindex to remove pages from Google, and don’t block noindexed pages.
• Allow search-related AI crawlers (OAI-SearchBot, PerplexityBot, Googlebot) if you want AI visibility. Decide on training crawlers separately.
• Crawl budget mainly matters for very large sites. Fix links and noindex thin pages before blocking patterns.
• Duplicate your theme, save your current file, get a developer to review, then test every change with Search Console’s robots.txt report and URL Inspection.

FAQs

Can you edit robots.txt in Shopify?

Yes. Shopify lets you customise robots.txt by adding a robots.txt.liquid template to your theme. It outputs Shopify’s default rules, and you can add or remove rules with Liquid. Shopify treats it as an unsupported customisation, so test changes carefully.

Where is the robots.txt file in Shopify?

Shopify serves it automatically at yourdomain.com/robots.txt. To change it, open your theme’s code editor, add a new template, choose robots, and edit the robots.txt.liquid file that Shopify creates.

Should I block AI bots on my Shopify store?

Only if you’ve decided you don’t want your content used by them. Blocking search-related crawlers like OAI-SearchBot or PerplexityBot can remove you from those AI answers. Many stores allow the search crawlers and decide separately on training crawlers like GPTBot or Google-Extended.

Does robots.txt remove pages from Google?

No. robots.txt stops crawling, not indexing. A blocked URL can still be indexed if other pages link to it. To keep a page out of Google, use a noindex meta tag and let Google crawl the page. If the page should be gone altogether, remove it and let the URL return a 404 or 410, which Google treats as an error and removes from its index.

Should I block Shopify collection filter URLs?

Shopify’s default robots.txt already blocks filtered and sorted collection URLs. Block more only if Search Console shows Google wasting crawls on them and they get no organic traffic. If a filtered view has real search demand, make it a proper collection instead.

Do small Shopify stores need to worry about crawl budget?

Usually not. Google’s crawl budget guide is aimed mainly at very large sites. Check Search Console’s Crawl stats before making changes.

Why does Search Console say a URL is “indexed, though blocked by robots.txt”?

Shopify’s Help Center calls this normal. It can mean Google’s data about your store is out of date, since updates can take days, weeks or months. Usually no action is needed.

How do I undo a robots.txt.liquid change?

Delete the robots.txt.liquid template so Shopify serves its current default file. In Online Store > Themes > Edit code, right-click robots.txt.liquid and choose Delete permanently. Then check yourdomain.com/robots.txt and the robots.txt report in Search Console. You can recreate your customisations later from a saved copy.

Need a second pair of eyes?

Robots.txt is one of the few SEO settings where a small mistake can hide your entire store. If you’re unsure whether you need to change it, you probably don’t. If you do, test carefully.

Fixing thin tag and filter pages and robots.txt rules is part of the technical work in our Shopify SEO service. Or book a free 30-minute diagnostic and we can talk through your robots.txt and crawl setup.