Control which AI crawlers can read your store
Pick allow or block for each major AI crawler and get a clean robots.txt block. For most stores the answer is allow, because a blocked crawler cannot recommend you.
# Generated free with UnderstoodBy.ai (https://www.understoodby.ai/tools) User-agent: OAI-SearchBot Allow: / User-agent: GPTBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: ClaudeBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / User-agent: Google-Extended Allow: /
Why a blocked crawler cannot recommend you
robots.txt is the first thing a crawler reads. If it blocks the AI crawlers, those assistants never see your products, and a brand they can read wins the recommendation instead. Many stores block these bots by accident, with a broad rule copied from an old template.
OAI-SearchBot is the one to watch: it powers ChatGPT Search and Shopping, the surfaces where shoppers actually buy. Blocking it is far more costly than blocking a training-only crawler. Unless you have a specific reason to keep a bot out, allowing them is the visibility-friendly choice.
How to install it
Shopify
- Shopify serves robots.txt from a theme template called robots.txt.liquid.
- In your theme code editor, add (or edit) robots.txt.liquid and include the rules below.
- Confirm the result at yourstore.com/robots.txt.
WordPress / WooCommerce
- Edit robots.txt via your SEO plugin (Yoast, Rank Math) or upload it to the site root over SFTP.
- Paste in the rules below.
- Confirm the result at yoursite.com/robots.txt.
Next.js / custom
- Add the rules to public/robots.txt or your robots route handler.
- Deploy.
- Confirm the result at yourdomain.com/robots.txt.
Questions
Should I block AI crawlers to protect my content?
Usually no, if you want to be found. Blocking the crawler that feeds ChatGPT Search means ChatGPT cannot represent you when shoppers ask. Block only if you have a clear reason, such as a paywalled catalogue.
What is the difference between GPTBot and OAI-SearchBot?
GPTBot gathers data to train OpenAI models. OAI-SearchBot powers live ChatGPT Search and Shopping results. For store visibility, OAI-SearchBot is the higher-impact one to allow.
Does allowing these bots slow my site down?
No meaningfully. They crawl at a measured pace, the same as any search crawler.
Will this replace my whole robots.txt?
No. These are crawler rules to add to your existing file. Keep your current Sitemap line and any other rules you rely on.
Want to know how AI actually reads your store?
This tool fixes one signal. Our free scan checks all of them and gives you a 0 to 100 AI Visibility Score in about a minute, no signup to see it.
Get my free score