Audit AI crawler access one path at a time
Inventory robot rules for search, training and user-initiated visits, then verify what reaches the storefront.
Decide what each crawler should reach
Write down your policy for search discovery, training and user-initiated visits separately. For OpenAI, OAI-SearchBot is used for search and GPTBot is used for training. ChatGPT-User can visit pages in response to a person's request. Check providers' documentation when deciding how to express each choice.
Sample URLs before editing
Use a public product path, a collection, an internal search page with a query string and a private account path. Retrieve robots.txt from the exact hostname shoppers use. If you operate alternate market domains, repeat the collection there, since each host publishes its own rules. Save the dated rules and the paths for comparison.
Inspect which group wins
If a named crawler has its own group, evaluate the group's rules for each path. With Disallow: /collections/ and Allow: /collections/daypacks, the more specific allow can matter for the daypack collection. In the checker, keep the full query string when testing a rule targeting a filtered or sorted URL.
Compare the file with actual access
An allowed path can still return a login page, rate-limit response or bot challenge. Review your CDN's access events for traffic attributable to the crawler and sample responses. Look for consistent successful responses with the correct content type. Keep private paths protected with authentication and apply bot rules narrowly to public catalog pages.
Monitor a small set of pages after changes
Record the result for the same pages before and after a rules change. If a single country storefront stops being discovered, check its robots file and canonical URLs first. Pair the crawl record with a storefront inspection to tell whether bots can request the URL and whether the target contains the intended product.