BLOG
A robots.txt audit can tell you whether a site’s stated crawling rules permit Googlebot and OpenAI’s search crawler to reach the pages you want discovered. It cannot prove that either system will index, cite or recommend the site. It also does not answer every question about AI training. Google Search access, ChatGPT Search access and OpenAI training controls involve different crawlers and separate decisions.
Google’s robots.txt documentation explains the file’s crawler-control role. OpenAI’s crawler overview identifies OAI-SearchBot for ChatGPT search features and GPTBot for content that may be used to train generative AI foundation models. Read the official descriptions before copying a rule from a blog post or a plugin preset.
What robots.txt does, and what it cannot do
The public file at /robots.txt gives cooperating crawlers instructions about paths they may request. A Disallow rule can prevent crawling of a path for a named user agent. It is not an access-control system for private data; anyone who knows a public URL can often still request it. Keep confidential pages behind authentication and proper server controls, not merely a robots rule.
Robots blocking also is not the same as a noindex directive. A crawler prevented from loading a page may not see a noindex instruction in that page. A blocked URL can still be known from other links even when its contents are not crawled. Decide whether your aim is to prevent crawling, prevent indexing, or restrict access, then use the mechanism that actually serves that aim.
A sitewide `Disallow: /` can be appropriate for a private development environment, but it is a serious mistake on a public site whose pages should appear in search. Conversely, removing every restriction from a public site’s file does not mean every URL should be indexed. Internal search results, duplicate filters and administrative paths often have separate reasons to stay out of search.
Googlebot and Google’s AI search features
Googlebot is the familiar crawler for Google Search. Google’s AI features and website guidance says a page must meet normal Search technical requirements to be eligible as a supporting link in AI Overviews or AI Mode. That includes allowing crawling through robots.txt and hosting or CDN infrastructure, and being indexable and eligible for a snippet. A separate “AI Overview bot” rule is not a documented shortcut to eligibility.
If a priority service page is blocked for Googlebot, inspect the exact rule and path. Do not edit the entire file based on one report row. The page may also have a noindex directive, an incorrect canonical, a redirect or a server error. Fixing a robots rule alone cannot resolve those other signals. A Search Console live test can help compare the current public response with a previous crawl.
Even when a page is accessible and indexed, Google chooses whether it appears for a query and whether an AI feature links to it. The robots file is an access prerequisite, not a ranking control. Keep that distinction in a report to a business owner.
OAI-SearchBot is for ChatGPT Search
OpenAI documents OAI-SearchBot as the crawler used to surface websites in ChatGPT search features. The documentation recommends allowing it in robots.txt and allowing requests from published IP ranges if a site wants to appear in search results. A CDN or firewall can still return 403 to a real crawler even when robots.txt says Allow, so the audit must include both the policy file and the server response.
OpenAI says a site that opts out of OAI-SearchBot will not be shown in ChatGPT search answers, though it can still appear as a navigational link. That distinction matters: the rule affects the documented search-crawling route, but not every possible user interaction with a URL. OpenAI also says its systems can take about a day to adjust after a robots.txt update. Do not treat an immediate manual prompt as a conclusive test.
Check the actual user-agent and IP information in the current official documentation when reviewing logs or firewall rules. A request that merely claims to be OAI-SearchBot may be spoofed. Security rules should be changed based on verified identification and the site’s risk policy, not because someone copied a sample user-agent string into a dashboard.
GPTBot is a different policy choice
OpenAI describes GPTBot as a crawler for content that may be used in training its generative AI foundation models. Its documentation says a site can allow OAI-SearchBot for search while disallowing GPTBot to signal that its crawled content should not be used for training. The settings are independent. A broad rule for every bot may therefore create a result the site owner did not intend.
ChatGPT-User is another documented agent used for certain visits triggered by users and custom GPT actions. OpenAI says it is not the automatic crawler that determines Search opt-outs. Do not use a ChatGPT-User rule as a substitute for the OAI-SearchBot decision. The site owner may choose different policies for search, training and user-triggered access, but should record them explicitly.
These are OpenAI-specific controls. Do not assume that another AI company’s crawler reads the same user-agent groups or applies the same training policy. Consult each provider’s current primary documentation before adding a new rule.
Separate the checks in one table
| Agent or mechanism | Documented role | What an allow rule does not prove |
|---|---|---|
| Googlebot | Google Search crawling | Indexing, ranking or AI Overview link |
| OAI-SearchBot | ChatGPT search crawling | Inclusion in a given answer |
| GPTBot | OpenAI model-training crawl control | ChatGPT Search visibility |
| ChatGPT-User | Some user-triggered visits | Automatic Search crawl policy |
| CDN or firewall | Actual request filtering | Whether robots.txt expresses the desired policy |
A site can show an Allow rule and still return 403. It can also return 200 and contain an unintended noindex directive. The full audit therefore checks policy, response and the page’s own search signals.
Audit a WordPress site’s actual file
Open `https://your-domain.example/robots.txt` anonymously and note the final URL and response. WordPress can serve a virtual robots.txt file when no physical file exists, while a plugin, hosting panel or CDN may override it. The public response is what a crawler sees; editing a file you assume is active may change nothing. Check both the root domain and any separate site or subdomain that has its own content.
Read each user-agent group and the paths affected. Identify important service pages, articles and sitemaps. If the site has an English domain and a separate Brazilian domain, audit each independently and do not copy rules blindly across them. Their Search Console properties and analytics should also remain separate. Check whether a host or CDN rule blocks the same pages after robots.txt permits them.
Save the current response and the intended change before editing. After a change, fetch the public file again, inspect the exact affected URL, and test in Search Console where applicable. A cached robots file may continue to be served briefly. Avoid making several unrelated edits at once; otherwise you cannot tell which one affected access.
An illustrative policy, not a paste-ready answer
A site owner who wants ordinary search access and ChatGPT Search access but chooses to opt out of OpenAI training might permit Googlebot and OAI-SearchBot while disallowing GPTBot. The exact syntax and interaction with existing groups must be checked for the site; a generic snippet can override broader rules in ways the owner did not intend. Start with the current file, not a blank template.
Do not block `/wp-content/` wholesale because it “looks technical.” Search systems may need images and CSS to understand the page. Likewise, do not allow administrative or private paths simply to clear a diagnostic warning. A robots.txt policy should reflect the site’s information architecture and privacy posture.
For any production edit, have a rollback copy and verify the response after deployment. A mistyped wildcard or an unintended `Disallow: /` could affect a large part of the public site. Keep the audit and the change log separate from promotional claims about AI visibility.
Measure the effect without overclaiming
Server logs can show whether a verified crawler requested robots.txt or a page. They do not establish that the page appeared in a user-visible answer. Search Console can report Google’s crawl and search outcomes for its own systems. Analytics can observe visits that arrive from a search result, though attribution may be incomplete. These are related signals, not interchangeable counts.
If the goal is ChatGPT Search visibility, check that OAI-SearchBot is permitted and the server responds normally, then monitor observed referral visits and relevant customer contacts. Do not promise a citation based on a bot visit. If the goal is Google AI features, check normal Search eligibility and available performance reporting. Do not add generative-feature impressions to the overall total as if they were separate traffic.
Review the policy when a platform’s documentation changes. A robots file copied years ago may reflect a crawler name or intention that no longer applies. Keep a short note of who approved the policy and why, particularly when search access and training use are intentionally different.
Frequently asked questions
Robots.txt permits the bot but the CDN returns 403. What should I check? Identify the actual requester using the provider’s published verification information, inspect the CDN rule and avoid a blanket security bypass. A robots Allow rule does not override a firewall.
WordPress shows a different robots.txt from the file I edited. Which one matters? The response at the public root URL matters. Check whether WordPress, a physical file, a plugin, hosting or a CDN controls it.
Will allowing OAI-SearchBot force ChatGPT to cite my page? No. It addresses the documented search-crawl route; answer selection is separate.
Can I block GPTBot and still allow ChatGPT Search? OpenAI documents these as independent controls, so a site can make that policy choice. Verify the final public rules and the current documentation.
Final checklist
Fetch the public robots.txt; identify important paths; review Googlebot, OAI-SearchBot and GPTBot groups separately; test public page and CDN responses; check noindex and canonical on the page; and record the policy decision. Treat crawl access as one condition for discoverability, not proof of indexing, citation or a lead.
