Robots.txt for AI Search Bots: Configure OAI-SearchBot

To configure robots.txt for ai search bots, add a clear, named rule group for each search crawler you want to permit, such as OAI-SearchBot and PerplexityBot, then check that your CDN and server allow their requests. Keep training-related crawlers separate if your policy requires it. Most importantly, robots.txt is a public crawler request, not a way to secure private pages or files.

This hands-on guide takes you from checking your current file to testing the result. If you want a quick initial check, SearchFIT’s free AI Crawler Checker checks which AI crawlers your robots.txt allows or blocks. You will still need to verify the actual page response and your site’s edge rules.

For more practical guidance on AI search, browse SearchFIT’s resources.

Goal and what you need

The goal is to make your public pages available for discovery by the AI search crawlers you choose, while keeping private material private and making sure your existing search rules still work. The example below allows OAI-SearchBot and PerplexityBot to crawl public pages, excludes a few example transactional paths, and separately blocks OpenAI’s GPTBot.

You will need:

• The public website’s root URL, including its exact host, such as https://www.example.com.
• Access to the existing root-level robots.txt file, either through your CMS, hosting platform, code repository or server.
• A list of public pages you want crawled and private paths that must be protected through authentication.
• Access to your CDN, firewall or web application firewall settings, if your site uses them.
• A way to check HTTP responses and, ideally, review server or CDN access logs.

Do not copy the example paths without checking your site. /account/, /checkout/ and /preview/ are placeholders. Your own private areas might use different URLs, but the correct protection for those areas is authentication and server-side authorisation, not a robots.txt rule.

What robots.txt does, and what it cannot do

A robots.txt file tells compatible crawlers which parts of a site they are asked to crawl. The Robots Exclusion Protocol standard describes these rules as crawler instructions, not access authorisation. A crawler may choose not to follow them, and the file itself is publicly readable.

That distinction matters when you configure AI search bots. A Disallow line does not make a page private, prevent someone from requesting it, or guarantee that a URL will stay out of search results. Google also warns that robots.txt is not a security mechanism and that blocked URLs can still appear in search without their page content. For information that must not be public, require a login, enforce permissions on the server, or remove the resource.

OpenAI documents OAI-SearchBot as the crawler used to surface websites in ChatGPT Search. It lists GPTBot separately as a crawler associated with content that may be used to train OpenAI’s generative AI models. OpenAI’s crawler documentation explains these roles. In practice, you can allow one and disallow the other with separate groups.

Perplexity documents PerplexityBot as its search-results crawler and says it is not used to crawl content for foundation-model training. Keep the decision about search discovery separate from any policy you have about model training. Do not assume every AI-related crawler has the same purpose or uses the same name.

The user-initiated fetchers are different from automatic search crawlers. OpenAI and Perplexity explain these distinctions in their respective crawler documentation. In particular, a robots.txt preference for an automatic crawler should not be treated as a guarantee that a user-requested page visit will be blocked.

Google says Google-Extended is a robots.txt control for Gemini-related training and grounding, and it does not affect inclusion in Google Search or act as a Search ranking signal. See Google’s documentation on its common crawlers. Choose rules by the specific crawler and purpose, rather than using one broad “AI bots” setting.

Step 1: Inspect the current robots.txt file

Open the file at the root of the host you want to check. For example:

https://www.example.com/robots.txt

Look for the following before editing:

• A wildcard group beginning with User-agent: *.
• Existing groups for OAI-SearchBot, PerplexityBot, GPTBot or other named crawlers.
• Broad rules such as Disallow: / that might prevent crawling across the site.
• Rules covering public sections, images, scripts, stylesheets or product content.
• A sitemap line and whether its URL is still correct.

Check each relevant host separately. https://www.example.com/robots.txt does not automatically set the rules for https://example.com/, a shop subdomain, a documentation subdomain, or a staging host. The Google robots.txt specification explains that the file applies to the protocol, host and port where it is served. Put it at the root, not in a subdirectory.

If you see a broad block, do not simply paste Allow: / into the file and assume everything is fixed. A crawler that matches a named group should not be assumed to inherit rules from User-agent: *. Put the rules you intend that named crawler to follow in its own group. The standard also says the most specific matching path rule takes precedence; where an Allow and Disallow rule have the same length, Allow takes precedence.

What you should see: a clear picture of the rules currently served for each host, and a short list of the crawlers and URL paths whose policies need attention. Save a copy of the original file before making changes so you can restore it if a site-wide rule behaves unexpectedly.

Step 2: Choose a policy for each crawler

Write down your decision for each crawler before editing. For example:

This avoids a common mistake: blocking every crawler with “AI” in its name when the actual decision is only about training-related use. It also avoids the opposite mistake, assuming that allowing a search crawler automatically grants access only to a safe subset of content. Your public pages remain crawlable unless your rules say otherwise, and private resources need a real access check.

If you are deciding whether to allow search crawlers, consider which pages should be discoverable, not just whether the homepage loads. A public product page, help article or category page may be appropriate; a logged-in account view or unpublished preview is not. For an overview of how technical access connects with broader AI search visibility, see SearchFIT’s AI search features. A crawl permission is only one part of the picture; it does not guarantee a citation, ranking or appearance in an answer.

Step 3: Add named crawler groups

Edit the root file and add a group for each crawler you have chosen to permit. Here is a sample for a public site:

User-agent: OAI-SearchBot
Allow: /
Disallow: /account/
Disallow: /checkout/
Disallow: /preview/

User-agent: PerplexityBot
Allow: /
Disallow: /account/
Disallow: /checkout/
Disallow: /preview/

# Optional: keep OpenAI's training crawler separate from its search crawler.
User-agent: GPTBot
Disallow: /

Sitemap: https://www.example.com/sitemap.xml

Replace www.example.com and the sitemap URL with your live site’s details. Replace or remove the sample paths to match your URL structure. If your site uses a different URL for customer accounts or checkout, the example will not restrict that path. Conversely, adding a real path to a Disallow group only asks compatible crawlers not to crawl it. It does not secure the content.

Allow: / makes the intention explicit: the named crawler may crawl the site unless a more specific rule in that group restricts a path. The path-specific Disallow entries are more specific than /, so they ask the crawler not to crawl those matching URLs. Keeping the private-path rules in both named groups makes their intended behaviour easier to review. If you want different rules for each crawler, write them separately rather than relying on a shared wildcard group.

The GPTBot group is optional. Include it only if it reflects your organisation’s policy. OpenAI describes GPTBot separately from OAI-SearchBot, so blocking GPTBot does not, according to its current documentation, mean that you have blocked ChatGPT Search. Likewise, disallowing OAI-SearchBot affects its eligibility to crawl pages for ChatGPT Search; it does not remove the need to manage other crawlers independently.

The sitemap line is optional if you already manage sitemaps another way, but it can identify the sitemap for crawlers that use it. Use the absolute public URL of the sitemap. The file should be plain text encoded as UTF-8, with directives on their own lines. The Google specification covers the file’s format and scope.

What you should see: each crawler you have made a decision about has a readable, distinct group. The rules for OAI-SearchBot and PerplexityBot do not depend on a wildcard group, and any training-crawler rule is clearly separate.

Step 4: Keep private assets and pages private

Do not use robots.txt as a list of secret URLs. Because the file is public, a line such as Disallow: /staff-payroll/ can reveal that the path exists. It still does not stop a person or crawler from requesting the URL directly. Google’s robots.txt introduction explains why the file is not a security measure and why blocked URLs may still appear in search results.

For anything confidential, apply access controls where the content is served. Depending on your setup, that means requiring a valid login, checking the user’s permissions on every protected request, or removing a resource that should not be available. Apply the same care to files, previews, product feeds, staging environments and APIs as you do to HTML pages. A robots.txt rule cannot make a public file private.

Also, do not use Disallow as a substitute for noindex. If a page should remain publicly accessible but should not appear in Google Search, Google recommends an appropriate noindex directive or removal method. A crawler must be able to access a page to read a page-level noindex, so blocking that page in robots.txt can prevent the directive from being seen. For confidential information, neither noindex nor robots.txt replaces authentication.

Avoid blocking scripts, stylesheets, images or other assets that public pages need to render or make sense. A broad asset restriction can make it harder for crawlers to interpret the page. Review each rule against the actual page and resources it uses instead of blocking all paths with names such as /assets/ or /api/ by default. Some APIs are public page data; others are private. Decide based on access requirements, not the path label alone.

What you should see: sensitive pages return an authentication requirement or an appropriate denial when requested without permission. Private files cannot be fetched anonymously. Public pages still load their necessary resources. These checks test actual protection, which is a different thing from a crawler’s decision to follow robots.txt.

Step 5: Publish the file and inspect what is served

Save and deploy the file to the site root. Then open the live URL in a browser, not just the local copy in your editor. Confirm that the response is the latest version and that it contains the expected groups. Check each host independently, including any www and non-www versions you serve.

You can also inspect the HTTP response from a terminal:

curl -i https://www.example.com/robots.txt

The response should successfully return the plain-text file rather than an error page, a login screen, a redirect to an unrelated host or a cached old version. Check the status and response body. If your site redirects from one hostname to another, test both URLs and confirm that the destination host has the intended root-level file too.

For a representative public URL, check that it remains available without a login. For a protected URL, check that an anonymous request cannot retrieve the content. Do not treat a crawler’s disallow instruction as the result of this security test. You are checking the actual response from the site, not just what the text file asks bots to do.

What you should see: the live robots.txt matches the file you intended to publish; public content returns normally; protected content is protected independently of crawler rules. If the response shows an unexpected version, inspect your deployment, caching and redirects before changing directives again.

Step 6: Check the CDN, firewall and server logs

A permissive robots.txt rule cannot override a block elsewhere. A CDN, web application firewall, bot-management system, IP restriction, authentication layer or server error can stop a crawler even when the file says Allow: /. Check those controls after publishing, especially if you have previously blocked automated traffic.

Look for requests to the public robots.txt file and to representative public pages. Review the status codes returned by your edge and origin systems. A successful robots.txt response does not prove that a crawler can fetch the rest of the site. If you see denied or challenged requests, check which rule caused the response and whether it applies to the crawler you intend to permit.

OpenAI and Perplexity publish information about their crawlers, including IP ranges. If you use IP allowlists, compare them with the provider’s current official documentation rather than copying an old list from a forum or an unrelated source. OpenAI’s crawler documentation is the relevant reference for its crawler names and published network information. Do not identify a crawler from a user-agent string alone if your security process requires stronger verification.

You can use SearchFIT’s AI Crawler Checker as a quick way to spot whether your robots.txt rules appear to permit or block named AI crawlers. Treat that as one part of verification, not proof that your CDN, origin server or authentication system serves every page correctly.

What you should see: the edge layer does not challenge or block the crawlers you chose to permit, and logs show normal responses for the pages you expect to be public. If logs show no crawler visits, that does not necessarily mean the configuration is wrong; recheck the rules, provider documentation and time since publication.

Step 7: Confirm the result and allow time for changes

After deployment, run through a short verification checklist:

• Visit the root robots.txt URL for every relevant host.
• Confirm the named groups are present and have the intended rules.
• Check that no broader or duplicate rules create an unexpected outcome.
• Test a public page and a genuinely protected page separately.
• Check CDN, firewall and server responses for blocks or challenges.
• Confirm that your sitemap URL, if included, is correct and public.
• Record the change and the date so you can compare future crawler activity.

OpenAI says a robots.txt change may take about 24 hours to affect ChatGPT Search. Perplexity says its changes can take up to 24 hours to take effect. These are provider-reported timeframes, not a promise that a page will be crawled or shown within that period. If you have just published a change, verify the live file first and allow the providers’ systems time to reflect it.

Permission to crawl is not a guarantee of inclusion, citation or ranking. OpenAI says sites that opt out of OAI-SearchBot will not be shown in ChatGPT Search answers, though they may still appear as navigational links. Allowing it makes your eligible pages available for that crawler; it does not guarantee they will be selected for a particular question. Search visibility also depends on whether pages can be fetched and understood, and on the systems that choose what to show.

For ongoing work, keep technical checks separate from performance monitoring. SearchFIT’s features cover broader visibility tracking and related workflows; a robots.txt check alone cannot tell you whether your brand is mentioned or cited. If you want to review plans for that kind of monitoring, see SearchFIT pricing. These tools are separate from the provider’s crawler rules and do not change your site’s permissions.

For a practical next step after checking crawler access, read the AEO content audit guide to review whether your public pages are ready to be understood and cited.

Common configuration mistakes to avoid

Adding a rule to the wrong host

A robots.txt file applies to the host, protocol and port where it is served. Publish or verify a file for each host you actively use, such as the main site, shop or documentation site. Do not expect the main domain’s file to govern a separate subdomain.

Mixing search and training preferences

Do not assume that a single crawler name represents every use by a provider. OpenAI’s documentation distinguishes OAI-SearchBot from GPTBot. State your preference for each separately, and review provider documentation when crawler names or purposes change.

Pasting rules without reading the existing file

A new group may not cancel a broad block in the way you expect, especially if you assume that a named group inherits every wildcard rule. Read the file as a whole, keep relevant rules together, and test the result on the live host.

Treating robots.txt as a privacy setting

A disallow line does not prevent direct access or hide the URL. Secure private pages and files at the server. Avoid publishing sensitive path names in a public file when you can protect the resource without advertising its location.

Blocking useful resources by default

Do not block all JavaScript, CSS, images or API paths without checking how public pages depend on them. Restrict only the resources that should not be crawled, and protect private data with access controls.

Expecting an allow rule to produce a citation

An allow rule permits a crawler to request content under the site’s crawler policy. It does not promise crawling, indexing, a citation, a particular answer or a ranking. Confirm that the page is public and accessible, then measure visibility separately from technical eligibility.