Decide Exactly
Who Crawls You.
Configure robots.txt, llms.txt, and crawl budget rules so GPTBot, PerplexityBot, Google-Extended, and the next AI crawler all get exactly the access you decide — no more and no less — while Googlebot keeps the access it already has.
Custom robots.txt & llms.txt configuration
Crawl budget protection for Googlebot
Ongoing monitoring as new bots launch
# Training crawlers: disallowed User-agent: GPTBot Disallow: / User-agent: Google-Extended Disallow: / # Answer-engine crawlers: allowed User-agent: PerplexityBot Allow: / User-agent: OAI-SearchBot Allow: /
PerplexityBot · Allowed
OAI-SearchBot · Allowed
Google-Extended · Disallowed
What Proper Crawler Access Actually Protects
A one-line “Disallow: /” or a wide-open default both cost you something. Here’s what’s actually at stake once AI crawlers are in the picture.
Keep the Citations You Want
Allow answer-engine crawlers like PerplexityBot and OAI-SearchBot to keep fetching and citing your pages, instead of losing referral traffic to an accidental blanket block.
Opt Out of Training, Not Search
Disallow training-only crawlers like GPTBot and Google-Extended with zero impact on your Google rankings or indexing — Google has confirmed the two are separate.
Protect Your Crawl Budget
Stop AI bots from burning server resources on faceted filters and internal search pages, so Googlebot keeps clear, fast access to the pages that actually drive revenue.
Your Default Crawler Settings Are Probably Working Against You
Almost no WordPress site, Shopify store, or custom-built site ships with an opinion about AI crawlers. Robots.txt was written for Googlebot and Bingbot years before GPTBot, ClaudeBot, PerplexityBot, Amazonbot, and Google-Extended existed, so by default every one of them is either let in everywhere, or — if a security plugin added a blanket “block all bots” rule after the ChatGPT panic of 2023 — locked out everywhere. Neither default is a decision. Both are accidents.
There is a second, quieter problem: crawl budget. AI crawlers aren’t bound by the crawl-rate courtesy search engines have built up over twenty-five years, and a handful of them hammering your faceted navigation or paginated search results can add real server load and compete with Googlebot for the pages that matter most.
This service pairs naturally with our broader AEO services and GEO services: getting your content structured and trusted enough to be cited only matters if the crawlers behind those citations can actually reach it.
Crawlers We Configure For
Every AI crawler that reaches your site does a different job. We sort them by what they actually do, then set robots.txt directives accordingly — instead of applying one blanket rule to all of them. Each card below links to that operator’s own published documentation, exactly what our audits are checked against.
PerplexityBot
Perplexity
Indexes pages so Perplexity’s answer engine can cite and link to them.
Google-Extended
Controls use of your content for Gemini and Vertex AI training; does not affect Google Search.
Shown statuses are our typical starting recommendation, not a fixed rule — every engagement starts with an audit and you decide what each bot sees. Crawler behavior and documentation change regularly; we verify current behavior against the source during every audit.
What’s Included in This Service
This is a technical engagement with a defined set of deliverables, not a vague “AI optimization” retainer. Here is exactly what you get.
Full Crawler Audit
We pull your server logs and identify every AI crawler actually hitting your site today: user agent, request volume, which paths it is reading, and what response codes it’s getting.
Robots.txt Configuration
A custom ruleset grouped by what each crawler actually does: training bots, answer-engine bots, and known scrapers get different directives, scoped by path where that makes sense.
llms.txt Implementation
We write and publish an /llms.txt file following the emerging llms.txt convention, giving AI agents a curated summary of what your site is and which pages matter most.
Crawl Budget Management
Directives and, where needed, server-level rules that stop AI bots from burning crawl budget on faceted filters, internal search, and admin paths.
Ongoing Monitoring
New AI crawlers launch on a rolling basis. We track changes to the major published crawler lists and come back to update your rules as new bots appear.
Verification & Reporting
After launch, we re-check the server logs to confirm the crawlers you targeted are actually respecting the new rules, and send a plain-language report.
Our Process
Every engagement follows the same five steps, in the same order, so you know what’s happening at each stage before it happens.
Discovery and log analysis
We review your server logs (or set up logging if it isn’t already capturing bot traffic) to see which AI crawlers are visiting now, how often, and what they’re reading.
Crawler inventory and classification
We sort every bot we find, plus the ones not yet seen but worth planning for, into training crawlers, answer-engine crawlers, and anything you’d rather block outright.
Strategy: deciding what each bot should see
Allowing OAI-SearchBot and PerplexityBot can earn citations and referral traffic; disallowing GPTBot or Google-Extended limits training use without touching search visibility. You make the call on each one.
Implementation
We write the robots.txt rules, build the llms.txt file, and add any server-level rules crawl budget management requires, then stage the changes for your review before anything goes live.
Verification and monitoring
Once live, we confirm in the server logs that each crawler is behaving the way the new rules intend, then keep watching as new crawlers launch.
A Before-and-After Example
Most sites we audit start with one of two robots.txt files: either nothing written for AI crawlers at all, or a single rule a plugin added that blocks everything indiscriminately. Here’s the kind of change we typically make.
User-agent: *
Disallow: /wp-admin/
# No rules for AI crawlers: every bot gets
# full, unscoped access to the entire site.User-agent: *
Disallow: /wp-admin/
Disallow: /?s=
Disallow: /*?*filter=
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
Sitemap: https://topaiseoexperts.com/sitemap.xmlTraining crawlers are disallowed, answer-engine crawlers that can send citations and traffic are explicitly allowed, and the internal search and filter paths that waste crawl budget are closed off for everyone.
Why Businesses Trust Us With Their Crawler Access
A lot of agencies bolt a “block AI bots” plugin onto a site and call it done. We treat crawler access as a deliberate, documented decision — not a default.
We Show Our Work
You see the current robots.txt, the proposed robots.txt, and a plain-English explanation of every line that changed, before it goes live.
We Cite Our Sources
Every recommendation ties back to a crawler operator’s own published documentation — the same pages linked earlier on this page — not a guess about what a bot “probably” does.
We’re Honest About the Limits
Robots.txt is an honor system. Reputable crawlers from OpenAI, Perplexity, Anthropic, and Google generally follow it, but it’s not a security control, and we’ll tell you when a goal needs firewall-level blocking instead.
One Person Owns It
The same specialist who runs your audit writes your rules and handles your ongoing updates, so nothing gets lost in a handoff.
⭐ 5.0 Clutch
⭐ 5.0 G2
⭐ 5.0 Google Reviews
Frequently Asked Questions
Will blocking AI crawlers hurt my SEO?
Not your Google rankings. Google has stated that Google-Extended is separate from Googlebot and does not affect Search ranking or indexing, and disallowing training-only crawlers like GPTBot works the same way. The real trade-off: blocking an answer-engine crawler like PerplexityBot or OAI-SearchBot can mean losing citations and referral clicks from that platform, which is why we classify bots by function before recommending a disallow.
How often do crawler rules need to be updated?
New AI crawlers have launched every few months since GPTBot first appeared in 2023, and existing ones occasionally change what they do or how they identify themselves. We recommend a review at least quarterly, and our ongoing monitoring deliverable covers exactly that.
Should I block AI crawlers or allow them?
There’s no single right answer; it depends on what you’re trying to protect and what you’re trying to gain. Publishers worried about content being used to train competing models often disallow training crawlers while leaving search-and-cite crawlers open, so they can still appear in AI-generated answers. We walk through the specific trade-offs for your content and your goals rather than applying a template.
What’s the difference between robots.txt and llms.txt?
Robots.txt is a set of rules that tells automated crawlers what they may access. Llms.txt is the opposite kind of file: a curated, human-written summary that helps an AI agent understand what a site is about and which pages matter most, used when an agent is already allowed in and needs direction rather than permission.
Can these rules actually be enforced, or can bots just ignore them?
Robots.txt is voluntary, and the major labs state publicly that they honor it, but it isn’t a technical barrier. For sites that need a harder guarantee, we can pair robots.txt and llms.txt configuration with server- or firewall-level blocking by verified IP range or user agent, which is a separate, enforceable layer on top of the honor-system one.
Do you make changes directly on my live site?
No. Every proposed change is documented and reviewed with you first. Nothing goes live until you’ve seen the before-and-after and agreed to it.
Ready to Decide Who Crawls You?
Stop leaving AI crawler access to whatever your CMS shipped with by default. Get a clear picture of who’s crawling you now and a configuration built around what you actually want.