Which AI Crawlers Should You Allow?

By Ryan Goering, founder and CEO of BaaDigi, a contractor marketing agency.

Last updated: September 2026. BaaDigi provides search and AI visibility services. This guide describes a review process for site owners and their administrators; it is not a claim that allowing every bot improves performance.

Which provider controls do different jobs?

Use the actual provider names and purposes in the change request. “Unblock AI” is too broad to approve or verify.

Sources: OpenAI crawler documentation, Perplexity crawler documentation and Google’s crawler controls. Google-Extended is a product-control token, not a separate HTTP user-agent you should expect to see in a request log.

What does robots.txt fail to protect?

Google’s robots.txt guide explains that the file controls compliant crawling, not access protection; a blocked URL can still be discovered and appear in results. Use authentication and authorization for private resources. A list of disallowed paths is not a password.

Ask the administrator to test sensitive material while signed out and with an account that lacks permission. The test should cover the actual files or endpoints, not only a login page. If a customer document is publicly downloadable, fix that exposure directly. Do not consider the issue closed because its directory appears in robots.txt.

Keep secrets, customer details and private project records out of public explanatory files such as sitemaps and llms.txt. A convenient navigation file should never become an index of information that was not intended for publication.

How do you inspect the current setup safely?

Begin with a read-only inventory. Save the current robots.txt, list important public URLs and identify the hosting, firewall or content-delivery settings that may affect requests. Include any plugin or framework that generates crawler rules so the team knows where the authoritative configuration lives.

Do not assume the basics are in place. Across 1,047 free website audits BaaDigi ran (Mar–Sep 2026), the audit found a robots.txt on 73.9% of sites and an XML sitemap on 72.9%; the rest had neither file to inspect, let alone an AI-specific rule. Those are self-selected prospects, not a random sample, but they are the sites whose owners are asking this question.

1. Choose representative public pages: homepage, a priority service page and an educational article.
2. Identify the exact crawler purpose the business wants to allow or restrict.
3. Inspect applicable rules and infrastructure responses together.
4. Record the current behavior and the intended change in plain language.
5. Assign one person to implement it and another to verify the result where practical.
6. Keep the previous configuration available for rollback.

This is our recommended operating checklist. Do not paste a generic replacement file over an existing configuration without understanding what it removes. An unrelated rule may serve a deliberate purpose elsewhere on the site.

How should the team test a change?

Confirm that the new configuration is actually served at the live site and that the affected public URLs respond as intended. Test through the normal delivery path, including the firewall, rather than checking only an origin server that visitors never reach directly.

Do not trust a claimed crawler name alone when changing security rules. Ask the administrator to use the provider’s verification guidance and published network information. Avoid a broad firewall exception that weakens protection for unrelated requests.

What should you ask an agency proposing bot changes?

Ask which bot, which pages, which existing rule and which business goal are involved. Require a before-and-after check. If the explanation jumps from “the request is allowed” to “AI will recommend you,” ask for the missing evidence.

The evidence gap is large. In BaaDigi’s Aug 2026 sample of 1,429 AI answers to 88 contractor-marketing prompts, reddit.com was cited in 657 answers and google.com in 310; being crawlable is the entry fee, not the reason a page gets quoted. What gets quoted is covered in the LLM SEO guide, and it starts after the crawler is allowed in.

A technical problem may justify search access and indexing work. It does not automatically justify a new website. The AI visibility check observes answers; it is a different measurement from whether a server accepted a request.

If you need the findings combined into a business decision, BaaDigi’s $497 AI Visibility Audit covers one business and one market. Keep security repairs, crawl preferences and search outcomes as separate lines in the plan.

What should the administrator hand back?

Keep the old and new rules, the intended crawler and purpose, the tested URLs and the observed response. Record any separate firewall change. That lets the next administrator distinguish an access repair from a change in training preferences. Use the llms.txt guide for public resource descriptions and the agent task checklist for interactive testing; neither replaces access control.