Can AI Access Your Brand? Travel's Hidden Crawler Problem

Why an overlooked technical issue may be undermining travel brands' AI strategies before the customer journey even begins.

A travel brand can invest in content, structured data, AI optimization, and direct distribution while inadvertently preventing AI systems from accessing the information.

That is not a hypothetical risk.

During the Future of AI Search panel at Align AI 2026, Stephen Burns, Web Intelligence Lead at Common Crawl, shared the results of an analysis he had conducted that morning. He tested the homepages of 20 leading hotel websites to determine whether Common Crawl's CCBot could reach them.

Six were blocking the crawler.

The response in the room was immediate. Senior leaders began questioning their own configurations and contacting colleagues to investigate.

The finding mattered because Common Crawl maintains one of the world's largest open repositories of web data. Its corpus has been used extensively in AI research and model development. A hotel that blocks CCBot is not necessarily invisible to every AI system, but it is excluding one important path through which information can enter the AI ecosystem.

More importantly, the exercise exposed a broader problem:

Many travel companies do not know which AI systems can access their content, which systems are being blocked, or whether those outcomes reflect an intentional policy.

One brand, several types of AI access

"AI crawler" is often treated as one category. It is not.

Different automated systems may access a website for different purposes:

• Search engines crawl pages to index and rank content.
• AI search systems retrieve content to answer current questions and provide citations.
• Training crawlers collect material that may contribute to future model development.
• User-directed agents visit a site to complete a task on someone's behalf.
• Commercial and monitoring services crawl pages for specific business functions.

Those distinctions matter because the business decision may be different for each one.

OpenAI, for example, distinguishes between OAI-SearchBot, which supports discovery and citation in ChatGPT search, and GPTBot, which is associated with potential model training. OpenAI advises publishers that want their content included in ChatGPT search summaries and snippets not to block OAI-SearchBot. It separately provides controls for sites that do not want content accessed by GPTBot for potential training. OpenAI's publisher guidance makes clear that AI access is not a single yes-or-no decision.

Google also separates the systems involved in search from some other uses. Its guidance states that the same foundational search requirements apply to AI Overviews and AI Mode, and that a page generally needs to be indexed and eligible to appear with a snippet to be used as a supporting link. Google's documentation also cautions that there is no special shortcut or additional technical requirement that replaces sound search fundamentals.

Travel brands therefore need a crawler policy, not a universal toggle.

The block may not be where you think it is

Most conversations about crawler access begin with robots.txt. That file remains important because it communicates which crawlers may access which parts of a site.

But it is only one layer.

A crawler can be allowed in robots.txt and still fail to reach the content because of:

• Content-delivery network rules
• Web application firewall settings
• Bot-management products
• IP or geographic restrictions
• Rate limits and automated challenges
• Authentication requirements
• Redirect failures
• Pages that depend heavily on client-side JavaScript
• Content that is visible to a user but absent from the initial HTML response

Common Crawl states that CCBot checks robots.txt, fetches pages through standard HTTP requests, and currently does not execute JavaScript or use cookies. Its crawler documentation also explains how operators can verify genuine CCBot requests rather than trusting a user-agent string that could be spoofed.

That creates an obvious failure mode for travel websites. A property detail, room attribute, policy, or offer may appear correctly in a browser after scripts run while remaining inaccessible to a crawler that reads only the server response.

Security products can add another layer of complexity. Cloudflare's documentation, for example, includes controls that can block known AI-training bots, manage robots.txt directives, challenge automated traffic, and distinguish verified bots from other requests. These capabilities can be valuable, but they also mean that a configuration chosen for security, infrastructure, or content protection can affect AI accessibility. Cloudflare's current guidance explicitly separates signaling preferences through robots.txt from technically enforcing a block.

This is why reading the public robots.txt file is not enough. Brands need to test the complete request path.

Stephen has since turned the conference exercise into a public tool. His AI Visibility Audit checks whether Common Crawl's CCBot can see a site across all three gates where access can fail, rather than stopping with a robots.txt check.

As Burns puts it:

"AI can't cite what it never read."

The audit gives travel leaders and their teams an immediate way to identify where a site may be invisible to this important source of AI training data. It is a practical first diagnostic, not a substitute for a complete crawler policy or page-level accessibility review.

Inaccessible content creates a commercial vacuum

Blocking one crawler does not automatically remove a brand from every AI response. Models may already contain information learned previously. Retrieval systems may use search indexes, third-party sources, licensed data, intermediary content, or other accessible pages.

That is precisely the problem.

When a brand's authorized content is difficult to access, AI does not necessarily stop answering. It may answer from whatever sources remain available.

For a travel supplier, that can mean:

• An intermediary becomes easier to understand than the direct channel.
• An old policy remains influential after the brand has updated it.
• A property is described through reviews rather than verified product information.
• Room-level distinctions disappear.
• Loyalty benefits are missing from the recommendation.
• A direct booking path is replaced by a generic page or third-party link.

Access does not guarantee that AI will get the brand right. But without access, the brand has less influence over what the system treats as current and authoritative. It affects brand representation, recommendation quality, direct distribution, customer expectations, and the ability to participate in future agent-led transactions.

Unrestricted access is not the answer

The lesson from Align AI was not that every travel company should allow every bot to crawl every page.

Some content should remain private. Some systems should never be exposed publicly. Rate, inventory, customer, loyalty, and payment data require controlled access through appropriate systems and permissions. Companies also have legitimate concerns about intellectual property, security, infrastructure cost, competitive use, and how their content contributes to model training.

The real issue is whether access decisions are intentional.

A defensible policy should answer four questions:

1. Who is requesting access? Verify the crawler or agent rather than relying solely on a claimed identity.
2. Why does it want the content? Search, current-answer retrieval, model training, monitoring, and user-directed action are different use cases.
3. Which information should it reach? Public brand and product content may warrant a different policy from transactional endpoints or customer data.
4. What commercial outcome does the company want? The access decision should support the brand's strategy for discovery, attribution, direct routing, and AI-mediated commerce.

Without those answers, a brand may be blocking strategically valuable access by accident or allowing strategically sensitive access by default.

A practical AI accessibility review

Travel leaders should ask their digital, security, infrastructure, content, and commercial teams to examine the issue together.

The review should include:

1. Inventory the crawlers and agents that matter

Document the systems associated with major search platforms, AI answer engines, training datasets, monitoring tools, and user-directed agents. Record the purpose of each one and the company's intended policy.

2. Review every control layer

Check robots.txt, meta directives, response headers, CDN settings, firewall rules, bot-management products, authentication gates, rate limits, and security challenges. Do not assume that these systems express the same policy.

3. Test representative pages

The homepage is only the beginning. Test brand pages, property pages, room and product details, amenities, policies, offers, loyalty information, destination content, and booking links.

4. Inspect what the crawler receives

Confirm that essential information appears in the returned content. A successful status code does not prove that the crawler received a meaningful page.

5. Review server and security logs

Measure which verified crawlers are arriving, what they request, where they fail, and which controls stop them. Logs reveal the operating reality that configuration reviews can miss.

6. Establish ownership and change control

Crawler policy often falls between SEO, security, legal, IT, e-commerce, and brand teams. Assign an accountable owner and require review when infrastructure or bot-management settings change.

7. Retest continuously

Crawler identities, platform policies, website architecture, and security tools change. Accessibility should be monitored as an operating condition, not checked once and forgotten.

Access is the first gate, not the finish line

The hidden crawler problem is easy to underestimate because it looks technical. In reality, it sits at the intersection of AI strategy, content rights, cybersecurity, customer acquisition, and distribution.

Travel brands need to decide where and how their information should be available. Then they need to confirm that their technical environment implements that decision.

If AI cannot reach the brand's authorized content, everything that follows becomes harder.

But access alone is not enough. Once AI can reach the information, it still must interpret that information in the context of what a traveler is really asking.

That is the subject of the next article in the Align AI 2026 Insights Series: From Keywords to Intent: Why AI Changes Travel Discovery.