Most travel companies who discover they are invisible in AI answers assume the problem is content. Usually it is access, and access is the cheapest thing on the list to check.
Here is the whole test. It takes about ten minutes and needs no tooling beyond a browser and a terminal.
Step one: read your own robots.txt
Open yourcompany.co.uk/robots.txt in a browser. You are looking for any block
naming an AI crawler:
User-agent: GPTBot
Disallow: /Rules like this are common and are almost never a current decision. They were often added in 2023, when blocking AI crawlers was the default advice, by somebody who has since moved on. If your business sells something and competes for consideration, this rule is now working against you.
Note what you find, but do not stop here. Roughly half the blocking we see never appears in robots.txt at all.
Step two: request a page as each crawler
This is the step that gets skipped, and it is the one that matters. Your CDN or web application firewall may be refusing AI crawlers at the edge, entirely independently of robots.txt.
curl -sI -A "GPTBot" https://yourcompany.co.uk/ | head -1
curl -sI -A "ClaudeBot" https://yourcompany.co.uk/ | head -1
curl -sI -A "PerplexityBot" https://yourcompany.co.uk/ | head -1You want HTTP/2 200. A 403 means something in front of your site is
refusing the request. A 503 with a challenge page usually means bot
protection has decided the crawler is a threat.
The important detail: a 403 from your CDN looks exactly like a deliberate block to the crawler, and nothing in your own configuration will show it. This is routinely switched on by default when bot protection is enabled or upgraded, and nobody is told.
Step three: turn off JavaScript
In Chrome, open developer tools, press Cmd+Shift+P, type “JavaScript”, and choose Disable JavaScript. Now reload one of your itinerary or product pages.
If the page is blank, or the prices, dates and descriptions have vanished, then most AI crawlers see the same thing you are looking at. Google’s crawler executes JavaScript. Most AI crawlers do not — they read whatever HTML the server returns and move on.
What to do with the results
If crawlers are blocked in robots.txt, that is an afternoon’s work to correct, and the only real task is agreeing internally that you want it corrected.
If they are blocked at the CDN, you need whoever administers that CDN. Ask them specifically to allow the AI crawler user agents through bot management rules, and to confirm with the curl commands above rather than by inspecting the dashboard.
If your pages are empty without JavaScript, that is a genuine engineering project rather than a configuration change. It is also the single most valuable thing on this list, because no amount of structured data helps a crawler that receives an empty page.
The honest caveat
Passing all three tests does not mean you will appear in AI answers. It means you are eligible to. Access is necessary rather than sufficient — what a model does with your site once it can read it depends on structure, clarity and what independent sources say about you.
But you cannot be cited by a system that cannot fetch you, and you would be surprised how often that is the entire problem.