When a business tells me that ChatGPT never mentions them, the first thing I open is not their website. It is their robots.txt file. More often than you would expect, the answer is sitting right there in a rule somebody added in 2023 to keep the AI scrapers out, which nobody has looked at since.
That is the awkward part of learning how to optimise for AI search. Almost everything written on the subject is about writing, and writing is the half of the job you only reach once the machines can actually get to your pages and read them. Two faults will hide a business from ChatGPT, Claude and Perplexity no matter how good its content is. Neither is a content problem, and neither gets mentioned by a single page currently ranking for this term.
So this guide runs in that order. Access, then readability, then the content and off site work, then how to tell whether any of it moved. Prices where prices exist, and a straight answer on what I think you can skip.
What does optimising for AI search actually involve?
It means making your business easy for an answer engine to reach, read and quote. The engines that matter are Google’s AI Overviews and AI Mode, ChatGPT, Perplexity, Gemini and Claude. The work splits four ways: letting the right crawlers in, serving your text in a form they can parse, answering real questions cleanly enough to be lifted out, and being described consistently on websites you do not control.
Google is blunt about the foundation. Its own guidance states that the best practices for SEO continue to be relevant, because its generative features are rooted in its core Search ranking and quality systems. If you have been doing sensible SEO, you have not wasted your time.
Here is what genuinely changed. Google used to be the only front door. Now there are six or seven, each operating its own crawler with its own rules, and the ones that are not Google are considerably fussier about how your site is built. You can pass Google’s tests and fail everyone else’s on the same afternoon.
Can the AI crawlers actually reach your website?
Check before you change a single word of content. It takes five minutes and it is the only step here that can be instantly fatal.
The confusion comes from the fact that each company runs more than one crawler, doing different jobs. OpenAI’s GPTBot collects training data. Blocking it costs you nothing in visibility. But OAI-SearchBot is the one that surfaces your site inside ChatGPT search results, and a blanket rule catches both. Anthropic splits the same way, with ClaudeBot gathering content for models and Claude-SearchBot described in Anthropic’s own documentation as navigating the web to improve search result quality for users. Perplexity states plainly that PerplexityBot exists to surface and link websites and is not used to crawl content for foundation models.
Two details catch people out. Anthropic requires the directives to be applied per subdomain, so a rule on your main domain does nothing for shop.yourdomain.com. And OpenAI notes that a robots.txt change takes around a day to register, so do not judge a fix an hour later.
Your robots.txt is also not the only gate. Cloudflare and similar services sit in front of a lot of small business sites with bot protection switched on, and that can refuse a crawler your robots.txt is perfectly happy to admit. Check both.
Can they read your pages once they arrive?
Only if your text is genuinely in the HTML. AI crawlers other than Googlebot do not execute JavaScript.
This is the single most consequential technical fact of 2026 and I could not find one page currently ranking for this keyword that mentions it. Log analyses published this year found no evidence of JavaScript execution by GPTBot, even in the small share of fetches where it downloaded the JavaScript files. The same pattern holds for Anthropic’s and Perplexity’s crawlers. The reason is cost and patience: these crawlers work to tight timeouts, often a few seconds, with no retries, and rendering pages at their scale would be enormously expensive.
The effect is bigger than it sounds. When one analysis examined major retail product pages this year, somewhere between 49 and 89 percent of the text a human could see was missing from the markup a crawler receives. On one retailer’s page, the price itself was not in the HTML.
So a modern site built as a single page application can rank respectably in Google, appear in AI Overviews because Googlebot renders it, and be a blank page to ChatGPT, Claude and Perplexity at the same moment. Same site. Same afternoon.
Here is the test, and anyone can run it. Right click your most important page, choose View Page Source, then search that source for a sentence from the middle of the page. Present means every crawler can read you. Absent means only Google can.
How to optimise for AI search, step by step
Work through it in this order. It is sorted by what is cheapest to fix and most damaging to leave, not by what makes the best slide.
- Open yourdomain.com/robots.txt in a browser and read it. Look for any line disallowing
OAI-SearchBot,Claude-SearchBotorPerplexityBot. If one is there and you did not put it there on purpose, take it out. - Check whatever sits in front of your site, usually Cloudflare, for bot protection that blocks those same crawlers independently of robots.txt.
- Run the View Page Source test on your three most commercially important pages. If your text is missing, that is now the biggest job on your list, and it is one for whoever built the site.
- Write down the ten questions customers ask you before they buy, in the wording from your inbox rather than the wording from your brochure.
- Give each question its own heading on a relevant page, with a direct answer of roughly forty to sixty words immediately underneath. These systems retrieve passages rather than whole pages, so the answer needs to survive being lifted out on its own.
- Make your business facts match everywhere they appear: your site, your Google Business Profile, LinkedIn, Companies House, and any trade directory you are listed in. Contradictions weaken the picture a model builds of you.
- Set up the measurement described further down before you change anything else, so you have a baseline worth comparing against.
Step four is where people cheat without noticing, writing the questions they wish customers asked. Your enquiry inbox already has the real wording in it. Use that.
Why do your citations come from websites you do not own?
Because the answer engines lean heavily on a small set of third party platforms, and your homepage is not on the list.
A study of 30 million citations published in March 2026 found Reddit was the most cited source across ChatGPT, Google AI Mode, Gemini, Perplexity and AI Overviews, with YouTube, LinkedIn, Wikipedia and Forbes making up the rest of the top five. That finding is repeated so often it has become a tactic, which brings me to the part nobody quotes.
Semrush tracked more than 230,000 prompts across thirteen weeks and watched Reddit appear in roughly 60 percent of ChatGPT responses in early August 2025, then fall to around 10 percent after mid September. Wikipedia dropped from roughly 55 percent to about 20 percent over the same stretch while holding steady on other engines. Semrush attributed it to deliberate de biasing by OpenAI.
Read that properly and the lesson is not “go and post on Reddit”. It is that anyone selling you a citation package on a specific platform is selling a position that halved in a fortnight.
What survives the volatility is duller and more useful. Be described accurately and consistently wherever you already appear. Claim and complete your Google Business Profile, which Google itself points to for local visibility in AI features. Get into the roundup articles in your field that already rank, because a listicle on a trusted site gets pulled into answers that your own service page never will. And note that Google explicitly warns against chasing inauthentic mentions across the web, so buying your way onto a hundred low quality sites is a spend with a downside attached. This cross engine work is most of what my search and AI visibility service actually consists of.
How do you check whether it is working?
For free, with three tools you already have access to. You do not need to buy anything in your first six months.
Start with Google Search Console. Its Search Generative AI performance report rolled out to every website worldwide on 31 August 2026, and it shows how often a link to your site appeared inside AI Overviews and AI Mode, split by page, country and device. The catch is important: impressions only. No queries, no clicks, no click through rate, no position. It answers whether you are visible, never whether that visibility earned you anything.
Then fix Google Analytics 4, which does not separate AI traffic on its own. Go to Admin, then Channel groups, create a new group with a channel called AI Referrals, and set the condition to source matching a regular expression along the lines of .*chatgpt.*|.*openai.*|.*perplexity.*|.*claude.*|.*gemini.*|.*copilot.*. Move that rule to the top so it claims the traffic before the default rules do. Fifteen minutes, and it works retroactively on your existing data.
One honest caveat on that number. A large share of AI referral sessions arrive without a referrer header and get filed under Direct, so whatever GA4 shows you is a floor rather than a total.
The third tool is your server access logs, and it is the only ground truth for crawler behaviour. Filter them by user agent and you can see whether OAI-SearchBot and Claude-SearchBot are fetching your pages at all, which pages they take, and how often. Because a user agent string is trivially spoofed, verify the IP addresses against the ranges OpenAI, Anthropic and Perplexity publish. Nobody ranking for this keyword tells you this, and it is free.
Is any of this actually worth the effort?
Partly yes and partly unproven, and I would rather say so than sell you a certainty that does not exist.
The pressure is real enough. In the United States 68.01 percent of Google searches ended without a click in the first four months of 2026, up from 60.45 percent in 2024, according to SparkToro working from Similarweb clickstream data. AI Overviews now appear on more than a fifth of searches.
On whether AI visitors are worth having, the evidence genuinely conflicts, and most articles quote only the half that suits them. Ahrefs published first party numbers showing AI search visitors made up 0.5 percent of its traffic but 12.1 percent of its signups, which is a striking multiple. Ahrefs also sells software to exactly the people most likely to be using AI search, so read it accordingly. Pointing the other way, a working paper covering 973 ecommerce sites and roughly 20 billion dollars of revenue compared more than 50,000 ChatGPT driven transactions against 164 million from traditional channels and found organic search converted better than ChatGPT did.
My position is that the split in the evidence maps neatly onto the split in this article. The access and readability work costs an afternoon, cannot backfire, and pays off in ordinary search too, so do it whatever you believe about AI. The heavier ongoing investment in content and off site presence is a real bet, and it is more defensible for a considered purchase than for a low value transaction. You can read more about how I think about this if you want the longer version.
The honest order of work
If you do one thing after reading this, open your robots.txt and then View Page Source on your best page. That is ten minutes, and for a decent number of businesses it turns up something that no amount of content spending would ever have fixed. I have sat with owners who had been paying for months of writing while a single line in a text file kept the crawler out of the whole site.
The rest is less dramatic. Ten real questions, answered plainly where a machine can lift them out, and your business facts saying the same thing everywhere they appear. It does not feel like a strategy, which is precisely why so few of your competitors have bothered.
And I would keep this in your own hands rather than renting it forever. The habit underneath it, answering a real question directly and making the answer easy to read, does not expire when the next model ships. If you want a second pair of eyes on where your site currently stands with the AI crawlers, get in touch and I will tell you honestly whether there is anything worth fixing.