At this point, there’s no escaping the fact that AI (which isn’t technically AI, but that’s the prevailing term so I’m going with it for ease) is going to play a big part in all of our lives for the foreseeable future.
Most of the ways it starts to appear won’t actually involve an appearance at all – it will start to handle tasks, systems and processes which were once the province of humans, and as end users we’ll be none the wiser.
Where it’s more obvious, however, is right in front of you. Many people use ChatGPT, Gemini or Claude on a daily basis, to search, research, and build their personal data pool until they can make a purchase decision on a product or service. AI search will soon become the default option when Googling, and even more people will begin using it as a result.
How do you optimise for AI search?
This is the $64,000 question. ChatGPT has been with us for a few years now, as has a form of Google’s AI in Bard, and then Gemini. Their capabilities constantly evolve, and as they become more embedded in the ecosystem that is search, companies are starting to realise that, particularly with ChatGPT, there is a new marketplace in which they want to be seen.
As with any desirable goal, services showing the route to achieving that success has become an industry in itself. There are already services available which will show you how to appear in more AI searches, and having tested some I’m here to say that… they don’t really work. Or at least, not yet.
The science of getting to the top of Google was decades in the making
It’s easy to forget that there was a time when we didn’t have reliable tools to show us keyword performance in Google. It’s slightly more difficult to forget that during those years Google would frequently move the goalposts, as they’re still doing it, but ask anyone who’s worked in SEO for long enough, and they’ll have a story about a site which was caught in a Google algorithm update and lost a large amount of their keyword rankings, possibly for reasons that remain unknown to this day.
That’s essentially where we are again. Companies are rushing to develop tracking capabilities, and some egos are already making promises that the body can’t cash.
What can AI search tracking currently do?
It can do a bit, although due to the emergent nature some of the price tags struggle to justify the outlay. We can see which keywords led to your company appearing in Google’s AI overviews. We can see which companies are appearing for certain desirable searches. We can even audit a site for “AI readiness”.
That sounds pretty good, for a new industry. Here’s how it actually worked for us, though.
Keywords in AI Overviews
This was helpful. Despite being a small part of the bigger Google AI picture, knowing when companies appear in AI Overviews is useful from a success point of view for your content, and it also can clarify either a lift or drop in traffic to the page as a result.
Which companies appear for which searches
This is where it falls apart a bit. Currently, the more reasonably priced end of the market relies on credits to check for results. In order to find the results, services will suggest searches related to what you want to be found for, but the issue is that unlike Google’s keyword approach, AI needs to run the full query in order to return a result. That means that you’ll need to anticipate every single variation in the way your question is asked, along with having enough credits available to service those searches.
That’s a lot of credits.
AI readiness audit
This is where it fell apart a bit. The particular SaaS we were trying, which I’m not going to shame publicly (but I’m happy to disclose to anyone that wants to drop me an email) carried out an audit on the same site we’d trialled with the company search. The results were that their site was overall in great shape, with a good level of optimisation that would help AI platforms to parse their content.
However, there was a recommendation to include more citations in content (AI has been generally more likely to cite a website which in turn cites other companies and organisations in its content), and a failure which was both telling and puzzling in equal measure – no LLMs.txt file was in place.
What is LLMs.txt?
Currently, most websites use a file called robots.txt to explain how the site is structured to Google’s search robots, which constantly crawl the internet looking for new websites and new content on existing sites.
AI Search platforms use similar technology to gather the data they use to answer questions (at least, they do now – earlier versions couldn’t access any data outside of the set they were trained on, while current models can conduct web searches so they’re up to date). The proposal is that every site would implement the LLMs.txt file as standard to explain to the AI what the site is about, and give answers to the questions it is trying to answer.
However, that model is open to abuse – you could vary the answer from the one on the website, and an AI wouldn’t double-check as the LLMs.txt file is designed to remove the need to visit the website page. This could lead to some gaming of the system in order to gain more mentions, and also makes it even more difficult to ensure that AI search returns accurate and trustworthy information.
The upshot of this is that no AI firm has adopted the LLMs.txt standard, and it’s difficult to imagine they ever will. Searching the live pages of a site provides the same information and reduces the risk of “gamed” information being given out.
And that leads us back to the reason for this article. Why would a service that promises to audit your website for AI-readiness recommend that you implement a file which achieves nothing?
The answer is that no one knows how to optimise for this technology yet.
It makes paying for something that promises the key to success a risky proposition, but hard to resist – if it really is the key, you get ahead of the competition.
How can you achieve success in AI Search?
At the moment, the “science” of this is the same as succeeding in old-school search optimisation. You need to have easy to read, comprehensive information on whatever it is you do, it needs to be accessible to users and search engines, and it needs expert insight that makes it worthwhile for both human and machine readers. We can help you to implement this, just get in touch.
