Technical GEO: how to find and fix AI access problems

Technical GEO checks whether search systems can reach and read your pages. Find access and indexing problems, fix the cause and verify the result.

Getting cited in AI answers6 min readLast updated 21 September 2026

In this post

A page can look fine in your browser while a search crawler receives a security challenge or an empty page. Start with a priority service page, a pricing page and one supporting article.

Check the right crawler

Use each provider’s current documentation to identify the search crawler:

Google AI Overviews and AI Mode: Googlebot.
ChatGPT search: OAI-SearchBot.
Perplexity search: PerplexityBot.
Claude search: Claude-SearchBot.

Check the rules for the exact page and hostname. Access to your homepage does not establish access to the rest of the site.

A successful visit from ChatGPT-User, which handles user requests, also does not confirm that OpenAI’s search crawler can access the page. OpenAI documents these separately.

Find the failure

Ask your developer or hosting provider to run these checks on the selected URLs.

Checks for finding and fixing AI crawler access problems
Check	Failure to look for	Fix	Verify
Read the applicable robots.txt rules	The search crawler is blocked from the page	Amend the relevant rule for public pages you want discoverable	Recheck the rules and subsequent crawler activity
Inspect server and security logs	Blocked requests, rate limits, server errors or challenge pages	Correct the responsible security rule, rate limit or server fault	Confirm verified crawler requests receive the actual page
Compare initial HTML with the browser's rendered page	Important service information appears only after JavaScript runs	Include essential copy in server-rendered or static HTML	Fetch again and confirm the response contains that copy
Inspect Google eligibility	Unintended noindex or nosnippet directives, or an incorrect canonical URL	Correct the instructions in page tags or response headers, and the preferred URL	Run a live URL Inspection test, then recheck indexing after recrawling
Check links and the sitemap	Important pages are unlinked or listed under old URLs	Add relevant internal links and update the sitemap	Crawl the site again and confirm the final URLs are reachable

A successful status code alone is insufficient. A 200 response can still contain a challenge page instead of your content. Check the body too.

Google can render JavaScript. Use Search Console to inspect what Google rendered; missing text in the initial HTML alone does not prove Google cannot read it. Providing essential copy in that HTML also supports bots that do not render JavaScript. Google explains the distinction.

When changing firewall rules, verify crawler identity using the provider’s published method. A request calling itself PerplexityBot is insufficient evidence. Perplexity recommends checking both the user-agent and its published IP ranges.

A worked example

Illustrative example.

A UK facilities company has a service page that loads normally for visitors. Its robots.txt permits PerplexityBot, but security logs show verified crawler requests receiving a 403 response, meaning access was refused.

The developer traces the block to a firewall rule and adjusts it for verified Perplexity traffic accessing public pages.

The verification is a subsequent verified request receiving a 200 response and the actual service content. The team records the affected URL, responsible rule and evidence that the block has been resolved.

That establishes successful access. Whether Perplexity later selects and cites the page needs separate measurement.

Keep it working

Record the affected URL, failed check, responsible person, fix date and verification evidence. Repeat the checks after changes to your website templates, hosting or security settings.

Once access is working, monitor the buyer questions you want to appear for. The next priorities may be content quality, clearer service information or stronger evidence from other sources.

Tilio’s ongoing GEO support brings technical implementation together with content, authority work and tracking. Book a call if you want the checks and fixes handled as part of that wider programme.

FAQ

Can I allow AI search while blocking training bots?

Some providers offer separate controls. OpenAI lets you allow OAI-SearchBot while blocking GPTBot. Check each provider's documentation before applying rules.

Do I need llms.txt or special AI schema?

Google says neither is required for AI Overviews or AI Mode. Supporting pages must be indexed and eligible for a search snippet. Fix access and eligibility issues first.

Can I check this without a developer?

Start with Tilio's free AI checker for an initial readiness scan. Server logs, security rules and rendering problems usually need your developer or hosting provider. The checker does not test whether AI recommends your business.