AI Search Optimization: How ChatGPT, Perplexity and Google Find and Cite Your Business

AI assistants like ChatGPT search, Perplexity, Microsoft Copilot and Google's AI Overviews and AI Mode don't keep a secret index you can pay to get into. They depend on web crawlers and search indexes. When someone asks a question, they pull the pages that answer it most clearly, write a response and link some of those pages as sources. For a small business, "generative engine optimization" (GEO) comes down to three jobs: let the right crawlers in, make your pages easy to understand and check, and get indexed by both Google and Bing. No one can promise that an AI assistant will cite you, and anyone who does is guessing. What you can do is remove the things that keep you out entirely.

How AI answer engines find your pages

Each assistant works a bit differently, but the pattern is the same. A crawler or search index collects pages. Then, when a user asks a question, the system retrieves relevant pages, writes an answer and cites some of them.

• Google AI Overviews and AI Mode run on Google Search's own index. Google says there are no extra requirements: the page only has to be indexed and eligible to appear in Search with a snippet. Google also says you don't need new machine-readable files, AI text files or special schema markup.
• ChatGPT search relies on OpenAI's search crawler, OAI-SearchBot. According to OpenAI, sites that opt out of OAI-SearchBot won't be shown in ChatGPT search answers, although they can still appear as plain navigational links.
• Perplexity uses PerplexityBot to surface and link websites in its results. It uses Perplexity-User when it visits a page because of a specific user question.
• Microsoft Copilot grounds many of its answers in Bing. If Bing hasn't indexed your page, Copilot has far less to work with.
• Claude (Anthropic) uses Claude-SearchBot to index content for search and Claude-User to fetch pages when a user asks.

The takeaway: a page that Google and Bing can't crawl or index is very unlikely to be cited.

Robots.txt: training crawlers vs. search crawlers

This is where many sites make a costly mistake. AI companies separate the crawler that collects training data from the crawler that powers search answers. A blanket "block all AI bots" rule can quietly remove you from AI search results. Allowing everything, on the other hand, may not match how you feel about model training.

Robots.txt token	Company	What it controls
GPTBot	OpenAI	Whether content may be used to train OpenAI's foundation models
OAI-SearchBot	OpenAI	Whether your site can appear in ChatGPT search answers
ChatGPT-User	OpenAI	Visits triggered by a user's request; OpenAI says robots.txt rules may not apply
PerplexityBot	Perplexity	Surfacing and linking your site in Perplexity results; Perplexity says it isn't used to train foundation models
Perplexity-User	Perplexity	User-initiated visits; Perplexity says this fetcher generally ignores robots.txt
Google-Extended	Google	Use of your content for training and grounding Google's Gemini models; Google says it doesn't affect inclusion or ranking in Google Search
ClaudeBot	Anthropic	Collection of content that may be used for model training
Claude-SearchBot	Anthropic	Indexing for Claude's search results
Googlebot / Bingbot	Google / Microsoft	Regular search crawling, which also feeds AI Overviews, AI Mode and Copilot

Three points are easy to miss:

1. Google-Extended is a token, not a crawler. Google crawls with its existing user agents. Blocking Google-Extended does not remove you from AI Overviews or AI Mode. For those, Google points to normal Googlebot rules and snippet controls such as nosnippet, data-nosnippet, max-snippet and noindex.
2. Decide on a policy before you edit the file. Some businesses want to appear in AI search but would rather not contribute to training. Others are happy to allow both. Both are legitimate choices.
3. Changes are not instant. OpenAI and Perplexity both say updates can take about 24 hours to take effect, and Google's recrawling can take much longer.

Here is an example for a business that wants AI search visibility but opts out of training by OpenAI and Google. It's an illustration, not a rule for every site:

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: *
Disallow: /admin

Keep in mind that robots.txt is a request that reputable crawlers honor. It is not access control. Anything private belongs behind a login.

Check your firewall and CDN, too

Bot protection on your host or CDN can block these crawlers even when robots.txt allows them. Google's guidance specifically mentions allowing crawling in robots.txt "and on any CDN or host". Perplexity recommends that firewall rules match both the user agent and its published IP ranges. If you use Cloudflare, review your bot settings alongside the rules in our Cloudflare WAF guide.

Make pages that are easy to quote

AI systems tend to cite pages that answer a question directly and are easy to verify. That is also good writing for human readers.

• Answer first. Open each page or section with the plain answer, then the detail. "We provide managed IT support for law firms in Miami-Dade and Broward" beats three paragraphs of slogans.
• One topic per page. Give each service, industry and area served its own page with a clear title and a single H1.
• Be specific and consistent. List your services, the areas you serve, the languages you support and how to reach you. Use the same business name and contact details on your site, your Google Business Profile and your directory listings.
• Show who is behind the business. A real About page, real credentials and a clear way to get in touch help both readers and search systems trust the content.
• Use text, not just images or PDFs. Google recommends textual content that supports images and video. Service details locked inside a scanned brochure are hard for any system to use.
• Keep it current. Update or remove outdated pages, and date your articles.
• Never fabricate. Fake reviews, invented statistics and made-up client stories break search policies and can surface in AI answers under your name.

Structured data: clarity, not magic

Schema.org markup such as Organization or LocalBusiness, Service and BreadcrumbList helps search engines understand who you are and what you offer. Google is clear on two points: no special schema is required to appear in its AI features, and structured data must match what is visible on the page. Add it to describe your business accurately, not to chase a ranking trick.

Get indexed by Google and Bing

1. Google Search Console. Verify your domain, submit your XML sitemap and check the Page indexing report for important pages that aren't indexed.
2. Bing Webmaster Tools. Verify your site (you can import it from Google Search Console) and submit the same sitemap. This matters for Copilot and for other services that use Bing's index.
3. IndexNow. This open protocol lets you notify participating search engines when a URL is added or changed. The IndexNow FAQ lists Amazon, Bing, Naver, Seznam.cz, Yandex and Yep as participants; Google isn't on that list. It also states that submitting a URL does not guarantee indexing.
4. Measure what you can. Google Search Console now has a Generative AI performance report that shows impressions from AI Overviews and AI Mode. Google says it rolled out to all websites as of August 31, 2026, but the report may not appear for sites with too few AI impressions. Bing Webmaster Tools launched an AI Performance report as a public preview in February 2026. It shows how often your pages are cited in Microsoft Copilot and Bing's AI answers. In your analytics, watch for referrals from chatgpt.com and perplexity.ai.

Fast, well-built pages help here too. Our guide to website speed and Core Web Vitals covers the technical basics.

What about llms.txt?

llms.txt is a Markdown file placed at /llms.txt that summarizes a website for large language models. Jeremy Howard proposed it in September 2024, and it is still a community proposal, not a formal standard. None of the major AI search providers we reviewed lists it as a requirement, and Google says outright that you don't need AI text files to appear in AI Overviews or AI Mode.

Our view: it's cheap to add and harmless if it stays accurate, so we publish one on our own site. But we don't count on it to earn citations, and you shouldn't either. If you add one, keep it consistent with what your pages actually say.

What not to do

• Don't buy "guaranteed AI citations" or "guaranteed AI rankings." No vendor controls what ChatGPT, Perplexity, Copilot or Google decide to cite.
• Don't mass-produce pages. Google's spam policies treat generating many low-value pages, including automated rewrites and translations, as scaled content abuse.
• Don't show bots different content than people. That is cloaking, and it violates search policies.
• Don't block Googlebot or Bingbot in the name of "blocking AI." You would lose both classic search and AI answers.

If your business is in Miami-Dade or elsewhere in South Florida and you want a second pair of eyes on your robots.txt, firewall rules, structured data and indexing, Core Florida IT can review them as part of our custom website and performance optimization work. Start by requesting a free IT assessment.

Sources

• AI features and your website (Google Search Central)
• Google's common crawlers and Google-Extended (Google Search Central)
• Overview of OpenAI crawlers (OpenAI)
• Perplexity crawlers (Perplexity docs)
• Does Anthropic crawl data from the web, and how can site owners block the crawler? (Anthropic)
• Generative AI performance report (Search Console Help)
• Introducing AI Performance in Bing Webmaster Tools Public Preview (Bing Webmaster Blog)
• IndexNow FAQ
• The /llms.txt file proposal (llmstxt.org)
• Spam policies for Google web search (Google Search Central)