Google-Extended
Google-Extended is the user-agent token Google publishes specifically so site owners can opt out of having their content used to train Google's Gemini and Vertex AI models — without affecting how Googlebot indexes the site for regular Search. It is controlled separately in robots.txt: a Disallow directive for Google-Extended blocks AI training use, while leaving Googlebot Allow rules untouched for SEO. Why it matters: Most brands should explicitly Allow Google-Extended — blocking it removes the brand from Gemini's future training data and weakens long-term AI recall, while delivering no SEO benefit. The correct stance for a B2B authority brand is open access for both Googlebot and Google-Extended.
Why Google-Extended matters
This bot determines whether your brand narrative becomes part of the large language model's permanent knowledge base. Maintaining access ensures that generative engines like Gemini can retrieve and synthesize your specific expertise, preventing your competitors from dominating the AI-generated answer space.
In practice
A digital publisher like The New York Times might use a robots.txt Disallow: / directive for this agent to protect intellectual property from being scraped by the Gemini API while keeping standard Googlebot open.
Common mistake
Assuming that blocking this agent improves privacy when it actually just ensures Gemini or Vertex AI cannot reference your specific expertise or brand history during future response synthesis.
How it connects
This technical directive directly impacts GenAI Optimization (GEO) strategies and sits alongside robots.txt protocols like Googlebot and GPTBot.
Learn more:
→ llms.txt GuideArticles About Google-Extended
Deep-dive guides and tactical breakdowns from our editorial team.
Frequently Asked Questions
What is Google-Extended?
In short: Google-Extended is google-Extended is the user-agent token Google publishes specifically so site owners can opt out of having their content used to train Google's Gemini and Vertex AI models — without affecting how Googlebot indexes the site for regular Search. See the full definition above for context.
Does blocking this crawler impact my site ranking in standard search results? My SEO is my main priority.
No, this specific token is isolated from traditional search rankings. While Googlebot handles indexing for the search results page, the Extended agent only governs permission for generative AI training datasets.
How do I specifically opt out if I want to protect my proprietary data from training sets?
You must add a dedicated User-agent: Google-Extended section to your robots.txt file with a Disallow directive. Keep in mind that doing so prevents Gemini from accurately citing your freshest content in its generated answers.
Is there any competitive advantage to leaving this access open for my brand?
Smart Money Media recommends allowing access because AI models serve as the new discovery layer for B2B buyers. If your data is excluded from training, AI engines may provide outdated or inaccurate information about your services to potential leads.
Related Terms
An AI sitemap is a machine-readable file — most commonly an llms.txt or llms-full.txt —…
llms.txtllms.txt is a proposed plain-text file placed at the root of a website (e.g. /llms.txt)…
GeminiGemini is Google's family of multimodal large language models that powers Google AI Mode,…
GPTBotGPTBot is OpenAI's web crawler used to gather training data for future GPT models. Site…
ClaudeBotClaudeBot is Anthropic's web crawler, used both for collecting training data and for…
PerplexityBotPerplexityBot is Perplexity AI's web crawler, used to fetch pages in real time when…