A new AI Overview study just published by Advanced Web Ranking found 33.4% of the linked documents ranked in the top 10 organic search results for that query. A whopping 46.5% ranked outside the top 50.
Other recent studies have shown variability in how often AI Overviews link to top-ranked documents. SE Ranking, for example, found that 84.72% of “AIOs linked to at least one domain from the top 10 organic search results.” An earlier study by Authoritas found a significantly higher number of documents (93.8%) outside the top ranking sites. This study was published in January 2024 before AI Overviews were available to all Google users. It was also based on the Google Labs version, Search Generative Experience.
This discrepancy raises a critical question:. Why would Google’s AI Overviews cite low ranked, or even unranked, sources for that query? Considering Google’s patent, which emphasizes trustworthiness in its criteria nine times, it seems counterintuitive that content ranking outside the top 50, or not at all, would meet Google’s standards for trustworthiness.
Could it be that these studies fail to account for how Google AI Overview works? After examining the patent underlying AI Overviews, I’ve identified several reasons why the system might cite such low or unranked sources. This article aims to explore these explanations in detail.
First, let’s review Advanced Web Ranking’s AI Overview rank findings then get into how the AI Overview selects and links its sources. This will lay the groundwork to better understand what we might be observing in the studies.
Note: Unless stated otherwise, quotes and images in this article are sourced from Google’s “Generative summaries for search results.”
Advanced Web Ranking AI Overview Rank Findings
Advanced Web Ranking (AWR) published a study earlier this month offering several intriguing insights into AI Overviews. One of the goals of the study was to determine if a document needed “to rank organically to appear in SGE (AI Overview) results.”
As mentioned in the introduction, their study found that “33.4% of AI Overview links rank in that query’s top 10 organic results” and “46.5% of the URLs included in AI Overviews rank outside the top 50 organic results.” These findings are significant as they relate to the specific query entered. Here is the breakdown of rankings
One of the questions AWR attempted to answer was whether you needed to rank highly to appear in the AI Overview. They suggested that, logically, the system should favor the “top organic ranking domains as sources for the AI Overviews.” The underlying assumption here is that the system should only seek out documents in that specific query’s search results.
I believe this overlooks how Google AI Overviews are intended to work. One of its purposes is to help searchers “quickly understand information from a range of sources, including information from across the web and Google’s Knowledge Graph.” It seeks a variety of sources and diverse information. If the top results for a query are all covering the same information, it looks elsewhere for more unique information.
AI Overview’s Comprehensive Selection Criteria
As we will explore below, the AI Overview process for selecting documents extends beyond just the entered query. The patent shows that high organic ranking is indeed an important factor for inclusion in an AI Overview, but not solely for the specific query initially searched. The selection process also considers high-ranking documents from related, recent, and potentially implied queries, emphasizing the system’s preference for diversity, quality, trustworthiness, and relevance alongside ranking. Furthermore, it seeks out high ranking and relevant documents to link to that verify the summary’s content.
Visit the Google AI Overview Library
Find in-depth articles, research, FAQs, and tracking tools to enhance your knowledge and improve your performance with AI Overviews.
How Google AI Overview Selects and Links to Sources
An important quality of the AI Overview system is that it can engage in a search result document (SRD) search at two different points in its process: first when it creates the initial summary, and second, when it adds links that verify the content. The system can also create the summary without a document search.
Curiously, the SRDs selected for the summary creation may not necessarily be the same SRDs selected during the linkification process. In my reading of the patent, there was no clarification that they were, or were not, the same sources.
A couple additional notes here. The AI Overview system has several measures it uses to select documents beyond what is discussed here. Additionally, it can access information to build the summaries from its training data (the patent refers to this as its “world knowledge”), the Google index, and from the Knowledge Graph.
Initial Document Search for Summary Creation
When the AI Overview system initially constructs the summary, it can either use its training data or employ Retrieval Augmented Generation (RAG) to enhance the process. When the system seeks out and evaluates sources relevant to the received query, it considers factors such as positional ranking, selection rate (click-through rate), geographical area, and language.
See the document selection process highlighted in FIG. 2 below.
If the system determines that the top results with a sufficient selection rate are not different or high quality enough, it will then seek out SRDs from related queries. Specifically, the patent states it will search for SRDs responsive to related queries when it determines SRDs responsive to the received query are not “diverse relative to one another, being of low quality, and/or having other characteristic(s)).” Diversity and low quality are mentioned together 6 times, highlighting the importance of these to the system.
The system can identify multiple related queries and select documents from each of the corresponding search results. How does the system determine what are related queries?
Correlational threshold.
It’s analyzing the strength of the relationship between the query and related queries. This is accomplished through embedding.
What Exactly is Embedding?
To understand words, phrases, and their relationships to each other, natural language models like search engines and large language models (LLMs) have to convert them to a numerical representation. Each specific word or symbol is represented as a number called a vector. Words and phrases that are similar are close together numerically. This placement of word vectors in relation to each other in a numerical space is called embedding.
Getting back to the correlational threshold, the selection of related-query-responsive SRDs “can be performed only when the query-responsive search result document(s) are of low quality and/or not diverse relative to one another, and the magnitude of the correlation satisfies a threshold.” The strength of the relationship between the query and the related queries — distance in the embedding space — has to reach a certain tipping point to be considered.
After seeking out related queries, the system can optionally evaluate SRDs responsive to relevant recent queries the searcher has made and implied queries following the same process for evaluating the quality of each SRD.
After the summary has been generated, whether from training data, RAG, or both, the system next seeks to add links that verify the summaries statements.
Document Selection for Linkification
With the summary created, the system next verifies the information and adds citations. You can see the process in the image below.
The system seeks out SRDs to verify each section of the summary. It analyzes the most relevant top search results for each portion. The patent states, “the candidate document can be determined based on its corresponding to the top search result for such a search, or being in the top N search results.”
Top Search Result or Top N Search Results
This implies that the system can limit how deep into the SERPs it goes to find and select documents. This could be the top 10, top 20, or even top 50 search results for that query.
In addition to ranking, the system also considers the embedding distance between the document content and the related content in the summary. From the patent, “in determining whether the document verifies the portion of the NL based content the system can determine a distance measure between the content embedding and the document content embedding and determining, based on the distance measure, whether the document verifies the portion of the NL-based content (e.g., verifies only if distance measure is less than a threshold).”
It can also evaluate documents with the same measures used during the initial summary creation, including positional ranking, selection rate, quality, trustworthiness, diversity, and more.
You can read about those additional measures in my article detailing how to optimize for AI Overviews.
Are the Studies Misinterpreting the Process?
With that overview of SRD selection and linking out of the way, let’s turn our attention back to the Advanced Web Ranking, SE Ranking, and Authoritas studies.
While these high-quality studies offer great insights into AI Overviews, they employ a linear approach in their analysis. This means they focus on direct query matching, analyzing the SERP position of the SRDs cited by the AI Overview for the exact query.
The linear approach assumes the AI Overview creation process looks something like this:
The user enters a query, the system seeks out documents in the SERPs responsive only to that query, and then creates the AI Overview linking to those same documents. At the end, the researchers note the SERP position of the direct match documents in relation to the specific query.
However, this linear approach does not fully capture the non-linear and dynamic nature of the AI Overview’s process as outlined in the patent.
As noted earlier, if employing RAG, the system first looks for high-ranked, high CTR, high-quality, and diverse SRDs responsive to the specific query. However, when it doesn’t find suitable SRDs, it expands its search to related queries that are high-ranked, high CTR, high-quality, trustworthy and offer unique information. If necessary, it then further expands to recent queries and implied queries, following the same process.
Additionally, when adding links to the summaries, the system selects SRDs based on both their ranking and their closeness in the vector space to the specific portions of the summary, ensuring relevance and accuracy in its source verification. This method highlights an approach that incorporates semantic and contextual analysis, potentially missed by the linear evaluation methods of these studies.
Here’s a simplified look at how the AI Overview system works:
To fully understand how AI Overviews are selecting SRDs, studies need to adopt a non-linear approach that considers the multi-dimensional criteria used by Google’s AI Overview, rather than focusing solely on direct query matches.
In this approach, the measurement of SERP positions focuses on the SERP positions not just of the input query, but of documents responsive to the related, recent, or implied queries.
Another consideration not evident in the non-linear diagram above is the fact that the documents used to build the summary may not be the same documents used to verify and link to. Embed distance and ranking still apply in this case. The system seeks out content in documents that is close in the embedding space to the statements in the summary while also ranking high. This adds another layer of complexity to consider when conducting these studies.
As an aside, it’s important to note that since release, we’ve seen AI Overviews fall short in terms of citing high-quality, relevant, and trustworthy SRDs. When I refer to documents as being high-quality and trustworthy, this is not necessarily my view. This is purely based on statements in the patent and the intention of the system. I’m sure it will be fine tuned to do so over time.
Update 9/9/24: I partnered with Laurence O’Toole, CEO of Authoritas, to research AI Overviews and related queries. For a deeper dive into how Google’s system might prioritize diverse and relevant content over purely top-ranked entries, read the complete study on Google AI Overview Link Selection Based on Related Queries.
Example: Understanding AI Overview’s Source Selection
Let’s take a look at an example AI Overview and how we can analyze it based on what we have discussed.
Query Used: “how to break in running shoes”
Generated AI Overview: This AI Overview included 9 citations, demonstrating a diverse selection process. Of these, only 3 were within the top 10 search results for the query, while 3 were low-ranked, and 2 did not rank for the query at all. The distribution aligns with the findings from Advanced Web Ranking, highlighting the AI Overview’s approach to sourcing.
Based on my hypothesis, if the system used RAG, the 3 documents in the top ten are the documents the system determined to be most relevant to the entered query. The other 6 documents were selected based on related queries.
See the URLs, their citation position and their SERP ranking in the table below. I used the Google AI Overview Impact Analysis extension to extract the URLs and the Google SERP Extractor Tool extension to grab the SERP positions for the query.
| URL | AI Overview Citation Position | SERP Ranking Position |
| Nordstrom | 1 | 1 |
| Academy Sports | 2 | 6 |
| Lucky Feet Shoes | 3 | 10 |
| Wrightsock | 4 | 88 |
| KURU Footwear | 5 | 46 |
| Vionic Shoes | 6 | Not Ranked |
| GoEnGo | 7 | 76 |
| Nike | 8 | Not Ranked |
This AI Overview did not include any personalization (See image below). So a recent or implied query search for documents likely did not occur. On my end, I have not seen any AI Overviews outside of Google Labs indicating personalization as of yet.
Let’s look at the top keywords from Semrush for 2 of the low or unranked cites, Vionic Shoes and Kuru Footwear.
Top 10 ranked keywords for the Vionic Shoes page:
Here are the top 10 ranked queries for the Kuru Footwear site:
For both of those sites, most of the top keywords are related and relevant to the main keyword. Keep in mind that as the system widens it’s search, it is looking for content that will add something relevant and new to the AI Overview. It might have to increase the distance in the embedding space between the primary and related queries to find those documents.
Analysis of AI Overview Citations and Their Selection Process
- Initial Search for Summary Creation:
- The AI Overview starts by evaluating the top-ranked Search Result Documents (SRDs) for the query “how to break in running shoes.”
- The initial search returned three top 10 ranking results: Nordstrom, Academy Sports, and Lucky Feet Shoes.
- Due to a lack of diversity or insufficient quality among the other top SRDs, the system expanded its search.
- Exploring Related Queries:
- The system used embedding distances to explore related queries, seeking content based on rank, relevance, trustworthiness, and diversity.
- For Kuru Footwear, keywords like “how to break in shoes” and “how to break in shoes faster” indicated a strong thematic fit.
- For Vionic Shoes, keywords such as “shoe comfort” and “how to make shoes more comfortable” highlight its relevance to enhancing shoe comfort, a more distantly related aspect of breaking in new shoes.
- Despite being low or unranked for the primary query, both Kuru Footwear and Vionic Shoes were selected due to their relevance to related topics.
- Summary Verification:
- The system re-evaluates documents by embed distance and document rank to verify the accuracy and relevance of the content used in the summary.
- For instance, content from Vionic Shoes was used to verify the summary statement about using a hair dryer to loosen tight areas of the shoes.
- It remains unclear if the linked documents were directly used to create or just to verify the summary.
The SERP position of each document should be measured based on the direct match keyword “how to break in running shoes” in addition to the related queries. As I mentioned earlier, at this time there doesn’t seem to be much personalization going on in the AI Overviews. Therefore, this example focuses solely on related queries rather than recent or implied queries.
There will definitely be much variability here depending on the query and category. For example, I noticed that for several health-related queries, such as “are forehead thermometers accurate,” “what is a cold,” and “when is the best time to exercise,” the cited URLs were mostly in the top 10 SERP results.
AI Overview Visibility: Challenges and Opportunities
What does this mean for establishing AI Overview visibility? First, let’s start with the good news.
For more competitive queries, you might have a better chance at gaining visibility for related queries that are less competitive. For example, the query “how to break in running shoes” might be too difficult to rank high in the search results for. However, a keyword like “breaking in shoes faster” might be less competitive and relevant to your site. If you can rank in the top 10 for this query, you will have a better chance than ranking in the top 50 for “how to break in running shoes.”
Now for the bad news. Your content might be selected to help generate an AI Overview summary, but it might not receive a link. While your content is relevant enough to contribute to the summary, during the linkification stage, the system might identify a higher-ranking SRD that confirms the specific text in the summary and is more closely aligned in the vector embedding space.
In this case, you are already likely competitive in the SERPs. To improve your chances of appearing in the AI Overview, you will need to take two steps: 1) adjust your content to more closely align with what is included in the AI Overview (I discussed this process in my article on how to optimize for SGE through reverse engineering) and 2) aim to rank higher than the page that is currently cited.
Cyrus Shephard recently did just this and shared his process on X. He adjusted the text on his site to closely match the text in the AI Overview. As his site was already ranking in the top 10, he only had to reduce the embedding distance between the content on his page and the content in the AI Overview. In doing so, he satisfied the 2 main criteria we discussed above for linkification: embedding distance and high rank.
It wasn’t all good news for Cyrus, though. He ended up losing his featured snippet for that search result. Before making any changes to your content, consider other potential impacts they might have on your organic search performance and other search features.
Conclusion: Rethinking AI Overview Link Studies
This article focused solely on exploring how the AI Overview works in its current form and how it aligns with the descriptions in the patent. The question of whether the AI Overview should exclusively select documents based on the entered query falls outside the scope of this discussion.
The study by Advanced Web Ranking provided significant insights but may overlook the full complexity of Google’s AI Overviews. Beyond SRDs responsive to the specific query, the system may incorporate SRDs from related, recent, and implied queries and evaluates each for positional ranking, relevance, diversity, and trustworthiness.
For future studies, understanding how to optimize for AI Overviews through testing queries and related queries on a large scale is critical and admittedly difficult. Advanced Web Ranking, SE Ranking, and Authoritas have made significant progress already in this area. They have curated extensive sets of queries known to trigger AI Overviews and analyzed where these are ranked in the SERPs by pulling citations for each query.
The next step, which is admittedly the most challenging to do at scale, involves:
- Identifying potential related queries through embedding distance analysis to see where low-ranked or unranked SRDs might be appearing in the search results.
I encourage Advanced Web Ranking, SE Ranking, and Authoritas to consider this analysis as the next phase in their research on how AI Overviews link to SRDs.
For SEOs, website owners, and content creators, this highlights the importance of producing content that is high in quality, diverse and relevant to a range of related queries.
