Bing has officially entered the AI-enhanced search arena with Bing Generative Search, their response to Google’s AI Overviews. This feature is still in the preliminary stages, visible only for a select few queries as part of its initial rollout.
Over the past few months, we have seen a mixed response to Google’s AI Overviews with summaries that range from inaccurate and false to downright disturbing and dangerous. Will we see the same from Bing or have they learned from Google’s mistakes and figured out how to roll out its product more effectively? It’s still too early to tell but worth observing.
In this article, we will explore what Bing Generative Search offers, examining its functionality, potential impacts on website visibility, and how it compares to Google’s AI Overviews. We’ll get into the technical workings, assess its unique features, and discuss its implications for users and SEO strategists.
Let’s get to it.
What is Bing Generative Search?
Bing Generative Search (BGS) is an AI-enhanced search result page that uses traditional search results, large language models (LLMs), and small language models. According to Bing, their generative AI-powered search results page “creates a bespoke and dynamic response to a user’s query.” When the searcher enters a query into the search box, the system determines whether standard organic results are sufficient or if the AI-enhanced experience provides a better response. Currently, BGS only activates for a small number of queries.
For instance, the press release features a representative search example for “How long can elephants live,” which showcases a layout markedly different from that of Google’s AI Overview (AIO).
The page features a Document Index that makes it easy to quickly navigate to Related Sections, enhancing user experience by making information more accessible. Organic search results are positioned to the right of the answer summary, ensuring they remain visible and relevant. Additionally, Source Citations are prominently displayed throughout the AI-generated experience, providing greater transparency and easy verification of the information presented.
You might be wondering how BGSs impact search ads. The answer summaries are shifted down the page to make room for ads at the top of the page for specific queries.
Bing Generative Search vs. Google AI Overview
Both BGS and Google’s AIOs aim to provide a summarized response to search queries, but they differ significantly in design and functionality.
Layout Differences
BGS adopts a more elaborate layout that takes up more real estate both horizontally and vertically on the search results page, offering detailed information about the query.
In contrast, the AIO sports a simpler and cleaner layout. While BGS features a document index and search results surrounding its summary on the left and right, the AIO opts for a minimalist approach with white space surrounding the summary. This design reduces visual clutter, enhancing user experience by making the summary more visually accessible.
Impact on Organic Search Results
AI summaries for both Google and Bing take up a significant amount of space on the search results page. This has a significant impact on the visibility of organic search results. While the BGS has visible search results next to the summary on desktop, on mobile the results are located below the summary.
There is a a significant difference between how organic results are presented in Microsoft Edge and other browsers. Both Chrome and Safari show organic results automatically, in Microsoft Edge you have to select See more on the right. See below:
In Google AIOs, the summary pushes the organic search results further down the page for both mobile and desktop.
Overall, Bing’s experience provides more (albeit mixed) visibility to the organic search results, which potentially helps lessen the impact of the summaries on clicks.
Citations and Links
Both BGS and AIO summaries provide links to URLs cited as sources for the summaries. The BGS experience provides annotations for each section of AI-generated content. This makes it very easy to identify the source document for the statement.
As you hover over each annotation and section of text, a box opens providing direct links to the source verifying the statement.
Below each section of the summary, Bing also provides the sourced documents. If you select the arrow next to ‘Learn more,’ the additional documents will display.
In AIOs, links may be provided in a carousel underneath the full summary or individual sections. To see the links, you have to select the arrow under each section.
A key distinction here is that Google uses text fragments in URLs to highlight relevant excerpts directly on the sourced document page, enhancing the specificity and transparency of the citations.
This provides more specificity and transparency regarding how the information in the summary is verified.
Accessibility and User Authentication: Signed-In and Incognito Mode
You do not have to be signed in to Bing for the AI summary to appear. You can use it in incognito mode no problem.
On the other hand, AIOs do not appear if you are not signed in or in incognito mode. This makes Bing’s version much more user friendly.
User Warnings Regarding AI Generated Content
We know that AI-created content potentially includes inaccurate or false information and the US Congress currently scrutinizing Google’s AI Overviews for these issues. How do Bing and Google warn users?
In BGS, you have to select the information icon to see any type of warning to the searcher. It states “This summary wan generated using AI based on multiple online sources. To view the original source information, use the ‘Sources’ links.”
Google includes a more visible warning to users which can vary. The warning states “Generative AI is experimental.” For health- and finance- related queries, the summar directs the searcher to seek out professional advice.
By selecting Learn more in the AIO, an additional and more specific warning is provided:
Overall, while both platforms seek to enhance the user search experience with AI-driven summaries, their approaches differ significantly in design, user interaction, and the handling of source citations and warnings.
Links in Bing Generative Search Result and Search Position
Bing has provided several examples of its generative search experience which help provide some first insights into the linked sources. This is a very small sample of summaries. I identified queries that trigger BGS results by entering the query “bing generative search” in the Bing search box. Including the 2 from Bing’s press release, here are 18 queries that trigger BGS:
- what is a spaghetti western
- how long can elephants live
- can you live with one lung
- norfolk island pine care
- sleep training methods
- body focused repetitive behaviors
- bladder stones in dogs
- what was the first anime
- epigastric hernia symptoms
- benefits of ginger water
- french independence day
- what do raccoons eat
- littermate syndrome
- air conditioner freezing
- lower back tightness
- nightshades and inflammation
- how to airbrush for beginners
- build your own kitchen cabinets
Note: “UNESCO’s Patagonian Jewel” was identified by BGS as a query but it did not trigger the AI experience on my end.
Link Diversity in Bing Generative Search
On average, BGS summaries linked to 6.6 URLs ranging from 4 on the low end to 12 on the high end. Each URL can appear multiple times in the various sections.
In the 18 examples observed, it appears that Bing’s Generative Search doesn’t just pull information from the top-ranking documents related to a query but also includes lower-ranking and unranked documents. 28.3% of URLs were found to not rank in the top 20 results for the query. That’s over 70% of URLs ranking in the top 20 and 59% ranked in the top 10. That’s a significantly higher percentage than what we’ve seen in AIO studies.
Specific Query Examples
For the query “what is a spaghetti western”, there were 6 URLs cited, and all of them are in the top 10 results for that query.
For the query, “how long can elephants live”, 3 URLs were in the top 10 results, 2 were out of the top 10, and 3 did not rank at all in the top 100 results for the query.
The fact that the summaries can include documents that don’t rank for the query provides some important clues as to how the system might work.
Comparisons with Google AI Overviews
If you’ve been following research into Google AIOs, this should sound familiar. Several studies from Advanced Web Ranking, SE Ranking, and Authoritas found that AIO summaries included a significant number of links that were low or unranked for that query. Google’s patent underlying the AIOs offered some clues as to why this might happen. When building summaries, the system evaluates documents responsive to the query but, if the top results are repetitive, it will explore documents responsive to related queries as well.
We are likely seeing the same thing occurring in Bing Generative Search. This can be good news for website owners if the query is competitive. You might be able to gain visibility for that query by targeting related queries. I explore this in an earlier article highlighting how A Overviews explore related queries for documents to provide unique information.
How Does Bing Generative Search Work?
What does all of this tell us about how BGS actually works? Based on the examples provided and our understanding of AIOs, we can make some very, very early educated guesses that may change as we see more examples.
When you enter a query in Bing, if the query is identified for an AI-enhanced search result, you will receive a BGS summary. Bing has stated that its answer summaries are created “by combining the power of generative AI and large language models (LLMs) with the search results page.”
The Role of Retrieval-Augmented Generation (RAG)
This indicates the system uses what’s known as a Retrieval-Augmented Generation (RAG) system. BGS leverages the underlying LLMs’ training data as well as Bing’s own search index. The system accesses additional documents to provide more accurate and up-to-date information, supplement, or verify information in the LLM’s training data.
Process Breakdown:
- Query Identification: When a query is entered, the system first identifies whether it is appropriate for an AI-enhanced summary.
- Document Retrieval: The system then pulls in documents that might confirm or supplement the information derived from the LLM’s initial data output.
- Summary Creation: Using a combination of sourced content and generative AI, Bing crafts a summary that is both accurate and informative.
The first step in creating the summary likely looks something like this:
Comparison with Google’s AI Overviews
This approach closely mirrors how Google’s AIOs operate, as described in its patent. From the analysis of link and search position data, we understand that the system does not solely select sources directly responsive to the query. It seems that the system either seeks out documents responsive to related queries or documents with text that verifies the statements in the summary. In both scenarios, this is likely done through embed distance.
Given this information, we can hypothesize that Bing’s methodology closely resembles that used in AIOs and looks something like this:
Hypothesized Bing Generative Search Process:
- Query Input: A user types a query into Bing.
- LLM Processing: The query is processed by LLMs, which utilizes its training data to generate an initial interpretation.
- RAG Decision: The system decides whether to employ Retrieval-Augmented Generation (RAG), determining if additional documents are needed to supplement or verify the LLM’s output.
- Document Retrieval: If RAG is used, Bing searches for direct match and related query documents to ensure comprehensive and accurate responses.
- Summary Creation: Bing generates a search summary, incorporating information verified by linked documents.
- Verification: Each point in the summary is supported by linked documents to ensure the information is credible and accurate.
As more information becomes available about the underlying mechanisms, I will update.
Speculation on Personalization
While it is not explicitly confirmed, there’s a possibility that Bing could personalize the generative search experience based on factors like search history, geographical location, language preferences, or user profile data. This aspect of personalization in search results could customize the information to more closely address user needs.
Optimizing for Visibility in Bing Generative Search
What insights can we gain about influencing Bing Generative Search (BGS) to display specific URLs in its summaries? Given the limited data from just 18 queries, it’s too soon for definitive conclusions, but early observations offer some insights.
The analysis of the SERP positions for the 18 URLs linked in the documents reveals a clear preference for top-ranked URLs, especially those in the top 2 positions. This suggests that for your target query, securing a spot within the top five organic search results dramatically increases your chances of visibility in BGS summaries.
Additionally, BGS’s inclusion of both low-ranked and unranked URLs presents an opportunity for enhanced visibility. By focusing on less competitive, related keywords and optimizing on-page content to align closely with relevant search queries (measured by embed distance), websites can improve their likelihood of being featured in BGS summaries. This approach not only aids in ranking for targeted keywords but also leverages the system’s broader criteria for content selection, potentially capturing additional traffic from related searches.
Conclusion
Bing Generative Search is the latest entry into AI-enhanced search results. This development promises to have a sizable impact on user experience as well as the visibility and engagement with the organic performance of content.
So far, we have seen the BGS experience for a small number of queries. This raises important questions: Is this the experience we will see across the board? Will each query have an AI-generated page experience as robust as these examples? The consistency of this feature’s implementation will significantly influence its effectiveness and reception.
Moreover, Bing has taken a much stronger stance than Google in providing visibility to organic results and transparency in sourcing summary content. Further, in previous features like Bing Copilot (formerly Chat), Bing integrated Bing Chat and Copilot data with web data in Bing Webmaster. This approach begs the question: Will Bing continue this trend or offer more specific insights into the performance of Bing Generative Search through Bing Webmaster?
Bing Generative Search was just released to the public on July 24, 2024. As more information becomes available, I will update the article.
