As anyone who’s taken a look at the Internet in recent weeks has probably noticed, Google’s radical AI upgrade to its search engine got off to a rocky start. Within days of the company launching AI-generated answers to search queries called AI Overviews, the feature was widely mocked for producing incorrect and sometimes far-fetched answers, such as recommendations for eating rocks or making pizza with glue.
New data from search engine optimization company BrightEdge suggests that Google has significantly reduced how often it shows people AI summaries since the feature was launched and, in fact, had already substantially limited the feature before the critics. The company has been tracking the appearance of Google’s AI responses in results for a list of tens of thousands of sample searches since the feature was first offered as a beta test last year.
When AI Overviews launched in English to registered US users after Google’s I/O conference on May 14, BrightEdge saw AI-generated responses in just under 27 percent of the queries it tracked. But its presence dropped precipitously a few days later, the week before screenshots of the AI Overviews errors went viral online. Late last week, when Google published a blog post acknowledging the flaws in its AI feature, BrightEdge saw AI overviews appear on just 11 percent of search results pages. Their prevalence was essentially the same on Monday.
Jim Yu, founder and CEO of BrightEdge, says the drop suggests Google has decided to take an increasingly cautious approach to this launch. “There are obviously some risks that they are trying to strictly manage,” he says. But Yu adds that he’s generally optimistic about how Google is approaching AI overviews and sees these early issues as a “small problem” rather than a feature.
“We continue to refine when and how we display AI overviews to make them as useful as possible, including a number of technical updates in the last week to improve the quality of the response,” says Google spokesperson Ned Adriance. Google declined to share its internal statistics on how often AI overviews appear in searches, but Adriance says BrightEdge’s numbers don’t reflect what the company sees internally.
It’s unclear why Google may have decided to significantly reduce the appearance of AI Overviews shortly after its launch, but the company’s blog post last week acknowledged that the fact that millions of people were using the feature provided new data on its performance and errors. The company’s head of search, Liz Reid, said Google had made “more than a dozen technical improvements”, such as limiting the appearance of satirical content in its results. Her post noted that these changes would create restrictions on when AI summaries were offered, but did not detail how exactly those restrictions would change how often AI results appeared.
BrightEdge began tracking AI overviews using its list of sample queries after Google allowed users to opt-in to a beta test of the feature late last year. The test queries covered nine categories, including e-commerce, insurance and education, and were designed to cover common but also rarer searches. They were tested again and again, in some cases several times a day.
In December 2023, BrightEdge found that summaries were appearing in 84 percent of its searches, but that number decreased over time. Google’s Adriance said he didn’t automatically turn on AI overviews in 84 percent of searches, but he didn’t clarify his internal measurements. After Google opened AI overviews to everyone, BrightEdge continued to track its appearance using a mix of accounts that had previously signed up for the beta test and others that hadn’t but didn’t see a significant difference between what users saw. two groups.
Google declined to share exactly how much the number of AI summaries it showed to the general public changed compared to people enrolled in the beta test, but Adriance said that people who had opted in to the test were shown summaries. of AI in a broader range of queries.
BrightEdge data also sheds light on the topics where Google believes AI overviews can be most useful. AI answers appeared in the most healthcare keyword searches, most recently at a frequency of 63 percent. Sample visits included in the BrightEdge data included “foot infection,” “intestinal bleeding,” and “telehealth urgent care.” By comparison, e-commerce queries return general AI descriptions at about 23 percent, while restaurants or travel rarely return general AI responses.
