So far, even Artificial intelligence companies have struggled to create tools that can reliably detect when a text was written. generated using a large language modelNow, a group of researchers has established a novel method for estimating LLM usage in a broad set of scientific writing by measuring which “surplus words” began to appear much more frequently during the LLM era (i.e., 2023 and 2024). The results “suggest that at least 10 percent of 2024 abstracts were processed using LLM,” according to the researchers.
In A preprint article published earlier this monthFour researchers from the University of Tübingen and Northwestern University in Germany said they were inspired by studies measuring the impact of the Covid-19 pandemic. Looking at the excess deaths Compared to the recent past, by similarly analyzing the “overuse of words” after LLM writing tools It became widely available in late 2022The researchers found that “the emergence of LLM caused an abrupt increase in the frequency of certain style words” that was unprecedented in both quality and quantity.”
Going deeper
To measure these vocabulary changes, the researchers analyzed 14 million abstracts of articles published in PubMed Between 2010 and 2024, they tracked the relative frequency of each word as it appeared throughout each year. They then compared the expected frequency of those words (based on the trend line before 2023) to the actual frequency of those words in the 2023 and 2024 abstracts, when LLMs were in widespread use.
The results found a number of words that were extremely uncommon in these scientific abstracts before 2023 and that suddenly increased in popularity after the introduction of the LLMs. The word “delves,” for example, appears in 25 times as many 2024 papers as would be expected from the pre-LLM trend; words like “showcasing” and “underscores” also increased in usage by ninefold. Other previously common words became noticeably more common in post-LLM abstracts: the frequency of “potential” increased by 4.1 percentage points, “findings” by 2.7 percentage points, and “crucial” by 2.6 percentage points, for example.
Of course, these kinds of changes in word usage could happen independently of LLM usage (the natural evolution of language means that words sometimes go in and out of fashion). However, the researchers found that in the pre-LLM era, such massive, sudden increases from one year to the next were only seen for words related to major global health events: “Ebola” in 2015; “Zika” in 2017; and words like “coronavirus,” “lockdown,” and “pandemic” in the period from 2020 to 2022.
In the post-master’s period, however, the researchers found hundreds of words with sudden, pronounced increases in scientific usage that had no common link to world events. Indeed, while the words in excess during the Covid pandemic were overwhelmingly nouns, the researchers found that the words with a post-master’s frequency increase were overwhelmingly “style words” like verbs, adjectives, and adverbs (a small sample: “across, additionally, comprehensive, crucial, enhancement, exhibited, insights, notably, personally, within”).
This is not a completely new finding: the increased prevalence of the word “deepen” in scientific articles It has been widely observed in the recent pastfor example. But earlier studies typically relied on comparisons with “real” human writing samples or lists of predefined LLM markers obtained outside the study. Here, the pre-2023 abstract set acts as its own effective control group to show how vocabulary choice has changed overall in the post-LLM era.
An intricate interaction
By highlighting hundreds of so-called “keywords” that became significantly more common in the post-LLM era, it’s sometimes easy to spot the telltale signs of LLM use. Take as an example this abstract line pointed out by the researchers, with the keywords highlighted: “A comprehensive understanding of the intricate interaction between (…) and (…) is essential for effective therapeutic strategies.”
After performing some statistical measurements of keyword occurrences in individual articles, the researchers estimate that at least 10 percent of the post-2022 articles in the PubMed corpus were written with at least some LLM assistance. The number could be even higher, the researchers say, because the aggregate could be missing LLM-assisted abstracts that do not include any of the keywords they identified.
