In a new study, University of Michigan School of Information assistant professor Misha Teplitskiy and his colleagues test if the number of citations a paper has reflects the paper’s influence on subsequent research.  

Every scholar wants to be seen as an expert in their field. A common way to measure the intellectual influence a scholar, or even a paper, has is to look at journal impact factors and h-indices. These impact factors are metrics that are derived from citations of a research paper. 

Citations are appealing and seemingly straightforward: one citation, one unit of influence. However, this simple measurement does not account for the fact that some citations are more influential than others. For example, authors might cite a study that greatly influenced their research choices or thought processes. On the other hand, authors may cite a study simply for general context or background. The former has more influence on authors’ research than the latter.

Teplitskiy says their goal was to distinguish between three things in a paper: quality, influence and citation count.

“If researchers do cite popular papers superficially, then it would mean that the lofty citation counts of popular papers aren’t very meaningful,”says Teplitskiy. “Those well-known papers would actually have less influence than they seem to and the more obscure papers are relatively more important that people sometimes think if they go by citation metrics.”

To measure influence directly, the team conducted a large-scale study of citations, including an international survey of 9,380 corresponding authors across 15 academic disciplines. They asked the researchers a number of questions, including how they discovered a reference, at what point in the project did they discover the citation, and how closely did the researcher read the paper they are citing. 

They found that popular, highly cited papers are not superficial sources in new research. In fact, citations of highly cited papers are two to three times more likely to reflect intellectual influence on researchers. 

“Our key finding would be that highly cited papers are engaged with, and cited more meaningfully, than obscure papers,” he says. “When you get to truly famous papers — those are really driving the bus,” says Teplitskiy. 

“To me, that seems like a big finding, because it goes against pretty much everybody's assumption,” he says. “While researchers often think or hope that even obscure papers have a chance to be influential, in practice researchers focus much of their attention on famous papers.. 

They found that famous papers were discovered earlier in the research project, compared to more obscure studies. “These more famous papers you discover early and they motivate the whole project,” Teplitskiy says, adding that more obscure papers tend to be found when researchers are doing last-minute literature searches and tend to be read more superficially.

Their findings have implications for widely used metrics like the h-factor. Teplitskiy explains that one paper with 1,000 citations or 100 papers with 10 citations technically have the same number of citations and, as widely assumed in practice, influence. “But, if you have 100 obscure papers, you’re almost invisible,” he says. “That always struck me as almost irrational for our publication system as a whole.”

He says their findings support the view that citations and other metrics are status signals that help drive the attention of researchers and the direction of their future studies. “Our work confirms the suspicion that those status signals really help drive attention,” says Teplitskiy. “Rather than just simply reflecting the interests of research communities, these metrics actually actively shape where attention goes in future work.”

—Sarah Derouin, UMSI public relations specialist

Read “How status of research papers affects the way they are read and cited,” published in Research Policy. 

Read more about assistant professor Misha Teplitskiy.