Evaluating scientific impact of publications: combining citation polarity and purpose

Citation counts are commonly used to evaluate the scientific impact of a publication on the general premise that more citations probably mean more endorsements. However, two questionable assumptions underpin this idea: a) that all authors contributed equally to the paper; and b) that the endorsement is positive. Obviously, neither of these assumptions hold true. Hence, with this study, we examine two components of citations—their purpose, i.e., the reason for the citation, and polarity, being the author’s attitude toward the cited work. Our findings provide a new perspective on the scientific impact of highly-cited publications. Our methodology consists of three steps. Firstly, a pre-trained model composed of a Word2Vec—a well-known word embedding approach—and a convolutional neural network (CNN) is used to identify citation polarity and purpose. Secondly, in a set of highly-cited papers, we compare eight categories of purpose from foundational to critical and three categories of polarity: positive, negative, and neutral. We further explore how different types of papers—those discussing discoveries or those discussing utilitarian topics—influence the evaluation of scientific impact of papers. Finally, we mine and discover the knowledge (e.g. method, concept, tool or data) to explain the actual scientific impact of a highly-cited paper. To demonstrate how combining citation polarity with purpose can provide far greater details of a paper’s scientific impact, we undertake a case study with 370 highly-cited journal articles spanning “Biochemistry & Molecular Biology” and “Genetics & Heredity”. The results yield valuable insights into the assumption about citation counts as a metric for evaluating scientific impact.

     

Ranking scientific publications considering the aging characteristics of citations

Ranking the significance of scientific publications has been a challenging topic for a long time. So far, many ranking methods have been proposed, one of which is the well-known PageRank algorithm. In this paper, we introduce aging characteristics to the PageRank algorithm via considering only the first 10 year citations when aggregating resource from different nodes. The validation of our new method is performed on the data of American Physical Society journals. The results indicate that taking into account aging characteristics improves the performance of the PageRank algorithm in terms of ranking accuracy for both papers and authors. Though our method is only applied to citation networks in this paper, it can be naturally used in many other real systems and similar improvements are expected.

     

Exploiting Structural Similarities to Classify Citations

Citations play an important role in the scientific community by assisting in measuring multifarious policies like the impact of journals, researchers, institutions, and countries. Authors cite papers for different reasons, such as extending previous work, comparing their study with the state-of-the-art, providing background of the field, etc. In recent years, researchers have tried to conceptualize all citations into two broad categories, important and incidental. Such a categorization is very important to enhance scientific output in multiple ways, for instance, (1) Helping a researcher in identifying meaningful citations from a list of 100 to 1000 citations (2) Enhancing the impact factor calculation mechanism by more strongly weighting important citations, and (3) Improving researcher, institutional, and university rankings by only considering important citations. All of these uses depend upon correctly identifying the important citations from the list of all citations in a paper. To date, researchers have utilized many features to classify citations into these broad categories: cue phrases, in-text citation counts, and metadata features, etc. However, contemporary approaches are based on identification of in-text citation counts, mapping sections onto the Introduction, Methods, Results, and Discussion (IMRAD) structure, identifying cue phrases, etc. Identifying such features accurately is a challenging task and is normally conducted manually, with the accuracy of citation classification demonstrated in terms of these manually extracted features. This research proposes to examine the content of the cited and citing pair to identify important citing papers for each cited paper. This content similarity approach was adopted from research paper recommendation approaches. Furthermore, a novel section-based content similarity approach is also proposed. The results show that solely using the abstract of the cited and citing papers can achieve similar accuracy as the state-of-the-art approaches. This makes the proposed approach a viable technique that does not depend on manual identification of complex features.     

Visualizing the context of citations referencing papers published by Eugene Garfield: a new type of keyword co-occurrence analysis

During Eugene Garfield’s (EG’s) lengthy career as information scientist, he published about 1500 papers. In this study, we use the impressive oeuvre of EG to introduce a new type of bibliometric networks: keyword co-occurrences networks based on the context of citations, which are referenced in a certain paper set (here: the papers published by EG). The citation context is defined by the words which are located around a specific citation. We retrieved the citation context from Microsoft Academic. To interpret and compare the results of the new network type, we generated two further networks: co-occurrence networks which are based on title and abstract keywords from (1) EG’s papers and (2) the papers citing EG’s publications. The comparison of the three networks suggests that papers of EG and citation contexts of papers citing EG are semantically more closely related to each other than to titles and abstracts of papers citing EG. This result accords with the use of citations in research evaluation that is based on the premise that citations reflect the cognitive influence of the cited on the citing publication.     

The citation impact of social sciences and humanities upon patentable technology

This paper examines the citation impact of papers published in scientific-scholarly journals upon patentable technology, as reflected in examiner- or inventor-given references in granted patents. It analyses data created by SCImago Research Group, linking PATSTAT’s scientific non-patent references (SNPRs) to source documents indexed in Scopus. The frequency of patent citations to journal papers is calculated per discipline, year, institutional sector, journal subject category, and for “top” journals. PATSTAT/Scopus-based statistics are compared to those derived from Web of Science/USPTO linkage. A detailed assessment is presented of the technological impact of research publications in social sciences and humanities (SSH). Several subject fields perform well in terms of the number of citations from patents, especially Library and Information Science, Language and Linguistics, Education, and Law, but many of the most cited journals find themselves in the interface between SSH and biomedical or natural sciences. Analyses of the titles of citing patents and cited papers are presented that shed light upon the cognitive content of patent citations. It is proposed to develop more advanced indicators of citation impact of papers upon patents, and ways to combine citation counts with citation content and context analysis.

     

A deep-learning based citation count prediction model with paper metadata semantic features

Predicting the impact of academic papers can help scholars quickly identify the high-quality papers in the field. How to develop efficient predictive model for evaluating potential papers has attracted increasing attention in academia. Many studies have shown that early citations contribute to improving the performance of predicting the long-term impact of a paper. Besides early citations, some bibliometric features and altmetric features have also been explored for predicting the impact of academic papers. Furthermore, paper metadata text such as title, abstract and keyword contains valuable information which has effect on its citation count. However, present studies ignore the semantic information contained in the metadata text. In this paper, we propose a novel citation prediction model based on paper metadata text to predict the long-term citation count, and the core of our model is to obtain the semantic information from the metadata text. We use deep learning techniques to encode the metadata text, and then further extract high-level semantic features for learning the citation prediction task. We also integrate early citations for improving the prediction performance of the model. We show that our proposed model outperforms the state-of-the-art models in predicting the long-term citation count of the papers, and metadata semantic features are effective for improving the accuracy of the citation prediction models.

     

Does cumulative advantage affect collective learning in science? An agent-based simulation

Agent-based simulation can model simple micro-level mechanisms capable of generating macro-level patterns, such as frequency distributions and network structures found in bibliometric data. Agent-based simulations of organisational learning have provided analogies for collective problem solving by boundedly rational agents employing heuristics. This paper brings these two areas together in one model of knowledge seeking through scientific publication. It describes a computer simulation in which academic papers are generated with authors, references, contents, and an extrinsic value, and must pass through peer review to become published. We demonstrate that the model can fit bibliometric data for a token journal, Research Policy. Different practices for generating authors and references produce different distributions of papers per author and citations per paper, including the scale-free distributions typical of cumulative advantage processes. We also demonstrate the model’s ability to simulate collective learning or problem solving, for which we use Kauffman’s NK fitness landscape. The model provides evidence that those practices leading to cumulative advantage in citations, that is, papers with many citations becoming even more cited, do not improve scientists’ ability to find good solutions to scientific problems, compared to those practices that ignore past citations. By contrast, what does make a difference is referring only to publications that have successfully passed peer review. Citation practice is one of many issues that a simulation model of science can address when the data-rich literature on scientometrics is connected to the analogy-rich literature on organisations and heuristic search.     

Exploring the analytical potential of comparing citing and cited source items

Summary Comparing properties of citing and cited source items opens a wide variety of analytical possibilities. In a study of citations among papers in the journal Scientometrics a number of analytical themes are identified. The analysis shows: the way in which a citation graph can be decomposed into different subparts; country specific citation patterns; the effects of self-citations and domestic citations; the mapping of cited author relationships using direct citation and co-citation links; and time slicing effects on impact ranking of countries and papers.     

Patents cited in the scientific literature: An exploratory study of 'reverse' citation relations

This paper reports on a new approach to study the linkage between science and technology. Unlike most contributions to this area we do not trace citations of scientific literature in patents but explore citations of patents in scientific literature. Our analysis is based on papers recorded in the 1996-2000 annual volumes of the CD-Edition of Science Citation Index (SCI) of the Institute for Scientific Information (ISI) and patent data provided by the US Patent and Trademark Office. Almost 30,000 US patents were cited by scientific research papers. We analysed the citation links by scientific fields and technological sectors. Chemistry-related subfields tended to cite patents more than other scientific area. Among technological sectors, chemical clearly dominates followed by drugs and medical patents as the most frequently cited categories. Further analyses included a country-ranking based on inventor-addresses of the cited patents, a more detailed inspection of the ten most cited patents, and an analysis of class-field transfers. The paper concludes with the suggestions for future research. One of them is to compare our 'reverse' citation data with 'regular' patent citation data within the same classification system to see whether citations occur, irrespectively of their directionality, in the same fields of science and technology. Another question is as to how one should interpret reverse citation linkages.     

Some reflections on being cited 10,000 times

This letter describes the author’s reactions to being cited 10,000 times in Google Scholar. Data are provided to illustrate differences between the numbers of citations for books and edited works and between papers on different topics and in different styles.

     

The citation advantage of foreign language references for Chinese social science papers

Contemporary scientific exchanges are international, yet language continues to be a persistent barrier to scientific communication, particularly for non-native English-speaking scholars. Since the ability to absorb knowledge has a strong impact on how researchers create new scientific knowledge, comprehensive access to and understanding of both domestic and international scientific publications is essential for scientific performance. This study explores the effect of absorbed knowledge on research impact by analyzing the relationship between the language diversity of cited references and the number of citations received by the citing paper. Chinese social sciences are taken as the research object, and the data, 950,302 papers published between 1998 and 2013 with 8,151,327 cited references, were collected from the Chinese Social Sciences Citation Index. Results show that there is a stark increase in the consumption of foreign language material within the Chinese social science community, and English material accounts for the vast majority of this consumption. Papers with foreign language references receive significantly more citations than those without, and the citation advantage of these internationalized work holds when we control for characteristics of the citing papers, such as the discipline, prestige of journal, prestige of institution, and scientific collaboration. However, the citation advantage has decreased from 1998 to 2008, largely as an artifact of the increased number of papers citing foreign language material. After 2008, however, the decline of the citation advantage subsided and became relatively stable, which suggests that incorporating foreign language literature continues to increase scientific impact, even as the scientific community itself becomes increasingly international. However, internationalization is not without concerns: the work closes with a discussion of the potential problems associated with the lack of linguistic diversity in scientific communication.

     

The lengthening of papers’ life expectancy: a diachronous analysis

The aging of scientific has generally been studied using synchronous approaches, i.e., based on references made by papers. This paper uses a diachronous model based on citations received by papers to study the changes in the life expectancy of three corpus of papers: papers from G6 and BRICS countries, papers published in Science, Nature, Physical Review and the Lancet and all papers divided into four broad fields: medical sciences, natural sciences and engineering, social sciences and arts and humanities. It shows that that: (i) life expectancy is extensively different from a corpus to another and may be either finite or infinite, meaning that the corpus would never be obsolete from a mathematical perspective; (ii) life expectancy for scientific literature has lengthened over the 1980–2000 period; (iii) life expectancy of developed countries’ (G6) literature is on average shorter than that of emerging countries (BRICS).     

Trends in the core literature on tropical medicine: a bibliometric analysis from 1952-2002

Summary We present empirical data on frequency and pattern of misprints in citations to twelve high-profile papers. We find that the distribution of misprints, ranked by frequency of their repetition, follows Zipf's law. We propose a stochastic model of citation process, which explains these findings, and leads to the conclusion that about 70-90% of scientific citations are copied from the lists of references used in other papers.     

The effect of keyword repetition in abstract and keyword frequency per journal in predicting citation counts

This paper investigates an association between two new variables and citations in papers. These variables include the abstract ratio (the sum of repetition of keywords in abstract divided by abstract length) and the weight ratio (the frequency of paper’s keyword per journal). The data consist of 5875 papers from 12 journals in education: three journals from each SCImago quartile. The researchers used semi-continuous regression to model the data and measure the impact of the proposed variables on citations. The results revealed that both abstract ratio and weight ratio are statistically significant predictors of citations in scientific articles in education.     

The utility of scientific papers

This article investigates the function of scientific papers in the production of scientific knowledge. For this production, the citations made of these papers in the scientific literature can be considered as economic utilities. The work of the scientist is described as the production of citations by means of citations. The number of citations received by a given paper can be used to measure the paper's formal utility. The formal utility of scientific papers is studied empirically. It is concluded that the references contained in a scientific paper are a major determinant of its future utility.     

Identification of important citations by exploiting research articles’ metadata and cue-terms from content

Citations play a pivotal role in indicating various aspects of scientific literature. Quantitative citation analysis approaches have been used over the decades to measure the impact factor of journals, to rank researchers or institutions, to discover evolving research topics etc. Researchers doubted the pure quantitative citation analysis approaches and argued that all citations are not equally important; citation reasons must be considered while counting. In the recent past, researchers have focused on identifying important citation reasons by classifying them into important and non-important classes rather than individually classifying each reason. Most of contemporary citation classification techniques either rely on full content of articles, or they are dominated by content based features. However, most of the time content is not freely available as various journal publishers do not provide open access to articles. This paper presents a binary citation classification scheme, which is dominated by metadata based parameters. The study demonstrates the significance of metadata and content based parameters in varying scenarios. The experiments are performed on two annotated data sets, which are evaluated by employing SVM, KLR, Random Forest machine learning classifiers. The results are compared with the contemporary study that has performed similar classification employing rich list of content-based features. The results of comparisons revealed that the proposed model has attained improved value of precision (i.e., 0.68) just by relying on freely available metadata. We claim that the proposed approach can serve as the best alternative in the scenarios wherein content in unavailable.

     

Technological Relevance of Science: An Assessment of Citation Linkages between Patents and Research Papers

Patent citations to the research literature offer a way for identifying and comparing contributions of scientific and technical knowledge to technological development. This case study applies this approach through a series of analyses of citations to Dutch research papers listed on Dutch-invented and foreign patents granted in the US during the years 1987–1996.First, we examined the general validity and utility of these data as input for quantitative analyses of science-technology interactions. The findings provide new empirical evidence in support of the general view that these citations reflect genuine links between science and technology. The results of the various analyses reveal several important features of industrially relevant Dutch science: (1) the international scientific impact of research papers that are also highly cited by patents, (2) the marked rise in citations to Dutch papers on foreign-invented patents; (3) the large share of author-inventor self-citations in Dutch-invented patents; (4) the growing relevance of the life sciences, (5) an increase in the importance of scientific co-operation. We also find significant differences between industrial sectors as well as major contributions of large science-based multinational enterprises, such as Philips, in domestic science-technology linkages.The paper concludes by discussing general benefits and limitations of this bibliometric approach for macro-level analysis of science bases in advanced industrialised countries like the Netherlands.     

What is Special about Patent Citations? Differences between Scientific and Patent Citations

The emergence of pattent bibliometrics as a new branch of scientometrics necessitates a deeper understanding of the relationship between patents and papers. As this connection is established through the linkage between patents and research papers, one must have a clear idea of similarities and differences between patent and paper citations. This paper will investigate to what extent one can not only apply bibliometric methods to patents but also extend the existing interpretative framework for citations in research papers to the field of patent citations. After pointing out some parallels in the debates about the nature of citations in patents and scientific articles, the paper outlines those parts of bibliometric theory covering scientific citations that could be relevant to patent citations too. Then it highlights the specialties and peculiarities of patent citations. One major conclusion is that the general nature of a common framework for both scientific and patent citations would severely limit its usefulness, but research on academic citations might still be a great source of inspiration to the study of patent citations.