The citation impact of social sciences and humanities upon patentable technology

This paper examines the citation impact of papers published in scientific-scholarly journals upon patentable technology, as reflected in examiner- or inventor-given references in granted patents. It analyses data created by SCImago Research Group, linking PATSTAT’s scientific non-patent references (SNPRs) to source documents indexed in Scopus. The frequency of patent citations to journal papers is calculated per discipline, year, institutional sector, journal subject category, and for “top” journals. PATSTAT/Scopus-based statistics are compared to those derived from Web of Science/USPTO linkage. A detailed assessment is presented of the technological impact of research publications in social sciences and humanities (SSH). Several subject fields perform well in terms of the number of citations from patents, especially Library and Information Science, Language and Linguistics, Education, and Law, but many of the most cited journals find themselves in the interface between SSH and biomedical or natural sciences. Analyses of the titles of citing patents and cited papers are presented that shed light upon the cognitive content of patent citations. It is proposed to develop more advanced indicators of citation impact of papers upon patents, and ways to combine citation counts with citation content and context analysis.

     

Microwave effect: analyzing citations from classic theories and their reinventions—a case study from a classic paper in aquatic ecology—Brooks & Dodson, 1965

Citations play an essential role in creating a knowledge network and recognizing relevant contributions during the process of scientific production. Despite the citations establishing the links between new evidence and the preceding ideas, classic articles may not be cited adequately. Our aim is to identify if classic studies are cited over time and if the recent studies are producing new knowledge or just “giving a new look” to pre-existing ideas. We evaluated whether the theory proposed by Brooks and Dodson (Science 150(3692): 28–35, 1965)-Size-efficiency Hypothesis was referenced in studies on the subject since its publication. Through the analysis of 1480 scientific papers, we quantified—from 1965 to 2018—the citation index (CI) of the original article considering the number of articles produced on the topic per year and the number of citations to other authors (intermediaries). We observed that 60% of the papers and 59% of the intermediaries do not refer to the original article. The CI was low and negatively affected by the age of the original article, showing that the frequency of citation was lower than the rate by which articles on the topic were published. There is a tendency to cite more recent articles and articles that corroborate their own findings. Our data illustrated the microwave effect, in which pre-existing ideas and theories are “reheated” by more recent articles where little of the original idea is modified. The microwave effect can create the impression of scientific advancement when there is little being added to the knowledge already produced.

     

Exploiting Structural Similarities to Classify Citations

Citations play an important role in the scientific community by assisting in measuring multifarious policies like the impact of journals, researchers, institutions, and countries. Authors cite papers for different reasons, such as extending previous work, comparing their study with the state-of-the-art, providing background of the field, etc. In recent years, researchers have tried to conceptualize all citations into two broad categories, important and incidental. Such a categorization is very important to enhance scientific output in multiple ways, for instance, (1) Helping a researcher in identifying meaningful citations from a list of 100 to 1000 citations (2) Enhancing the impact factor calculation mechanism by more strongly weighting important citations, and (3) Improving researcher, institutional, and university rankings by only considering important citations. All of these uses depend upon correctly identifying the important citations from the list of all citations in a paper. To date, researchers have utilized many features to classify citations into these broad categories: cue phrases, in-text citation counts, and metadata features, etc. However, contemporary approaches are based on identification of in-text citation counts, mapping sections onto the Introduction, Methods, Results, and Discussion (IMRAD) structure, identifying cue phrases, etc. Identifying such features accurately is a challenging task and is normally conducted manually, with the accuracy of citation classification demonstrated in terms of these manually extracted features. This research proposes to examine the content of the cited and citing pair to identify important citing papers for each cited paper. This content similarity approach was adopted from research paper recommendation approaches. Furthermore, a novel section-based content similarity approach is also proposed. The results show that solely using the abstract of the cited and citing papers can achieve similar accuracy as the state-of-the-art approaches. This makes the proposed approach a viable technique that does not depend on manual identification of complex features.     

A quantitative analysis of determinants of non-citation using a panel data model

The term “non-citation factor” refers to the percentage of never-cited papers in a citation time window, a common phenomenon in the science world. Some scholars have qualitatively explored the reasons for not citing a publication, and quantitatively analyzed the mathematical functional relations between the “non-citation factor” and “impact factor of a journal.” This study simultaneously considers the mutual relations and closeness degree between the “non-citation factor” and different influencing factors from a novel perspective—that of using a more structuralized panel data model. The analysis revealed that the determinants, including “impact factor of journal,” “age of journal,” “average number of references per paper in journal,” and “issues of journal,” exerted an extremely small but positive influence (<?0.025) on the decline of “percentage of never-cited papers in the citation time window of publication year or 3 years.” That means the improvement of these determinants can decrease the percentage of never-cited papers. The “impact factor of the journal” always had the biggest positive influence, while the “average number of references per paper in journal” always had the smallest positive influence. In wider citation time windows of 3 or 6 years, two determinants—“number of publications in journal” and “amount of interdisciplinarity in journal”—began to exert a negative effect with a positive correlation coefficient on the decline of the “non-citation factor.” That means the improvement of these two determinants cannot decrease the value of the “non-citation factor,” even though they can increase its value. It is worth noting that the “impact factor of the journal” had a positive influence on the decline of the percentages of never-cited papers in the citation time window of publication year or 3 years, and began to play a negative role in the decline of percentage of never-cited papers in the citation time window of 6 years. Finally, three variables—“average number of authors per paper in journal,” “average number of references per paper in journal,” and “issues of journal”—no longer exerted an influence on the decline of percentages of never-cited papers in the citation time window of 6 years, while “age of journal” and “average number of pages per paper in journal” still made a positive contribution. Our findings could help research institutions, researchers, editors, and publishers understand the positively or negatively influential factors that lead to non-citation, thus improving the chance of papers being cited and having some academic influence.     

Institution bias in the New England Journal of Medicine? A bibliometric analysis of publications (1997–2016)

There are three main reasons for retraction: (1) ethical misconduct (e.g. duplicate publication, plagiarism, missing credit, no IRB, ownership issues, authorship issues, interference in the review process, citation manipulation); (2) scientific distortion (e.g. data manipulation, fraudulent data, unsupported conclusions, questionable data validity, non-replicability, data errors—even if unintended); (3) administrative error (e.g. article published in wrong issue, not the final version published, publisher errors). The first category, although highly deplorable has almost no effect on the advancement of science, the third category is relatively minor. The papers belonging to the second category are most troublesome from the scientific point of view, as they are misleading and have serious negative implications not only on science but also on society. In this paper, we explore some temporal characteristics of retracted articles, including time of publication, years to retract, growth of post retraction citations over time and social media attention by the three major categories. The data set comprises 995 retracted articles retrieved in October 2014 from Elsevier’s ScienceDirect. Citations and Mendeley reader counts were retrieved four times within 4 years, which allowed us to examine post-retraction longitudinal trends not only for citations, but also for Mendeley reader counts. The major findings are that both citation counts and Mendeley reader counts continue to grow after retraction.     

Citations to the “Introduction to informetrics” indexed by WOS,Scopus and Google Scholar

Google Scholar and Scopus are recent rivals to Web of Science. In this paper we examined these three citation databases through the citations of the book “Introduction to informetrics” by Leo Egghe and Ronald Rousseau. Scopus citations are comparable to Web of Science citations when limiting the citation period to 1996 and onwards (the citation coverage of Scopus)—each database covered about 90% of the citations located by the other. Google Scholar missed about 30% of the citations covered by Scopus and Web of Science (90 citations), but another 108 citations located by Google Scholar were not covered either by Scopus or by Web of Science. Google Scholar performed considerably better than reported in previous studies, however Google Scholar is not very “user-friendly” as a bibliometric data collection tool at this point in time. Such “microscopic” analysis of the citing documents retrieved by each of the citation databases allows us a deeper understanding of the similarities and the differences between the databases.     

A deep-learning based citation count prediction model with paper metadata semantic features

Predicting the impact of academic papers can help scholars quickly identify the high-quality papers in the field. How to develop efficient predictive model for evaluating potential papers has attracted increasing attention in academia. Many studies have shown that early citations contribute to improving the performance of predicting the long-term impact of a paper. Besides early citations, some bibliometric features and altmetric features have also been explored for predicting the impact of academic papers. Furthermore, paper metadata text such as title, abstract and keyword contains valuable information which has effect on its citation count. However, present studies ignore the semantic information contained in the metadata text. In this paper, we propose a novel citation prediction model based on paper metadata text to predict the long-term citation count, and the core of our model is to obtain the semantic information from the metadata text. We use deep learning techniques to encode the metadata text, and then further extract high-level semantic features for learning the citation prediction task. We also integrate early citations for improving the prediction performance of the model. We show that our proposed model outperforms the state-of-the-art models in predicting the long-term citation count of the papers, and metadata semantic features are effective for improving the accuracy of the citation prediction models.

     

Citation curves of “all-elements-sleeping-beauties”: “flash in the pan” first and then “delayed recognition”

“Delayed recognition” refers to the phenomenon where papers did not achieve recognition in terms of citations until some years after their original publication. A paper with delayed recognition was termed a “sleeping beauty”: a princess sleeps (goes unnoticed) for a long time and then, almost suddenly, is awakened (receives a lot of citations) by a prince (another article). There are a sleeping period and an awakening period in the definition of a “sleeping beauty”. Apart from and prior to the two periods, an awaking period was found in citation curves of some publications, “sleeping beauties” was hence expanded to “all-elements-sleeping-beauties”. The opposite effect of “delayed recognition” was described as “flash in the pan”: documents that were noticed immediately after publication but did not seem to have a lasting impact. In this work, we briefly discussed the citation curves of two remarkable “all-elements-sleeping-beauties”. We found they appeared “flash in the pan” first and then “delayed recognition”. We also found happy endings of sleeping beauties and princes, and hence suggest the citation curve of an “all-elements-sleeping-beauty” include an awaking period, a sleeping period, an awakening period and a happy ending.     

The citation advantage of foreign language references for Chinese social science papers

Contemporary scientific exchanges are international, yet language continues to be a persistent barrier to scientific communication, particularly for non-native English-speaking scholars. Since the ability to absorb knowledge has a strong impact on how researchers create new scientific knowledge, comprehensive access to and understanding of both domestic and international scientific publications is essential for scientific performance. This study explores the effect of absorbed knowledge on research impact by analyzing the relationship between the language diversity of cited references and the number of citations received by the citing paper. Chinese social sciences are taken as the research object, and the data, 950,302 papers published between 1998 and 2013 with 8,151,327 cited references, were collected from the Chinese Social Sciences Citation Index. Results show that there is a stark increase in the consumption of foreign language material within the Chinese social science community, and English material accounts for the vast majority of this consumption. Papers with foreign language references receive significantly more citations than those without, and the citation advantage of these internationalized work holds when we control for characteristics of the citing papers, such as the discipline, prestige of journal, prestige of institution, and scientific collaboration. However, the citation advantage has decreased from 1998 to 2008, largely as an artifact of the increased number of papers citing foreign language material. After 2008, however, the decline of the citation advantage subsided and became relatively stable, which suggests that incorporating foreign language literature continues to increase scientific impact, even as the scientific community itself becomes increasingly international. However, internationalization is not without concerns: the work closes with a discussion of the potential problems associated with the lack of linguistic diversity in scientific communication.

     

Early citation counts correlate with accumulated impact

Summary The present paper addresses the objective of developing forward indicators of research performance using bibliometric information on the UK science base. Most research indicators rely primarily on historical time series relating to inputs to, activity within and outputs from the research system. Policy makers wish to be able to monitor changing research profiles in a more timely fashion, the better to determine where new investment is having the greatest effect. Initial (e.g. 12 months from publication) citation counts might be useful as a forward indicator of the long-term (e.g. 10 years from publication) quality of research publications, but - although there is literature on citation-time functions - no study to evaluate this specifically has been carried out by Thomson ISI or any other analysts. Here, I describe the outcomes of a preliminary study to explore these citation relationships, drawing on the UK National Citation Report held by Evidence Ltd under licence from Thomson ISI for OST policy use. Annual citation counts typically peak at around the third year after publication. I show that there is a statistically highly significant correlation between initial (years 1-2) and later (years 3-10) citations in six research categories across the life and physical sciences. The relationship holds over a wide range of initial citation counts. Papers that attract more than a definable but field dependent threshold of citations in the initial period after publication are usually among the top 1&percnt; (the most highly cited papers) for their field and year. Some papers may take off slowly but can later join the high impact group. It is important to recognise that the statistical relationship is applicable to groups of publications. The citation profiles of individual articles may be quite different. Nonetheless, it seems reasonable to conclude that leading indicators of research excellence could be developed. This initial study should now be extended across a wider range fields to test the initial outcomes: earlier papers suggest the model holds in economics. Additional statistical tests should be applied to explore and model the relationship between initial, later and total citation counts and thus to create a general tool for policy application.     

Citation analysis of Ph.D. theses with data from Scopus and Google Books

This study investigates the potential of citation analysis of Ph.D. theses to obtain valid and useful early career performance indicators at the level of university departments. For German theses from 1996 to 2018 the suitability of citation data from Scopus and Google Books is studied and found to be sufficient to obtain quantitative estimates of early career researchers’ performance at departmental level in terms of scientific recognition and use of their dissertations as reflected in citations. Scopus and Google Books citations complement each other and have little overlap. Individual theses’ citation counts are much higher for those awarded a dissertation award than others. Departmental level estimates of citation impact agree reasonably well with panel committee peer review ratings of early career researcher support.

     

Trends in the core literature on tropical medicine: a bibliometric analysis from 1952-2002

Summary We present empirical data on frequency and pattern of misprints in citations to twelve high-profile papers. We find that the distribution of misprints, ranked by frequency of their repetition, follows Zipf&apos;s law. We propose a stochastic model of citation process, which explains these findings, and leads to the conclusion that about 70-90&percnt; of scientific citations are copied from the lists of references used in other papers.     

Is citation analysis a legitimate evaluation tool?

A comprehensive discussion on the use of citation analysis to rate scientific performance and the controversy surrounding it. The general adverse criticism that citation counts include an excessive number of negative citations (citations to incorrect results worthy of attack), self-citations (citations to the works of the citing authors), and citations to methodological papers is analyzed. Included are a discussion of measurement problems such as counting citations for multiauthored papers, distinguishing between more than one person with the same last name (homographs), and what it is that citation analysis actually measures. It is concluded that as the scientific enterprise becomes larger and more complex, and its role in society more critical, it will become more difficult, expensive and necessary to evaluate and identify the largest contributors. When properly used, citation analysis can introduce a useful measure of objectivity into the evaluation process at relatively low financial cost.Modification of a chapter in E. Garfield:Citation Indexing: Its Theory and Application in Science, Technology and the Humanities, New York, Wiley, 1979.     

The use and misuse of citation analysis in research evaluation

The present paper addresses some of the many possible uses of citations, including bookmark, intellectual heritage, impact tracker, and self-serving purposes. The main focus is on the applicability of citation analysis as an impact or quality measure. If a paper's bibliography is viewed as consisting of a directed (research impact or quality) component related to intellectual heritage and random components related to specific self-interest topics, then for large numbers of citations from many different citing paper, the most significant intellectual heritage (research impact or quality) citations will aggregate and the random author-specific self-serving citations will be scattered and not accumulate. However, there are at least two limitations to this model of citation analysis for stand-alone use as a measure of research impact of quality. First, the reference to intellectual heritage could be positive or negative. Second, there could be systemic biases which affect the aggregate results, and one of these, the “Pied Piper Effect”, is described in detail. Finally, the results of a short citation study comparing Russian and American papers in different technical fields are presented. The questions raised in interpreting this data highlight a few of the difficulties in attempting to interpret citation results without supplementary information. Leydesdorff (Leydesdorff, 1998) addresses the history of citations and citation analysis, and the transformation of a reference mechanism into a purportedly quantitive measure of research impact/quality. The present paper examines different facets of citations and citation analysis, and discusses the validity of citation analysis as a useful measure of research impact/quality.     

Early Mendeley readers correlate with later citation counts

Counts of the number of readers registered in the social reference manager Mendeley have been proposed as an early impact indicator for journal articles. Although previous research has shown that Mendeley reader counts for articles tend to have a strong positive correlation with synchronous citation counts after a few years, no previous studies have compared early Mendeley reader counts with later citation counts. In response, this first diachronic analysis compares reader counts within a month of publication with citation counts after 20 months for ten fields. There are moderate or strong correlations in eight out of ten fields, with the two exceptions being the smallest categories (n?=?18, 36) with wide confidence intervals. The correlations are higher than the correlations between later citations and early citations, showing that Mendeley reader counts are more useful early impact indicators than citation counts.     

A multi-view method of scientific paper classification via heterogeneous graph embeddings

The classification task of scientific papers can be implemented based on contents or citations. In order to improve the performance on this task, we express papers as nodes and integrate scientific papers’ contents and citations into a heterogeneous graph. It has two types of edges. One type represents the semantic similarity between papers, derived from papers’ titles and abstracts. The other type represents the citation relationship between papers and the journals or proceedings of conferences of their references. We utilize a contrastive learning method to embed the nodes in the heterogeneous graph into a vector space. Then, we feed the paper node vectors into classifiers, such as the decision tree, multilayer perceptron, and so on. We conduct experiments on three datasets of scientific papers: the Microsoft Academic Graph with 63,211 scientific papers in 20 classes, the Proceedings of the National Academy of Sciences with 38,243 scientific papers in 18 classes, and the American Physical Society with 443,845 scientific papers in 5 classes. The experimental results on the multi-class task show that our multi-view method scores the classification accuracy up to 98%, outperforming state-of-the-arts.

     

Positive results receive more citations, but only in some disciplines

Negative results are commonly assumed to attract fewer readers and citations, which would explain why journals in most disciplines tend to publish too many positive and statistically significant findings. This study verified this assumption by counting the citation frequencies of papers that, having declared to “test” a hypothesis, reported a “positive” (full or partial) or a “negative” (null or negative) support. Controlling for various confounders, positive results were cited on average 32 % more often. The citation advantage, however, was unequally distributed across disciplines (classified as in the Essential Science Indicators database). Using Space Science as the reference category, the citation differential was positive and formally statistically significant only in Neuroscience & Behaviour, Molecular Biology & Genetics, Clinical Medicine, and Plant and Animal Science. Overall, the effect was significantly higher amongst applied disciplines, and in the biological compared to the physical and the social sciences. The citation differential was not a significant predictor of the actual frequency of positive results amongst the 20 broad disciplines considered. Although future studies should attempt more fine-grained assessments, these results suggest that publication bias may have different causes and require different solutions depending on the field considered.     

Usage metrics vs classical metrics: analysis of Russia’s research output

This paper discusses the results of a pilot project investigating Russian scholarly publications using the altmetric indicators “Usage Count Last 180 days” (U1) and “Usage Count Since 2013” (U2) introduced by Web of Science. We explored the relationship between citation impact and both types of usage counts. The data set consisted of 37,281 records (publications) indexed by SCI-E in 2015. Seven broad research areas were selected to observe citation patterns and usage counts. A significant difference was found between mean citations and mean usage counts (U2) in a few research areas. We discovered a significant Kendall rank correlation between the citation metrics and usage metrics at the article level. This correlation is particularly strong for the longer period usage metric (U2). We also analyzed the relationship between usage metrics and traditional journal-level citation metrics. Very weak correlation was observed.