Usage metrics vs classical metrics: analysis of Russia’s research output

This paper discusses the results of a pilot project investigating Russian scholarly publications using the altmetric indicators “Usage Count Last 180 days” (U1) and “Usage Count Since 2013” (U2) introduced by Web of Science. We explored the relationship between citation impact and both types of usage counts. The data set consisted of 37,281 records (publications) indexed by SCI-E in 2015. Seven broad research areas were selected to observe citation patterns and usage counts. A significant difference was found between mean citations and mean usage counts (U2) in a few research areas. We discovered a significant Kendall rank correlation between the citation metrics and usage metrics at the article level. This correlation is particularly strong for the longer period usage metric (U2). We also analyzed the relationship between usage metrics and traditional journal-level citation metrics. Very weak correlation was observed.     

Understanding the impact change of a highly cited article: a content-based citation analysis

Researchers tend to cite highly cited articles, but how these highly cited articles influence the citing articles has been underexplored. This paper investigates how one highly cited essay, Hirsch’s “h-index” article (H-article) published in 2005, has been cited by other articles. Content-based citation analysis is applied to trace the dynamics of the article’s impact changes from 2006 to 2014. The findings confirm that citation context captures the changing impact of the H-article over time in several ways. In the first two years, average citation mention of H-article increased, yet continued to decline with fluctuation until 2014. In contrast with citation mention, average citation count stayed the same. The distribution of citation location over time also indicates three phases of the H-article “Discussion,” “Reputation,” and “Adoption” we propose in this study. Based on their locations in the citing articles and their roles in different periods, topics of citation context shifted gradually when an increasing number of other articles were co-mentioned with the H-article in the same sentences. These outcomes show that the impact of the H-article manifests in various ways within the content of these citing articles that continued to shift in nine years, data that is not captured by traditional means of citation analysis that do not weigh citation impacts over time.     

Trends in and contributions to entrepreneurship research: a broad review of literature from 1996 to June 2012

This article, which began as an effort to gauge trends in and contributions to the broad field of “entrepreneur/entrepreneurship,” reviews 5,476 academic articles on entrepreneurship that were published in 522 Social Sciences Citation Index and Science Citation Index journals from 1996 to June 2012. This survey identifies keywords and conducts a review to search for and identify related articles in the Institute for Scientific Information Web of Science database. We then present our findings, including the number of publications by year, categorization of article types, main academic journals, authors, and most-cited articles. The citation counts for authors, journals, and articles are also analyzed. This study indicates that the number of articles related to the keyword entrepreneur increased from 1996 to the end of 2011, which is a sign of an upward trend in the influence of entrepreneurs. Entrepreneur research fascinated numerous scholars during the study period covering 16.5 years. In particular, researchers from the USA, England, Canada, Germany, and the Netherlands have made the most contributions to this field. This literature review provides evidence that the concept of entrepreneur attracted academic researchers, resulting in significant contributions to the field of entrepreneur research.     

A quantitative analysis of Antarctic related articles in humanities and social sciences appearing in the world core journals

To demonstrate the importance and the actual research situation of Antarctic studies in the humanities and social sciences, we collected data from the SSCI and A&HCI covering a period of over 100 years and focused on the number of articles published each year, major journals, types of document, authors and their countries publishing the most articles, collaboration, the major research subjects covered, and citations. Comparisons were also made with the Arctic studies to show some similarities and differences. The results suggest that the research in the fields of humanities and social sciences has been in the long-run developing without interruption over 100 years. With regard to the number of articles in high-capacity journals, Geographical Journal performs best, followed by the Petermanns Geographische Mitteilungen and Scottish Geographical Magazine. The documentation is rather scattered without a strong cohesion, while book review and article are the two most common types of document. There haven’t many stable collaborated teams on Antarctic topics. Joyner, Savours, and Beck are the three authors having the highest number of publications. USA is the most active country while the most active research institute is University of Tasmania in Australia. The Antarctic expedition has been the main theme lasted for centuries. In addition, the research in the fields of humanities and social sciences has generated a lot of high-impact articles, among which the article entitled “Chemical concentrations of pollutant lead aerosols, terrestrial dusts and sea salts in Greenland and Antarctic snow strata” enjoys the highest citation counts.     

Benchmarking Google Scholar with the New Zealand PBRF research assessment exercise

Google Scholar was used to generate citation counts to the web-based research output of New Zealand Universities. Total citations and hits from Google Scholar correlated with the research output as measured by the official New Zealand Performance-Based Research Fund (PBRF) exercise. The article discusses the use of Google Scholar as a cybermetric tool and methodology issues in obtaining citation counts for institutions. Google Scholar is compared with other tools that provide web citation data: Web of Science, SCOPUS, and the Wolverhampton Cybermetric Crawler.     

Selecting scientific excellence through committee peer review - A citation analysis of publications previously published to approval or rejection of post-doctoral research fellowship applicants

Summary We investigated committee peer review for awarding long-term fellowships to post-doctoral researchers as practiced by the Boehringer Ingelheim Fonds (B.I.F.) - a foundation for the promotion of basic research in biomedicine. Assessing the validity of selection decisions requires a generally accepted criterion for research impact. A widely used approach is to use citation counts as a proxy for the impact of scientific research. Therefore, a citation analysis for articles published previous to the applicants' approval or rejection for a B.I.F. fellowship was conducted. Based on our model estimation (negative binomial regression model), journal articles that had been published by applicants approved for a fellowship award (n = 64) prior to applying for the B.I.F. fellowship award can be expected to have 37% (straight counts of citations) and 49% (complete counts of citations) more citations than articles that had been published by rejected applicants (n = 333). Furthermore, comparison with international scientific reference values revealed (a) that articles published by successful and non-successful applicants are cited considerably more often than the “average” publication and (b) that excellent research performance can be expected more of successful than non-successful applicants. The findings confirm that the foundation is not only achieving its goal of selecting the best junior scientists for fellowship awards, but also successfully attracting highly talented young scientists to apply for B.I.F. fellowships.     

The expansion of Google Scholar versus Web of Science: a longitudinal study

Web of Science (WoS) and Google Scholar (GS) are prominent citation services with distinct indexing mechanisms. Comprehensive knowledge about the growth patterns of these two citation services is lacking. We analyzed the development of citation counts in WoS and GS for two classic articles and 56 articles from diverse research fields, making a distinction between retroactive growth (i.e., the relative difference between citation counts up to mid-2005 measured in mid-2005 and citation counts up to mid-2005 measured in April 2013) and actual growth (i.e., the relative difference between citation counts up to mid-2005 measured in April 2013 and citation counts up to April 2013 measured in April 2013). One of the classic articles was used for a citation-by-citation analysis. Results showed that GS has substantially grown in a retroactive manner (median of 170 % across articles), especially for articles that initially had low citations counts in GS as compared to WoS. Retroactive growth of WoS was small, with a median of 2 % across articles. Actual growth percentages were moderately higher for GS than for WoS (medians of 54 vs. 41 %). The citation-by-citation analysis showed that the percentage of citations being unique in WoS was lower for more recent citations (6.8 % for citations from 1995 and later vs. 41 % for citations from before 1995), whereas the opposite was noted for GS (57 vs. 33 %). It is concluded that, since its inception, GS has shown substantial expansion, and that the majority of recent works indexed in WoS are now also retrievable via GS. A discussion is provided on quantity versus quality of citations, threats for WoS, weaknesses of GS, and implications for literature research and research evaluation.     

A bibliometric analysis of NOAA’s Office of Ocean Exploration and Research

Bibliometric analysis techniques are increasingly being used to analyze and evaluate scientific research produced by institutions and grant funding agencies. This article uses bibliometric methods to analyze journal articles funded by NOAA’s Office of Ocean Exploration and Research (OER), an extramural grant-funding agency focused on the scientific exploration of the world’s oceans. OER-supported articles in this analysis were identified through grant reports, personal communication, and acknowledgement of OER support or grant numbers. The articles identified were analyzed to determine the number of publications and citations received per year, subject, and institution. The productivity and citation impact of institutions in the US receiving OER grant funding were mapped geographically. Word co-occurrence and bibliographic coupling networks were created and visualized to identify the research topics of OER-supported articles. Finally, article citation counts were evaluated by means of percentile ranks. This article demonstrates that bibliometric analysis can be useful for summarizing and evaluating the research performance of a grant funding agency.     

Methodological quality in clinical trials and bibliometric indicators: no evidence of correlations

Citation frequencies and journal impact factors (JIFs) are being used more and more to assess the quality of research and allocate research resources. If these bibliometric indicators are not an adequate predictor of research quality, there could be severe negative consequences for research. To analyse to which extent citation frequencies and journal impact factors correlate with the methodological quality of clinical research articles included in an SBU systematic review of antibiotic prophylaxis in surgery. All 212 eligible original articles were extracted from the SBU systematic review “Antibiotic Prophylaxis in Surgery” and categorized according to their methodological rigourness as high, moderate or low quality articles. Median of citation frequencies and JIFs were compared between the methodological quality groups using Kruskal–Wallis non-parametric test. An in-depth study of low-quality studies with higher citation frequencies/JIFs was also conducted. No significant differences were found in median citation frequencies (p = 0.453) or JIFs (p = 0.185) between the three quality groups. Studies that had high citation frequencies/JIFs but were assessed as low-quality lacked control groups, had high dropout rates or low internal validity. This study of antibiotic prophylaxis in surgery does not support the hypothesis that bibliometric indicators are a valid instrument for assessing methodological quality in clinical trials. This is a worrying observation, since bibliometric indicators have a major influence on research funding. However, further studies in other areas are needed.     

Why was this cited? Explainable machine learning applied to COVID-19 research literature

Multiple studies have investigated bibliometric factors predictive of the citation count a research article will receive. In this article, we go beyond bibliometric data by using a range of machine learning techniques to find patterns predictive of citation count using both article content and available metadata. As the input collection, we use the CORD-19 corpus containing research articles—mostly from biology and medicine—applicable to the COVID-19 crisis. Our study employs a combination of state-of-the-art machine learning techniques for text understanding, including embeddings-based language model BERT, several systems for detection and semantic expansion of entities: ConceptNet, Pubtator and ScispaCy. To interpret the resulting models, we use several explanation algorithms: random forest feature importance, LIME, and Shapley values. We compare the performance and comprehensibility of models obtained by “black-box” machine learning algorithms (neural networks and random forests) with models built with rule learning (CORELS, CBA), which are intrinsically explainable. Multiple rules were discovered, which referred to biomedical entities of potential interest. Of the rules with the highest lift measure, several rules pointed to dipeptidyl peptidase4 (DPP4), a known MERS-CoV receptor and a critical determinant of camel to human transmission of the camel coronavirus (MERS-CoV). Some other interesting patterns related to the type of animal investigated were found. Articles referring to bats and camels tend to draw citations, while articles referring to most other animal species related to coronavirus are lowly cited. Bat coronavirus is the only other virus from a non-human species in the betaB clade along with the SARS-CoV and SARS-CoV-2 viruses. MERS-CoV is in a sister betaC clade, also close to human SARS coronaviruses. Thus both species linked to high citation counts harbor coronaviruses which are more phylogenetically similar to human SARS viruses. On the other hand, feline (FIPV, FCOV) and canine coronaviruses (CCOV) are in the alpha coronavirus clade and more distant from the betaB clade with human SARS viruses. Other results include detection of apparent citation bias favouring authors with western sounding names. Equal performance of TF-IDF weights and binary word incidence matrix was observed, with the latter resulting in better interpretability. The best predictive performance was obtained with a “black-box” method—neural network. The rule-based models led to most insights, especially when coupled with text representation using semantic entity detection methods. Follow-up work should focus on the analysis of citation patterns in the context of phylogenetic trees, as well on patterns referring to DPP4, which is currently considered as a SARS-Cov-2 therapeutic target.

     

An empirical investigation of the influence of collaboration in Finance on article impact

We investigate the impact of collaborative research in academic Finance literature to find out whether and to what extent collaboration leads to higher impact articles (6,667 articles across 2001–2007 extracted from the Web of Science). Using the top 5 % as ranked by the 4-year citation counts following publication, we also follow related secondary research questions such as the relationships between article impact and author impact; collaboration and average author impact of an article; and, the nature of geographic collaboration. Key findings indicate: collaboration does lead to articles of higher impact but there is no significant marginal value for collaboration beyond three authors; high impact articles are not monopolized by high impact authors; collaboration and the average author impact of high-impact articles are positively associated, where collaborative articles have a higher mean author impact in comparison to single-author articles; and collaboration among the authors of high impact articles is mostly cross-institutional.     

Evaluating scientific impact of publications: combining citation polarity and purpose

Citation counts are commonly used to evaluate the scientific impact of a publication on the general premise that more citations probably mean more endorsements. However, two questionable assumptions underpin this idea: a) that all authors contributed equally to the paper; and b) that the endorsement is positive. Obviously, neither of these assumptions hold true. Hence, with this study, we examine two components of citations—their purpose, i.e., the reason for the citation, and polarity, being the author’s attitude toward the cited work. Our findings provide a new perspective on the scientific impact of highly-cited publications. Our methodology consists of three steps. Firstly, a pre-trained model composed of a Word2Vec—a well-known word embedding approach—and a convolutional neural network (CNN) is used to identify citation polarity and purpose. Secondly, in a set of highly-cited papers, we compare eight categories of purpose from foundational to critical and three categories of polarity: positive, negative, and neutral. We further explore how different types of papers—those discussing discoveries or those discussing utilitarian topics—influence the evaluation of scientific impact of papers. Finally, we mine and discover the knowledge (e.g. method, concept, tool or data) to explain the actual scientific impact of a highly-cited paper. To demonstrate how combining citation polarity with purpose can provide far greater details of a paper’s scientific impact, we undertake a case study with 370 highly-cited journal articles spanning “Biochemistry & Molecular Biology” and “Genetics & Heredity”. The results yield valuable insights into the assumption about citation counts as a metric for evaluating scientific impact.

     

Current use of effect size or confidence interval analyses in clinical and biomedical research

The isolated use of the statistical hypothesis testing for two group comparison has limitations, and its combination with effect size or confidence interval analysis as complementary statistical tests is recommended. In the present work, we estimate the use of these complementary statistical tests (i.e. effect size or confidence interval) in recently published in research articles in clinical and biomedical areas. Methods: The ProQuest database was used to search published studies in academic journals between 2019 and 2020. The analysis was carried out using terms that represent five areas of clinical and biomedical research: “brain”, “liver”, “heart”, “dental”, and “covid-19”. A total of 119,558 published articles were retrieved. Results: The relative use of complementary statistical tests in clinical and biomedical publications was low. The highest frequency usage of complementary statistical tests was among articles that also used statistical hypothesis testing for two-sample comparison. Publications with the term “covid-19” showed the lowest usage rate of complementary statistical tests when all article were analyzed but presented the highest rate among articles that used hypothesis testing. Conclusion: The low use of effect size or confidence interval in two-sample comparison suggests that coordinate measures should be taken in order to increase the use of this analysis in clinical and biomedical research. Their use should be emphasized in statistical disciplines for college and graduate students, become a routine procedure in research laboratories, and recommended by reviewers and editors of scientific journals.

     

Prediction and application of article potential citations based on nonlinear citation-forecasting combined model

As the number of academic articles rapidly increases, a reasonable evaluation method for the articles is highly required in the current academic research. Meanwhile, a faster access to the high-quality academic articles for the researchers is also of critical significance. This paper first improves the AVG model and presents a new Nonlinear Citation-Forecasting Combined Model (NCFCM) based on a neural network to predict the potential increase of citation counts. Then, the NCFCM is used to analyze and rank the academic articles in online databases. The results of NCFCM model are compared to the results from other existing methods. Empirical analysis and comparisons demonstrate that the NCFCM model is of high accuracy and robustness in forecasting potential citation counts and ranking academic articles. Ranking academic articles according to the potentional citation counts can help researchers retrieve the desired articles efficiently in a short time.

     

Article length and citation outcomes

Scientometrics - We examine the relationship between article length and citation counts. Focusing on articles published between 2010 and 2014 in the top five economics journals and their citation...     

The story behind Oncotarget? A bibliometric analysis

Being the most proliferative journal of oncology a cancer research of the past decade, the Open Access journal Oncotarget had reached more than 20,000 publications and a relatively high impact factor score in the past years. In 2018, the journal citation report decided to withdraw the status of an impact factor journal. Since there was a large discussion in the scientific community and specific reasons for the withdrawal were not stated, this bibliometric analysis was performed to assess if Oncotarget exhibits any differences in its bibliometric structure compared to other journals. For this purpose, we used the “New Quality and Quantity Indices in Sciences” platform and analyzed 20,000 Oncotarget articles. Density equalizing mapping technique helps to construct maps of cancer research in Oncotarget and shows that it has led to a unique global landscape which is not asymmetrically dominated by the Western hemisphere but exhibits a publishing architecture with a pronounced emphasis on Chinese articles.

     

Early Mendeley readers correlate with later citation counts

Counts of the number of readers registered in the social reference manager Mendeley have been proposed as an early impact indicator for journal articles. Although previous research has shown that Mendeley reader counts for articles tend to have a strong positive correlation with synchronous citation counts after a few years, no previous studies have compared early Mendeley reader counts with later citation counts. In response, this first diachronic analysis compares reader counts within a month of publication with citation counts after 20 months for ten fields. There are moderate or strong correlations in eight out of ten fields, with the two exceptions being the smallest categories (n?=?18, 36) with wide confidence intervals. The correlations are higher than the correlations between later citations and early citations, showing that Mendeley reader counts are more useful early impact indicators than citation counts.     

Response to Dr. Copiello’s comments on “The impact of video abstract on citation counts”

Scientometrics - This letter is a response to Dr. Copiello’s comments on “The impact of video abstract on citation counts”. Citation counts (with self-citations and without...     

Characteristics and impact of grant-funded research: a case study of the library and information science field

This paper reports on a bibliometric study of the characteristics and impact of research in the library and information science (LIS) field which was funded through research grant programs, and compares it with research that received no extra funding. Seven core LIS journals were examined to identify articles published in 1998 that acknowledge research grant funding. The distribution of these articles by various criteria (e.g., topic, affiliation, funding agency) was determined. Their impact as indicated by citation counts during 1998–2008 was evaluated against that of articles without acknowledging extra funding and published in the same journals in the same year using citation data collected from Scopus’ Citation Tracker. The impact of grant-funded research as measured by citation counts was substantially higher than that of other research, both overall and in each journal individually. Scholars from outside LIS core institutions contributed heavily to grant-funded research. The two highest-impact publications by far reported non-grant-based research, and grant-based funding of research reported in core LIS journals was biased towards the information retrieval (IR) area, particularly towards research on IR systems. The percentage of articles reporting grant-funded research was substantially higher in information-oriented journals than in library-focused ones.     

Differences between journals and years in the proportions of students,researchers and faculty registering Mendeley articles

This article contains two investigations into Mendeley reader counts with the same dataset. Mendeley reader counts provide evidence of early scholarly impact for journal articles, but reflect the reading of a relatively young subset of all researchers. To investigate whether this age bias is constant or varies by narrow field and publication year, this article compares the proportions of student, researcher and faculty readers for articles published 1996–2016 in 36 large monodisciplinary journals. In these journals, undergraduates recorded the newest research and faculty the oldest, with large differences between journals. The existence of substantial differences in the composition of readers between related fields points to the need for caution when using Mendeley readers as substitutes for citations for broad fields. The second investigation shows, with the same data, that there are substantial differences between narrow fields in the time taken for Scopus citations to be as numerous as Mendeley readers. Thus, even narrow field differences can impact on the relative value of Mendeley compared to citation counts.