Citation time window choice for research impact evaluation

This paper aims to inform choice of citation time window for research evaluation, by answering three questions: (1) How accurate is it to use citation counts in short time windows to approximate total citations? (2) How does citation ageing vary by research fields, document types, publication months, and total citations? (3) Can field normalization improve the accuracy of using short citation time windows? We investigate the 31-year life time non-self-citation processes of all Thomson Reuters Web of Science journal papers published in 1980. The correlation between non-self-citation counts in each time window and total non-self-citations in all 31 years is calculated, and it is lower for more highly cited papers than less highly cited ones. There are significant differences in citation ageing between different research fields, document types, total citation counts, and publication months. However, the within group differences are more striking; many papers in the slowest ageing field may still age faster than many papers in the fastest ageing field. Furthermore, field normalization cannot improve the accuracy of using short citation time windows. Implications and recommendations for choosing adequate citation time windows are discussed.     

Validation of bibliometric indicators in the field of mathematics

Bibliometric analyses of scientific publications provide quantitative information that enables evaluators to obtain a useful picture of a team's research visibility. In combination with peer judgements and other qualitative background knowledge, these analyses can serve as a basis for discussions about research performance quality. However, many mathematicians are not convinced that citation counts do in fact provide useful information in the field of mathematics. According to these mathematicians, citation and publication habits differ completely from scholarly fields such as chemistry or physics. Therefore, it is impossible to derive valid information regarding research performance from citation counts. The aim of this study is to obtain more insight into the significance of citation-based indicators in the field of mathematics. To which extent do citation-scores mirror to the opinions of experts concerning the quality of a paper or a journal? A survey was conducted to answer this question.Top journals, as qualified by experts, receive significantly higher citation rates thangood journals. Thesegood journals, in turn, have significantly higher scores than journals with the qualificationless good. Top publications, recorded in the ISI database, receive on the average 15 times more citations than the mean score within the field of mathematics as a whole. In conclusion, the experts' views on top publications or top journals correspond very well to bibliometric indicators based on citation counts.     

Citation frequency: A biased measure of research impact significantly influenced by the geographical origin of research articles

Context. The use of citation frequency and impact factor as measures of research quality and journal prestige is being criticized. Citation frequency is augmented by self-citation and for most journals the majority of citations originate from a minority of papers. We hypothesized that citation frequency is also associated with the geographical origin of the research publication. Objective. We determined whether citations originate more frequently from institutes that are located in the same country as the authors of the cited publication than would be expected by chance. Design. We screened citations referring to 1200 cardiovascular publications in the 7 years following their publication. For the 1200 citation recipient publications we documented the country where the research originated (9 countries/regions) and the total number of received citations. For a selection of 8864 citation donor papers we registered the country/region where the citing paper originated. Results. Self-citation was common in cardiovascular journals (n = 1534, 17.8%). After exclusion of self-citation, however, the number of citations that originated from the same country as the author of the citation recipient was found to be on average 31.6% higher than would be expected by chance (p<0.01 for all countries/regions). In absolute numbers, nation oriented citation bias was most pronounced in the USA, the country with the largest research output (p<0.001). Conclusion. Citation frequency was significantly augmented by nation oriented citation bias. This nation oriented citation behaviour seems to mainly influence the cumulative citation number for papers originating from the countries with a larger research output.     

Scientometrics of laser research in India during 1970-1994

An analysis of 952 publications published by Indian scientists and abstracted by Journal of Current Laser Abstracts during 1970-1994 indicates that laser research in India picked up during 1978-1994 and reached its peak in 1980. The Indian output in the field of laser research forms an integral part of the mainstream science as reflected by the pattern of publications and their citations in the international literature. Laser research performed in India improved considerably during 1985-1994 as compared to 1970-1984 as seen by different impact indicators such as citation per paper, proportion of high quality papers, and publication effective index. The publication output is concentrated among few institutions and there is a similarity in the activity and attractively profile of the highly productive institutions. India"s citation rate per paper for highly productive authors is at par with the world citation rate per paper. The study indicates that the proportion of mega authored papers increased during 1990-1994 and the international collaboration is mainly with the USA.     

Using content-based and bibliometric features for machine learning models to predict citation counts in the biomedical literature

The most popular method for judging the impact of biomedical articles is citation count which is the number of citations received. The most significant limitation of citation count is that it cannot evaluate articles at the time of publication since citations accumulate over time. This work presents computer models that accurately predict citation counts of biomedical publications within a deep horizon of 10 years using only predictive information available at publication time. Our experiments show that it is indeed feasible to accurately predict future citation counts with a mixture of content-based and bibliometric features using machine learning methods. The models pave the way for practical prediction of the long-term impact of publication, and their statistical analysis provides greater insight into citation behavior.     

Do types of collaboration change citation? Collaboration and citation patterns of South African science publications

Bibliographic records are extensively used in the study of citations. Based on ISI data, this paper examines citation patterns of the publications of South African scientists in recent years. In particular, the focus of this paper is on citations as to the collaborative dimensions of South African scientists in their publications. The study reveals that the number of citations received by a publication varies not only according to the collaboration but also to the types of collaboration of the authors who are involved in its production. Furthermore, it emerges that the impact of citations on publications differs from discipline to discipline, and affiliating sector to sector, regardless of collaboration.     

How well developed are altmetrics? A cross-disciplinary analysis of the presence of ‘alternative metrics’ in scientific publications

In this paper an analysis of the presence and possibilities of altmetrics for bibliometric and performance analysis is carried out. Using the web based tool Impact Story, we collected metrics for 20,000 random publications from the Web of Science. We studied both the presence and distribution of altmetrics in the set of publications, across fields, document types and over publication years, as well as the extent to which altmetrics correlate with citation indicators. The main result of the study is that the altmetrics source that provides the most metrics is Mendeley, with metrics on readerships for 62.6 % of all the publications studied, other sources only provide marginal information. In terms of relation with citations, a moderate spearman correlation (r = 0.49) has been found between Mendeley readership counts and citation indicators. Other possibilities and limitations of these indicators are discussed and future research lines are outlined.     

Publication types,citation rates and evaluation

In order to resolve questions frequently raised in the context of research evaluation about the citation rates of journal publications in relation to other types of publications, the total research output of substantial institutions or systems has to be brought under bibliographic control. That precondition has rarely been met: there are few published studies of the total range of publications of major research institutions, including books, book chapters, technical reports and published conference proceedings. The Research Evaluation and Policy Project (REPP) at the Australian National University (ANU) has established a database covering all the publications from the Institute of Advanced Studies (IAS), a fulltime research institution at the ANU, and has examined in detail citations in the journal literature accruing to all types of publications. The database contains a significant number of publications, nearly 30 000 items, and covers the sciences and the social sciences and humanities. This data enables us to examine whether the citation record of research publications appearing in journals indexed by the Institute for Scientific Information (ISI) is a useable surrogate for the citation record within ISI journals of all model of publication. We contend that, if certain preconditions are met, the choice of citation rate is not critical.     

A study on the citation situation within the citing paper: citation distribution of references according to mention frequency

Citation impact indicators play a significant role in evaluating the scientific research activity. Most of citation impact indicators are based on the citation count that the publication is cited as a reference in the other publications, but the difference between each citation situation was not considered. Normally, the number of citations that a publication is cited in the other publications may represent the formal quality of the publication. Similarly, the number of times that a publication is really mentioned within the citing publication, it may also represent the formal quality of the citation. We have examined about how many times each reference was really mentioned within the citing publications and studied about the citation situation within the citing publications. We verified that the citation distribution of references according to the mention frequency follows the Generalized Pareto distribution. The results showed that about 20% of total references were mentioned three and more times, and the number of citation mentions for the about 50% of total references were from about 20% of the total references in the citing publications.     

Extending citation analysis to non-source items

Summary This paper reports the first results of the extension of citation analysis to 'non-source' items, which is one strand of an extensive study of quantitative performance indicators used in the assessment of research. It would be presumptuous to draw firm conclusions from this first foray into the realm of non-source citations, however our analysis is based on an extensive experimental database of over 30,000 publications, so the results can be viewed as strong pointers to possible generalised outcomes. We show that it is possible to mine ISI databases for references to a comprehensive oeuvre of items from whole institutions. Many types of publications are visible in the ISI data - books, book chapters, journals not indexed by ISI, and some conference publications. When applied to the assessment of university departments, they can have a significant effect on rankings, though this does not follow in all cases. The investment of time, effort, and money in a significantly extended analysis will not be equally beneficial in all fields. However, a considerable amount of testing is required to confirm our initial results.     

Publication activity, citation impact and bi-directional links between publications and patents in biotechnology

The study focuses on publication activity, citation impact and citation links between publications and patents in biotechnology. The European Union (EU), US, Japan and China are the most important global players. However, the landscape is changing since the EU and the US are losing ground because of challenges from a group of emerging economies. National profiles differ between the two groups of main players and upcoming countries; the focus on red biotechnology in the US and Europe is contrasted by propensity for white and green technology in Asia. Furthermore, the subject profile of biotechnology papers citing patents and cited by patents as well as the relationship between patent citations and citation impact in scientific literature is explored. Papers that cite patents tend to reflect propensity towards white biotechnology while patent-cited publications have a higher relative share in red biotechnology. No significant difference concerning the citation impact of publications ??citing patents?? and ??not citing patents?? can be found. This is contrasted by the observation that patent-cited papers perform distinctly better in terms of standard bibliometric indicators than comparable publications that are not linked to technology in this direction.     

Growing trend of China’s contribution to tissue engineering

This study examines China’s performance on tissue engineering using scientometrics measures such as China’s global publication share, rank, growth rate and citation impact, its publications in various sub-fields, top journals in terms of national share based on last 5 years (2008–2012) publications data obtained from ISI Science citation index expanded database. We have also determined Chinese share with international collaborative papers at the national level, as well as h-core papers and high-cited papers, etc.     

A macro study of self-citation

This study investigates the role of self-citation in the scientific production of Norway (1981-1996). More than 45,000 publications have been analysed. Using a three-year citation window we find that 36% of all citations represent author self-citations. However, this percentage is decreasing when citations are traced for longer periods. We find the highest share of self-citation among the least cited papers. There is a strong positive correlation between the number of self-citations and the number of authors of the publications. Still, only a minor part of the overall increase in citation rates that can be found for multi-authored papers is due to self-citations. Also, the share of self-citation shows significant variations among different scientific disciplines. The results are relevant for the discussion concerning use of citation indicators in research assessments. This revised version was published online in August 2006 with corrections to the Cover Date.     

The citation impact of social sciences and humanities upon patentable technology

This paper examines the citation impact of papers published in scientific-scholarly journals upon patentable technology, as reflected in examiner- or inventor-given references in granted patents. It analyses data created by SCImago Research Group, linking PATSTAT’s scientific non-patent references (SNPRs) to source documents indexed in Scopus. The frequency of patent citations to journal papers is calculated per discipline, year, institutional sector, journal subject category, and for “top” journals. PATSTAT/Scopus-based statistics are compared to those derived from Web of Science/USPTO linkage. A detailed assessment is presented of the technological impact of research publications in social sciences and humanities (SSH). Several subject fields perform well in terms of the number of citations from patents, especially Library and Information Science, Language and Linguistics, Education, and Law, but many of the most cited journals find themselves in the interface between SSH and biomedical or natural sciences. Analyses of the titles of citing patents and cited papers are presented that shed light upon the cognitive content of patent citations. It is proposed to develop more advanced indicators of citation impact of papers upon patents, and ways to combine citation counts with citation content and context analysis.

     

Usage metrics vs classical metrics: analysis of Russia’s research output

This paper discusses the results of a pilot project investigating Russian scholarly publications using the altmetric indicators “Usage Count Last 180 days” (U1) and “Usage Count Since 2013” (U2) introduced by Web of Science. We explored the relationship between citation impact and both types of usage counts. The data set consisted of 37,281 records (publications) indexed by SCI-E in 2015. Seven broad research areas were selected to observe citation patterns and usage counts. A significant difference was found between mean citations and mean usage counts (U2) in a few research areas. We discovered a significant Kendall rank correlation between the citation metrics and usage metrics at the article level. This correlation is particularly strong for the longer period usage metric (U2). We also analyzed the relationship between usage metrics and traditional journal-level citation metrics. Very weak correlation was observed.     

Early Mendeley readers correlate with later citation counts

Counts of the number of readers registered in the social reference manager Mendeley have been proposed as an early impact indicator for journal articles. Although previous research has shown that Mendeley reader counts for articles tend to have a strong positive correlation with synchronous citation counts after a few years, no previous studies have compared early Mendeley reader counts with later citation counts. In response, this first diachronic analysis compares reader counts within a month of publication with citation counts after 20 months for ten fields. There are moderate or strong correlations in eight out of ten fields, with the two exceptions being the smallest categories (n?=?18, 36) with wide confidence intervals. The correlations are higher than the correlations between later citations and early citations, showing that Mendeley reader counts are more useful early impact indicators than citation counts.     

Research performance of Turkish astronomers in the period of 1980–2010

We investigated the development of astronomy and astrophysics research productivity in Turkey in terms of publication output and their impacts as reflected in the science citation index for the period 1980–2010. Our study involves 838 refereed publications, including 801 articles, 16 letters, 15 reviews, and six research notes. The number of papers were prominently increased after 2000 and the average number of papers per researcher is calculated as 0.89. Total number of received citations for 838 papers is 6938, while number of citations per papers is approximately 8.3 in 30 years. Publication performance of Turkish astronomers and astrophysicists was compared with those of seven countries that have similar gross domestic expenditures on research and development, and members of organization for economic co-operation and development. Our study reveals that the output of astronomy and astrophysics research in Turkey has gradually increased over the years.     

Do citations and readership identify seminal publications?

This work presents a new approach for analysing the ability of existing research metrics to identify research which has strongly influenced future developments. More specifically, we focus on the ability of citation counts and Mendeley reader counts to distinguish between publications regarded as seminal and publications regarded as literature reviews by field experts. The main motivation behind our research is to gain a better understanding of whether and how well the existing research metrics relate to research quality. For this experiment we have created a new dataset which we call TrueImpactDataset and which contains two types of publications, seminal papers and literature reviews. Using the dataset, we conduct a set of experiments to study how citation and reader counts perform in distinguishing these publication types, following the intuition that causing a change in a field signifies research quality. Our research shows that citation counts work better than a random baseline (by a margin of 10%) in distinguishing important seminal research papers from literature reviews while Mendeley reader counts do not work better than the baseline.     

Sleeping beauties in Computer Science: characterization and early identification

Scientometrics - While a large majority of scientific publications get most of their citations within the initial few years after publication, there is an interesting number of papers—termed...     

A bibliometric analysis of publications by the School of Biological Sciences, Seoul National University, South Korea

This study presents an in-depth survey of research and citation performance of the School of Biological Sciences (SBS) 39-member faculty at Seoul National University (SNU), the most prestigious university in South Korea, for the years 2004–2009. Thirty-nine faculty members published a total of 640 publications during the period, representing an average of 16.4 publications per scientist. Among the 640 publications, 521 (81.4 %) were cited 9,204 times, an average of 14.4 citations per publication. More publications co-authored by the SBS faculty with foreign researchers (mostly from the U.S.A.) were published in mainstream journals than publications by three other co-authorship types. Accordingly, publications by international co-authorships received more citations compared to citation levels of three other co-authorship types in terms of the average citations per publication. The study has found a concentration effect, whereby quite a small number of publications received approximately one-third of the citation performance generated by the SBS faculty at SNU. The results demonstrate that the citation performance of the SBS at SNU can be influenced considerably by the presence and productivity of ‘star’ scientists.