Citation frequency: A biased measure of research impact significantly influenced by the geographical origin of research articles

Context. The use of citation frequency and impact factor as measures of research quality and journal prestige is being criticized. Citation frequency is augmented by self-citation and for most journals the majority of citations originate from a minority of papers. We hypothesized that citation frequency is also associated with the geographical origin of the research publication. Objective. We determined whether citations originate more frequently from institutes that are located in the same country as the authors of the cited publication than would be expected by chance. Design. We screened citations referring to 1200 cardiovascular publications in the 7 years following their publication. For the 1200 citation recipient publications we documented the country where the research originated (9 countries/regions) and the total number of received citations. For a selection of 8864 citation donor papers we registered the country/region where the citing paper originated. Results. Self-citation was common in cardiovascular journals (n = 1534, 17.8%). After exclusion of self-citation, however, the number of citations that originated from the same country as the author of the citation recipient was found to be on average 31.6% higher than would be expected by chance (p<0.01 for all countries/regions). In absolute numbers, nation oriented citation bias was most pronounced in the USA, the country with the largest research output (p<0.001). Conclusion. Citation frequency was significantly augmented by nation oriented citation bias. This nation oriented citation behaviour seems to mainly influence the cumulative citation number for papers originating from the countries with a larger research output.     

Can applied science be ‘good science’? Exploring the relationship between patent citations and citation impact in nanoscience

There is a rich literature on how science and technology are related to each other. Patent citation analysis is amongst the most frequently used to tool to track the strengths of links. In this paper we explore the relationship between patent citations and citation impact in nanoscience. Our observations indicate that patent-cited papers perform better in terms of standard bibliometric indicators than comparable publications that are not linked to technology in this way. More specifically, we found that articles cited in patents are more likely to be cited also by other papers. The share of highly cited papers is the most striking result. Instead of the average of 4% of all papers, 13.8% of the papers cited once or twice in patents fall into this category and even 23.5% of the papers more frequently cited in patents receive citation rates far above the standard. Our analyses further demonstrate the presence and the relevance of bandwagon effects driving the development of science and technology.     

Citation life cycles of ten citation classics

Not all highly cited papers have the same citation life cycle curves, i.e. curves of frequency of citations received vs. time. The citation life of ten randomly selected Citation Classics, five in medicine and five in biochemistry, are studied longitudinally in time and compared with a random sample of ten non-classics of the same cohort. There are pronounced differences in the life cycle curves; two distinct types are suggested. Type A, comprised of both high and low cited papers in both fields, has an early peak of citation rate and may be approximated by a bilinear cumulative citation curve with a break at six years of age, when three quarters of the total number of citations have occurred. Type B, in this study comprised of extremely highly cited methodological Citation Classics, exhibit a constant or slowly accelerating growth rate with a vigorous citation life extending over the entire period studied and typically one third or less of the total citations accumulated at six years of age.     

How much do different ways of calculating percentiles influence the derived performance indicators? A case study

Bibliometric indicators can be determined by comparing specific citation records with the percentiles of a reference set. However, there exists an ambiguity in the computation of percentiles because usually a significant number of papers with the same citation count are found at the border between percentile rank classes. The present case study of the citations to the journal Europhysics Letters (EPL) in comparison with all physics papers from the Web of Science shows the deviations which occur due to the different ways of treating the tied papers in the evaluation of the percentage of highly cited publications. A strong bias can occur, if the papers tied at the threshold number of citations are all considered as highly cited or all considered as not highly cited.     

A quasi-quantitative citation model

On the basis of investigating author's opinion on citing motivations of chemistry papers aquasi-quantitative model for citing is suggested. The model selects professional and nonprofessional motivations of citing and introduces thecitation threshold concept which tries to characterize the effect of citing motivations quantitatively. Possible reasons for missing citations are also treated. Mean ages of real and of self-citations were calculated by subtracting the average of the publication years of cited papers from the publication year of the citing publication. The difference between the mean ages may characterize thesynchronity of the author's research in comparison with those working on similar topics. The paper introduces thecitation strategy indicator which relates impact factors of cited periodicals with the mean impact factor of periodicals in the corresponding research subfield.     

The citation impact of social sciences and humanities upon patentable technology

This paper examines the citation impact of papers published in scientific-scholarly journals upon patentable technology, as reflected in examiner- or inventor-given references in granted patents. It analyses data created by SCImago Research Group, linking PATSTAT’s scientific non-patent references (SNPRs) to source documents indexed in Scopus. The frequency of patent citations to journal papers is calculated per discipline, year, institutional sector, journal subject category, and for “top” journals. PATSTAT/Scopus-based statistics are compared to those derived from Web of Science/USPTO linkage. A detailed assessment is presented of the technological impact of research publications in social sciences and humanities (SSH). Several subject fields perform well in terms of the number of citations from patents, especially Library and Information Science, Language and Linguistics, Education, and Law, but many of the most cited journals find themselves in the interface between SSH and biomedical or natural sciences. Analyses of the titles of citing patents and cited papers are presented that shed light upon the cognitive content of patent citations. It is proposed to develop more advanced indicators of citation impact of papers upon patents, and ways to combine citation counts with citation content and context analysis.

     

Citation time window choice for research impact evaluation

This paper aims to inform choice of citation time window for research evaluation, by answering three questions: (1) How accurate is it to use citation counts in short time windows to approximate total citations? (2) How does citation ageing vary by research fields, document types, publication months, and total citations? (3) Can field normalization improve the accuracy of using short citation time windows? We investigate the 31-year life time non-self-citation processes of all Thomson Reuters Web of Science journal papers published in 1980. The correlation between non-self-citation counts in each time window and total non-self-citations in all 31 years is calculated, and it is lower for more highly cited papers than less highly cited ones. There are significant differences in citation ageing between different research fields, document types, total citation counts, and publication months. However, the within group differences are more striking; many papers in the slowest ageing field may still age faster than many papers in the fastest ageing field. Furthermore, field normalization cannot improve the accuracy of using short citation time windows. Implications and recommendations for choosing adequate citation time windows are discussed.     

Patents cited in the scientific literature: An exploratory study of 'reverse' citation relations

This paper reports on a new approach to study the linkage between science and technology. Unlike most contributions to this area we do not trace citations of scientific literature in patents but explore citations of patents in scientific literature. Our analysis is based on papers recorded in the 1996-2000 annual volumes of the CD-Edition of Science Citation Index (SCI) of the Institute for Scientific Information (ISI) and patent data provided by the US Patent and Trademark Office. Almost 30,000 US patents were cited by scientific research papers. We analysed the citation links by scientific fields and technological sectors. Chemistry-related subfields tended to cite patents more than other scientific area. Among technological sectors, chemical clearly dominates followed by drugs and medical patents as the most frequently cited categories. Further analyses included a country-ranking based on inventor-addresses of the cited patents, a more detailed inspection of the ten most cited patents, and an analysis of class-field transfers. The paper concludes with the suggestions for future research. One of them is to compare our 'reverse' citation data with 'regular' patent citation data within the same classification system to see whether citations occur, irrespectively of their directionality, in the same fields of science and technology. Another question is as to how one should interpret reverse citation linkages.     

The use of deciles of the citation impact to evaluate different fields of research in Israel

One often uses the average citation impact factor in order to perform international comparisons between the levels of scientific performance within given disciplines. In averaging over all (or all cited) papers one may gives undue weight to papers with few citations while, in fact, the standing of a country within a given field would be better assessed by looking only at the “successful” papers in that discipline. The present papers suggests that one should do so by averaging citations only over the ten (or twenty) percent of the most cited papers in a discipline and use these in order to establish a ranking between countries. The case of Israel is used as an illustration of this approach.     

The highly cited papers of professors as an indicator of a research group's scientific performance

In the first part of the paper the citations in 1986 and 1987 of 3938 papers published in 1985 by 324 research groups in the faculties of science and of medicine of eight universities in the Netherlands are analyzed. Because of the large statistical spread of (1) the number of short-term citations of papers cited equally frequently over a long period, and (2) the number of citations over a long period of papers by the same author, short-term citation scores appear to be an unreliable indicator of a research group's contribution to science. In the second part of the paper an alternative approach is presented, based on a subdivision of the 3938 papers in papers authored by professors with 0–2, 3–8, or 9 highly cited papers (HCPs, 25 citations) to their name. Very large citation score differences were found for the three categories. For example: for papers first-authored by a professor, the average number of citations per person in 1986 and 1987 for 1985 papers was for 161 professors with 9 HCPs a factor 14 larger than for 575 professors with only 0–2 HCPs; for papers co-authored by professors, this factor was 6.6. These findings justify the conclusion that the number of HCPs scored by the professors (and other senior scientists) during their entire career is a much more reliable predictor of the performance of a research group than the number of short-term citations of the articles published by the group within a short period. A research group's contribution to science is primarily determined by the individual scientifictalents of its members.     

Visualizing the context of citations referencing papers published by Eugene Garfield: a new type of keyword co-occurrence analysis

During Eugene Garfield’s (EG’s) lengthy career as information scientist, he published about 1500 papers. In this study, we use the impressive oeuvre of EG to introduce a new type of bibliometric networks: keyword co-occurrences networks based on the context of citations, which are referenced in a certain paper set (here: the papers published by EG). The citation context is defined by the words which are located around a specific citation. We retrieved the citation context from Microsoft Academic. To interpret and compare the results of the new network type, we generated two further networks: co-occurrence networks which are based on title and abstract keywords from (1) EG’s papers and (2) the papers citing EG’s publications. The comparison of the three networks suggests that papers of EG and citation contexts of papers citing EG are semantically more closely related to each other than to titles and abstracts of papers citing EG. This result accords with the use of citations in research evaluation that is based on the premise that citations reflect the cognitive influence of the cited on the citing publication.     

Sustained impact of publications of C.V. Raman

C.V. Raman is being acknowledged by worldwide physics community for his classic works. The present study has made an effort to analyze how much impact in number of citation receiving for his publications. Of course, there was a lack of tools for such a study some years back. The study has limited to the database Science Citation Index for the period 1982–2005. The noteworthy results are: One third of his research papers have been cited at least once; The research papers published during 1918–1940 could make remarkable impact; Three of his papers have shown an upward growth in number of citations receiving; The total citations to papers of age 46 and 54 as on the year 1982 accounted for more than 50 per cent of the total citations received; Research works in the ‘Acoustics’ area have been cited more than any other area of his works; Eponymal citations are to be explored and analysed to understand the real impact of his works.     

Using ISI's 'Highly Cited Researchers' to obtain a country level indicator of citation excellence

Summary A high level of citation to an author's work is, in general, a testimony to the fact that the author's work has been noted and used by his peers. High citation is seen to be correlated with other forms of recognition and rewards, and is a key indicator of research performance, among other bibliometric indicators. The Institute for Scientific Information (ISI) defines a 'highly cited researcher' (HCR) as one of 250 most cited authors of journal papers in any discipline. Citation data for 20 years (1981-1999) is used to calculate the share of HCRs for countries in 21 subject areas. We find that the US dominates in all subject areas (US share ~ 40-90%). Based on the number of highly cited researchers in a country, an index of citation excellence is proposed. We find that rank order of countries based on this index is in conformity with our general understanding of research excellence, whereas the more frequently used indicator, citations per paper, gave an unacceptable rank order due to an inherent bias toward very small countries. Additionally, a high value of the index of citation excellence was found to be associated with higher concentration of highly cited researchers in affiliating organizations.     

Exploiting Structural Similarities to Classify Citations

Citations play an important role in the scientific community by assisting in measuring multifarious policies like the impact of journals, researchers, institutions, and countries. Authors cite papers for different reasons, such as extending previous work, comparing their study with the state-of-the-art, providing background of the field, etc. In recent years, researchers have tried to conceptualize all citations into two broad categories, important and incidental. Such a categorization is very important to enhance scientific output in multiple ways, for instance, (1) Helping a researcher in identifying meaningful citations from a list of 100 to 1000 citations (2) Enhancing the impact factor calculation mechanism by more strongly weighting important citations, and (3) Improving researcher, institutional, and university rankings by only considering important citations. All of these uses depend upon correctly identifying the important citations from the list of all citations in a paper. To date, researchers have utilized many features to classify citations into these broad categories: cue phrases, in-text citation counts, and metadata features, etc. However, contemporary approaches are based on identification of in-text citation counts, mapping sections onto the Introduction, Methods, Results, and Discussion (IMRAD) structure, identifying cue phrases, etc. Identifying such features accurately is a challenging task and is normally conducted manually, with the accuracy of citation classification demonstrated in terms of these manually extracted features. This research proposes to examine the content of the cited and citing pair to identify important citing papers for each cited paper. This content similarity approach was adopted from research paper recommendation approaches. Furthermore, a novel section-based content similarity approach is also proposed. The results show that solely using the abstract of the cited and citing papers can achieve similar accuracy as the state-of-the-art approaches. This makes the proposed approach a viable technique that does not depend on manual identification of complex features.     

The motivation for citing specific references by social scientists in Korea: The phenomenon of co-existing references

This study, based on the premise that references are a social product that reflects the social environment of a society, is an attempt to explore the co-existence of Korean and non-Korean literature in the references to Korean papers. 321 authors (papers) who published in 43 issues of 24 Korean journals focused on the social sciences were surveyed about their research channels and citation motivations, and the 11,358 references in the papers were analyzed. The findings were as follows : (1) The extent of the co-existence was that non-Korean literature was cited 1.9 times (65.3%) more often than Korean literature; (2) Research channel was the most common non-Korean channel orientation (55.8%); (3) The motivation for citations was significantly dependent on whether the literature cited was Korean or non-Korean. Non-Korean literature was chiefly cited for conceptual (20.7%), perfunctory (16.0%), and persuasive (15.1%) motivations; (4) The citations and citation motivations behind non-Korean literature were significantly influenced by research channel, discipline, focus of research, publishing career, and type of paper. Of these variables, research channel was frequently related to the citation of non-Korean literature. Finally, this study is very suggestive on two counts: (1) Citation motivation might constitute a new approach for exploring the production of knowledge by researchers. (2) This study has demonstrated, in particular, an empirical relationship between knowledge produced by Korean social scientists and non-Korean knowledge through the analysis of citation motivation.     

A macro study of self-citation

This study investigates the role of self-citation in the scientific production of Norway (1981-1996). More than 45,000 publications have been analysed. Using a three-year citation window we find that 36% of all citations represent author self-citations. However, this percentage is decreasing when citations are traced for longer periods. We find the highest share of self-citation among the least cited papers. There is a strong positive correlation between the number of self-citations and the number of authors of the publications. Still, only a minor part of the overall increase in citation rates that can be found for multi-authored papers is due to self-citations. Also, the share of self-citation shows significant variations among different scientific disciplines. The results are relevant for the discussion concerning use of citation indicators in research assessments. This revised version was published online in August 2006 with corrections to the Cover Date.     

The relevance of short communication in scholarly journals: An empirical study

This study aims to examine whether rapid communications exert more influence/impact on subsequent research. Citation analysis of Short Communications (SCs) and Main Articles (MAs) from 1983 and 1990 for 5 high impact biomedical journals was carried out for a five year period following publication. The mean citations cumulated for the five year period showed no consistent trend. Some journals showed more citations for SCs while some showed more for MAs. The mean citations (range) for SCs and MAs for the 1983 and 1990 papers respectively were as follows:Gene: 14.13 (0-61) and 38.79 (0-677), 9.73 (0-93) and 13.17 (0-44);Journal of Clinical Investigation (JCI): 79.77 (3-202) and 27.52 (0-86), 50.52 (0-254) and 33.53 (0-151);Journal of Experimental Medicine (JEM): 39.80 (0-200) and 49.20 (0-403), 47.26 (0-258) and 50.27 (0-173); andJournal of Biological Chemistry (JBC); 36.21 (0-380) and 19.67 (0-53), 37.19 (0-273) and 26.84 (0-185). SCs ofJournal of Cell Biology (JCB) had a mean citation of 25.84 per article with a range of 0-98, while the MAs had a mean citation of 33.13 with the range 4-122 during 1983-87. The citation peak was seen about three years after publication for all the journals during both the periods. The mean cumulative citations showed a progressive increase over the five years for both types of papers, in all journals and for both the 5 year periods. The initial differences observed persisted even four years after the year of publication. No significant differences were observed in the distribution of the cumulative 5 year citations between the SCs and MAs. An index of speed of citation per article showed no substantial differences between SCs and MAs with MAs showing an edge over SCs. Both MAs and SCs of all the journals showed nearly same average time per citation per article further confirming that the SCs do not enjoy the advantage of speedier citation. The results show that the generally perceived feeling of SCs getting cited more frequently and faster does not appear to be valid. Hence, the practice of publishing SCs on a priority basis is perhaps not warranted.     

Is citation analysis a legitimate evaluation tool?

A comprehensive discussion on the use of citation analysis to rate scientific performance and the controversy surrounding it. The general adverse criticism that citation counts include an excessive number of negative citations (citations to incorrect results worthy of attack), self-citations (citations to the works of the citing authors), and citations to methodological papers is analyzed. Included are a discussion of measurement problems such as counting citations for multiauthored papers, distinguishing between more than one person with the same last name (homographs), and what it is that citation analysis actually measures. It is concluded that as the scientific enterprise becomes larger and more complex, and its role in society more critical, it will become more difficult, expensive and necessary to evaluate and identify the largest contributors. When properly used, citation analysis can introduce a useful measure of objectivity into the evaluation process at relatively low financial cost.Modification of a chapter in E. Garfield:Citation Indexing: Its Theory and Application in Science, Technology and the Humanities, New York, Wiley, 1979.     

A bibliometric analysis of physics publications in Korea, 1994-1998

This study examined research performance of Korean physicists, comparing Korean-authoredpapers versus internationally co-authored papers, indexed in SCI, 1994-1998, and using thenumber of citations received by internationally co-authored papers covered by the SCI CD-ROM.For the study, 4,665 papers published from the researchers affiliated with the physics departmentsor physics-associated laboratories at Korean universities and indexed by SCI were analyzed.Korean authored papers tended to be published in Korean, Japanese, and UK journals, whileinternationally co-authored papers were more likely to appear in German, Dutch, and Swissjournals. Among the 18 authorship countries (on the basis of first author), 93 internationally co-authored papers by U.S. researchers had the highest citation rate, an average 15.9 citations perpaper. Of the eight countries that published over 5 papers, there was no correlation between theaverage number of citations per paper and the total number of citations. However, an ANOVAindicated a significant difference between the average number of citations per paper according tocountry (F = 5.84, p < 0.0005). In other words, papers by the U.S. and French researchers tendedto be cited more frequently than papers by the Italian, Japanese, Korean, Russian, and Germanresearchers.