On the validity of citation counting in science evaluation: Content analyses of references and citations in psychological publications

In reference to the increasing significance of citation counting in evaluations of scientists and science institutes as well as in science historiography, it is analyzed empirically what is cited in which frequency and what types of citations in scientific texts are used. Content analyses refer to numbers of references, self-references, publication language of references cited, publication types of references cited, and type of citation within the texts. Validity of citation counting is empirically analyzed with reference to random samples of English and German journal articles as well as German textbooks, encyclopedias, and test-manuals from psychology. Results show that 25% of all citations are perfunctory, more than 50% of references are journal articles and up to 40% are books and book-chapters, 10% are self-references. Differences between publications from various psychological sub-disciplines, publication languages, and types of publication are weak. Thus, validity of evaluative citation counting is limited because at least one quarter refers to perfunctory citations exhibiting a very low information utility level and by the fact that existing citation-databases refer to journal articles only.     

Publication activity, citation impact and bi-directional links between publications and patents in biotechnology

The study focuses on publication activity, citation impact and citation links between publications and patents in biotechnology. The European Union (EU), US, Japan and China are the most important global players. However, the landscape is changing since the EU and the US are losing ground because of challenges from a group of emerging economies. National profiles differ between the two groups of main players and upcoming countries; the focus on red biotechnology in the US and Europe is contrasted by propensity for white and green technology in Asia. Furthermore, the subject profile of biotechnology papers citing patents and cited by patents as well as the relationship between patent citations and citation impact in scientific literature is explored. Papers that cite patents tend to reflect propensity towards white biotechnology while patent-cited publications have a higher relative share in red biotechnology. No significant difference concerning the citation impact of publications ??citing patents?? and ??not citing patents?? can be found. This is contrasted by the observation that patent-cited papers perform distinctly better in terms of standard bibliometric indicators than comparable publications that are not linked to technology in this direction.     

A quasi-quantitative citation model

On the basis of investigating author's opinion on citing motivations of chemistry papers aquasi-quantitative model for citing is suggested. The model selects professional and nonprofessional motivations of citing and introduces thecitation threshold concept which tries to characterize the effect of citing motivations quantitatively. Possible reasons for missing citations are also treated. Mean ages of real and of self-citations were calculated by subtracting the average of the publication years of cited papers from the publication year of the citing publication. The difference between the mean ages may characterize thesynchronity of the author's research in comparison with those working on similar topics. The paper introduces thecitation strategy indicator which relates impact factors of cited periodicals with the mean impact factor of periodicals in the corresponding research subfield.     

The citation impact of social sciences and humanities upon patentable technology

This paper examines the citation impact of papers published in scientific-scholarly journals upon patentable technology, as reflected in examiner- or inventor-given references in granted patents. It analyses data created by SCImago Research Group, linking PATSTAT’s scientific non-patent references (SNPRs) to source documents indexed in Scopus. The frequency of patent citations to journal papers is calculated per discipline, year, institutional sector, journal subject category, and for “top” journals. PATSTAT/Scopus-based statistics are compared to those derived from Web of Science/USPTO linkage. A detailed assessment is presented of the technological impact of research publications in social sciences and humanities (SSH). Several subject fields perform well in terms of the number of citations from patents, especially Library and Information Science, Language and Linguistics, Education, and Law, but many of the most cited journals find themselves in the interface between SSH and biomedical or natural sciences. Analyses of the titles of citing patents and cited papers are presented that shed light upon the cognitive content of patent citations. It is proposed to develop more advanced indicators of citation impact of papers upon patents, and ways to combine citation counts with citation content and context analysis.

     

Renewable energy research 1995–2009: a case study of wind power research in EU, Spain, Germany and Denmark

The paper reports the developments and citation patterns over three time periods of research on Renewable Energy generation and Wind Power 1995–2011 in EU, Spain, Germany and Denmark. Analyses are based on Web of Science and incorporate journal articles as well as conference proceeding papers. Scientometric indicators include publication collaboration ratios, top-player distribution as well as citedness and correspondence analyses of citing publications, relative citation impact, distributions of top-cited as well as top-citing institutions and publication sources and cluster analysis of citing title terms to map knowledge export areas. Findings show an increase in citation impact for Renewable Energy and Wind Power research albeit hampered by scarcely cited conference papers. Although EU maintains its global top position in producing Renewable Energy and Wind Power research the developments of EU and German world shares as well as citation impact are negative during the most recent 7 year period. During the same time the citation impact of Spain and Denmark increase and place both nations among the top-ranking countries in Wind Power research. Spain is the only EU country that increases its world production share from 2000. China is currently ranked three after EU and USA in research output, however with a very low citation impact. Spain, Denmark and Germany each demonstrates distinct collaboration patterns and publication source and citation distribution profiles. More than half the citations to EU Wind Power research are EU-self citations. An expected intensified EU collaboration in the Wind Energy field does not come about. The most productive research institutions in Denmark and Spain are also the most cited ones.     

Universality of performance indicators based on citation and reference counts

We find evidence for the universality of two relative bibliometric indicators of the quality of individual scientific publications taken from different data sets. One of these is a new index that considers both citation and reference counts. We demonstrate this universality for relatively well cited publications from a single institute, grouped by year of publication and by faculty or by department. We show similar behaviour in publications submitted to the arXiv e-print archive, grouped by year of submission and by sub-archive. We also find that for reasonably well cited papers this distribution is well fitted by a lognormal with a variance of around ??²?=?1.3 which is consistent with the results of Radicchi et al. (Proc Natl Acad Sci USA 105:17268?C17272, 2008). Our work demonstrates that comparisons can be made between publications from different disciplines and publication dates, regardless of their citation count and without expensive access to the whole world-wide citation graph. Further, it shows that averages of the logarithm of such relative bibliometric indices deal with the issue of long tails and avoid the need for statistics based on lengthy ranking procedures.     

The citation advantage of foreign language references for Chinese social science papers

Contemporary scientific exchanges are international, yet language continues to be a persistent barrier to scientific communication, particularly for non-native English-speaking scholars. Since the ability to absorb knowledge has a strong impact on how researchers create new scientific knowledge, comprehensive access to and understanding of both domestic and international scientific publications is essential for scientific performance. This study explores the effect of absorbed knowledge on research impact by analyzing the relationship between the language diversity of cited references and the number of citations received by the citing paper. Chinese social sciences are taken as the research object, and the data, 950,302 papers published between 1998 and 2013 with 8,151,327 cited references, were collected from the Chinese Social Sciences Citation Index. Results show that there is a stark increase in the consumption of foreign language material within the Chinese social science community, and English material accounts for the vast majority of this consumption. Papers with foreign language references receive significantly more citations than those without, and the citation advantage of these internationalized work holds when we control for characteristics of the citing papers, such as the discipline, prestige of journal, prestige of institution, and scientific collaboration. However, the citation advantage has decreased from 1998 to 2008, largely as an artifact of the increased number of papers citing foreign language material. After 2008, however, the decline of the citation advantage subsided and became relatively stable, which suggests that incorporating foreign language literature continues to increase scientific impact, even as the scientific community itself becomes increasingly international. However, internationalization is not without concerns: the work closes with a discussion of the potential problems associated with the lack of linguistic diversity in scientific communication.

     

Identification of referencing and citation processes of scientific journals based on the citation distribution model

In this article, we firstly analyze the referencing process and the citation process of a scientific journal in theory, and find that the observed referencing or citation process includes the diffusing process and the aging process of cited literature and the publishing process of citing literature, thereby it is illuminated why the identified average publication delay ( [`(T)] = T_s + t ) \left( {\bar{T} = T_{s} + \tau } \right) was longer than the observed value. Secondly, we compare the transfer function model of the observed citing process with other classical citation distribution models and find that the model is superior to others. Finally, using the model, we identify parameters of actual referencing and citation processes from data of age distributions of references and citations of 38 journals of neurology and applied mathematics in JCR, respectively; and then compare differences of identified parameters and obtain some interesting conclusions.     

Understanding the impact change of a highly cited article: a content-based citation analysis

Researchers tend to cite highly cited articles, but how these highly cited articles influence the citing articles has been underexplored. This paper investigates how one highly cited essay, Hirsch’s “h-index” article (H-article) published in 2005, has been cited by other articles. Content-based citation analysis is applied to trace the dynamics of the article’s impact changes from 2006 to 2014. The findings confirm that citation context captures the changing impact of the H-article over time in several ways. In the first two years, average citation mention of H-article increased, yet continued to decline with fluctuation until 2014. In contrast with citation mention, average citation count stayed the same. The distribution of citation location over time also indicates three phases of the H-article “Discussion,” “Reputation,” and “Adoption” we propose in this study. Based on their locations in the citing articles and their roles in different periods, topics of citation context shifted gradually when an increasing number of other articles were co-mentioned with the H-article in the same sentences. These outcomes show that the impact of the H-article manifests in various ways within the content of these citing articles that continued to shift in nine years, data that is not captured by traditional means of citation analysis that do not weigh citation impacts over time.     

Revisiting citation aging: a model for citation distribution and life-cycle prediction

The study of citation distribution provides retrospective and prospective picture of the evolving impact of a corpus of publications on knowledge community. All distribution models agree on the rise of the number of citations in the first years following the publication to reach a peak and then tend to be less cited when time passes. However, questions such as how long it will continue being cited and what is objectively the rate of the decline remain unanswered. Built up of simple polynomial function, the proposed model is proven to be suitable to represent the observed citation distribution over time and to interestingly identify with accuracy when the major loss of citations happens. I calculate from the model the ‘residual citations’ representing the citations kept after a long time period after publication year. I demonstrate that the residual citations may be greater than or equal to zero, meaning that the ‘life-cycle’ of the corpus is infinite, contrary to what some researches termed to be around 21 years. This model fits the observed data from SCI according to R-sq which is greater than 98.9%. Rather, it is very simple and easy to implement and can be used by not highly-skilled scientometric users. Finally, the model serves as a citation predictive tool for a corpus by determining the citations that would obtain at any time of its life-cycle.     

Source normalized indicators of citation impact: an overview of different approaches and an empirical comparison

Different scientific fields have different citation practices. Citation-based bibliometric indicators need to normalize for such differences between fields in order to allow for meaningful between-field comparisons of citation impact. Traditionally, normalization for field differences has usually been done based on a field classification system. In this approach, each publication belongs to one or more fields and the citation impact of a publication is calculated relative to the other publications in the same field. Recently, the idea of source normalization was introduced, which offers an alternative approach to normalize for field differences. In this approach, normalization is done by looking at the referencing behavior of citing publications or citing journals. In this paper, we provide an overview of a number of source normalization approaches and we empirically compare these approaches with a traditional normalization approach based on a field classification system. We also pay attention to the issue of the selection of the journals to be included in a normalization for field differences. Our analysis indicates a number of problems of the traditional classification-system-based normalization approach, suggesting that source normalization approaches may yield more accurate results.     

Early citation counts correlate with accumulated impact

Summary The present paper addresses the objective of developing forward indicators of research performance using bibliometric information on the UK science base. Most research indicators rely primarily on historical time series relating to inputs to, activity within and outputs from the research system. Policy makers wish to be able to monitor changing research profiles in a more timely fashion, the better to determine where new investment is having the greatest effect. Initial (e.g. 12 months from publication) citation counts might be useful as a forward indicator of the long-term (e.g. 10 years from publication) quality of research publications, but - although there is literature on citation-time functions - no study to evaluate this specifically has been carried out by Thomson ISI or any other analysts. Here, I describe the outcomes of a preliminary study to explore these citation relationships, drawing on the UK National Citation Report held by Evidence Ltd under licence from Thomson ISI for OST policy use. Annual citation counts typically peak at around the third year after publication. I show that there is a statistically highly significant correlation between initial (years 1-2) and later (years 3-10) citations in six research categories across the life and physical sciences. The relationship holds over a wide range of initial citation counts. Papers that attract more than a definable but field dependent threshold of citations in the initial period after publication are usually among the top 1% (the most highly cited papers) for their field and year. Some papers may take off slowly but can later join the high impact group. It is important to recognise that the statistical relationship is applicable to groups of publications. The citation profiles of individual articles may be quite different. Nonetheless, it seems reasonable to conclude that leading indicators of research excellence could be developed. This initial study should now be extended across a wider range fields to test the initial outcomes: earlier papers suggest the model holds in economics. Additional statistical tests should be applied to explore and model the relationship between initial, later and total citation counts and thus to create a general tool for policy application.     

Quotation error, citation copying, and ant extinctions in Madeira

Many authors have written about how exotic ants invaded the Atlantic islands of Madeira and negatively impacted or even completely exterminated its native ants, despite the lack of firsthand observations concerning such impact.I examine howquotation error (misrepresentation of previous work) and citation copying (citing unexamined publications referred to by others) led to the origin and spread of the erroneous story of ant extinctions in Madeira.Quotation errorand citation copying may be more common than most scientists realize, particularly when authors cite references that are written in languages they do not understand.     

Scientific evaluations of citation quality of international research articles in the SCI database: Thailand case study

Summary Quantitative and qualitative scientific evaluations of the research performance of Thai researchers were carried out with regards to their international publications and citations in four different subject categories; namely Clinical Medicine, Chemistry, Material Sciences, and Engineering. This work used citations to publications of Thai researchers in the Science Citation Index (SCI) database during 1998-2002 as a data source. The calculations and comparisons of article impact factors (AIF), position impact factors (PIF) and journal impact factors (JIF) were attempted for quantitative evaluation.The positions and significance levels (cited contents) of the citations were considered for qualitative assessment.For quantitative evaluation, the highest article quantity and number of times cited were given by Thai researchers in Clinical Medicine, the lowest being for Material Sciences. Clinical Medicine had the highest AIF value, while Engineering exhibited the lowest. Each article by Thai researchers was found to be cited more than once within a citing article, especially articles in Clinical Medicine. For qualitative assessment, most articles from Thai scholars were cited in Introduction and Results & Discussion sections of the citing articles. Only non-Thai researchers in Clinical Medicine preferred to use Discussion from Thais' articles for discussion of their work whereas those in Chemistry, Material Sciences and Engineering were referred as general references. Less than 1.5% of research works of Thai scholars were cited as “the pioneer”for the research communities of the subject categories of interest.     

Use of citation per publication as an indicator to evaluate pentachlorophenol research

The objective of the study was to perform a bibliometric analysis of all pentachlorophenolrelated publications in the Science Citation Index (SCI). Analyzed parameters included document type, language of publication, page count, publication output, authorship, keywords plus, publication pattern, citation and country of publication. The US produced 29% of the total single country publications where the seven major industrial countries accounted for the majority of the total production (66%). An indicator citation per publication was successfully applied in this study to evaluate the impact of number of authors, countries, and journals. The mean value of citation per publication of collaborative papers was higher than that of single country publications. In addition analysis of keywords plus in different period was applied to indicate a research trend.     

Journals of general &; internal medicine and surgery: An analysis and comparison of citation

Summary The purpose of this study is to analyze and compare journal citation data, from Journal Citation Reports on the Web 2000, of general and internal medicine and surgery. The source items and five kinds of citation data, i.e. citation counts, impact factor, immediacy index, citing half-life and cited half-life are examined and the correlation between each of the fifteen pairs of citation data is determined based on the Pearson correlation tests. The Fishers Z-transform was employed to test the significant difference between the Pearson correlation coefficient for each pair of citation data of these two subject areas. The following results of this work reveal: the frequently published journals are cited more frequently and also with high impact factor and immediacy index, in addition, they are usually accompanied with short citing half-life (i.e., usually cite current literature). The impact factor and immediacy index has significant correlation with citation counts. A significant correlation also exists between impact factor and immediacy index. However there is no correlation between cited half-life and other citation data, except citing half-life. For journals of general and internal medicine and surgical medicine, there are no significant difference of the Pearson correlation coefficient for the following pair of citation data: source items and citation counts, source items and impact factor, source items and citing half-life, citation counts and citing half-life, impact factor and citing half-life, immediacy index and citing half-life, and cited half-life and citing half-life.     

Exploration into the evolution and historical roots of citation analysis by referenced publication year spectroscopy

In this study, we investigated the evolution process and historical roots of citation analysis study by reference publication year spectroscopy (RPYS), which is an advanced research method recently introduced in the field of Scientometrics. Through analyzing the publication year and citation frequency of cited references in a knowledge domain, RPYS can identify the citation peak of such cited references. We collected 2543 articles including 56,392 references regarding citation analysis in SCI-E and SSCI databases between 1970 and July 2016 as our data source. Based on the RPYS method by the CRExplorer program, the results showed that the peak citation publication years by chronological order occurred in 1955, 1963, 1973, 1979, 1981, 1990, 2005 and 2008 in the field of citation analysis study. According to the overall distribution of peak citation publication years, the RPYS for citation analysis was divided into five time periods for the convenience of comparison in this paper: that is, before 1900, 1901–1950, 1951–1970, 1971–2000, and 2001–2016. Pre 1950, especially during the 1900s–1950s, before the citation analysis method was introduced, there were three rather high peak citation publication years, Lotka’s law and Bradford’s law laid the knowledge foundation for citation analysis. 1950s–1970s was the forming period of citation analysis, among the three citation peaks in this period, Garfield (Science 122(3159):1127–1128, 1955), Price (Von Der Studierstube Zur 7(3–6):443–458, 1963, Science 149:510–515, 1965), and Kessler (1963) established an important knowledge base for the formation of citation analysis study. 1971–2000 was a developing period of citation analysis. Document Co-citation Analysis and Author Co-citation Analysis methods laid the foundation of the development of citation analysis study. Since 2000, citation analysis study has been expanding rapidly. By the number of published papers and number of cited references with highly cited frequency, Garfield E., White H.D., Small H., MacRoberts M.H., Price D.D. have played an important role in promoting the evolution of citation analysis study.     

Modeling the obsolescence of research literature in disciplinary journals through the age of their cited references

There are different citation habits in the research fields that influence the obsolescence of the research literature. We analyze the distinctive obsolescence of research literature in disciplinary journals in eight scientific subfields based on cited references distribution, as a synchronous approach. We use both negative binomial (NB) and Poisson distributions to capture this obsolescence. The corpus being examined is published in 2019 and covers 22,559 papers citing 872,442 references. Moreover, three measures to analyze the tail of the distribution are proposed: (i) cited reference survival rate, (ii) cited reference mortality rate, and (iii) cited reference percentile. These measures are interesting because the tail of the distribution collects the behavior of the citations at the time when the document starts to get obsolete in the sense that it is little cited (used). As main conclusion, the differences observed in obsolescence are so important even between disciplinary journals in the same subfield, that it would be necessary to use some measure for the tail of the citation distribution, such as those proposed in this paper, when analyzing in an appropriate way the long time impact of a journal.

     

Distance and velocity measures: using citations to determine breadth and speed of research impact

Research that integrates the social and natural sciences is vital to address many societal challenges, yet is difficult to arrange, conduct, and disseminate. This paper compares diffusion of the research supported by a unique U.S. National Science Foundation program on Human and Social Dynamics (“HSD”) with a matched group of heavily cited papers. We offer a measure of the distance of cites between the Web of Science Category (“WoSC”) in which a publication appears and the WoSC of the journal citing it, and find that HSD publications are cited more distantly than are comparison publications. We provide another measure—citation velocity—finding that HSD publications are cited with similar lag times as are the comparison papers. These basic citation distance and velocity measures enrich analyses of research knowledge diffusion patterns.     

Research and citation impact of publications by the Chemistry Division at Bhabha Atomic Research Centre

The paper analyses the citations to 1733 publications published during 1970–1999 by the Chemistry Division at Bhabha Atomic Research Centre, using Science Citation Index 1982–2003 as the source data. The extent of citations received, in terms of the number of citations per paper, yearwise break up of citations, domainwise citations, self-citations and citations by others, diachronous self-citation rate, citing authors, citing institutions, highly cited papers, the categories of citing documents, citing journals and distribution of citations among them etc. are determined. During 1982–2003 chemistry Division publications have received a total of 11041 citations. The average number of citations per year was 501.86. The average number of citations per publication was 6.37. The highest number of citations received were 877 in 2001. The citation rate was peaked during 1990–2003 as maximum 9145 (82.82%) citations were received during the period. Total self-citations were 3716 (33.66%) and citations by others were 7325 (66.34%). Mean diachronous self-citation rate was 36.16. Citation time lag was zero for 144 (15.52%) papers and one year for 350 (37.72%) papers. Single authored publications (168) have received 456 (4.13%) citations and 1565 multi-authored publications have received 10585 (95.87%) citations. The core citing authors were: J. P. Mittal (695) followed by V. K. Jain (524), H. Mohan (471), T. Mukherjee (307), R. M Iyer (253), H. Pal (251), J. V. Yakhmi (211), A. V. Sapre (174), D. K. Palit (161), N. M. Gupta (128), and S. K. Kulshrestha (116). Citation life cycles of four highly cited papers was discussed. The core journals citing Chemistry Division publications were: J. Phys. Chem.-A (436 citations), Chem. Phys. Lett. (372), J. Phys. Chem. (355), J. Chem. Phys. (353), J. Organomet. Chem. (285), J. Phys. Chem.-B (279), J. Photochem. Photobiol.-A (263), Langmuir (245), J. Am. Chem. Soc. (226), Physica-C (225), Radiat. Phys. Chem. (217), Inorg. Chem. (215) and Indian J. Chem.-A (207).