Bias in the journal impact factor

The ISI journal impact factor (JIF) is based on a sample that may represent half the whole-of-life citations to some journals, but a small fraction (<10%) of the citations accruing to other journals. This disproportionate sampling means that the JIF provides a misleading indication of the true impact of journals, biased in favour of journals that have a rapid rather than a prolonged impact. Many journals exhibit a consistent pattern of citation accrual from year to year, so it may be possible to adjust the JIF to provide a more reliable indication of a journal’s impact.     

The effect of additional citations in the stability of Journal Citation Report categories

We use a new approach to study the ranking of journals in JCR categories. The objectives of this study were to empirically evaluate the effect of increases in citations on the computation of the journal impact factor (JIF) for a large set of journals as measured by changes in JIF, and to ascertain the influence of additional citations on the rank order of journals according their new JIFs within JCR groups. To do so, modified JIFs were computed by adding additional citations to the number used by Thomson-Reuters to compute the JIF of journals listed in the JCR for 2008. We considered the effect on rank order of a given journal of adding 1, 2, 3 or more citations to the number used to compute the JIF, keeping everything else equal (i.e., without changing the JIF of other journals in a given group). The effect of additional citations on the internal structure of rankings in JCR groups increased with the number of citations added. In about one third of JCR groups, about half the journals changed their rank order when 1–5 citations were added. However, in general the rank order tended to be relatively stable after small increases in citations.     

The Journal’s Integrated Impact Index: a new indicator for journal evaluation

Journal impact factor (JIF) has been used for journal evaluation over a long time, but also accompanied by the continuing controversy. In this study, a new indicator, the Journal’s Integrated Impact Index (JIII) has been proposed for journal evaluation. In the JIII, one journal’s average citations per paper, total citations, and all journals’ average level of average citations per paper and total citations have been used to characterize the integrated impact of journals. Some contrastive analyses were carried out between JIII and JIF. The results show some interesting properties of the new indicator, and also reveal some relevant relationships among JIII, JIF, and other bibliometric indicators.     

A modified method for calculating the Impact Factors of journals in ISI Journal Citation Reports: Polymer Science Category in 1997–2001

This article introduces a new modified method for calculating the impact factor of journals based on the current ISI practice in generating journal impact factor values. The impact factor value for a journal calculated by the proposed method, the so-called Cited Half-Life Impact Factor (CHAL) method, which is based on the ratio of the number of current year citations of articles from the previous X years to that of articles published in the previous X years, the X value being equal to the value of the cited half-life of the journal in the current year. Thirty-four journals in the Polymer Science Category from the ISI Subject Heading Categories were selected and examined. Total citations, impact factors and cited half-life of the 34 journals during the last five years (1997-2001) were retrieved from the ISI Journal Citation Reports and were used as the data source for the calculations in this work, the impact factor values from ISI and CHAL methods then being compared. The positions of the journals ranked by impact factors obtained from the ISI method were different from those from the CHAL method. It was concluded that the CHAL method was more suitable for calculating the impact factor of the journals than the existing ISI method. This revised version was published online in June 2006 with corrections to the Cover Date.     

Effect on the journal impact factor of the number and document type of citing records: a wide-scale study

We studied the effect on journal impact factors (JIF) of citations from documents labeled as articles and reviews (usually peer reviewed) versus citations coming from other documents. In addition, we studied the effect on JIF of the number of citing records. This number is usually different from the number of citations. We selected a set of 700 journals indexed in the SCI section of JCR that receive a low number of citations. The reason for this choice is that in these instances some citations may have a greater impact on the JIF than in more highly-cited journals. After excluding some journals for different reasons, our sample consisted of 674 journals. We obtained data on citations that contributed to the JIF for the years 1998?C2006. In general, we found that most journals obtained citations that contribute to the impact factor from documents labeled as articles and reviews. In addition, in most of journals the ratio between citations that contributed to the impact factor and citing records was greater than 80% in all years. Thus, in general, we did not find evidence that citations that contributed to the impact factor were dependent on non-peer reviewed documents or only a few citing records.     

Journal self-citations that contribute to the impact factor: Documents labeled “editorial material” in journals covered by the Science Citation Index

Summary We investigated the distribution of citations included in documents labeled by the ISI as “editorial material” and how they contribute to the impact factor of journals in which the citing items were published. We studied all documents classified by the ISI as “editorial material” in the Science Citation Index between 1999 and 2004 (277,231 records corresponding to editorial material published in 6141 journals). The results show that most journals published only a few documents that included 1 or 2 citations that contributed to the impact factor, although a few journals published many such documents. The data suggest that manipulation of the impact factor by publishing large amounts of editorial material with many citations to the journal itself is not a widely used strategy to increase the impact factor.     

The generalized propensity score methodology for estimating unbiased journal impact factors

The journal impact factor (JIF) proposed by Garfield in the year 1955 is one of the most commonly used and prominent citation-based indicators of the performance and significance of a scientific journal. The JIF is simple, reasonable, clearly defined, and comparable over time and, what is more, can be easily calculated from data provided by Thomson Reuters, but at the expense of serious technical and methodological flaws. The paper discusses one of the core problems: The JIF is affected by bias factors (e.g., document type) that have nothing to do with the prestige or quality of a journal. For solving this problem, we suggest using the generalized propensity score methodology based on the Rubin Causal Model. Citation data for papers of all journals in the ISI subject category ??Microscopy?? (Journal Citation Report) are used to illustrate the proposal.     

Article-count impact factor of materials science journals in SCI database

This article proposed a new index, so-called “Article-Count Impact Factor” (ACIF) for evaluating journal quality in light of citation behaviour in comparison with the ISI journal impact factors. The ACIF index was the ratio of the number of articles that were cited in the current year to the source items published in that journal during the previous two years. In this work, we used 171 journal titles in materials categories published in the years of 2001–2004 in international journals indexed in the Science Citation Index Expanded (SCI) database as data source. It was found that ACIF index could be used as an alternative tool in assessing the journal quality, particularly in the case where the assessed journals had the same (equal or similar) JIF values. The experimental results suggested that the higher the ACIF value, the more the number of articles being cited. The changes in ACIF values were more dependent on the JIF values rather than the total number of articles. Polymer Science had the greatest ACIF values, suggesting that the articles in Polymer Science had greater “citation per article” than those in Metallurgical Engineering and Ceramics. It was also suggested that in order to increase a JIF value of 1.000, Ceramics category required more articles to be cited as compared to Metallurgical Engineering and Polymer Science categories.     

Journal self-citation rates in ecological sciences

Impact factors are a widely accepted means for the assessment of journal quality. However, journal editors have possibilities to influence the impact factor of their journals, for example, by requesting authors to cite additional papers published recently in that journal thus increasing the self-citation rate. I calculated self-citation rates of journals ranked in the Journal Citation Reports of ISI in the subject category “Ecology” (n = 107). On average, self citation was responsible for 16.2 ± 1.3% (mean ± SE) of the impact factor in 2004. The self-citation rates decrease with increasing journal impact, but even high impact journals show large variation. Six journals suspected to request for additional citations showed high self-citation rates, which increased over the last seven years. To avoid further deliberate increases in self-citation rates, I suggest to take journal-specific self-citation rates into account for journal rankings.     

Structure of the impact factor of academic journals in the field of Education and Educational Psychology: Citations from editorial board members

Summary We present a new approach to study the structure of the impact factor of academic journals. This new method is based on calculation of the fraction of citations that contribute to the impact factor of a given journal that come from citing documents in which at least one of the authors is a member of the cited journal's editorial board. We studied the structure of three annual impact factors of 54 journals included in the groups “Education and Educational Research” and “Psychology, Educational” of the Social Sciences Citation Index. The percentage of citations from papers authored by editorial board members ranged from 0% to 61%. In 12 journals, for at least one of the years analysed, 50% or more of the citations that contributed to the impact factor were from documents published in the journal itself. Given that editorial board members are considered to be among the most prestigious scientists, we suggest that citations from papers authored by editorial board members should be given particular consideration.     

Journals that increase their impact factor at least fourfold in a few years: The role of journal self-citations

The aim of this study was to ascertain the possible effect of journal self-citations on the increase in the impact factors of journals in which this scientometric indicator rose by a factor of at least four in only a few years. Forty-three journals were selected from the Thomson—Reuters (formerly ISI) Journal Citation Reports as meeting the above criterion. Eight journals in which the absolute number of citations was lower than 20 in at least two years were excluded, so the final sample consisted of 35 journals. We found no proof of widespread manipulation of the impact factor through the massive use of journal self-citations.     

What makes a great journal great in the sciences? Which came first,the chicken or the egg?

The paper is concerned with analysing what makes a great journal great in the sciences, based on quantifiable Research Assessment Measures (RAM). Alternative RAM are discussed, with an emphasis on the Thomson Reuters ISI Web of Science database (hereafter ISI). Various ISI RAM that are calculated annually or updated daily are defined and analysed, including the classic 2-year impact factor (2YIF), 5-year impact factor (5YIF), Immediacy (or 0-year impact factor (0YIF)), Eigenfactor, Article Influence, C3PO (Citation Performance Per Paper Online), h-index, Zinfluence, PI-BETA (Papers Ignored—By Even The Authors), Impact Factor Inflation (IFI), and three new RAM, namely Historical Self-citation Threshold Approval Rating (H-STAR), 2 Year Self-citation Threshold Approval Rating (2Y-STAR), and Cited Article Influence (CAI). The RAM data are analysed for the 6 most highly cited journals in 20 highly-varied and well-known ISI categories in the sciences, where the journals are chosen on the basis of 2YIF. The application to these 20 ISI categories could be used as a template for other ISI categories in the sciences and social sciences, and as a benchmark for newer journals in a range of ISI disciplines. In addition to evaluating the 6 most highly cited journals in each of 20 ISI categories, the paper also highlights the similarities and differences in alternative RAM, finds that several RAM capture similar performance characteristics for the most highly cited scientific journals, determines that PI-BETA is not highly correlated with the other RAM, and hence conveys additional information regarding research performance. In order to provide a meta analysis summary of the RAM, which are predominantly ratios, harmonic mean rankings are presented of the 13 RAM for the 6 most highly cited journals in each of the 20 ISI categories. It is shown that emphasizing THE impact factor, specifically the 2-year impact factor, of a journal to the exclusion of other informative RAM can lead to a distorted evaluation of journal performance and influence on different disciplines, especially in view of inflated journal self citations.     

Measuring the influence of clinical trials citations on several bibliometric indicators

The practice of publishing clinical trials in scientific journals is common, although not without its critics. This study aims to measure the effect of clinical trials citations on several bibliometric indicators: citations per document (CD); journal impact factor (JIF); relative h-index (RhI) and strike rate index (SRI). We select all the citable documents published in the NEJM, Lancet, JAMA, AIM and BMJ, for the period 2000-2004, and record the citations received by those papers from 2000 to 2005. Our results show that clinical trials have a CD significantly higher than those for conventional papers; JIF is lower when clinical trials are excluded, especially for NEJM, Lancet and JAMA. Finally, both RhI and SRI seem to be unaffected by clinical trials citations.     

Towards appropriate indicators of journal impact

This paper reviews a range of studies conducted by the authors on indicators reflecting scholarly journal impact. A critical examination of the journal impact data in theJournal Citation Reports (JCR), published by the Institute for Scientific Information (ISI) has shown that the JCR impact factor is inaccurate and biased towards journals revealing a rapid maturing or decline in impact. In addition, it was found that the JCR cited half life is an inappropriate measure of decline of journal impact. More appropriate impact measures of scholarly journals are proposed. A new classification system is explored, describing both maturing and decline of journal impact as measured through citations. Suggestions for future research are made, analysing in more detail the distribution of citations among papers in a journal.     

Online citation analysis

The paper investigates the online citation analysis possibilities and limitations. The following online processing tools: RANK, MAP, and TARGET, provided by Dialog, are incorporated in order to perform analyses of citations to and from isolated sets of documents as well as to carry out diachrone journal analyses. These, analyses imply further to determine journal impact factors of ISI journals. Measures of the scope of internationalisation of journals are proposed and demonstrated. By the combined application of the RANK and TARGET commands we demonstrate a hitherto overlooked possibility of working with bibliographic coupling online and mapping of scientific fields.     

The citation impacts and citation environments of Chinese journals in mathematics

Based on the citation data of journals covered by the China Scientific and Technical Papers and Citations Database (CSTPCD), we obtained aggregated journal-journal citation environments by applying routines developed specifically for this purpose. Local citation impact of journals is defined as the share of the total citations in a local citation environment, which is expressed as a ratio and can be visualized by the size of the nodes. The vertical size of the nodes varies proportionally to a journal’s total citation share, while the horizontal size of the nodes is used to provide citation information after correction for the within-journal (self-) citations. In the “citing” environment, the equivalent of the local citation performance can also be considered as a citation activity index. Using the “citing” patterns as variables one is able to map how the relevant journal environments are perceived by the collective of authors of a journal, while the “cited” environment reflects the impact of journals in a local environment. In this study, we analyze citation impacts of three Chinese journals in mathematics and compare local citation impacts with impact factors. Local citation impacts reflect a journal’s status and function better than (global) impact factors. We also found that authors in Chinese journals prefer international instead of domestic ones as sources for their citations.     

An impact of Croatian journals measured by citation analysis from SCI-expanded database in time span 1975–2001

The aim of this research is to gain an insight into international recognition of the STM (Science, Technology, and Medicine) Croatian journals measured by citations in SCI-expanded database. The sample for the research was a citation analysis of 142 journals in time span 1975–2001 for papers published in 1975–1998. More than 90% of those journals are not indexed by SCI-expanded. For the purpose of this research we introduced a new scientometric indicator Normalized number of Citations per 100 Papers (NCP) that allows us direct comparison of the journals from various categories (NCP = 100C/P / IF₁₉₈₉). We chose the year 1989 as a mean value for time span 1975–2001. By citation analysis we established the influence of errors on recognition of Croatian journals and their articles. Obtained results show that an article-to-article link is not found for 32% of cited items. The most frequent type of error is journal title, 37%, which indicates that approximately one third of Croatian journals can not be found when searching by journal title only. Some Croatian journals, even not indexed by SCI-expanded, showed relatively high rank in an impact, i.e. their NCP is higher than 100, and number of citations per paper is higher than 1.     

On first quartile journals which are not of highest impact

Here we study the relationship between journal quartile rankings of ISI impact factor (at the 2010) and journal classification in four impact classes, i.e., highest impact, medium highest impact, medium lowest impact, and lowest impact journals in subject category computer science artificial intelligence. To this aim, we use fuzzy maximum likelihood estimation clustering in order to identify groups of journals sharing similar characteristics in a multivariate indicator space. The seven variables used in this analysis are: (1) Scimago Journal Ranking (SJR); (2) H-Index (H); (3) ISI impact factor (IF); (4) 5-Year Impact Factor (5IF); (5) Immediacy Index (II); (6) Eigenfactor Score (ES); and (7) Article Influence Score (AIS). The fuzzy clustering allows impact classes to overlap, thereby accommodating for uncertainty related to the confusion about the impact class attribution for a journal and vagueness in impact classes definition. This paper demonstrates the complex relationship between quartiles of ISI impact factor and journal impact classes in the multivariate indicator space. And that several indicators should be used for a distinct analysis of structural changes at the score distribution of journals in a subject category. Here we propose it can be performed in a multivariate indicator space using a fuzzy classifier.     

Citation patterns of editorial gatekeepers in international chemistry journals

A significant correlation was found between the mean number of citations to the editors of international chemistry journals and the impact factor of the journals in question. A much weaker correlation was found if citations to the editor(s)-in-chief only were considered; this suggests that the professional profile of the journal is determined by the editorial board rather than the person of the editor(s)-in-chief. The number of citations to the editors of international chemistry journals may be used for characterizing a country's chemical research activity.