Methodological quality in clinical trials and bibliometric indicators: no evidence of correlations

Citation frequencies and journal impact factors (JIFs) are being used more and more to assess the quality of research and allocate research resources. If these bibliometric indicators are not an adequate predictor of research quality, there could be severe negative consequences for research. To analyse to which extent citation frequencies and journal impact factors correlate with the methodological quality of clinical research articles included in an SBU systematic review of antibiotic prophylaxis in surgery. All 212 eligible original articles were extracted from the SBU systematic review “Antibiotic Prophylaxis in Surgery” and categorized according to their methodological rigourness as high, moderate or low quality articles. Median of citation frequencies and JIFs were compared between the methodological quality groups using Kruskal–Wallis non-parametric test. An in-depth study of low-quality studies with higher citation frequencies/JIFs was also conducted. No significant differences were found in median citation frequencies (p = 0.453) or JIFs (p = 0.185) between the three quality groups. Studies that had high citation frequencies/JIFs but were assessed as low-quality lacked control groups, had high dropout rates or low internal validity. This study of antibiotic prophylaxis in surgery does not support the hypothesis that bibliometric indicators are a valid instrument for assessing methodological quality in clinical trials. This is a worrying observation, since bibliometric indicators have a major influence on research funding. However, further studies in other areas are needed.     

Does the higher citation of collaborative research differ from region to region? A case study of Economics

Many studies have found that collaborative research is, in general, more highly cited than non-collaborative research. This paper describes an investigation into the extent to which the association between high citation and collaboration for Economics articles published in 2000 varies from region to region and depends on the choice of indicator of citation level. Using data from the Social Science Citation Index (SSCI) for 18 countries, 17 American states and four indicators of citation level the citation levels of the collaborative articles are compared with the citation levels of the non-collaborative articles. The main findings are that: (a) for every country and every indicator the mean citation level of the collaborative articles was at least as high as that for the non-collaborative articles, but for five US states and for at least one other indicator the citation level of collaborative articles was lower than that of non-collaborative articles, and (b) the extent to which collaborative articles were more highly cited varied considerably from country to country, from state to state, and from indicator to indicator. This indicates the importance of using multiple indicators when investigating citation advantage since the choice of indicator can change the results.     

Characterizing a scientific elite (B): publication and citation patterns of the most highly cited scientists in environmental science and ecology

Science is principally driven by the efforts of a vanishingly small fraction of researchers publishing the majority of scientific research and garnering the majority of citations. Despite this well-established trend, knowledge of exactly how many articles these researchers publish, how highly they are cited, and how they achieved their distinctive accomplishments is meager. This article examines the publication and citation patterns of the world’s most highly cited environmental scientists and ecologists, inquiring into their levels of scientific productivity and visibility, examining relationships between scientific productivity and quality within their research programs, and considering how different publication strategies contribute to these distinctive successes. Generally speaking, highly cited researchers are also highly productive, publishing on average well over 100 articles each. Furthermore, articles published by this group are more highly cited on average than articles published in premier generalist journal like Nature and Science, and their citation to publication ratios are more equitably distributed than is typical. Research specialization and primacy of authorship are important determinants of citation frequency, while geographic differences and collaborative propensity matter less. The article closes with a set of suggestions for those wishing to increase the use of their research by the scientific community.     

Visualizing the context of citations referencing papers published by Eugene Garfield: a new type of keyword co-occurrence analysis

During Eugene Garfield’s (EG’s) lengthy career as information scientist, he published about 1500 papers. In this study, we use the impressive oeuvre of EG to introduce a new type of bibliometric networks: keyword co-occurrences networks based on the context of citations, which are referenced in a certain paper set (here: the papers published by EG). The citation context is defined by the words which are located around a specific citation. We retrieved the citation context from Microsoft Academic. To interpret and compare the results of the new network type, we generated two further networks: co-occurrence networks which are based on title and abstract keywords from (1) EG’s papers and (2) the papers citing EG’s publications. The comparison of the three networks suggests that papers of EG and citation contexts of papers citing EG are semantically more closely related to each other than to titles and abstracts of papers citing EG. This result accords with the use of citations in research evaluation that is based on the premise that citations reflect the cognitive influence of the cited on the citing publication.     

Weak evidence for determinants of citation frequency in ecological articles

Citation frequency has been considered a biased surrogate of publication merit. However, previous studies on this subject were based on small sample sizes and were entirely based on null-hypothesis significance testing. Here we evaluated the relative effects of different predictors on citation frequency of ecological articles using an information theory framework designed to evaluate multiple competing hypotheses. Supposed predictors of citation frequency (e.g., number of authors, length of articles) accounted for a low fraction of the total variation. We argue that biases concerning citation are minor in ecology and further studies that attempt to quantify the scientific relevance of an article, aiming to make further relationships with citation, are needed to advance our understanding of why an article is cited.     

Comparison of the Hirsch-index with standard bibliometric indicators and with peer judgment for 147 chemistry research groups

In this paper we present characteristics of the statistical correlation between the Hirsch (h-) index and several standard bibliometric indicators, as well as with the results of peer review judgment. We use the results of a large evaluation study of 147 university chemistry research groups in the Netherlands covering the work of about 700 senior researchers during the period 1991–2000. Thus, we deal with research groups rather than individual scientists, as we consider the research group as the most important work floor unit in research, particularly in the natural sciences. Furthermore, we restrict the citation period to a three-year window instead of ‘life time counts’ in order to focus on the impact of recent work and thus on current research performance. Results show that the h-index and our bibliometric ‘crown indicator’ both relate in a quite comparable way with peer judgments. But for smaller groups in fields with ‘less heavy citation traffic’ the crown indicator appears to be a more appropriate measure of research performance.     

The most highly cited Library and Information Science articles: Interdisciplinarity,first authors and citation patterns

Highly cited articles are interesting because of the potential association between high citation counts and high quality research. This study investigates the 82 most highly cited Information Science and Library Science’ (IS&LS) articles (the top 0.1%) in the Web of Science from the perspectives of disciplinarity, annual citation patterns, and first author citation profiles. First, the relative frequency of these 82 articles was much lower for articles solely in IS&LS than for those in IS&LS and at least one other subject, suggesting that that the promotion of interdisciplinary research in IS&LS may be conducive to improving research quality. Second, two thirds of the first authors had an h-index in IS&LS of less than eight, show that much significant research is produced by researchers without a high overall IS&LS research productivity. Third, there is a moderate correlation (0.46) between citation ranking and the number of years between peak year and year of publication. This indicates that high quality ideas and methods in IS&LS often are deployed many years after being published.     

Why with bibliometrics the Humanities does not need to be the weakest link

In this study an attempt is made to establish new bibliometric indicators for the assessment of research in the Humanities. Data from a Dutch Faculty of Humanities was used to provide the investigation a sound empirical basis. For several reasons (particularly related to coverage) the standard citation indicators, developed for the sciences, are unsatisfactory. Target expanded citation analysis and the use of oeuvre (lifetime) citation data, as well as the addition of library holdings and productivity indicators enable a more representative and fair assessment. Given the skew distribution of population data, individual rankings can best be determined based on log transformed data. For group rankings this is less urgent because of the central limit theorem. Lifetime citation data is corrected for professional age by means of exponential regression.     

The use of citation speed to understand the effects of a multi-institutional science center

The extent to which an article attracts citations has long been of interest. However, recent research has emphasized not just the receipt but also the pacing of citation. Citation speed has been shown to be affected by journal prestige and self-citation but also public funding of research. Amidst these viewpoints, this paper explores the speed of article citation of a multi-institutional, multi-disciplinary publicly funded research center relative to that of a comparison group of articles. Results indicate that articles by authors affiliated with the center are significantly more likely to have early-cited papers within the year of publication than the random comparison group, with controls by field also being significant. Implications for the ability of a publicly funded center to attract attention toward articles are discussed.     

Zero impact: a large-scale study of uncitedness

This paper presents a large-scale study of the phenomenon ‘uncitedness’. A literature review indicates that uncitedness is related to at least three factors: Field, document type, and time. To explore these factors and their mutual influence further, and at much larger scale than previous studies, the paper focuses on seven subject areas (arts and humanities; social sciences; computer science; mathematics; engineering; medicine; physics and astronomy), seven document types (articles; reviews; notes; letters; conference papers; books; book chapters), and a 20-year publication window (1996–2015). Documents are searched in Scopus, and retrieved year-by-year, discipline-by-discipline, and for each individual document type (total: 29,472,184 documents; 7,508,741 uncited documents). The results show great variance in uncitedness ratios between subject areas and document types. This is probably caused by a somewhat tacitly agreed upon genre hierarchy existing in all subject areas, yet with important local traits and differences. The importance of the time-dimension is documented. Time to first citation varies a great deal between subject areas, and the uncitedness ratio is consequently shown to be quite sensitive to the length of citation windows.

     

CAD: an algorithm for citation-anchors detection in research papers

Citations are very important parameters and are used to take many important decisions like ranking of researchers, institutions, countries, and to measure the relationship between research papers. All of these require accurate counting of citations and their occurrence (in-text citation counts) within the citing papers. Citation anchors refer to the citation made within the full text of the citing paper for example: ‘[1]’, ‘(Afzal et al, 2015)’, ‘[Afzal, 2015]’ etc. Identification of citation-anchors from the plain-text is a very challenging task due to the various styles and formats of citations. Recently, Shahid et al. highlighted some of the problems such as commonality in content, wrong allotment, mathematical ambiguities, and string variations etc in automatically identifying the in-text citation frequencies. The paper proposes an algorithm, CAD, for identification of citation-anchors and its in-text citation frequency based on different rules. For a comprehensive analysis, the dataset of research papers is prepared: on both Journal of Universal Computer Science (J.UCS) and (2) CiteSeer digital libraries. In experimental study, we conducted two experiments. In the first experiment, the proposed approach is compared with state-of-the-art technique over both datasets. The J.UCS dataset consists of 1200 research papers with 16,000 citation strings or references while the CiteSeer dataset consists of 52 research papers with 1850 references. The total dataset size becomes 1252 citing documents and 17,850 references. The experiments showed that CAD algorithm improved F-score by 44% and 37% respectively on both J.UCS and CiteSeer dataset over the contemporary technique (Shahid et al. in Int J Arab Inf Technol 12:481–488, 2014). The average score is 41% on both datasets. In the second experiment, the proposed approach is further analyzed against the existing state-of-the-art tools: CERMINE and GROBID. According to our results, the proposed approach is best performing with F1 of 0.99, followed by GROBID (F1 0.89) and CERMINE (F1 0.82).     

International and domestic co-publishing and their citation impact in different disciplines

This paper studies disciplinary differences in citation impacts of different types of co-publishing. The citation impacts of international, domestic inter-organizational and domestic intra-organizational co-publications, and single-authored publications, are compared. In particular, we examine the extent to which the number of authors explains the potential differences in citation impacts when compared to the influence of different types of international and domestic collaborations. The analysis is based on Finland’s publications in Thomson Reuters Web of Science database in 1990–2008. Finland is a small country, thus, it has fewer opportunities to find collaborators inside own country when compared to larger countries. Finland’s science policy has underlined internationalization and research collaboration as key means to increase the quality and impact of Finnish research. This study indicates that both international and domestic co-publishing have steadily increased during the past two decades in all disciplinary groups. International co-publications gain on average more citations than domestic co-publications. In natural sciences and engineering, co-authorship explains only a small proportion of variability in publications’ citation rates. When the effect of the number of authors is taken into account there are no big differences in citation impacts between international and domestic co-publications. However, international co-publications by ten authors or more gather significantly more citations than other publications. In humanities, the difference in citation impacts between co-authored publications in relation to single-authored publications is significant. However, international co-publications are not on average more highly cited in relation to domestic co-publications in humanities.     

Do men and women differ in their use of tables and graphs in academic publications?

In psychological research there is huge literature on differences between the sexes. Typically it used to be thought that women were more verbally and men more spatially oriented. These differences now seem to be waning. In this article we present three studies on sex differences in the use of tables and graphs in academic articles. These studies are based on data mining from approximately 2,000 articles published in over 200 peer-reviewed journals in the sciences and social sciences. In Study 1 we found that, in the sciences, men used 26 % more graphs and figures than women, but that there were no significant differences between them in their use of tables. In Study 2 we found no significant differences between men and women in their use of graphs and figures or tables in social science articles. In Study 3 we found no significant differences between men and women in their use of what we termed ‘data’ and ‘text’ tables in social science articles. It is possible that these findings indicate that academic writing is now becoming a genre that is equally undertaken by men and women.     

Understanding the impact change of a highly cited article: a content-based citation analysis

Researchers tend to cite highly cited articles, but how these highly cited articles influence the citing articles has been underexplored. This paper investigates how one highly cited essay, Hirsch’s “h-index” article (H-article) published in 2005, has been cited by other articles. Content-based citation analysis is applied to trace the dynamics of the article’s impact changes from 2006 to 2014. The findings confirm that citation context captures the changing impact of the H-article over time in several ways. In the first two years, average citation mention of H-article increased, yet continued to decline with fluctuation until 2014. In contrast with citation mention, average citation count stayed the same. The distribution of citation location over time also indicates three phases of the H-article “Discussion,” “Reputation,” and “Adoption” we propose in this study. Based on their locations in the citing articles and their roles in different periods, topics of citation context shifted gradually when an increasing number of other articles were co-mentioned with the H-article in the same sentences. These outcomes show that the impact of the H-article manifests in various ways within the content of these citing articles that continued to shift in nine years, data that is not captured by traditional means of citation analysis that do not weigh citation impacts over time.     

Multiple regression analysis of a patent’s citation frequency and quantitative characteristics: the case of Japanese patents

Although many studies have been conducted to clarify the factors that affect the citation frequency of “academic papers,” there are few studies where the citation frequency of “patents” has been predicted on the basis of statistical analysis, such as regression analysis. Assuming that a patent based on a variety of technological bases tends to be an important patent that is cited more often, this study examines the influence of the number of cited patents’ classifications and compares it with other factors, such as the numbers of inventors, classifications, pages, and claims. Multiple linear, logistic, and zero-inflated negative binomial regression analyses using these factors are performed. Significant positive correlations between the number of classifications of cited patents and the citation frequency are observed for all the models. Moreover, the multiple regression analyses demonstrate that the number of classifications of cited patents contributes more to the regression than do other factors. This implies that, if confounding between factors is taken into account, it is the diversity of classifications assigned to backward citations that more largely influences the number of forward citations.     

National research contributions: A case study on Finnish biomedical research

The long-term influence and contribution of research can be evaluated relatively reliably by bibliometric citation analysis. Previously, productivity of nations has been estimated by using either the number of published articles or journal impact factors and/or citation data. These studies show certain trends, but detailed analysis is not possible due to the assumption that all articles in a journal were equally cited. Here we describe the first comprehensive, longterm, nationwide analysis of scientific performance. We studied the lifetime research output of 748 Finnish principal investigators in biomedicine during the years 1966–2000, analysed national trends, and made a comparison with international research production. Our results indicate that analyses of the scientific contribution of persons, disciplines, or nations should be based on actual publication and citation counts rather than on derived information like impact factors. 51% of the principal investigators have published altogether 75% of the articles; however, the whole scientific community has contributed to the growth of biomedical research in Finland since the Second World War.     

Does the human capital of teams of natural science authors predict citation frequency?

This study examines the relationship between citation frequency and the human capital of teams of authors. Analysis of a random sample of articles published in top natural science journals shows that articles co-authored by teams including frequently cited scholars and teams whose members have diverse disciplinary backgrounds have greater citation frequency. The institutional prestige, the percentage of team members at U. S. institutions and the variety of disciplines represented by team member backgrounds do not influence citation frequency. The study introduces a method for evaluating the extent of multidisciplinarity that accounts for the relatedness of disciplines or authors.     

Assessing non-standard article impact using F1000 labels

Faculty of 1000 (F1000) is a post-publishing peer review web site where experts evaluate and rate biomedical publications. F1000 reviewers also assign labels to each paper from a standard list or article types. This research examines the relationship between article types, citation counts and F1000 article factors (FFa). For this purpose, a random sample of F1000 medical articles from the years 2007 and 2008 were studied. In seven out of the nine cases, there were no significant differences between the article types in terms of citation counts and FFa scores. Nevertheless, citation counts and FFa scores were significantly different for two article types: “New finding” and “Changes clinical practice”: FFa scores value the appropriateness of medical research for clinical practice and “New finding” articles are more highly cited. It seems that highlighting key features of medical articles alongside ratings by Faculty members of F1000 could help to reveal the hidden value of some medical papers.