Microwave effect: analyzing citations from classic theories and their reinventions—a case study from a classic paper in aquatic ecology—Brooks & Dodson, 1965

Citations play an essential role in creating a knowledge network and recognizing relevant contributions during the process of scientific production. Despite the citations establishing the links between new evidence and the preceding ideas, classic articles may not be cited adequately. Our aim is to identify if classic studies are cited over time and if the recent studies are producing new knowledge or just “giving a new look” to pre-existing ideas. We evaluated whether the theory proposed by Brooks and Dodson (Science 150(3692): 28–35, 1965)-Size-efficiency Hypothesis was referenced in studies on the subject since its publication. Through the analysis of 1480 scientific papers, we quantified—from 1965 to 2018—the citation index (CI) of the original article considering the number of articles produced on the topic per year and the number of citations to other authors (intermediaries). We observed that 60% of the papers and 59% of the intermediaries do not refer to the original article. The CI was low and negatively affected by the age of the original article, showing that the frequency of citation was lower than the rate by which articles on the topic were published. There is a tendency to cite more recent articles and articles that corroborate their own findings. Our data illustrated the microwave effect, in which pre-existing ideas and theories are “reheated” by more recent articles where little of the original idea is modified. The microwave effect can create the impression of scientific advancement when there is little being added to the knowledge already produced.

     

Observations and hypotheses on the phenomenon of multiple citation to a research group's oeuvre

This paper analyses the phenomenon when a publication referring to the oeuvre of a research group (i.e. all the articles published by its members) cites several articles rather than one article from that oeuvre (multiple citations, MC). It is shown that significant differences exist between research groups with respect to the frequency at which MC to their respective oeuvres occur, and that these differences affect to some extent rankings of these groups based on citation counts. In order to find an explanation for our results, four factors are discussed: (1) the impact of a research group; (2) mutual multiple citing arrangements; (3) the size of a group's oeuvre and (4): the degree of common intellectual interest between the research activities in a group. No definite conclusions can be drawn yet on the extent to which these factors are responsible for the observed patterns in the MC frequency. We conclude however that attempts to identify ‘top’ or ‘sub-top’ groups in comparative evaluations based on citation analysis should be performed with the greatest care.     

Relativity of citation performance and excellence measures: From cross-field to cross-scale effects of field-normalisation

Summary As citation practices strongly depend on fields, field normalisation is recognised as necessary for fair comparison of figures in bibliometrics and evaluation studies. However fields may be defined at various levels, from small research areas to broad academic disciplines, and thus normalisation values are expected to vary. The aim of this project was to test the stability of citation ratings of articles as the level of observation - hence the basis of normalisation - changes. A conventional classification of science based on ISI subject categories and their aggregates at various scales was used, namely at five levels: all science, large academic discipline, sub-discipline, speciality and journal. Among various normalisation methods, we selected a simple ranking method (quantiles), based on the citation score of the article in each particular aggregate (journal, speciality, etc.) it belonged to at each level. The study was conducted on articles in the full SCI range, for publication year 1998 with a four-year citation window. Stability is measured in three ways: overall comparison of article rankings; individual trajectory of articles; survival of the top-cited class across levels. Overall rank correlations on the observed empirical structure are benchmarked against two fictitious sets that keep the same embedded structure of articles but reassign citation scores either in a totally ordered or in a totally random distribution. These sets act respectively as a 'worst case' and 'best case' for the stability of citation ratings. The results show that: (a) the average citation rankings of articles substantially change with the level of observation (b) observation at the journal level is very particular, and the results differ greatly in all test circumstances from all the other levels of observation (c) the lack of cross-scale stability is confirmed when looking at the distribution of individual trajectories of articles across the levels; (d) when considering the top-cited fractions, a standard measure of excellence, it is found that the contents of the 'top-cited' set is completely dependent on the level of observation. The instability of impact measures should not be interpreted in terms of lack of robustness but rather as the co-existence of various perspectives each having their own form of legitimacy. A follow-up study will focus on the micro levels of observation and will be based on a structure built around bibliometric groupings rather than conventional groupings based on ISI subject categories.     

Does the higher citation of collaborative research differ from region to region? A case study of Economics

Many studies have found that collaborative research is, in general, more highly cited than non-collaborative research. This paper describes an investigation into the extent to which the association between high citation and collaboration for Economics articles published in 2000 varies from region to region and depends on the choice of indicator of citation level. Using data from the Social Science Citation Index (SSCI) for 18 countries, 17 American states and four indicators of citation level the citation levels of the collaborative articles are compared with the citation levels of the non-collaborative articles. The main findings are that: (a) for every country and every indicator the mean citation level of the collaborative articles was at least as high as that for the non-collaborative articles, but for five US states and for at least one other indicator the citation level of collaborative articles was lower than that of non-collaborative articles, and (b) the extent to which collaborative articles were more highly cited varied considerably from country to country, from state to state, and from indicator to indicator. This indicates the importance of using multiple indicators when investigating citation advantage since the choice of indicator can change the results.     

Use of citation per publication as an indicator to evaluate contingent valuation research

This is the first article using bibliometrics to study the field of contingent valuation research. The purpose of this study was to evaluate the contingent valuation research performance based on all the related articles in SCI and SSCI databases from 1991 to 2005. An indicator named citation per publication (CPP) was presented in this study to assess the impact of article output per year, different countries, institutes, and authors from the worldwide. Publication per institute (PPI) in a country was used to be an indicator to compare institute’s research performance by country. Citation analysis was made to select the most frequently cited articles since publication to 2005 of each year. A citation model was applied to describe the relationship between the cumulative number of citations and article life. The results indicate that with the increase article output per year, the CPP decreased slightly since 1997. The USA produced 55% of all pertinent articles. Institutes from the UK had a higher PPI. The most prolific institutes and authors, and the most frequently cited articles per year were all listed. In addition, a citation model was successfully applied to evaluate performance of each year, and the most frequently cited articles of each year were also compared by the model.     

Understanding the impact change of a highly cited article: a content-based citation analysis

Researchers tend to cite highly cited articles, but how these highly cited articles influence the citing articles has been underexplored. This paper investigates how one highly cited essay, Hirsch’s “h-index” article (H-article) published in 2005, has been cited by other articles. Content-based citation analysis is applied to trace the dynamics of the article’s impact changes from 2006 to 2014. The findings confirm that citation context captures the changing impact of the H-article over time in several ways. In the first two years, average citation mention of H-article increased, yet continued to decline with fluctuation until 2014. In contrast with citation mention, average citation count stayed the same. The distribution of citation location over time also indicates three phases of the H-article “Discussion,” “Reputation,” and “Adoption” we propose in this study. Based on their locations in the citing articles and their roles in different periods, topics of citation context shifted gradually when an increasing number of other articles were co-mentioned with the H-article in the same sentences. These outcomes show that the impact of the H-article manifests in various ways within the content of these citing articles that continued to shift in nine years, data that is not captured by traditional means of citation analysis that do not weigh citation impacts over time.     

What factors are associated with increasing co-authorship in the social sciences? A case study of Danish Economics and Political Science

We examine the relative strength of short-term citation counts, bibliometric measures such as journal impact factors, and journal rankings in terms of predicting long-run citations. Using a set of articles published in sixty of the highest reputation economics journals in 1994, we find that citations received over fairly short windows (between 1 and 2 years after publication) are much stronger predictors of long-run citation counts compared with journal impact factors or other journal rankings. Our results are robust to a series of robustness checks. These findings suggest department heads and tenure and promotion committees should place greater weight on short-term citations as opposed to article placement when making personnel decisions.     

What makes an article influential? Predicting impact in social and personality psychology

Factors contributing to citation impact in social-personality psychology were examined in a bibliometric study of articles published in the field’s three major journals. Impact was operationalized as citations accrued over 10 years by 308 articles published in 1996, and predictors were assessed using multiple databases and trained coders. Predictors included author characteristics (i.e., number, gender, nationality, eminence), institutional factors (i.e., university prestige, journal prestige, grant support), features of article organization (i.e., title characteristics, number of studies, figures and tables, number and recency of references), and research approach (i.e., topic area, methodology). Multivariate analyses demonstrated several strong predictors of impact, including first author eminence, having a more senior later author, journal prestige, article length, and number and recency of references. Many other variables — e.g., author gender and nationality, collaboration, university prestige, grant support, title catchiness, number of studies, experimental vs. correlational methodology, topic area — did not predict impact.     

Characterizing a scientific elite (B): publication and citation patterns of the most highly cited scientists in environmental science and ecology

Science is principally driven by the efforts of a vanishingly small fraction of researchers publishing the majority of scientific research and garnering the majority of citations. Despite this well-established trend, knowledge of exactly how many articles these researchers publish, how highly they are cited, and how they achieved their distinctive accomplishments is meager. This article examines the publication and citation patterns of the world’s most highly cited environmental scientists and ecologists, inquiring into their levels of scientific productivity and visibility, examining relationships between scientific productivity and quality within their research programs, and considering how different publication strategies contribute to these distinctive successes. Generally speaking, highly cited researchers are also highly productive, publishing on average well over 100 articles each. Furthermore, articles published by this group are more highly cited on average than articles published in premier generalist journal like Nature and Science, and their citation to publication ratios are more equitably distributed than is typical. Research specialization and primacy of authorship are important determinants of citation frequency, while geographic differences and collaborative propensity matter less. The article closes with a set of suggestions for those wishing to increase the use of their research by the scientific community.     

Methodological quality in clinical trials and bibliometric indicators: no evidence of correlations

Citation frequencies and journal impact factors (JIFs) are being used more and more to assess the quality of research and allocate research resources. If these bibliometric indicators are not an adequate predictor of research quality, there could be severe negative consequences for research. To analyse to which extent citation frequencies and journal impact factors correlate with the methodological quality of clinical research articles included in an SBU systematic review of antibiotic prophylaxis in surgery. All 212 eligible original articles were extracted from the SBU systematic review “Antibiotic Prophylaxis in Surgery” and categorized according to their methodological rigourness as high, moderate or low quality articles. Median of citation frequencies and JIFs were compared between the methodological quality groups using Kruskal–Wallis non-parametric test. An in-depth study of low-quality studies with higher citation frequencies/JIFs was also conducted. No significant differences were found in median citation frequencies (p = 0.453) or JIFs (p = 0.185) between the three quality groups. Studies that had high citation frequencies/JIFs but were assessed as low-quality lacked control groups, had high dropout rates or low internal validity. This study of antibiotic prophylaxis in surgery does not support the hypothesis that bibliometric indicators are a valid instrument for assessing methodological quality in clinical trials. This is a worrying observation, since bibliometric indicators have a major influence on research funding. However, further studies in other areas are needed.     

A bibliometric analysis of NOAA’s Office of Ocean Exploration and Research

Bibliometric analysis techniques are increasingly being used to analyze and evaluate scientific research produced by institutions and grant funding agencies. This article uses bibliometric methods to analyze journal articles funded by NOAA’s Office of Ocean Exploration and Research (OER), an extramural grant-funding agency focused on the scientific exploration of the world’s oceans. OER-supported articles in this analysis were identified through grant reports, personal communication, and acknowledgement of OER support or grant numbers. The articles identified were analyzed to determine the number of publications and citations received per year, subject, and institution. The productivity and citation impact of institutions in the US receiving OER grant funding were mapped geographically. Word co-occurrence and bibliographic coupling networks were created and visualized to identify the research topics of OER-supported articles. Finally, article citation counts were evaluated by means of percentile ranks. This article demonstrates that bibliometric analysis can be useful for summarizing and evaluating the research performance of a grant funding agency.     

Assessing non-standard article impact using F1000 labels

Faculty of 1000 (F1000) is a post-publishing peer review web site where experts evaluate and rate biomedical publications. F1000 reviewers also assign labels to each paper from a standard list or article types. This research examines the relationship between article types, citation counts and F1000 article factors (FFa). For this purpose, a random sample of F1000 medical articles from the years 2007 and 2008 were studied. In seven out of the nine cases, there were no significant differences between the article types in terms of citation counts and FFa scores. Nevertheless, citation counts and FFa scores were significantly different for two article types: “New finding” and “Changes clinical practice”: FFa scores value the appropriateness of medical research for clinical practice and “New finding” articles are more highly cited. It seems that highlighting key features of medical articles alongside ratings by Faculty members of F1000 could help to reveal the hidden value of some medical papers.     

Do lead articles signal higher quality in the digital age? Evidence from finance journals

Citations are regarded as measures of quality yet citation rates vary widely within each of the top finance journals. Since article ordering is at the discretion of editors, lead articles can be interpreted as signals of quality that academics can use to allocate their attention and assert the value of their publications. Advances in electronic journal access allow researchers to directly access articles, suggesting article ordering may be less relevant today. We confirm the past importance of lead articles by examining citation rates from published papers as well as the wider source of papers that are listed in Google Scholar. Our findings also confirm using Google Scholar as a citation source provides congruent results to using citations from articles published in ISI-listed journals, with the additional benefit of it potentially being more timely since it includes wider citation sources, inclusive of working and conference papers.     

Does the arXiv lead to higher citations and reduced publisher downloads for mathematics articles?

An analysis of 2, 765 articles published in four math journals from 1997 to 2005 indicate that articles deposited in the arXiv received 35% more citations on average than non-deposited articles (an advantage of about 1.1 citations per article), and that this difference was most pronounced for highly-cited articles. Open Access, Early View, and Quality Differential were examined as three non-exclusive postulates for explaining the citation advantage. There was little support for a universal Open Access explanation, and no empirical support for Early View. There was some inferential support for a Quality Differential brought about by more highly-citable articles being deposited in the arXiv. In spite of their citation advantage, arXiv-deposited articles received 23% fewer downloads from the publisher’s website (about 10 fewer downloads per article) in all but the most recent two years after publication. The data suggest that arXiv and the publisher’s website may be fulfilling distinct functional needs of the reader.     

A bibliometric analysis on rural studies in human geography and related disciplines

Although the world has experienced rapid urbanization, rural areas have always been and are still an important research field in human geography. This paper performed a bibliometric analysis on rural geography studies based on the peer-reviewed articles concerning rural geography published in the SSCI-listed journals from 1990 to 2012. Our analysis examines publication patterns (document types and publishing languages, article outputs and their categories, major journals and their publication, most productive authors, geographic distribution and international collaboration) and demonstrates the evolution of intellectual development of rural geography by studying highly cited papers and their citation networks and temporal evolution of keywords. Our research findings include: The article number has been increasing since the 1900s, and went through three phases, and the rural geography research is dominated in size by UK and USA. The USA is the most productive in rural geography, but the UK had more impact than other countries in the terms of the average citation of articles. Three distinct but loosely linked research streams of rural geography were identified and predominated by the UK rural geographers. The keywords frequencies evolved according to contexts of rural development and academic advances of human geography, but they were loosely and scattered since the rural researches in different regions or different systems faced with different problems.     

Analysis of the influence of the two types of the journal articles; theoretical and empirical on the impact factor of a journal

The study discusses the necessity to analyze the influence of theoretical and empirical types of journal articles on the citation impact of Spanish psychology journals. Three of the most representative Spanish psychology journals were selected for the purposes of this study: Papeles del Psicólogo, Análisis y Modificación de Conducta and Psicothema. Twenty-three psychology journals in Spanish were used as source journals. Altogether, there were sixty-seven issues reviewed for the references and ninety-three issues for the articles. The bibliometricanalysis was conducted by six highly trained psychologists. The results demonstrated differences regarding the percentages of empirical and theoretical articles published in the three examined journals and the number of citations received by them based on the article type. When normalizing the results according to the number of theoretical and empirical articles that were published, it becomes evident that the theoretical articles receive on average twice as many references as the empirical ones. We discuss the importance of this effect on the comparison of journals based on their citation impact and show the evidence that it is only valid to compare journals which publish a similar percentage of theoretical and empirical articles.     

EM-index: a new measure to evaluate the scientific impact of scientists

During the last decade, several scientometrics as well as bibliometrics indices were proposed to quantify the scientific impact of individual. The h-index gives a breakthrough in scientific evaluation, but this index suffers with big hit problem, i.e., once a paper is selected in h-core publication, further citation of h-core articles is not considered in scientific evaluation. To overcome this limitation of h-index, the e-index was proposed, but it does not consider the core citation count. It considers only the excess citation count. To overcome this limitation, the EM-index is proposed in this article. The EM-index is the extension of h-index and e-index, which uses the concept of multidimensional h-index. The EM-index uses all citation counts of h-core articles at multi-level to quantify the scientific impact of the individual. But this index does not consider all publication citations. To overcome this limitation of EM-index a multidimensional extension of the EM-index is also proposed called EM′-index. To validate the proposed indicators, an experimental analysis has been done on 82 scientist’s publication citation count, who are working in scientiometrics field. In such a way, we found a more balanced and fine-grained approach to evaluate the scientific impact of individual as well as to compare the scientific impact of two different researchers/scientists.

     

Can scientific impact be judged prospectively? A bibliometric test of Simonton"s model of creative productivity

Simonton"s (1997) model of creative productivity, based on a blind variation-selection process, predicts scientific impact can only be evaluated retrospectively, after recognition has been achieved. We test this hypothesis using bibliometric data from the Human Factors journal, which gives an award for the best paper published each year. If Simonton"s model is correct, award winning papers would not be cited much more frequently than non-award winning papers, showing that scientific success cannot be judged prospectively. The results generally confirm Simonton"s model. Receipt of the award increases the citation rate of articles, but accounts for only 0.8% to 1.2% of the variance in the citation rate. Consistent with Simonton"s model, the influence of the award on citation rate may reflect a selection process of an elite group of reviewers who are representative of the larger peer group that eventually determines the citation rate of the article. Consistent with Simonton"s model, author productivity accounts for far more variance in the authors" total citation rate (58.9%) and in the citation rate of the authors" most cited article (12.6%) than does award receipt. This revised version was published online in August 2006 with corrections to the Cover Date.     

Does the human capital of teams of natural science authors predict citation frequency?

This study examines the relationship between citation frequency and the human capital of teams of authors. Analysis of a random sample of articles published in top natural science journals shows that articles co-authored by teams including frequently cited scholars and teams whose members have diverse disciplinary backgrounds have greater citation frequency. The institutional prestige, the percentage of team members at U. S. institutions and the variety of disciplines represented by team member backgrounds do not influence citation frequency. The study introduces a method for evaluating the extent of multidisciplinarity that accounts for the relatedness of disciplines or authors.