The big challenge of Scientometrics 2.0: exploring the broader impact of scientific research in public health

In the present study we discuss the challenge of “Scientometrics 2.0” as introduced by Priem and Hemminger (2010) in the light of possible applications to research evaluation. We use the Web of Science subject category public, environmental and occupational health to illustrate how indicators similar to those used in traditional scientometrics can be built, and we also discuss their opportunities and limitations. The discipline under study combines life sciences and social sciences in a unique manner and provides usable metrics reflecting both scholarly and wider impact. Nonetheless, metrics reflecting social media attention like tweets, retweets and Facebook likes, shares or comments are still subject to limitations in this research discipline as well. Furthermore, Usage metrics clearly point to the manipulation proneness of this measure. Although the counterparts of important bibliometric indicators proved to work for several altmetrics too, their interpretation and application to research assessment requires proper context analysis.

     

Future publication success in science is better predicted by traditional measures than by the <Emphasis Type="Italic">h</Emphasis> index

Although the use of bibliometric indicators for evaluations in science is becoming more and more ubiquitous, little is known about how future publication success can be predicted from past publication success. Here, we investigated how the post-2000 publication success of 85 researchers in oncology could be predicted from their previous publication record. Our main findings are: (i) Rates of past achievement were better predictors than measures of cumulative achievement. (ii) A combination of authors’ past productivity and the past citation rate of their average paper was most successful in predicting future publication success (R ² ≈ 0.60). (iii) This combination of traditional bibliographic indicators clearly outperformed predictions based on the rate of the h index (R ² between 0.37 and 0.52). We discuss implications of our findings for views on creativity and for science evaluation.     

The difficulty of achieving full coverage of international social science literature and the bibliometric consequences

This review of social science bibliometric literature seeks to establish characteristics of the social science literature and to understand their consequences for the coverage of literature databases and for interpretation of bibliometric social science indicators based on such databases. The paper reviews what we know about social science publishing and database coverage of it. It examines the main reasons why social science bibliometrics are problematic, namely: the centrality of books in social science literature and their high citation rate; and the national orientation of social science literatures. The paper then looks at reasons why social science bibliometrics holds increasing promise, namely: increasing internationalization; and good coverage of scholarly journals.     

Using ego-network analyses to examine journal citations: a comparative study of public administration,political science,and business management

This study demonstrates how social network analysis, specifically ego-network analysis, can be used as a methodology to investigate the degrees of the isolation, insularity, and prestige of academic journals in a particular field in relation to other fields. Using the example of journals in public administration for illustrative purposes, we analyze the network of connections between journals in this field and the ones in two related fields: political science and business/management. Earlier studies found that the research in public administration was largely isolated from the three disciplines that were its foundations: law, management, and political science. Using ego-network analyses and ratio calculations from the Web of Science citation data, we found that the top journals of public administration were indeed relatively isolated in the years 2005, 2010, and 2015. Of the public administration journals, Public Administration Review (PAR) and the Journal of Public Administration Research and Theory (JPART) were increasingly more recognized (less isolated and became more prestigious) among the journals in other fields. This methodology can be applied to the journals in other fields to study the relations between fields, particularly the relative academic prestige of the fields, in terms of journal citations.

     

Impact of bibliometrics upon the science system: Inadvertent consequences?

Summary Ranking of research institutions by bibliometric methods is an improper tool for research performance evaluation, even at the level of large institutions. The problem, however, is not the ranking as such. The indicators used for ranking are often not advanced enough, and this situation is part of the broader problem of the application of insufficiently developed bibliometric indicators used by persons who do not have clear competence and experience in the field of quantitative studies of science. After a brief overview of the basic elements of bibliometric analysis, I discuss the major technical and methodological problems in the application of publication and citation data in the context of evaluation. Then I contend that the core of the problem lies not necessarily at the side of the data producer. Quite often persons responsible for research performance evaluation, for instance scientists themselves in their role as head of institutions and departments, science administrators at the government level and other policy makers show an attitude that encourages &apos;quick and dirty&apos; bibliometric analyses whereas better quality is available. Finally, the necessary conditions for a successful application of advanced bibliometric indicators as support tool for peer review are discussed.     

Which role do non-source items play in the social sciences? A case study in political science in Germany

Publications that are not indexed by citation indices such as Web of Science (WoS) or Scopus are called “non-source items”. These have so far been neglected by most bibliometric analyses. The central issue of this study is to investigate the characteristics of non-source items and the effect of their inclusion in bibliometric evaluations in the social sciences, specifically German political science publications. The results of this study show that non-source items significantly increase the number of publications (+1,350 %) and to a lesser extent the number of citations from SCIE, SSCI, and A&HCI (+150 %) for evaluated political scientists. 42 % of non-source items are published as book chapters. Edited books and books are cited the most among non-source items. About 40 % of non-source items are in English, while 80 % of source items are in English. The citation rates of researchers taking non-source items into account are lower than those from source items, partially as a result of the limited coverage of WoS. In contrast, the H-indices of researchers taking only non-source items into account are higher than those from source items. In short, the results of this study show that non-source items should be included in bibliometric evaluations, regardless of their impact or the citations from them. The demand for a more comprehensive coverage of bibliometric database in the social sciences for a higher quality of evaluations is shown.     

Challenges in the study of Cuban scientific output

Cuban scientific output at macro level has not been frequently studied in the literature on scientometrics. The current paper explores the different metric approaches to the Cuban scientific activity carried out by national and international authors. Also, the article develops a scientometric study of the Cuban scientific production as included in Scopus during the period 1996–2007, using socio-economic indicators combined with bibliometric indicators supported by the SCImago Journal & Country Rank. Web of Science and Scopus are compared as information sources. Results confirm the possibility to use Scopus to obtain an objective picture of the Cuban science behaviour during the end of the 1990s and the beginning of the XXI century. The SCImago Journal & Country Rank, in this case, offers an important set of indicators. The combination of these indicators with those related to socio-economic aspects of activities in Science and Technology, allow the authors to show a perspective of the Cuban science system evolution during the period analyzed. The inclusion in Scopus of less-cited journals published in Spanish language and its impact on productivity and citation-based indicators is also discussed. Our investigation found an increasing growth of the Cuban scientific production during the whole period, which is in correspondence to the country efforts and expenditures in Research and Development activities.     

Trends in Russian research output in post-Soviet era

Recently, the Russian government has ordered evaluation and reform of the basic research system. As a consequence, the number of research staff at the Russian Academy of Sciences will be reduced by 20% by 2007. The basis for research evaluation and institute budgeting will be bibliometric indicators. In view of these changes we look at the Russian publication output and argue that

Career advancement and scientific performance in universities

Many governments have placed priority on excellence in higher education as part of their policy agendas. Processes for recruitment and career advancement in universities thus have a critical role. The efficiency of faculty selection processes can be evaluated by comparing the subsequent performance of competition winners against that of the losers and the pre-existing staff of equal academic rank. Our study presents an empirical analysis concerning the recruitment procedures for associate professors in the Italian university system. The results of a bibliometric analysis of the hard science areas reveal that new associate professors are on average more productive than the incumbents. However a number of crucial concerns emerge, in particular concerning occurrence of non-winner candidates that are more productive than the winners over the subsequent triennium, and cases of winners that are completely unproductive. Beyond the implications for the Italian case, the analysis offers considerations for all decision-makers regarding the ex post evaluation of the efficiency of the recruitment process and the desirability of providing selection committees with bibliometric indicators in support of evaluation (i.e. informed peer review).     

The politics of strategic environmental assessment indicators: weak recognition found in Chinese guidelines

The choice and use of indicators is not only technical and science-led, but also a value-laden social process, and thus concerns public participation and political judgement. This article approaches the Chinese strategic environmental assessment (SEA) indicator system from a science–policy interface perspective and aims to: (1) contribute to the general recognition of indicators functioning in SEA; and (2) analyse, through a Chinese case study, to what extent national guidelines address this science–policy interaction. The overall finding is a strong emphasis on technical/science aspects in the Chinese SEA guidance, and a weak explicit recognition that policy plays a role in choosing and using indicators. Recent development, however, indicates a growing recognition of the politics involved and thus also leads to greater involvement of stakeholders.     

Some Thoughts on the Interaction between Scientometrics and Science and Technology Policy

In 1988 Le Pair postulated the existence of a citation gap for technological research. Several cases were studied, which confirmed his hypothesis. In the same period the use of bibliometric indicators for policy purposes increased. Here we saw the citation gap causing a disadvantage for application-oriented research groups. This is not merely an injustice, it also leads to suboptimum use of available funds, to the detriment of science as a whole. In addition, it may, in the long term, undermine the reputation of scientometrics as a science in its own right.     

Specialization and size of scientific activities: A bibliometric analysis of advanced countries

The relationship between the size of national scientific activities of advanced countries and the degree of specialization by fields of science is examined using bibliometric indicators of the number of papers and of paper citations. A negative relation between the amount of scientific activity and the degree of scientific specialization has emerged, with Japan and, to a lesser extent Italy, showing a specialization degree higher than expected. Countries with established scientific traditions (such as the US, the UK, the Netherlands, and Switzerland) have a lower than expected specialization degree, suggesting a more diversified range of research activities. Over time, however, most countries have reduced their scientific specialization, a pattern which is in contrast with recent research on patents and technological specialization.Rotating first authorship. This paper is part of a research on the Scientific and Technological Specialization of advanced countries jointly financed by the Commission of European Communities, D.G. XII, Science, Research and Development, Service Research Evaluation, and the Italian National Research Council. We are grateful to PatriziaPrincipessa and RobertoSimonetti for research assistance.     

An impact indicator for researchers

The assessment of individual researchers using bibliometric indicators is more complex than that of a region, country or university. For large scientific bodies, averages over a large number of researchers and their outputs is generally believed to give indication of the quality of the research work. For an individual, the detailed peer evaluation of his research outputs is required and, even this, may fail in the short term to make a final, long term assessment of the relevance and originality of the work. Scientometrics assessment at individual level is not an easy task not only due to the smaller number of publications that are being evaluated, but other factors can influence significantly the bibliometric indicators applied. Citation practices vary widely among disciplines and sub disciplines and this may justify the lack of good bibliometric indicators at individual level. The main goal of this study was to develop an indicator that considers in its calculation some of the aspects that we must take into account on the assessment of scientific performance at individual level. The indicator developed, the h _nf index, considers the different cultures of citation of each field and the number of authors per publication. The results showed that the h _nf index can be used on the assessment of scientific performance of individual researchers and for following the performance of a researcher.     

Mimicry in science?

Since bibliometric indicators have obtained a general acceptance in science policy and attained applied relevance in research evaluation, feedback effects on scientists?? behaviour resulting from the use of these indicators for science funding decisions have been reported. These adaptation strategies could be called mimicry in science. Scientists apply strategies that should enable them to comply to bibliometric accountability and to secure funds to their own research.     

Any publicity is better than none: newspaper coverage increases citations, in the UK more than in Italy

A citation advantage for research covered by the mass media is a plausible, but poorly studied phenomenon. Two previous studies, both conducted in the United States, found a positive correlation between media reporting and citations. Only one of these studies was able to conclude that the correlation was caused by a real “publicity effect” rather than by the media highlighting papers that are intrinsically destined to have greater scientific impact (called the ‘earmark’ hypothesis). This study assessed the relative importance of the publicity effect outside the US, by comparing studies published in 2008 and 2009 in the Proceedings of the National Academy of Sciences that had been featured in newspapers in Italy and the United Kingdom. Newspapers in the two countries covered a similar range of topics, and tended to over-represent local (national) research. Compared to studies not appearing in any of the newspapers considered, those featured in British newspapers had around 63 % more citations, whilst in Italian newspapers 16 %. The proportion of citations from Italian authors, however, was significantly increased by newspapers, particularly by those in Italian. The equivalent effect on citations from the UK was smaller and only marginally significant. Studies accompanied by a press release did not receive, overall, significantly more citations. In sum, results suggest that the publicity effect is strongest for English-speaking media, whilst non-English reporting has mostly a local influence. These effects might represent a confounding factor in citation-based research assessment and might contribute to the many biases known to affect the scientific literature.     

Testing bibliometric indicators by their prediction of scientists promotions

We have developed a method to obtain robust quantitative bibliometric indicators for several thousand scientists. This allows us to study the dependence of bibliometric indicators (such as number of publications, number of citations, Hirsch index...) on the age, position, etc. of CNRS scientists. Our data suggests that the normalized h-index (h divided by the career length) is not constant for scientists with the same productivity but different ages. We also compare the predictions of several bibliometric indicators on the promotions of about 600 CNRS researchers. Contrary to previous publications, our study encompasses most disciplines, and shows that no single indicator is the best predictor for all disciplines. Overall, however, the Hirsch index h provides the least bad correlations, followed by the number of papers published. It is important to realize however that even h is able to recover only half of the actual promotions. The number of citations or the mean number of citations per paper are definitely not good predictors of promotion. Due to space constraints, this paper is a short version of a more detailed article. [JENSEN & AL., 2008B]     

Correcting glasses help fair comparisons in international science landscape: Country indicators as a function of ISI database delineation

The increasing use of bibliometric indicators in science policy calls for a reassessment of their robustness and limits. The perimeter of journal inclusion within ISI databases will determine variations in the classic bibliometric indicators used for international comparison, such as world shares of publications or relative impacts. We show in this article that when this perimeter is adjusted using a natural criterion for inclusion of journals, the journal impact, the variation of the most common country indicators (publication and citation shares; relative impacts) with the perimeter chosen depends on two phenomena. The first one is a bibliometric regularity rooted in the main features of competition in the open space of science, that can be modeled by bibliometric laws, the parameters of which are “coverage-independent” indicators. But this regularity is obscured for many countries by a second phenomenon, the presence of a sub-population of journals that does not reflect the same international openness, the nationally-oriented journals. As a result indicators based on standard SCI or SCISearch perimeters are jeopardized to a certain extent by this sub-population which creates large irregularities. These irregularities often lead to an over-estimation of share and an under-estimation of the impact, for countries with national editorial tradition, while the impact of a few mainstream countries arguably benefits from the presence of this sub-population. This revised version was published online in August 2006 with corrections to the Cover Date.     

Do journals flipping to gold open access show an OA citation or publication advantage?

The effects of Open Access (OA) upon journal performance are investigated. The key research question holds: How does the citation impact and publication output of journals switching (“flipping”) from non-OA to Gold-OA develop after their switch to Gold-OA? A review is given of the literature, with an emphasis on studies dealing with flipping journals. Two study sets with 119 and 100 flipping journals, derived from two different OA data sources (DOAJ and OAD), are compared with two control groups, one based on a standard bibliometric criterion, and a second controlling for a journal’s national orientation. Comparing post-switch indicators with pre-switch ones in paired T-tests, evidence was obtained of an OA Citation advantage but not of an OA Publication Advantage. Shifts in the affiliation countries of publishing and citing authors are characterized in terms of countries’ income class and geographical world region. Suggestions are made for qualitative follow-up studies to obtain more insight into OA flipping or reverse-flipping.

     

Has Price's dream come true: Is scientometrics a hard science?

At the occasion of the completion of the 25th volume ofScientometrics, we present a combined bibliometric and social network analysis of this journal. In more than one respect,Scientometrics displays the characteristics of a social science journal. Its Price Index amounts to 43.0 percent, and is remarkably stable over time. The majority of the published items inScientometrics has been written by a single author. Moreover, the network of co-authorships is highly fragmented: most authors cooperate with no more than one or two colleagues. Both the citation networks of the authors and the network of title words indicate that the field is nonetheless highly cohesive. In this sense, a specific identity seems to have developed, indeed. Some indications concerning the character of this identity are discussed.