National research assessment exercises: a comparison of peer review and bibliometrics rankings

Development of bibliometric techniques has reached such a level as to suggest their integration or total substitution for classic peer review in the national research assessment exercises, as far as the hard sciences are concerned. In this work we compare rankings lists of universities captured by the first Italian evaluation exercise, through peer review, with the results of bibliometric simulations. The comparison shows the great differences between peer review and bibliometric rankings for excellence and productivity.     

National research assessment exercises: the effects of changing the rules of the game during the game

National research evaluation exercises provide a comparative measure of research performance of the nation’s institutions, and as such represent a tool for stimulating research productivity, particularly if the results are used to inform selective funding by government. While a school of thought welcomes frequent changes in evaluation criteria in order to prevent the subjects evaluated from adopting opportunistic behaviors, it is evident that the “rules of the game” should above all be functional towards policy objectives, and therefore be known with adequate forewarning prior to the evaluation period. Otherwise, the risk is that policy-makers will find themselves faced by a dilemma: should they reward universities that responded best to the criteria in effect at the outset of the observation period or those that result as best according to rules that emerged during or after the observation period? This study verifies if and to what extent some universities are penalized instead of rewarded for good behavior, in pursuit of the objectives of the “known” rules of the game, by comparing the research performances of Italian universities for the period of the nation’s next evaluation exercise (2004–2008): first as measured according to criteria available at the outset of the period and next according to those announced at the end of the period.     

National research assessment exercises: a measure of the distortion of performance rankings when labor input is treated as uniform

Measuring the efficiency of scientific research activity presents critical methodological aspects, many of which have not been sufficiently studied. Although many studies have assessed the relation between quality and research productivity and academic rank, not much is known about the extent of distortion in national university performance rankings when academic rank and the other labor factors are not considered as a factor of normalization. This work presents a comparative analysis that aims to quantify the sensitivity of bibliometric rankings to the choice of input, with input considered as only the number of researchers on staff, or alternatively where their cost is also considered. The field of observation consists of all 69 Italian universities active in the hard sciences. Performance measures are based on the 81,000 publications produced during the 2004–2006 triennium by all 34,000 research staff, with analysis carried out at the level of individual disciplines, 187 in total. The effect of the switch from labor to cost seems to be minimal except for a few outliers.     

The impact of basic research in the social sciences: The case of education

Recent expert evaluations of the condition and funding of educational research in the USA assume that basic research in the social sciences is a crucial factor in increasing our understanding and ultimately improving the practice of education (see, for example, the 1977 report¹ of the National Academy of Sciences' Committee on Fundamental Research Relevant to Education). Past tests of this assumption, however, have generally relied upon argument by example or anecdote. In the present study, we analyze citation patterns in the education literature to test the corollary proposition that basic research in the social sciences has had a substantial impact on the literature in education. Empirical data collected on citation patterns in the education literature are found to be largely consistent with this proposition.     

How 'do's' and 'dont's' can be of significant importance in radiation protection: a case report

The Norwegian Radiation Protection Authority (NRPA) was contacted by a cardiology department because of a suspicious radiation burn on a patient. The patient had undergone two bi-ventricular pacemaker implantations and the lesion was recognised as radiation dermatitis. The NRPA sent sets of thermoluminescense dosemeters to the department to measure the skin dose on eight successive patients undergoing bi-ventricular pacemaker implantations. The average maximum entrance surface dose for the eight patients was 5.3 Gy, ranging from 2.0 to 13.1 Gy. A site audit was performed during a procedure, with the aim to observe the general skills in radiation protection. Based on the findings in the audit, a few very simple 'Do's' and 'Don'ts' were highlighted in a meeting directly after the audit. After the audit, dose measurements were performed on six new patients showing an average maximum entrance surface dose of 0.4 Gy, ranging from 0.2 to 0.8 Gy. The aim of this work was to illustrate the dose reductions that are possible to achieve with a very few basic advices, especially when the operator has suboptimal competence.     

Interventions targeting non-symptomatic cases can be important to prevent local outbreaks: SARS-CoV-2 as a case study

During infectious disease epidemics, an important question is whether cases travelling to new locations will trigger local outbreaks. The risk of this occurring depends on the transmissibility of the pathogen, the susceptibility of the host population and, crucially, the effectiveness of surveillance in detecting cases and preventing onward spread. For many pathogens, transmission from pre-symptomatic and/or asymptomatic (together referred to as non-symptomatic) infectious hosts can occur, making effective surveillance challenging. Here, by using SARS-CoV-2 as a case study, we show how the risk of local outbreaks can be assessed when non-symptomatic transmission can occur. We construct a branching process model that includes non-symptomatic transmission and explore the effects of interventions targeting non-symptomatic or symptomatic hosts when surveillance resources are limited. We consider whether the greatest reductions in local outbreak risks are achieved by increasing surveillance and control targeting non-symptomatic or symptomatic cases, or a combination of both. We find that seeking to increase surveillance of symptomatic hosts alone is typically not the optimal strategy for reducing outbreak risks. Adopting a strategy that combines an enhancement of surveillance of symptomatic cases with efforts to find and isolate non-symptomatic infected hosts leads to the largest reduction in the probability that imported cases will initiate a local outbreak.     

Adoption readiness in service innovation: the case of digital money

One challenge for firms seeking to develop new services is to understand the conditions likely to affect the rates of adoption. Understanding relative degrees of ‘adoption readiness’ provides innovators with information to choose market segments and indicates opportunities to influence adoption environments. However, there is a little research into the adoption readiness of countries upon which firms can base their new service development decisions. This paper considers these issues through the case of digital money, a service innovation some claim to have the potential to provide major economic and social benefits. Defined as ‘currency exchange by electronic means’, we conceptualise digital money as a socio-technical system, and propose a Digital Money Readiness Index. This composite index integrates institutional, financial, technological, economic, industrial and social attributes to measure adoption readiness. We identify four stages of readiness and systematically analyse the factors that drive under or over adoption of digital money technologies.     

Mapping the research landscape of Covid-19 from social sciences perspective: a bibliometric analysis

Scientometrics - COVID-19 has emerged as a widely researched topic and the academia has taken interest in the effects of COVID-19 in various sectors of human life and society. Most of the...     

Research quality and diversity of funding: A model for relating research money to output of research

We analyze the relation between funding and output using bibliometric methods with field normalized data. Our approach is to connect individual researcher data on funding from Swedish university databases to data on incoming grants using the specific personal ID-number. Data on funding include the person responsible for the grant. All types of research income are considered in the analysis yielding a project database with a high level of precision. Results show that productivity can be explained by background variables, but that quality of research is more or less un-related to background variables.     

Indicators for the dynamics of research organizations: a biomedical case study

This paper reports results on a bibliometric case study of the long-term development of research organizations, using an internationally leading biomedical institute as example. Using scientometric concepts, small group theory, organizational ecology, and process-based organizational theory, we developed a life cycle based theoretical model for analyzing long-term development of research groups and institutes. Three bibliometric indicators are proposed for growth, activity profile stability, and focus. With these, the research dynamics of the case institute are described. First, overall output growth matches developments internationally in developmental biology and stem cell research, and, in line with this, journal article output increasingly dominates the institute’s activity profile. Second, superposed on the overall growth curve, a stepwise development is observed, consisting of long phases of growth and stabilisation. These steps reflect local conditions and events. Historical sources from the Institutes’ archive and interviews with the current staff of the institute suggest that the pattern of life cycles reflects a strong influence of pioneering individuals. But once settled, pioneering directors who remain in function for many years delay adaptation of the institutes’ mission to field developments. Furthermore, national science policies on PhD training, and on priority areas have influenced the life cycles, as did merging with other institutes. As in a social science case, also in this case study stabilized local conditions lead to adaptation to research field dynamics in a delayed fashion. In the present case stable output periods lasted at most 15 years, when local impulses led to new growth of research output and thus prevented onset of a lifecycle decline. The continued growth in the larger field both promoted and legitimized these local impulses.     

Test case of the long term preliminary performance assessment for the L&IL radioactive waste repository Baita Bihor, Romania

The paper presents the assessment methodology developed in the framework of the long-term preliminary performance assessment of Baita Bihor Romanian national low-level and intermediate-level radioactive waste repository. Conservative assumptions, input parameters and tools used in this approach reflect limitations imposed by the current assessment capability, as well as limited knowledge and understanding of the disposal system. The resultant dose rates are presented and analysed. Conclusions are drawn relating to the safety of the repository.     

On the use of journal classification in social sciences and humanities: evidence from an Italian database

Scientometrics - In social sciences and humanities, a two-tier journal classification is currently used in Italy in the context of the National Habilitation programme; peer review is also available...     

Scientometric impact assessment of a research policy instrument: the case of rating researchers on scientific outputs in South Africa

The influence of the National Research Foundation’s (NRF) rating system on the productivity of the South African social science researchers is investigated scientometrically for the period from 1981 to 2006. Their output performance is mainly indicated by their research publications. Following international best practice in scientometrics as well as the behavioural reinforcement theory, we employed the “before/after control impact (BACI) method”, as well as the well known econometric breakpoint test as proposed by Chow. We use as control group the publications in the field of clinical medicine. The field is not supported by NRF and hence clinical medicine researchers are not affected by the evaluation and rating system. The findings show a positive impact of the NRF programme on the research outputs of social sciences researchers and the implementation of the programme has increased the relevant population of research articles by an average of 24.5% (during the first 5 years) over the expected number of publication without the programme. The results confirm the scientometric findings of other studies (e.g. that of Nederhof) that ratings promulgate research productivity.     

Reviewer recommendation method for scientific research proposals: a case for NSFC

Scientometrics - Peer review is one of the important procedures to determine which research proposals are to be funded and to evaluate the quality of scientific research. How to find suitable...     

Measuring institutional research productivity for the life sciences: the importance of accounting for the order of authors in the byline

Accurate measurement of institutional research productivity should account for the real contribution of the research staff to the output produced in collaboration with other organizations. In the framework of bibliometric measurement, this implies accounting for both the number of co-authors and each individual’s real contribution to scientific publications. Common practice in the life sciences is to indicate such contribution through the order of author names in the byline. In this work, we measure the distortion introduced to university-level bibliometric productivity rankings when the number of co-authors or their position in the byline is ignored. The field of observation consists of all Italian universities active in the life sciences (Biology and Medicine). The analysis is based on the research output of the university staff over the period 2004–2008. Based on the results, we recommend against the use of bibliometric indicators that ignore co-authorship and real contribution of each author to research outputs.     

Size effects in the assessment of discipline-contribution scores: An example from the social sciences

Pichappan's formulation of the Discipline-Contribution Score (DCS) in the identification of core disciplinary journals is applied to social science literature. Modifications are made to Pichappan's method to cater for problems associated with low citation counts. Further analysis is undertaken to verify Pichappan's claims concerning the size-independent nature of the DCS score. The proposed modified formulation of the DCS calculation facilitates research into small research fields, and those characterised by low citation rates. The modified equation is tested on business and management literature.     

Profile of the Mexican health sciences elite: A bibliometric analysis of research performance

The most prestigious award in Mexico, the “National Prize for Science and Art” has been awarded to 33 health scientists. An exercise was carried out to assess their performance to answer the question: why them? The laureates' profile was based on data retrieved from MEDLINE andScience Citation Index Expanded available on the WWW as well as the ISI's 15-year (1981–1995) cumulative impact factor lists. The laureates published 2,049 papers and were cited 50,834 times. Our results showed the scientific pre-eminence of laureates. We concluded that bibliometric data could complement other indicators of research performance. Bibliometrics could insure the Prize committee against error and the operationalization of the Matthew Effect could be minimized to honor only the most creative researchers.