The dangers of performance-based research funding in non-competitive higher education systems

An increasing number of nations allocate public funds to research institutions on the basis of rankings obtained from national evaluation exercises. Therefore, in non-competitive higher education systems where top scientists are dispersed among all the universities, rather than concentrated among a few, there is a high risk of penalizing those top scientists who work in lower-performance universities. Using a 5 year bibliometric analysis conducted on all Italian universities active in the hard sciences from 2004 to 2008, this work analyzes the distribution of publications and relevant citations by scientists within the universities, measures the research performance of individual scientists, quantifies the intensity of concentration of top scientists at each university, provides performance rankings for the universities, and indicates the effects of selective funding on the top scientists of low-ranked universities.     

The pursuit of academic excellence and business engagement: is it irreconcilable?

Universities currently need to satisfy the demands of different audiences. In light of the increasing policy emphasis on “third mission” activities, universities are attempting to incorporate these into their traditional missions of teaching and research. University strategies to accomplishing its traditional missions are well-honed and routinized, but the incorporation of the third mission is posing important strategic and managerial challenges for universities. This study explores the relationship between university–business collaborations and academic excellence in order to examine the extent to which academic institutions can balance these objectives. Based on data from the UK Research Assessment Exercise 2001 at the level of the university department, we find no systematic positive or negative relationship between scientific excellence and engagement with industry. Across the disciplinary fields reported in the 2001 Research Assessment Exercise (i.e. engineering, hard sciences, biomedicine, social sciences and the humanities) the relationship between academic excellence and engagement with business is largely contingent on the institutional context of the university department. This paper adds to the growing body of literature on university engagement with business by examining this activity for the social sciences and the humanities. Our findings have important implications for the strategic management of university departments and for higher education policy related to measuring the performance of higher education research institutions.     

Where do Italian universities stand? An in-depth statistical analysis of national and international rankings

In a previous article (Degli Esposti and Geraci. Bulletin of Italian Politics, 2011), we presented an historical survey of the university reform laws that took place in Italy in the last 30 years. On that occasion, we stressed how important is merit evaluation for academics and their institutions, especially in view of the much debated but not yet implemented ‘Gelmini’ reform with its long awaited new regulation for accessing academic positions (concorsi) and for determining individual weight in financial resource allocation among universities. Here, we present and compare several rankings used to evaluate the prestige and merit of Italian universities. We also consider alternative approaches to academic rankings that highlight peculiar aspects of the universities in Italy which cannot be reasonably accounted for by other international rankings. Finally, we propose a new approach that combines both national and international standing of Italian universities. It is hoped that this study will provide practical guidance to policy makers for establishing the criteria upon which merit should be assessed.     

What do global university rankings really measure? The search for the X factor and the X entity

Most academic rankings attempt to measure the quality of university education and research. However, previous studies that examine the most influential rankings conclude that the variables they use could be an epiphenomenon of an X factor that has little to do with quality. The aim of this study is to investigate the existence of this hidden factor or profile in the two most influential global university rankings in the world: the Academic Ranking of World Universities (ARWU) of the University of Shanghai Jiao Tong, and the Times Higher Education (THE) ranking. Results support the existence of an underlying entity profile, characterized by institutions normally from the US that enjoy a high reputation. Results also support the idea that rankings lack the capacity to assess university quality in all its complexity, and two strategies are suggested in relation to the vicious circle created between institutional reputation and rankings.     

An insight into the importance of national university rankings in an international context: the case of the I-UGR rankings of Spanish universities

The great importance international rankings have achieved in the research policy arena warns against many threats consequence of the flaws and shortcomings these tools present. One of them has to do with the inability to accurately represent national university systems as their original purpose is only to rank world-class universities. Another one has to do with the lack of representativeness of universities’ disciplinary profiles as they usually provide a unique table. Although some rankings offer a great coverage and others offer league tables by fields, no international ranking does both. In order to surpass such limitation from a research policy viewpoint, this paper analyzes the possibility of using national rankings in order to complement international rankings. For this, we analyze the Spanish university system as a study case presenting the I-UGR Rankings for Spanish universities by fields and subfields. Then, we compare their results with those obtained by the Shanghai Ranking, the QS Ranking, the Leiden Ranking and the NTU Ranking, as they all have basic common grounds which allow such comparison. We conclude that it is advisable to use national rankings in order to complement international rankings, however we observe that this must be done with certain caution as they differ on the methodology employed as well as on the construction of the fields.     

The suitability of h and g indexes for measuring the research performance of institutions

It is becoming ever more common to use bibliometric indicators to evaluate the performance of research institutions, however there is often a failure to recognize the limits and drawbacks of such indicators. Since performance measurement is aimed at supporting critical decisions by research administrators and policy makers, it is essential to carry out empirical testing of the robustness of the indicators used. In this work we examine the accuracy of the popular “h” and “g” indexes for measuring university research performance by comparing the ranking lists derived from their application to the ranking list from a third indicator that better meets the requirements for robust and reliable assessment of institutional productivity. The test population is all Italian universities in the hard sciences, observed over the period 2001–2005. The analysis quantifies the correlations between the three university rankings (by discipline) and the shifts that occur under changing indicators, to measure the distortion inherent in use of the h and g indexes and their comparative accuracy for assessing institutions.     

National research assessment exercises: a measure of the distortion of performance rankings when labor input is treated as uniform

Measuring the efficiency of scientific research activity presents critical methodological aspects, many of which have not been sufficiently studied. Although many studies have assessed the relation between quality and research productivity and academic rank, not much is known about the extent of distortion in national university performance rankings when academic rank and the other labor factors are not considered as a factor of normalization. This work presents a comparative analysis that aims to quantify the sensitivity of bibliometric rankings to the choice of input, with input considered as only the number of researchers on staff, or alternatively where their cost is also considered. The field of observation consists of all 69 Italian universities active in the hard sciences. Performance measures are based on the 81,000 publications produced during the 2004–2006 triennium by all 34,000 research staff, with analysis carried out at the level of individual disciplines, 187 in total. The effect of the switch from labor to cost seems to be minimal except for a few outliers.     

Comparative study of international academic rankings of universities

International academic rankings that compare world universities have proliferated recently. In accordance with latter conceptual and methodological advances in academic rankings approaches, five selection criteria are defined and four international university rankings are selected. A comparative analysis of the four rankings is presented taking into account both the indicators frequency and its weights. Results show that, although some indicators differ considerably across selected rankings and even many indicators are unique, indicators referred to research and scientific productivity from university academic staff have a prominent role across all approaches. The implications of obtained data for main rankings consumers are discussed.     

National peer-review research assessment exercises for the hard sciences can be a complete waste of money: the Italian case

There has been ample demonstration that bibliometrics is superior to peer-review for national research assessment exercises in the hard sciences. In this paper we examine the Italian case, taking the 2001–2003 university performance rankings list based on bibliometrics as benchmark. We compare the accuracy of the first national evaluation exercise, conducted entirely by peer-review, to other rankings lists prepared at zero cost, based on indicators indirectly linked to performance or available on the Internet. The results show that, for the hard sciences, the costs of conducting the Italian evaluation of research institutions could have been completely avoided.     

Mapping research spotlights for different regions in China

To reveal China’s regional disparity both in research output and preferential research areas is the main purpose of this study. For this study, we investigated the research outputs of all 31 regions (27 provinces and 4 municipalities) in mainland China. The investigated dataset was sourced from CNKI, one of China’s largest domestic academic databases. To measure research preferences between regions, we used the function of cosine distance rather than Euclidean distance. Clustering method was employed to classify the regions according to their similarity/disparity. In the end, six clusters were generated. Each cluster is different in research preferences. For example, Inner Mongolia in Cluster D is featured with the emphasis on animal handcraft; while Hubei province in Cluster A is characterized by a wide range of research areas.     

Benchmarking research performance at the university level with information theoretic measures

This paper presents a new method for comparing universities based on information theoretic measures. The research output of each academic institution is represented statistically by an impact-factor histogram. To this aim, for each academic institution we compute the probability of occurrence of a publication with impact factor in different intervals. Assuming the probabilities associated with a pair of academic institutions our objective is to measure the Information Gain between them. To do so, we develop an axiomatic characterization of relative information for predicting institution-institution dissimilarity. We use the Spanish university system as our scenario to test the proposed methodology for benchmarking three universities with the rest as a case study. For each case we use different scientific fields such as Information and Communication Technologies, Medicine and Pharmacy, and Economics and Business as we believe comparisons must take into account their disciplinary context. Finally we validate the Information Gain values obtained for each case with previous studies.     

Comparing university rankings

Recently there is increasing interest in university rankings. Annual rankings of world universities are published by QS for the Times Higher Education Supplement, the Shanghai Jiao Tong University, the Higher Education and Accreditation Council of Taiwan and rankings based on Web visibility by the Cybermetrics Lab at CSIC. In this paper we compare the rankings using a set of similarity measures. For the rankings that are being published for a number of years we also examine longitudinal patterns. The rankings limited to European universities are compared to the ranking of the Centre for Science and Technology Studies at Leiden University. The findings show that there are reasonable similarities between the rankings, even though each applies a different methodology. The biggest differences are between the rankings provided by the QS-Times Higher Education Supplement and the Ranking Web of the CSIC Cybermetrics Lab. The highest similarities were observed between the Taiwanese and the Leiden rankings from European universities. Overall the similarities are increased when the comparison is limited to the European universities.     

Uncovering excellence in academic rankings: a closer look at the Shanghai ranking

In this paper, we examine whether the quality of academic research can be accurately captured by a single aggregated measure such as a ranking. With Shanghai University’s Academic Ranking of World Universities as the basis for our study, we use robust principal component analysis to uncover the underlying factors measured by this ranking. Based on a sample containing the top 150 ranked universities, we find evidence that, for the majority of these institutions, the Shanghai rankings reflect not one but in fact two different and uncorrelated aspects of academic research: overall research output and top-notch researchers. Consequently, the relative weight placed upon these two factors determines to a large extent the final ranking.     

Measuring institutional research productivity for the life sciences: the importance of accounting for the order of authors in the byline

Accurate measurement of institutional research productivity should account for the real contribution of the research staff to the output produced in collaboration with other organizations. In the framework of bibliometric measurement, this implies accounting for both the number of co-authors and each individual’s real contribution to scientific publications. Common practice in the life sciences is to indicate such contribution through the order of author names in the byline. In this work, we measure the distortion introduced to university-level bibliometric productivity rankings when the number of co-authors or their position in the byline is ignored. The field of observation consists of all Italian universities active in the life sciences (Biology and Medicine). The analysis is based on the research output of the university staff over the period 2004–2008. Based on the results, we recommend against the use of bibliometric indicators that ignore co-authorship and real contribution of each author to research outputs.     

A methodology for Institution-Field ranking based on a bidimensional analysis: the <Emphasis Type="Italic">IFQ</Emphasis><Superscript>2</Superscript><Emphasis Type="Italic">A</Emphasis> index

The problem of comparing academic institutions in terms of their research production is nowadays a priority issue. This paper proposes a relative bidimensional index that takes into account both the net production and the quality of it, as an attempt to provide a comprehensive and objective way to compare the research output of different institutions in a specific field, using journal contributions and citations. The proposed index is then applied, as a case study, to rank the top Spanish universities in the fields of Chemistry and Computer Science in the period ranging from 2000 until 2009. A comparison with the top 50 universities in the ARWU rankings is also made, showing the proposed ranking is better suited to distinguish among non-elite universities.     

National research assessment exercises: a comparison of peer review and bibliometrics rankings

Development of bibliometric techniques has reached such a level as to suggest their integration or total substitution for classic peer review in the national research assessment exercises, as far as the hard sciences are concerned. In this work we compare rankings lists of universities captured by the first Italian evaluation exercise, through peer review, with the results of bibliometric simulations. The comparison shows the great differences between peer review and bibliometric rankings for excellence and productivity.     

基于异质性投入DEA模型的高校人文社科R&D效率研究——以江苏省公办本科高校为例

为了探究江苏省各高校人文社科R&D效率影响因素和各高校人文社科R&D效率水平的差异,并考虑到评价单元投入的异质性问题,采用改进的DEA-CCR模型,对江苏省46所公办本科高校的人文社科R&D数据进行了实证分析。结果表明,江苏省46所公办本科高校在人文社会科学领域的科研效率水平尚可,但仍存在很大的进步空间,特别是“985”、“211”高校在人文社会科学的科研工作中需要取得突破性的进步,才能提高本校人文社会科学的科研效率。最后,根据所得结果,本文提出了各高校进一步提高在人文社会科学领域科研效率的针对性建议。     

Combining reference point based composite indicators with data envelopment analysis: application to the assessment of universities

In our knowledge society, where universities are key players, the assessment of higher education institutions should meet the new demands of the present complex environment. This calls for the use of techniques that are able to manage this complexity. In this paper, we propose a novel combination of methodologies, jointly using a multi-criteria reference point scheme and the data envelopment analysis (DEA) for the assessment of universities. This combination allows us to take into account all the aspects regarded as relevant to assess university performance, and use them as outputs in the efficiency analysis. Our findings highlight the convenience to assess the university performance by using both compensatory and non-compensatory schemes. This way, the information provided allows to detect the actions needed to improve the performances of the universities, rather than just giving an overall performance measure. Furthermore, combining the use of composite indicators with the DEA analysis provides a more complete picture of the institutions assessed, allowing universities to check their efficiency and to detect their weaknesses and strengths accordingly. The approach is illustrated using data of 47 Spanish public universities for the academic year, 2016–2017.

     

What about excellence in teaching? A benevolent ranking of universities

Existing university rankings apply fixed and exogenous weights based on a theoretical framework, stakeholder or expert opinions. Fixed weights cannot embrace all requirements of a ‘good ranking’ according to the Berlin Principles. As the strengths of universities differ, the weights on the ranking should differ as well. This paper proposes a fully nonparametric methodology to rank universities. The methodology is in line with the Berlin Principles. It assigns to each university the weights that maximize (minimize) the impact of the criteria where university performs relatively well (poor). The method accounts for background characteristics among universities and evaluates which characteristics have an impact on the ranking. In particular, it accounts for the level of tuition fees, an English speaking environment, size, research or teaching orientation. In general, medium sized universities in English speaking countries benefit from the benevolent ranking. On the contrary, we observe that rankings with fixed weighting schemes reward large and research oriented universities. Especially Swiss and German universities significantly improve their position in a more benevolent ranking.     

URAP-TR: a national ranking for Turkish universities based on academic performance

This study describes the basic methodological approach and the results of URAP-TR, the first national ranking system for Turkish universities. URAP-TR is based on objective bibliometric data resources and includes both size-dependent and size-independent indicators that balance total academic performance with performance per capita measures. In the context of Turkish national university rankings, the paper discusses the implications of employing multiple size-independent and size-dependent indicators on national university rankings. Fine-grained ranking categories for Turkish universities are identified through an analysis of ranking results across multiple indicators.