Comparing university rankings

Recently there is increasing interest in university rankings. Annual rankings of world universities are published by QS for the Times Higher Education Supplement, the Shanghai Jiao Tong University, the Higher Education and Accreditation Council of Taiwan and rankings based on Web visibility by the Cybermetrics Lab at CSIC. In this paper we compare the rankings using a set of similarity measures. For the rankings that are being published for a number of years we also examine longitudinal patterns. The rankings limited to European universities are compared to the ranking of the Centre for Science and Technology Studies at Leiden University. The findings show that there are reasonable similarities between the rankings, even though each applies a different methodology. The biggest differences are between the rankings provided by the QS-Times Higher Education Supplement and the Ranking Web of the CSIC Cybermetrics Lab. The highest similarities were observed between the Taiwanese and the Leiden rankings from European universities. Overall the similarities are increased when the comparison is limited to the European universities.     

Visibility in international rankings. Strategies for enhancing the competitiveness of Spanish universities

The world-wide popularity of university rankings has spurred the debate about the quality and performance of higher education systems and has had a considerable impact on global society in light of the internationalisation of higher education. While useful for policy makers, such rankings also furnish information on an institution’s “prestige”, which may in turn contribute to more effective resource capture (students, funding, projects). Certain university profiles and missions may prevent many universities from climbing to higher positions, however. One important question in this regard is: how many of a country’s universities can stand at the top of international rankings? The present article attempts to answer this question on the grounds of a study of the Spanish higher education system, and more specifically of an institutional alliance consisting of four high quality universities. A series of research activity indicators drawn from the IUNE Observatory are used to compare this alliance to leading Spanish and international universities and explore whether their visibility and consequently their position in international rankings would be enhanced if they were able to appear under a joint identity. This prospective study also addresses a series of strategies that the Spanish higher education system might implement to successfully rise to the challenges posed by future scenarios.     

What about excellence in teaching? A benevolent ranking of universities

Existing university rankings apply fixed and exogenous weights based on a theoretical framework, stakeholder or expert opinions. Fixed weights cannot embrace all requirements of a ‘good ranking’ according to the Berlin Principles. As the strengths of universities differ, the weights on the ranking should differ as well. This paper proposes a fully nonparametric methodology to rank universities. The methodology is in line with the Berlin Principles. It assigns to each university the weights that maximize (minimize) the impact of the criteria where university performs relatively well (poor). The method accounts for background characteristics among universities and evaluates which characteristics have an impact on the ranking. In particular, it accounts for the level of tuition fees, an English speaking environment, size, research or teaching orientation. In general, medium sized universities in English speaking countries benefit from the benevolent ranking. On the contrary, we observe that rankings with fixed weighting schemes reward large and research oriented universities. Especially Swiss and German universities significantly improve their position in a more benevolent ranking.     

Performance evaluation of research universities in Mainland China,Hong Kong and Taiwan: based on a two-dimensional approach

For a long time, rankings overused in evaluating Chinese universities’ research performance. The relationship between research production and research quality hasn’t been taken seriously in ranking systems. Most university rankings in China put more weight on research production rather than research quality. Recently, the developmental strategy of Chinese universities has shifted from ‘quantity’ to ‘quality’. As a result, a two-dimensional approach was developed in this article to balance ‘quantity’ and ‘quality’. The research production index and the research quality index were produced to locate research universities (RU) from Mainland China, Hong Kong (HK) and Taiwan (TW) in the two-dimensional graph. Fifty-nine RU were classified into three categories according to their locations, which indicated the relevant level of research performance. University of Hong Kong, National Taiwan University, Tsing Hua University and Peking University appeared to be leading universities in research performance. The result showed that the mainland universities were generally of higher research production and lower research quality than HK and TW universities, and proved that the merging tides of Chinese universities enlarged their research production while causing a low level of research quality as well.     

The decline of university patenting and the end of the Bayh–Dole effect

University patenting has been heralded as a symbol of changing relations between universities and their social environments. The Bayh–Dole Act of 1980 in the USA was eagerly promoted by the OECD as a recipe for the commercialization of university research, and the law was imitated by a number of national governments. However, since the 2000s university patenting in the most advanced economies has been on the decline both as a percentage and in absolute terms. In addition to possible saturation effects and institutional learning, we suggest that the institutional incentives for university patenting have disappeared with the new regime of university ranking. Patents and spin-offs are not counted in university rankings. In the new arrangements of university–industry–government relations, universities have become very responsive to changes in their relevant environments.     

Standardization problem of author affiliations in citation indexes

Academic effectiveness of universities is measured with the number of publications and citations. However, accessing all the publications of a university reveals a challenge related to the mistakes and standardization problems in citation indexes. The main aim of this study is to seek a solution for the unstandardized addresses and publication loss of universities with regard to this problem. To achieve this, all Turkey-addressed publications published between 1928 and 2009 were analyzed and evaluated deeply. The results show that the main mistakes are based on character or spelling, indexing and translation errors. Mentioned errors effect international visibility of universities negatively, make bibliometric studies based on affiliations unreliable and reveal incorrect university rankings. To inhibit these negative effects, an algorithm was created with finite state technique by using Nooj Transducer. Frequently used 47 different affiliation variations for Hacettepe University apart from “Hacettepe Univ” and “Univ Hacettepe” were determined by the help of finite state grammar graphs. In conclusion, this study presents some reasons of the inconsistencies for university rankings. It is suggested that, mistakes and standardization issues should be considered by librarians, authors, editors, policy makers and managers to be able to solve these problems.     

Where do Italian universities stand? An in-depth statistical analysis of national and international rankings

In a previous article (Degli Esposti and Geraci. Bulletin of Italian Politics, 2011), we presented an historical survey of the university reform laws that took place in Italy in the last 30 years. On that occasion, we stressed how important is merit evaluation for academics and their institutions, especially in view of the much debated but not yet implemented ‘Gelmini’ reform with its long awaited new regulation for accessing academic positions (concorsi) and for determining individual weight in financial resource allocation among universities. Here, we present and compare several rankings used to evaluate the prestige and merit of Italian universities. We also consider alternative approaches to academic rankings that highlight peculiar aspects of the universities in Italy which cannot be reasonably accounted for by other international rankings. Finally, we propose a new approach that combines both national and international standing of Italian universities. It is hoped that this study will provide practical guidance to policy makers for establishing the criteria upon which merit should be assessed.     

Using Google Scholar institutional level data to evaluate the quality of university research

In recent years, the extent of formal research evaluation, at all levels from the individual to the multiversity has increased dramatically. At the institutional level, there are world university rankings based on an ad hoc combination of different indicators. There are also national exercises, such as those in the UK and Australia that evaluate research outputs and environment through peer review panels. These are extremely costly and time consuming. This paper evaluates the possibility of using Google Scholar (GS) institutional level data to evaluate university research in a relatively automatic way. Several citation-based metrics are collected from GS for all 130 UK universities. These are used to evaluate performance and produce university rankings which are then compared with various rankings based on the 2014 UK Research Excellence Framework (REF). The rankings are shown to be credible and to avoid some of the obvious problems of the REF ranking, as well as being highly efficient and cost effective. We also investigate the possibility of normalizing the results for the university subject mix since science subjects generally produce significantly more citations than social science or humanities.     

Proposal for a multilevel university cybermetric analysis model

Universities’ online seats have gradually become complex systems of dynamic information where all their institutions and services are linked and potentially accessible. These online seats now constitute a central node around which universities construct and document their main activities and services. This information can be quantitative measured by cybermetric techniques in order to design university web rankings, taking the university as a global reference unit. However, previous research into web subunits shows that it is possible to carry out systemic web analyses, which open up the possibility of carrying out studies which address university diversity, necessary for both describing the university in greater detail and for establishing comparable ranking units. To address this issue, a multilevel university cybermetric analysis model is proposed, based on parts (core and satellite), levels (institutional and external) and sublevels (contour and internal), providing a deeper analysis of institutions. Finally the model is integrated into another which is independent of the technique used, and applied by analysing Harvard University as an example of use.     

URAP-TR: a national ranking for Turkish universities based on academic performance

This study describes the basic methodological approach and the results of URAP-TR, the first national ranking system for Turkish universities. URAP-TR is based on objective bibliometric data resources and includes both size-dependent and size-independent indicators that balance total academic performance with performance per capita measures. In the context of Turkish national university rankings, the paper discusses the implications of employing multiple size-independent and size-dependent indicators on national university rankings. Fine-grained ranking categories for Turkish universities are identified through an analysis of ranking results across multiple indicators.     

Comparative study of international academic rankings of universities

International academic rankings that compare world universities have proliferated recently. In accordance with latter conceptual and methodological advances in academic rankings approaches, five selection criteria are defined and four international university rankings are selected. A comparative analysis of the four rankings is presented taking into account both the indicators frequency and its weights. Results show that, although some indicators differ considerably across selected rankings and even many indicators are unique, indicators referred to research and scientific productivity from university academic staff have a prominent role across all approaches. The implications of obtained data for main rankings consumers are discussed.     

Quality assessment of Spanish universities’ web sites focused on the European Research Area

Summary This work has analyzed and evaluated the dissemination of research done at Spanish universities through the World Wide Web (WWW) in order to obtain a map of the visibility of the information available on this research and to propose measures for improving the quality of this diffusion, all within the social and institutional context of the European Area for Higher Education. The methodology applied in the study has used both qualitative and quantitative research methods to obtain some quality indicators on the dissemination of university research. The object of study consists of a sample of 19 Spanish universities, chosen according to their representativeness by Autonomous Community and their administrative and scientific weight. The process of defining indicators, both qualitative and quantitative, as well as the collection and analysis of data, are explained. The results give us a detailed panorama of the state of the art of the visibility of information on research in the web pages of selected universities. This has allowed us to make certain proposals for improvement that can contribute to the excellence of its dissemination.     

Country-specific determinants of world university rankings

This paper examines country-specific factors that affect the three most influential world university rankings (the Academic Ranking of World Universities, the QS World University Ranking, and the Times Higher Education World University Ranking). We run a cross sectional regression that covers 42–71 countries (depending on the ranking and data availability). We show that the position of universities from a country in the ranking is determined by the following country-specific variables: economic potential of the country, research and development expenditure, long-term political stability (freedom from war, occupation, coups and major changes in the political system), and institutional variables, including government effectiveness.     

An educational resource for information literacy in higher education: Functional and users analyses of the e-COMS academic portal

As in today’s knowledge society the Internet is playing an important role in the information literacy of university students the goal of this paper is to analyse, after its first year on the Web, the informational impact of an e-learning resource developed by Granada’s University lecturers (the e-COMS educational portal), a pioneer in Spain for training in information literacy. From the objective and subjective data provided by the own portal and by it users, two different and complementary kinds of analysis (functional and users’) are performed. Assessment of various capabilities, among which visibility and usability stand out, is provided. The highly positive but improvable results offer a detailed analysis of the functional aspects of the portal itself and of the users’ relations with this information resource. From these analyses strengths and weaknesses are extracted and some proposals for improvement are derived.     

The dangers of performance-based research funding in non-competitive higher education systems

An increasing number of nations allocate public funds to research institutions on the basis of rankings obtained from national evaluation exercises. Therefore, in non-competitive higher education systems where top scientists are dispersed among all the universities, rather than concentrated among a few, there is a high risk of penalizing those top scientists who work in lower-performance universities. Using a 5 year bibliometric analysis conducted on all Italian universities active in the hard sciences from 2004 to 2008, this work analyzes the distribution of publications and relevant citations by scientists within the universities, measures the research performance of individual scientists, quantifies the intensity of concentration of top scientists at each university, provides performance rankings for the universities, and indicates the effects of selective funding on the top scientists of low-ranked universities.     

An insight into the importance of national university rankings in an international context: the case of the I-UGR rankings of Spanish universities

The great importance international rankings have achieved in the research policy arena warns against many threats consequence of the flaws and shortcomings these tools present. One of them has to do with the inability to accurately represent national university systems as their original purpose is only to rank world-class universities. Another one has to do with the lack of representativeness of universities’ disciplinary profiles as they usually provide a unique table. Although some rankings offer a great coverage and others offer league tables by fields, no international ranking does both. In order to surpass such limitation from a research policy viewpoint, this paper analyzes the possibility of using national rankings in order to complement international rankings. For this, we analyze the Spanish university system as a study case presenting the I-UGR Rankings for Spanish universities by fields and subfields. Then, we compare their results with those obtained by the Shanghai Ranking, the QS Ranking, the Leiden Ranking and the NTU Ranking, as they all have basic common grounds which allow such comparison. We conclude that it is advisable to use national rankings in order to complement international rankings, however we observe that this must be done with certain caution as they differ on the methodology employed as well as on the construction of the fields.     

World-wide university rankings: A Scandinavian approach

Although universities’ world rankings are popular, their design and methods still request considerable elaborations. The paper demonstrates some shortcomings in the Academic World Ranking of Universities (ARWU, Shanghai Jiao Tong University) ranking methods. One deficiency is that universities’ scale differences are neglected due to omitting the whole input side. By resampling and reanalyzing the ARWU data, the paper proposes an input-output analysis for measuring universities’ scientific productivity with special emphasis on those universities which meet the productivity threshold (i.e. share of output exceeds share of input) in a certain group of universities. The productivity analysis on Scandinavian universities evaluates multidisciplinary and specialized universities on their own terms; consequently the ranking based on scientific productivity deviates significantly from the ARWU.     

What makes a journal international? A case study using conservation biology journals

The qualitative label ‘international journal’ is used widely, including in national research quality assessments. We determined the practicability of analysing internationality quantitatively using 39 conservation biology journals, providing a single numeric index (IIJ) based on 10 variables covering the countries represented in the journals’ editorial boards, authors and authors citing the journals’ papers. A numerical taxonomic analysis refined the interpretation, revealing six categories of journals reflecting distinct international emphases not apparent from simple inspection of the IIJs alone. Categories correlated significantly with journals’ citation impact (measured by the Hirsch index), with their rankings under the Australian Commonwealth’s ‘Excellence in Research for Australia’ and with some countries of publication, but not with listing by ISI Web of Science. The assessments do not reflect on quality, but may aid editors planning distinctive journal profiles, or authors seeking appropriate outlets.     

The role of Chinese universities in enterprise–university research collaboration

In this paper the role of Chinese universities in enterprise–university research collaboration is investigated. This study focuses on a special aspect of the collaboration—co-authored articles. The two cases are analyzed: (1) research collaboration between Baosteel Group Corporation and Chinese universities; (2) research collaboration between China Petroleum & Chemical Corporation and Chinese universities. The co-authorship data over the period 1998–2007 were searched from CNKI database, the largest Chinese publication and citation database. The main findings are as follows: the number of articles co-authored by enterprise and university scientists has been increasing rapidly; the share of co-authored articles has been growing; the authors from universities are more possible to be the first authors; as a whole, enterprise–university co-authored articles tend to receive more citations and get downloaded more frequently; a mathematical orientation emerges in the enterprise–university articles. To reveal and describe such a trend the methods of keywords analysis and co-occurrence analysis are applied. The Chinese government’s policy instruments and substantial supports for pushing and improving enterprise–university research collaboration are introduced and analyzed.