Comparing university rankings

Recently there is increasing interest in university rankings. Annual rankings of world universities are published by QS for the Times Higher Education Supplement, the Shanghai Jiao Tong University, the Higher Education and Accreditation Council of Taiwan and rankings based on Web visibility by the Cybermetrics Lab at CSIC. In this paper we compare the rankings using a set of similarity measures. For the rankings that are being published for a number of years we also examine longitudinal patterns. The rankings limited to European universities are compared to the ranking of the Centre for Science and Technology Studies at Leiden University. The findings show that there are reasonable similarities between the rankings, even though each applies a different methodology. The biggest differences are between the rankings provided by the QS-Times Higher Education Supplement and the Ranking Web of the CSIC Cybermetrics Lab. The highest similarities were observed between the Taiwanese and the Leiden rankings from European universities. Overall the similarities are increased when the comparison is limited to the European universities.     

What do global university rankings really measure? The search for the X factor and the X entity

Most academic rankings attempt to measure the quality of university education and research. However, previous studies that examine the most influential rankings conclude that the variables they use could be an epiphenomenon of an X factor that has little to do with quality. The aim of this study is to investigate the existence of this hidden factor or profile in the two most influential global university rankings in the world: the Academic Ranking of World Universities (ARWU) of the University of Shanghai Jiao Tong, and the Times Higher Education (THE) ranking. Results support the existence of an underlying entity profile, characterized by institutions normally from the US that enjoy a high reputation. Results also support the idea that rankings lack the capacity to assess university quality in all its complexity, and two strategies are suggested in relation to the vicious circle created between institutional reputation and rankings.     

World-wide university rankings: A Scandinavian approach

Although universities’ world rankings are popular, their design and methods still request considerable elaborations. The paper demonstrates some shortcomings in the Academic World Ranking of Universities (ARWU, Shanghai Jiao Tong University) ranking methods. One deficiency is that universities’ scale differences are neglected due to omitting the whole input side. By resampling and reanalyzing the ARWU data, the paper proposes an input-output analysis for measuring universities’ scientific productivity with special emphasis on those universities which meet the productivity threshold (i.e. share of output exceeds share of input) in a certain group of universities. The productivity analysis on Scandinavian universities evaluates multidisciplinary and specialized universities on their own terms; consequently the ranking based on scientific productivity deviates significantly from the ARWU.     

Country-specific determinants of world university rankings

This paper examines country-specific factors that affect the three most influential world university rankings (the Academic Ranking of World Universities, the QS World University Ranking, and the Times Higher Education World University Ranking). We run a cross sectional regression that covers 42–71 countries (depending on the ranking and data availability). We show that the position of universities from a country in the ranking is determined by the following country-specific variables: economic potential of the country, research and development expenditure, long-term political stability (freedom from war, occupation, coups and major changes in the political system), and institutional variables, including government effectiveness.     

Quantifying the publication preferences of leading research universities

Research universities have a strong devotion and advocacy for research in their core academic mission. This is why they are widely recognized for their excellence in research which make them take the most renowned positions in the different worldwide university leagues. In order to examine the uniqueness of this group of universities we analyze the scientific production of a sample of them in a 5 year period of time. On the one hand, we analyze their preferences in research measured with the relative percentage of publications in the different subject areas, and on the other hand, we calculate the similarity between them in research preferences. In order to select a set of research universities, we studied the leading university rankings of Shanghai, QS, Leiden, and Times Higher Education (THE). Although the four rankings own well established and developed methodologies and hold great prestige, we choose to use THE because data were readily available for doing the study we had in mind. Having done that, we selected the twenty academic institutions ranked with the highest score in the last edition of THE World University Rankings 2020 and to contrast their impact, we also, we compared them with the twenty institutions with the lowest score in this ranking. At the same time, we extracted publication data from Scopus database for each university and we applied bibliometrics indicators from Elsevier’s SciVal. We applied the statistical techniques cosine similarity and agglomerative hierarchical clustering analysis to examine and compare affinities in research preferences among them. Moreover, a cluster analysis through VOSviewer was done to classify the total scientific production in the four major fields (health sciences, physical sciences, life sciences and social sciences). As expected, the results showed that top universities have strong research profiles, becoming the leaders in the world in those areas and cosine similarity pointed out that some are more affine among them than others. The results provide clues for enhancing existing collaboration, defining and re-directing lines of research, and seeking for new partnerships to face the current pandemic to find was to tackle down the covid-19 outbreak.

     

Why it has become more difficult to predict Nobel Prize winners: a bibliometric analysis of nominees and winners of the chemistry and physics prizes (1901–2007)

We propose a comprehensive bibliometric study of the profile of Nobel Prize winners in chemistry and physics from 1901 to 2007, based on citation data available over the same period. The data allows us to observe the evolution of the profiles of winners in the years leading up to—and following—nominations and awarding of the Nobel Prize. The degree centrality and citation rankings in these fields confirm that the Prize is awarded at the peak of the winners’ citation history, despite a brief Halo Effect observable in the years following the attribution of the Prize. Changes in the size and organization of the two fields result in a rapid decline of predictive power of bibliometric data over the century. This can be explained not only by the growing size and fragmentation of the two disciplines, but also, at least in the case of physics, by an implicit hierarchy in the most legitimate topics within the discipline, as well as among the scientists selected for the Nobel Prize. Furthermore, the lack of readily-identifiable dominant contemporary physicists suggests that there are few new paradigm shifts within the field, as perceived by the scientific community as a whole.     

Should you believe in the Shanghai ranking?

This paper proposes a critical analysis of the “Academic Ranking of World Universities”, published every year by the Institute of Higher Education of the Jiao Tong University in Shanghai and more commonly known as the Shanghai ranking. After having recalled how the ranking is built, we first discuss the relevance of the criteria and then analyze the proposed aggregation method. Our analysis uses tools and concepts from Multiple Criteria Decision Making (MCDM). Our main conclusions are that the criteria that are used are not relevant, that the aggregation methodology is plagued by a number of major problems and that the whole exercise suffers from an insufficient attention paid to fundamental structuring issues. Hence, our view is that the Shanghai ranking, in spite of the media coverage it receives, does not qualify as a useful and pertinent tool to discuss the “quality” of academic institutions, let alone to guide the choice of students and family or to promote reforms of higher education systems. We outline the type of work that should be undertaken to offer sound alternatives to the Shanghai ranking.     

A reassessment of Asian pacific excellence programs in higher education: the Taiwan experience

With the growth of competition between nations in our knowledge-based world economy, excellence programs are becoming a national agenda item in developing as well as developed Asian countries. The main purpose of this paper is to compare the goals, funding policies and selection criteria of excellence programs in China, Japan, Korea and Taiwan and to analyze the academic achievement of their top ranked universities in three areas: research output, internationalization, and excellence, by using data from the Shanghai Jiao Tong, QS, and HEEACT rankings. The effectiveness of Taiwan??s ??Development Plan for World Class Universities and Research Centers of Excellence?? was assessed as a case study in the paper via a survey targeting on 138 top administrators from 11 Taiwan??s universities and 30 reviewers. The study found that more funding nations had, the more outputs and outcomes they would gain, for example China. The Taiwan case demonstrates that world-class universities and research centers are needed in Asian nations despite the concerns for inequality which they raise.     

University patenting activities and their link to the quantity and quality of scientific publications

Integrating data from three independent data sources––USPTO patenting data, Shanghai Jiao Tong University’s Academic Ranking of World Universities (ARWU) and the Times Higher Education Supplement’s World University Ranking (WUR), we examine the possible link between patenting output and the quantity and quality of scientific publications among 281 leading universities world-wide. We found that patenting by these universities, as measured by patents granted by the USPTO, has grown consistently faster than overall US patenting over 1977–2000, although it has grown more slowly over the last 5 years (2000–2005). Moreover, since the mid-1990s, patenting growth has been faster among universities outside North America than among those within North America. We also found that the patenting output of the universities over 2003–2005 is significantly correlated with the quantity and quality of their scientific publications. However, significant regional variations are found: for universities in North America, both the quantity and quality of scientific publications matter, but for European and Australian/NZ universities, only the quantity of publications matter, while for other universities outside North America and Europe/Australia/NZ, only quality of publications matter. We found similar findings when using EPO patenting data instead of USPTO data. Additionally, for USPTO data only, the degree of internationalization of faculty members is found to reduce patenting performance among North American universities, but to increase that of universities outside North America. Plausible explanations for these empirical observations and implications for future research are discussed.     

Academic ranking of world universities using scientometrics - A comment to the “Fatal Attraction”

Summary The Institute of Higher Education, Shanghai Jiao Tong University published on the web the Academic Ranking of World Universities and attracted wide attentions worldwide. 60% of their criteria are based on the databases using scientometrics. They were aware of all possible technical problems, have gone through clean up processes and made necessary corrections. Highly cited researchers and articles published in Nature and Science were identified one by one and attributed to the correct institutions. They are confident that errors including human ones in their data are less than two percent. They will continue their ranking efforts, improve their ranking methodologies and provide more choices on the ranking lists.     

A comparative study on world university rankings: a bibliometric survey

Recently there are many organizations conducting projects on ranking world universities from different perspectives. These ranking activities have made impacts and caused controversy. This study does not favor using bibliometric indicators to evaluate universities?? performances, but not against the idea either. We regard these ranking activities as important phenomena and aim to investigate correlation of different ranking systems taking bibliometric approach. Four research questions are discussed: (1) the inter-correlation among different ranking systems; (2) the intra-correlation within ranking systems; (3) the correlation of indicators across ranking systems; and (4) the impact of different citation indexes on rankings. The preliminary results show that 55?% of top 200 universities are covered in all ranking systems. The rankings of ARWU and PRSPWU show stronger correlation. With inclusion of another ranking, WRWU (2009?C2010), these rankings tend to converge. In addition, intra-correlation is significant and this means that it is possible to find out some ranking indicators with high degree of discriminativeness or representativeness. Finally, it is found that there is no significant impact of using different citation indexes on the ranking results for top 200 universities.     

Uncovering excellence in academic rankings: a closer look at the Shanghai ranking

In this paper, we examine whether the quality of academic research can be accurately captured by a single aggregated measure such as a ranking. With Shanghai University’s Academic Ranking of World Universities as the basis for our study, we use robust principal component analysis to uncover the underlying factors measured by this ranking. Based on a sample containing the top 150 ranked universities, we find evidence that, for the majority of these institutions, the Shanghai rankings reflect not one but in fact two different and uncorrelated aspects of academic research: overall research output and top-notch researchers. Consequently, the relative weight placed upon these two factors determines to a large extent the final ranking.     

Relation of early career performance and recognition to the probability of winning the Nobel Prize in economics

To explore the relation between early career performance or recognition and receiving the Nobel Prize in Economic Sciences, we compare winners of the John Bates Clark Medal, the most prestigious early career recognition for economists, with other successful scholars. The initial comparison combines JBCM winners with scholars published in leading economics journals, controlling for educational background (institution conferring the Ph.D.) and publication and citation success. We then narrow the comparison group down to those given relatively early recognition (based on age category) in the form of other major awards. Lastly, we compare the JBCM awardees with synthetic counterfactuals that best resemble their pre-award academic career performance. All three analyses provide strong support for the notion that winning the JBCM is related to receiving the Nobel Prize, the award of which is also correlated with early career performance success as measured by number of publications and citations.     

Exploring the web visibility of world-class universities

With the rapid development of the Internet, there is a need for evaluating the public visibility of universities on the Internet (i.e., web visibility) in terms of its implications for university management, planning, and governance. The data were collected in December 2010 by using Yahoo, one of the most widely used search engines. Specifically, we gathered “Single Mention” data to measure the number of times that each university was mentioned on websites. In addition, we collected network-based data on Single Mentions. We obtained another data set based on the 2010 world university rankings by Shanghai Jiao Tong University (SJTU). We employed several analytical methods for the analysis, including correlations, nonparametric tests (e.g., the Mann–Whitney test), and multidimensional scaling (MDS). The significant positive correlation between university rankings and web visibility suggests that indicators of web visibility can function as a proxy measure of conventional university rankings. Another distinctive implication can be drawn from the pattern of a disparity in web visibility stemming from the linguistic divide, that is, universities in English-speaking countries dominated the central positions in various network structures of web visibility, whereas those in non-English-speaking countries were located in the periphery of these structures. In this regard, further research linking web visibility to university management, planning, and governance is needed.     

On the internal dynamics of the Shanghai ranking

The Academic Ranking of World Universities (ARWU) published by researchers at Shanghai Jiao Tong University has become a major source of information for university administrators, country officials, students and the public at large. Recent discoveries regarding its internal dynamics allow the inversion of published ARWU indicator scores to reconstruct raw scores for 500 world class universities. This paper explores raw scores in the ARWU and in other contests to contrast the dynamics of rank-driven and score-driven tables, and to explain why the ARWU ranking is a score-driven procedure. We show that the ARWU indicators constitute sub-scales of a single factor accounting for research performance, and provide an account of the system of gains and non-linearities used by ARWU. The paper discusses the non-linearities selected by ARWU, concluding that they are designed to represent the regressive character of indicators measuring research performance. We propose that the utility and usability of the ARWU could be greatly improved by replacing the unwanted dynamical effects of the annual re-scaling based on raw scores of the best performers.     

A fresh approach to evaluating the academic ranking of world universities

The aim of this article is to present new ideas in evaluating Shanghai University’s Academic Ranking of World Universities (ARWU). One issue frequently put forth in various publications is that the Shanghai rankings are sensitive to the relative weight they attribute to each variable. As a possible remedy to this issue, the statistical I-distance method is proposed to be used. Based on a sample containing the top 100 ranked universities, the results show a significant correlation with the official ARWU list. However, some inconsistencies concerning European universities have been noticed and elaborated upon.     

A new methodology for ranking scientific institutions

This paper examines policy-relevant effects of a yearly public ranking of individual researchers and their institutes in economics by means of their publication output in international top journals. In 1980, a grassroots ranking (‘Top 40’) of researchers in the Netherlands by means of their publications in international top journals started a competition among economists. The objective was to improve economics research in the Netherlands to an internationally competitive level. The ranking lists did stimulate output in prestigious international journals. Netherlands universities tended to perform well compared to universities elsewhere in the EU concerning volume of output in ISI source journals, but their citation impact was average. Limitations of ranking studies and of bibliometric monitoring in the field of economics are discussed.     

A scientometric analysis of the effectiveness of Taiwan’s educational research projects

The seeking of evidence for revealing the research performance of Education in Taiwan, in response to the stimulus by the national research projects, is presented and interpreted. More than 70,000 publication records over the years 1990–2011 from Web of Science were downloaded and analyzed. The overview analysis by data aggregation and country ranking shows that Taiwan has significantly improved its publication productivity and citation impact over the last decade. The drill-down analysis based on journal bibliographic coupling, information visualization, and diversity and trend indexes, reveals that e-Learning and Science Education are two fast growing subfields that attract global interests and that Taiwan is among the top-ranked countries in these two fields in terms of research productivity. Implications of the analysis are discussed with an emphasis on the subfield characteristics from which more insightful interpretations can be obtained, such as the regional or cultural characteristics that may affect the performance ranking.     

$$C^3$$-index: a PageRank based multi-faceted metric for authors’ performance measurement

Ranking scientific authors is an important but challenging task, mostly due to the dynamic nature of the evolving scientific publications. The basic indicators of an author’s productivity and impact are still the number of publications and the citation count (leading to the popular metrics such as h-index, g-index etc.). H-index and its popular variants are mostly effective in ranking highly-cited authors, thus fail to resolve ties while ranking medium-cited and low-cited authors who are majority in number. Therefore, these metrics are inefficient to predict the ability of promising young researchers at the beginning of their career. In this paper, we propose \(C^3\)-index that combines the effect of citations and collaborations of an author in a systematic way using a weighted multi-layered network to rank authors. We conduct our experiments on a massive publication dataset of Computer Science and show that—(1) \(C^3\)-index is consistent over time, which is one of the fundamental characteristics of a ranking metric, (2) \(C^3\)-index is as efficient as h-index and its variants to rank highly-cited authors, (3) \(C^3\)-index can act as a conflict resolution metric to break ties in the ranking of medium-cited and low-cited authors, (4) \(C^3\)-index can also be used to predict future achievers at the early stage of their career.     

Physics publication productivity in South African universities

Publication productivity during 2009–2011 was studied for physicists who teach in South African universities, using data from departmental websites and Thomson Reuters’ Web of Science. The objective was to find typical ranges of two measures of individual productivity: number of papers and sum of author share, where author share per n-author paper is 1/n author units (AU). All values given below are average output per year. Median productivity was 1.33 papers (inter-quartile range 0.33–2.33) and 0.3 AU (inter-quartile range 0.1–0.5 AU). The lowest 10 % did not publish, and the top 10 % produced above four papers and above 1 AU. Productivity varied with rank, ranging from medians of 0.67 papers and 0.2 AU for lecturers to 1.67 papers and 0.4 AU for full professors. Productivity of South African professors was similar to that of a sample of USA professors in a comparable mid-ranked bracket in the Shanghai Jiao Tong world ranking of universities, and about half that of professors in the six top-ranked departments in the world, which had medians of four papers and 1 AU.