A mathematical model to optimize decisions to impact multi-attribute rankings

We formulate the problem of how to climb in multi-attribute rankings with known weights using mathematical optimization. A model is derived based on familiar practices used in rankings in higher education where several attributes are combined using known weights to obtain a score. The method applies in any situation where multiple attributes are used to rank entities. We invoke several assumptions such as independence among attributes and that administrators can affect the values of some of the attributes and know the cost of doing so. Our results suggest that a strategy to advance in the rankings is to focus on modifying the value of fewer rather than more attributes. The model is generalized to allow for synergies and antagonisms among the attributes.     

Exploring the web visibility of world-class universities

With the rapid development of the Internet, there is a need for evaluating the public visibility of universities on the Internet (i.e., web visibility) in terms of its implications for university management, planning, and governance. The data were collected in December 2010 by using Yahoo, one of the most widely used search engines. Specifically, we gathered “Single Mention” data to measure the number of times that each university was mentioned on websites. In addition, we collected network-based data on Single Mentions. We obtained another data set based on the 2010 world university rankings by Shanghai Jiao Tong University (SJTU). We employed several analytical methods for the analysis, including correlations, nonparametric tests (e.g., the Mann–Whitney test), and multidimensional scaling (MDS). The significant positive correlation between university rankings and web visibility suggests that indicators of web visibility can function as a proxy measure of conventional university rankings. Another distinctive implication can be drawn from the pattern of a disparity in web visibility stemming from the linguistic divide, that is, universities in English-speaking countries dominated the central positions in various network structures of web visibility, whereas those in non-English-speaking countries were located in the periphery of these structures. In this regard, further research linking web visibility to university management, planning, and governance is needed.     

An insight into the importance of national university rankings in an international context: the case of the I-UGR rankings of Spanish universities

The great importance international rankings have achieved in the research policy arena warns against many threats consequence of the flaws and shortcomings these tools present. One of them has to do with the inability to accurately represent national university systems as their original purpose is only to rank world-class universities. Another one has to do with the lack of representativeness of universities’ disciplinary profiles as they usually provide a unique table. Although some rankings offer a great coverage and others offer league tables by fields, no international ranking does both. In order to surpass such limitation from a research policy viewpoint, this paper analyzes the possibility of using national rankings in order to complement international rankings. For this, we analyze the Spanish university system as a study case presenting the I-UGR Rankings for Spanish universities by fields and subfields. Then, we compare their results with those obtained by the Shanghai Ranking, the QS Ranking, the Leiden Ranking and the NTU Ranking, as they all have basic common grounds which allow such comparison. We conclude that it is advisable to use national rankings in order to complement international rankings, however we observe that this must be done with certain caution as they differ on the methodology employed as well as on the construction of the fields.     

Quality perceptions of design journals: The design scholars’ perspective

The primary objectives of this study were to identify a set of journals that report on industrial design research and to propose quality rankings of those journals. Based on an online survey, design journals were assessed in terms of two quality metrics: popularity and indexed average rank position. We find that both general and specialized design journals are highly valued and that geographic origin and academic background can be related with journal rankings. The results of the study offer a guide to both evaluators and those evaluated when judging or selecting research outlets.     

National peer-review research assessment exercises for the hard sciences can be a complete waste of money: the Italian case

There has been ample demonstration that bibliometrics is superior to peer-review for national research assessment exercises in the hard sciences. In this paper we examine the Italian case, taking the 2001–2003 university performance rankings list based on bibliometrics as benchmark. We compare the accuracy of the first national evaluation exercise, conducted entirely by peer-review, to other rankings lists prepared at zero cost, based on indicators indirectly linked to performance or available on the Internet. The results show that, for the hard sciences, the costs of conducting the Italian evaluation of research institutions could have been completely avoided.     

Reverse-engineering conference rankings: what does it take to make a reputable conference?

In recent years, several national and community-driven conference rankings have been compiled. These rankings are often taken as indicators of reputation and used for a variety of purposes, such as evaluating the performance of academic institutions and individual scientists, or selecting target conferences for paper submissions. Current rankings are based on a combination of objective criteria and subjective opinions that are collated and reviewed through largely manual processes. In this setting, the aim of this paper is to shed light into the following question: to what extent existing conference rankings reflect objective criteria, specifically submission and acceptance statistics and bibliometric indicators? The paper specifically considers three conference rankings in the field of Computer Science: an Australian national ranking, a Brazilian national ranking and an informal community-built ranking. It is found that in all cases bibliometric indicators are the most important determinants of rank. It is also found that in all rankings, top-tier conferences can be identified with relatively high accuracy through acceptance rates and bibliometric indicators. On the other hand, acceptance rates and bibliometric indicators fail to discriminate between mid-tier and bottom-tier conferences.     

Comparative study of international academic rankings of universities

International academic rankings that compare world universities have proliferated recently. In accordance with latter conceptual and methodological advances in academic rankings approaches, five selection criteria are defined and four international university rankings are selected. A comparative analysis of the four rankings is presented taking into account both the indicators frequency and its weights. Results show that, although some indicators differ considerably across selected rankings and even many indicators are unique, indicators referred to research and scientific productivity from university academic staff have a prominent role across all approaches. The implications of obtained data for main rankings consumers are discussed.     

Visibility in international rankings. Strategies for enhancing the competitiveness of Spanish universities

The world-wide popularity of university rankings has spurred the debate about the quality and performance of higher education systems and has had a considerable impact on global society in light of the internationalisation of higher education. While useful for policy makers, such rankings also furnish information on an institution’s “prestige”, which may in turn contribute to more effective resource capture (students, funding, projects). Certain university profiles and missions may prevent many universities from climbing to higher positions, however. One important question in this regard is: how many of a country’s universities can stand at the top of international rankings? The present article attempts to answer this question on the grounds of a study of the Spanish higher education system, and more specifically of an institutional alliance consisting of four high quality universities. A series of research activity indicators drawn from the IUNE Observatory are used to compare this alliance to leading Spanish and international universities and explore whether their visibility and consequently their position in international rankings would be enhanced if they were able to appear under a joint identity. This prospective study also addresses a series of strategies that the Spanish higher education system might implement to successfully rise to the challenges posed by future scenarios.     

What do global university rankings really measure? The search for the X factor and the X entity

Most academic rankings attempt to measure the quality of university education and research. However, previous studies that examine the most influential rankings conclude that the variables they use could be an epiphenomenon of an X factor that has little to do with quality. The aim of this study is to investigate the existence of this hidden factor or profile in the two most influential global university rankings in the world: the Academic Ranking of World Universities (ARWU) of the University of Shanghai Jiao Tong, and the Times Higher Education (THE) ranking. Results support the existence of an underlying entity profile, characterized by institutions normally from the US that enjoy a high reputation. Results also support the idea that rankings lack the capacity to assess university quality in all its complexity, and two strategies are suggested in relation to the vicious circle created between institutional reputation and rankings.     

Using Google Scholar institutional level data to evaluate the quality of university research

In recent years, the extent of formal research evaluation, at all levels from the individual to the multiversity has increased dramatically. At the institutional level, there are world university rankings based on an ad hoc combination of different indicators. There are also national exercises, such as those in the UK and Australia that evaluate research outputs and environment through peer review panels. These are extremely costly and time consuming. This paper evaluates the possibility of using Google Scholar (GS) institutional level data to evaluate university research in a relatively automatic way. Several citation-based metrics are collected from GS for all 130 UK universities. These are used to evaluate performance and produce university rankings which are then compared with various rankings based on the 2014 UK Research Excellence Framework (REF). The rankings are shown to be credible and to avoid some of the obvious problems of the REF ranking, as well as being highly efficient and cost effective. We also investigate the possibility of normalizing the results for the university subject mix since science subjects generally produce significantly more citations than social science or humanities.     

A ranking of supply chain management journals based on departmental lists

The issue of journal rankings has been an issue of frequent debate, likely due to the effect rankings can have on academic careers. While many rankings have been done over the years, few, if any, take into account what really matters to academics’ careers, how their own departments evaluate their research. This study collected department journal ranking lists from top international supply chain programmes. The resulting data, approximately 70% US and 30% international, presents a diverse perspective on the journals considered when analysing research productivity. A cluster analysis is performed and the results are compared with impact factors for select journals; we also discuss the differences highlighted between perceived rankings based on the departments’ geographic location. The results show that the field still has some divisions.     

Comparing university rankings

Recently there is increasing interest in university rankings. Annual rankings of world universities are published by QS for the Times Higher Education Supplement, the Shanghai Jiao Tong University, the Higher Education and Accreditation Council of Taiwan and rankings based on Web visibility by the Cybermetrics Lab at CSIC. In this paper we compare the rankings using a set of similarity measures. For the rankings that are being published for a number of years we also examine longitudinal patterns. The rankings limited to European universities are compared to the ranking of the Centre for Science and Technology Studies at Leiden University. The findings show that there are reasonable similarities between the rankings, even though each applies a different methodology. The biggest differences are between the rankings provided by the QS-Times Higher Education Supplement and the Ranking Web of the CSIC Cybermetrics Lab. The highest similarities were observed between the Taiwanese and the Leiden rankings from European universities. Overall the similarities are increased when the comparison is limited to the European universities.     

Relative measure index: a metric to measure the quality of journals

Journal impact factors (JIF) have been an accepted indicator of ranking journals. However, there has been increasing arguments against the fairness of using the JIF as the sole ranking criteria. This resulted in the creation of many other quality metric indices such as the h-index, g-index, immediacy index, Citation Half-Life, as well as SCIMago journal rank (SJR) to name a few. All these metrics have their merits, but none include any great degree of normalization in their computations. Every citation and every publication is taken as having the same importance and therefore weight. The wealth of available data results in multiple different rankings and indexes existing. This paper proposes the use of statistical standard scores or z-scores. The calculation of the z-scores can be performed to normalize the impact factors given to different journals, the average of z-scores can be used across various criteria to create a unified relative measurement (RM) index score. We use the 2008 JCR provided by Thompson Reuters to demonstrate the differences in rankings that would be affected if the RM-index was adopted discuss the fairness that this index would provide to the journal quality ranking.     

A comparison of bibliometric indicators for computer science scholars and journals on Web of Science and Google Scholar

Given the current availability of different bibliometric indicators and of production and citation data sources, the following two questions immediately arise: do the indicators’ scores differ when computed on different data sources? More importantly, do the indicator-based rankings significantly change when computed on different data sources? We provide a case study for computer science scholars and journals evaluated on Web of Science and Google Scholar databases. The study concludes that Google scholar computes significantly higher indicators’ scores than Web of Science. Nevertheless, citation-based rankings of both scholars and journals do not significantly change when compiled on the two data sources, while rankings based on the h index show a moderate degree of variation.     

The Patent Asset Index – A new approach to benchmark patent portfolios

Patent metrics are increasingly used to assess the competitive position of technology-oriented firms. Patent rankings and patent scoreboards are popular methods to benchmark patent portfolios of firms against each other. Existing rankings, however, have methodological limitations that significantly reduce the meaningfulness of these benchmarks for managers, investors and other stakeholders. In this paper, we develop a new benchmarking methodology that overcomes limitations of existing approaches and offers a more accurate assessment of a firm’s patent portfolio vis-à-vis its competitors. Firms are ranked according to the Patent Asset Index, which is derived from a set of newly developed patent indicators. These indicators are empirically validated and reflect more accurately the value of patents. We apply the new benchmarking method in the global chemical industry and contrast our findings with those of other existing patent portfolio rankings.     

Journal status

Summary The status of an actor in a social context is commonly defined in terms of two factors: the total number of endorsements the actor receives from other actors and the prestige of the endorsing actors. These two factors indicate the distinction between popularity and expert appreciation of the actor, respectively. We refer to the former as popularity and to the latter as prestige. These notions of popularity and prestige also apply to the domain of scholarly assessment. The ISI Impact Factor (ISI IF) is defined as the mean number of citations a journal receives over a 2 year period. By merely counting the amount of citations and disregarding the prestige of the citing journals, the ISI IF is a metric of popularity, not of prestige. We demonstrate how a weighted version of the popular PageRank algorithm can be used to obtain a metric that reflects prestige. We contrast the rankings of journals according to their ISI IF and their Weighted PageRank, and we provide an analysis that reveals both significant overlaps and differences. Furthermore, we introduce the Y-factor which is a simple combination of both the ISI IF and the weighted PageRank, and find that the resulting journal rankings correspond well to a general understanding of journal status.     

Uncertainty and the ranking of economics journals

Scientometrics - Journal rankings often show significant changes compared to previous rankings. This gives rise to the question of how well estimated the rank of a journal is. In this contribution,...     

What do university rankings by fields rank? Exploring discrepancies between the organizational structure of universities and bibliometric classifications

University rankings by fields are usually based on the research output of universities. However, research managers and rankings consumers expect to see in such fields a reflection of the structure of their own organizational institution. In this study we address such misinterpretation by developing the research profile of the organizational units of two Spanish universities: University of Granada and Pompeu Fabra University. We use two classification systems, the subject categories offered by Thomson Scientific which are commonly used on bibliometric studies, and the 37 disciplines displayed by the Spanish I-UGR Rankings which are constructed from an aggregation of the former. We also describe in detail problems encountered when working with address data from a top down approach and we show differences between universities structures derived from the interdisciplinary organizational forms of new managerialism at universities. We conclude by highlighting that rankings by fields should clearly state the methodology for the construction of such fields. We indicate that the construction of research profiles may be a good solution for universities for finding out levels of discrepancy between organizational units and subject fields.     

What factors are associated with increasing co-authorship in the social sciences? A case study of Danish Economics and Political Science

We examine the relative strength of short-term citation counts, bibliometric measures such as journal impact factors, and journal rankings in terms of predicting long-run citations. Using a set of articles published in sixty of the highest reputation economics journals in 1994, we find that citations received over fairly short windows (between 1 and 2 years after publication) are much stronger predictors of long-run citation counts compared with journal impact factors or other journal rankings. Our results are robust to a series of robustness checks. These findings suggest department heads and tenure and promotion committees should place greater weight on short-term citations as opposed to article placement when making personnel decisions.     

Mapping library and information science in China: a coauthorship network analysis

This paper aims to identify the collaboration pattern and network structure of the coauthorship network of library and information science (LIS) in China. Using data from 18 core source LIS journals in China covering 6 years, we construct the LIS coauthorship network. We analyze the network from both macro and micro perspectives and identify some key features of this network: this network is a small-world network, and follows the scale-free character. In the micro-level, we calculate each author’s centrality values and compare them with citation counts. We find that centrality rankings are highly correlated with citation rankings. We also discuss the limitation of current centrality measures for coauthorship network analysis.