Calibrating the zoom — a test of Zitt’s hypothesis

Bibliometric indicators are widely used to compare performance between units operating in different fields of science. For cross-field comparisons, article citation rates have to be normalised to baseline values because citation practices vary between fields, in respect of timing and volume. Baseline citation values vary according to the level at which articles are aggregated (journal, sub-field, field). Consequently, the normalised citation performance of each research unit will depend on the level of aggregation, or ‘zoom’, that was used when the baselines were calculated. Here, we calculate the citation performance of UK research units for each of three levels of article-aggregation. We then compare this with the grade awarded to that unit by external peer review. We find that the correlation between average normalised citation impact and peerreviewed grade does indeed vary according to the selected level of zoom. The possibility that the level of ‘zoom’ will affect our assessment of relative impact is an important insight. The fact that more than one view and hence more than one interpretation of performance might exist would need to be taken into account in any evaluation methodology. This is likely to be a serious challenge unless a reference indicator is available and will generally require any evaluation to be carried out at multiple levels for a reflective review.     

Uncovering excellence in academic rankings: a closer look at the Shanghai ranking

In this paper, we examine whether the quality of academic research can be accurately captured by a single aggregated measure such as a ranking. With Shanghai University’s Academic Ranking of World Universities as the basis for our study, we use robust principal component analysis to uncover the underlying factors measured by this ranking. Based on a sample containing the top 150 ranked universities, we find evidence that, for the majority of these institutions, the Shanghai rankings reflect not one but in fact two different and uncorrelated aspects of academic research: overall research output and top-notch researchers. Consequently, the relative weight placed upon these two factors determines to a large extent the final ranking.     

Performance evaluation of research universities in Mainland China,Hong Kong and Taiwan: based on a two-dimensional approach

For a long time, rankings overused in evaluating Chinese universities’ research performance. The relationship between research production and research quality hasn’t been taken seriously in ranking systems. Most university rankings in China put more weight on research production rather than research quality. Recently, the developmental strategy of Chinese universities has shifted from ‘quantity’ to ‘quality’. As a result, a two-dimensional approach was developed in this article to balance ‘quantity’ and ‘quality’. The research production index and the research quality index were produced to locate research universities (RU) from Mainland China, Hong Kong (HK) and Taiwan (TW) in the two-dimensional graph. Fifty-nine RU were classified into three categories according to their locations, which indicated the relevant level of research performance. University of Hong Kong, National Taiwan University, Tsing Hua University and Peking University appeared to be leading universities in research performance. The result showed that the mainland universities were generally of higher research production and lower research quality than HK and TW universities, and proved that the merging tides of Chinese universities enlarged their research production while causing a low level of research quality as well.     

Aggregation properties of relative impact and other classical indicators: Convexity issues and the Yule-Simpson paradox

Among classical bibliometric indicators, direct and relative impact measures for countries or other players in science are appealing and standard. Yet, as shown in this article, they may exhibit undesirable statistical properties, or at least ones that pose questions of interpretation in evaluation and benchmarking contexts. In this article, we address two such properties namely sensitivity to the Yule-Simpson effect, and a problem related to convexity. The Yule-Simpson effect can occur for direct impacts and, in a variant form, for relative impact, causing an apparent incoherence between field values and the aggregate (all-fields) value. For relative impacts, it may result in a severe form of ‘out-range’ of aggregate values, where a player’s relative impact shifts from ‘good’ to ‘bad’, or conversely. Out-range and lack of convexity in general are typical of relative impact indicators. Using empirical data, we suggest that, for relative impact measures, ‘out-range’ due to lack of convexity is not exceptional. The Yule-Simpson effect is less frequent, and especially occurs for small players with particular specialisation profiles.     

Crack growth in structural adhesive joints in aqueous environments

The adhesive fracture energy, G _c , of metallic joints, bonded with a rubber-toughened epoxy adhesive, has been measured using monotonically-loaded tests. Such tests have been conducted in various relative humidities and in water, at 21 °C. Two surface pretreatments have been employed for the substrates prior to bonding: a simple grit-blast and degrease (‘GBD’) pretreatment or a grit-blast, degrease and silane primer (‘GBS’) pretreatment. The joints were formed using metallic substrates which were either (a) aluminium-alloy substrates, (b) steel substrates, or (c) ‘dissimilar’ substrates (i.e. one substrate being aluminium-alloy with the other one being steel). For both test environments, when G _c was plotted against the crack velocity, three regions of fracture behaviour could be distinguished. At low rates of displacement the crack grew in a stable manner, visually along the interface, and relatively low crack velocities could be readily measured. This was termed ‘Region I’ and here the value of G _c measured in the aqueous environment was relatively low compared to that measured in a relatively dry environment of 55% relative humidity. On the other hand, at relatively high rates of displacement the crack always grew in a stick-slip manner mainly cohesively in the adhesive layer at approximately 20 km/min. This was termed ‘Region III’, and here the value of G _c was relatively high and independent of the environmental test conditions employed. In this region the crack was considered to grow faster than the water molecules were able to reach the crack tip, which explains the independence of G _c upon the test environment. In between ‘Region I’ and ‘Region III’, a transition region was observed which was designated as ‘Region II’. The major effect of the ‘GBS’ pretreatment, compared to the ‘GBD’ pretreatment, was to increase the value of G _c both in ‘Regions I and III’, although the presence of the silane primer had the far greater effect in ‘Region I’.     

The evaluation of university departments and their scientists: Some general considerations with reference to exemplary bibliometric publication and citation analyses for a Department of psychology

In reference to an exemplary bibliometric publication and citation analysis for a University Department of Psychology, some general conceptual and methodological considerations on the evaluation of university departments and their scientists are presented. Data refer to publication and citation-by-others analyses (PsycINFO, PSYNDEX, SSCI, and SCI) for 36 professorial and non-professorial scientists from the tenure staff of the department under study, as well as confidential interviews on self-and colleagues-perceptions with seven of the sample under study. The results point at (1) skewed (Pareto-) distributions of all bibliometric variables demanding nonparametrical statistical analyses, (2) three personally identical outliers which must be excluded from some statistical analyses, (3) rather low rank-order correlations of publication and citation frequencies having approximately 15% common variance, (4) only weak interdependences of bibliometric variables with age, occupational experience, gender, academic status, and engagement in basic versus applied research, (5) the empirical appropriateness and utility of a normative typological model for the evaluation of scientists’ research productivity and impact, which is based on cross-classifications with reference to the number of publications and the frequency of citations by other authors, and (6) low interrater reliabilities and validity of ad hoc evaluations within the departments’ staff. Conclusions refer to the utility of bibliometric data for external peer reviewing and for feedback within scientific departments, in order to make colleague-perceptions more reliable and valid.     

A new reader trial approach to peer review in funding research grants: An Australian experiment

Summary Peer reviews are highly valued in academic life, but are notoriously unreliable. A major problem is the substantial measurement error due to the idiosyncratic responses when large numbers of different assessors each evaluate only a single or a few submissions (e.g., journal articles, grants, etc.). To address this problem, the main funding body of academic research in Australia trialed a “reader system” in which each of a small number of senior academics read all proposals within their subdiscipline. The traditional peer review process for 1996 (2,989 proposals, 6,233 assessors) resulted in unacceptably low reliabilities comparable with those found in other research (0.475 for research project, 0.572 for researcher). For proposals from psychology and education in 1997, the new reader system resulted in substantially higher reliabilities: 0.643 and 0.881, respectively. In comparison to the traditional peer review approach, the new reader system is substantially more reliable, timely, and cost efficient - and applicable to many peer review situations.     

Establishment of paper assessment system based on academic disciplinary benchmarks

An article assessment system based on both Tianjin University and nine key Chinese Universities’ academic disciplinary benchmarks was established to evaluate researcher’s published papers. With this scientific benchmarking system, the quality of a researcher’s papers could be easily located in a percentile scale in corresponding field within certain groups. Several factors, including total number of papers, order of authors, impact of journals, citation count, h-index, e-index, a-index, m-quotient, etc., were also utilized for both quantity and quality analysis. Furthermore, the novel proposed weighted citation analysis was introduced to judge a researcher’s contribution to his/her research outcomes. The convenient application and comprehensive evaluation property of this assessment system was thoroughly discussed via a given example.     

Life sciences research in Venezuela

The development, current status and dynamics of research in biology related domains in Venezuela is examined through the study of demographic, academic distribution, scientific output and productivity, for two sets of investigators that fit a profile outlined for life sciences researchers or scientists. The first group corresponds to biologists extracted from the ranks of the official Program for the Promotion of Researchers (PPI), the other, pulled out from those that publish in biologically oriented journals, indexed by the Institute of Scientific Information (ISI). Both sets of biology scientists, PPI researchers or Web of Science/ISI scientists, show similar characteristics. The number (absolute and relative) of PPI member that are supposedly dedicated to biological research but do not publish in ISI indexed journals was found to be very similar to the number of supposedly non biologist members of the PPI Program that do publish biological articles in ISI indexed journals. There is also an ongoing feminization process, of academic hierarchies. Female biologists predominate in lower academic ranks and in research cadres, as many as 70% in some areas of biology. This contrasts with the pattern of male predominance observed during the second half of twentieth century in the country. Productivity of Venezuelan biologists seems to depend on gender; men are more productive that their female counterparts. From the bibliometric standpoint, it is found that, on average, 30% of all publications produced in the country are related to biology (or life sciences). The Venezuelan biologists network qualifies neither as a ‘Small World’ nor it follows the ‘Scale Free’ model. Finally, in a country rich in renewable natural resources, it seems that the Venezuelan community of researchers in biology is in decline, despite the fact that they constitute its most productive group of investigators.     

Reputation,research and reality: The publication records of UK Departments of Politics, 1978–1984

This article contributes some data on the relative research performance of university departments, a topic of growing interest. It analyses the total published andper capita publication rates of 52 UK Politics Departments from 1978 to 1984. The main findings are that 1) departmentalper capita publication rates vary enormously: 2) a department's relative productivity is strongly correlated across all types of publication; 3) a few departments are substantially more productive then the rest; 4) among highly productive departments, one can usefully distinguish between those with collective and those with individual strength 5) a department's productivity is not related to its size. Various rankings are compiled and compared with theTHES peer review and the UGC's classification of research quality.     

Africa’s contribution to the worldwide research literature: New analytical perspectives,trends, and performance indicators

This paper examines general characteristics of African science from a quantitative ‘scientometric’ perspective. More specifically, that of research outputs of Africa-based authors published in the scientific literature during the years 1980–2004, either within the international journals representing ‘mainstream’ science, or within national and regional journals reflecting ‘indigenous science’. As for the international journals, the findings derived from Thomson Scientific’s Citation Indexes show that while Africa’s share in worldwide science has steadily declined, the share of international co-publications has increased very significantly, whereas low levels of international citation impact persist. A case study of South African journals reveals the existence of several journals that are not processed for these international databases but nonetheless show a distinctive citation impact on international research communities.     

Citation bias in medical journals

Multivariate statistical analysis of the citation profiles of urology and related journals (i.e. the relative extent to which each journal cites itself and other journals within a set) has highlighted hidden correlations. We reveal the existence of a ‘transatlantic’ rift in citation practice and of a confined discipline-oriented world which interfaces weakly with many other disciplines. We also interpret the results of our analyses in terms of basic and clinical research and examine whether there is a time-related selectivity in citation. Taken together, our results call for a serious appraisal of present-day research trends and of their evaluation. The open question is how to create a terrain that will foster original, possibly interdisciplinary, research in developed nations whilst maintaining cultural individuality.     

Design Support Using Distributed Web-Based AI Tools

Currently, designers are faced with searching through a ‘sea’ of on-line knowledge to support their decision making activities. This paper describes WebCADET, which is a reimplementation of the stand-alone CADET – a Knowledge-Based System (KBS) for product design evaluation. WebCADET aims to provide effective and efficient support for designers during their searches for design knowledge. WebCADET uses the ‘AI as text’ approach, where KBSs can be seen as a medium to facilitate the communication of design knowledge between designers. The development of WebCADET to include practical support via World Wide Web-based functionality, which illustrates the potential of the ‘AI as text’ approach, is described in the paper.     

Influence of crystallographic textures on tensile properties of 316L austenitic stainless steel

Role of cold rolling texture on the tensile properties of the cold rolled and cold rolled and annealed AISI 316L austenitic stainless steel is described here. The solution-annealed stainless steel plates were unidirectionally cold rolled to 50, 70 and 90% of reduction in thickness. The cold rolled material was annealed at 500–900 °C annealing temperatures. X-ray diffraction technique was employed to study the texture evolution in cold rolled as well as cold rolled and annealed conditions. The texture components that evolved were translated into slip transmission number ‘λ’ and Schmid factor ‘μ’. These two parameters were correlated with the tensile properties of the material. The tensile properties were evaluated under all processing conditions. Softening of the cold rolled material was observed after annealing with increasing annealing temperatures. From the stress–strain curves, strain hardening coefficient ‘n’ and strain hardening rate ‘θ’ were determined. It was found that the effect of texture on tensile behaviour could be understood clearly by strain hardening rate. Out of the two parameters, ‘n’ and ‘θ’, strain hardening rate was found to be more sensitive to type of texture in the material.     

Mapping inventive activity and technological change through patent analysis: A case study of India and China

The characteristics of Indian and Chinese patenting activity in the US patent system are examined by delineating two categories of patents; ‘nationally assigned’, and ‘invented not nationally assigned’ patents (not-nationally assigned patents in short). Further within the above two categories, patents are distinguished and analysed in terms of patent types: utility, design, and plant patents. Indian patents are mainly of utility type whereas China's activity falls in both utility and design.In the ‘nationally assigned’ patents, the different types of institutions involved and linkages are much higher for China. However, ‘not-nationally assigned’ patents of both the countries are dominated by industry and inter-institutional collaborations are sparse. Patents addressing technology sectors (analysis based on utility patents) do not exhibit major differences between the two categories in Chinese patents and address with varying degree all technology sectors. Unlike China, India's ‘nationally assigned’ patents are concentrated in chemical and drugs & medical whereas their ‘not-nationally assigned’ patents are similar to that of China in addressing technology sectors. In design patents, Chinese ‘nationally assigned’ patents mainly cover ornamental design of lighting equipments whereas their ‘not-nationally assigned’ patents are mainly in design equipment for production, distribution or transformation of energy. Further, few firms are active in design patents in both the categories. India's design activity is insignificant in both the categories. The paper concludes by examining the results in the policy context. This revised version was published online in June 2006 with corrections to the Cover Date.     

A new methodology for ranking scientific institutions

This paper examines policy-relevant effects of a yearly public ranking of individual researchers and their institutes in economics by means of their publication output in international top journals. In 1980, a grassroots ranking (‘Top 40’) of researchers in the Netherlands by means of their publications in international top journals started a competition among economists. The objective was to improve economics research in the Netherlands to an internationally competitive level. The ranking lists did stimulate output in prestigious international journals. Netherlands universities tended to perform well compared to universities elsewhere in the EU concerning volume of output in ISI source journals, but their citation impact was average. Limitations of ranking studies and of bibliometric monitoring in the field of economics are discussed.     

Does cumulative advantage affect collective learning in science? An agent-based simulation

Agent-based simulation can model simple micro-level mechanisms capable of generating macro-level patterns, such as frequency distributions and network structures found in bibliometric data. Agent-based simulations of organisational learning have provided analogies for collective problem solving by boundedly rational agents employing heuristics. This paper brings these two areas together in one model of knowledge seeking through scientific publication. It describes a computer simulation in which academic papers are generated with authors, references, contents, and an extrinsic value, and must pass through peer review to become published. We demonstrate that the model can fit bibliometric data for a token journal, Research Policy. Different practices for generating authors and references produce different distributions of papers per author and citations per paper, including the scale-free distributions typical of cumulative advantage processes. We also demonstrate the model’s ability to simulate collective learning or problem solving, for which we use Kauffman’s NK fitness landscape. The model provides evidence that those practices leading to cumulative advantage in citations, that is, papers with many citations becoming even more cited, do not improve scientists’ ability to find good solutions to scientific problems, compared to those practices that ignore past citations. By contrast, what does make a difference is referring only to publications that have successfully passed peer review. Citation practice is one of many issues that a simulation model of science can address when the data-rich literature on scientometrics is connected to the analogy-rich literature on organisations and heuristic search.     

Problems with “natural selection of academic papers”

In this short communication we give critical comments on the paper of Perakakis et al. (Scientometrics 85(2):553–559, 2010) on “Natural selection of academic papers”. The criticism mainly focusses on their unbalanced criticism of peer review and their negative evaluation of the link of peer review with commercial publishing.