The influence of the applicants’ gender on the modeling of a peer review process by using latent Markov models

In the grant peer review process we can distinguish various evaluation stages in which assessors judge applications on a rating scale. Bornmann & al. [2008] show that latent Markov models offer a fundamentally good opportunity to model statistically peer review processes. The main objective of this short communication is to test the influence of the applicants’ gender on the modeling of a peer review process by using latent Markov models. We found differences in transition probabilities from one stage to the other for applications for a doctoral fellowship submitted by male and female applicants.     

Journal peer review: a bar or bridge? An analysis of a paper’s revision history and turnaround time,and the effect on citation

Journal peer review lies at the heart of academic quality control. This article explores the journal peer review process and seeks to examine how the reviewing process might itself contribute to papers, leading them to be more highly cited and to achieve greater recognition. Our work builds on previous observations and views expressed in the literature about (a) the role of actors involved in the research and publication process that suggest that peer review is inherent in the research process and (b) on the contribution reviewers themselves might make to the content and increased citation of papers. Using data from the journal peer review process of a single journal in the Social Sciences field (Business, Management and Accounting), we examine the effects of peer review on papers submitted to that journal including the effect upon citation, a novel step in the study of the outcome of peer review. Our detailed analysis suggests, contrary to initial assumptions, that it is not the time taken to revise papers but the actual number of revisions that leads to greater recognition for papers in terms of citation impact. Our study provides evidence, albeit limited to the case of a single journal, that the peer review process may constitute a form of knowledge production and is not the simple correction of errors contained in submitted papers.     

Mechanism change in a simulation of peer review: from junk support to elitism

Peer review works as the hinge of the scientific process, mediating between research and the awareness/acceptance of its results. While it might seem obvious that science would regulate itself scientifically, the consensus on peer review is eroding; a deeper understanding of its workings and potential alternatives is sorely needed. Employing a theoretical approach supported by agent-based simulation, we examined computational models of peer review, performing what we propose to call redesign, that is, the replication of simulations using different mechanisms. Here, we show that we are able to obtain the high sensitivity to rational cheating that is present in literature. In addition, we also show how this result appears to be fragile against small variations in mechanisms. Therefore, we argue that exploration of the parameter space is not enough if we want to support theoretical statements with simulation, and that exploration at the level of mechanisms is needed. These findings also support prudence in the application of simulation results based on single mechanisms, and endorse the use of complex agent platforms that encourage experimentation of diverse mechanisms.     

The ability of different peer review procedures to flag problematic publications

There is a mounting worry about erroneous and outright fraudulent research that gets published in the scientific literature. Although peer review’s ability to filter out such publications is contentious, several peer review innovations attempt to do just that. However, there is very little systematic evidence documenting the ability of different review procedures to flag problematic publications. In this article, we use survey data on peer review in a wide range of journals to compare the retraction rates of specific review procedures, using the Retraction Watch database. We were able to identify which peer review procedures were used since 2000 for 361 journals, publishing a total of 833,172 articles, of which 670 were retracted. After addressing the dual character of retractions, signalling both a failure to identify problems prior to publication, but also the willingness to correct mistakes, we empirically assess review procedures. With considerable conceptual caveats, we were able to identify peer review procedures that seem able to detect problematic research better than others. Results were verified for disciplinary differences and variation between reasons for retraction. This leads to informed recommendations for journal editors about strengths and weaknesses of specific peer review procedures, allowing them to select review procedures that address issues most relevant to their field.

     

Institutional research rankings via bibliometric analysis and direct peer review: A comparative case study with policy implications

Recent years have seen enormously increased interest in the comparative evaluation of research quality in the UK, with considerable resources devoted to ranking the output of academic institutions relative to one another at the sub-discipline level, and the disposition of even greater resources dependent on the outcome of this process. The preferred methodology has been that of traditional peer review, with expert groups of academics tasked to assess the relative worth of all research activity in ‘their’ field. Extension toinstitutional evaluation of a recently refined technique ofjournal ranking (Discipline Contribution Scoring) holds out the possibility of ‘automatic’ evaluation within a time-frame considerably less than would be required using methods based directly on citation counts within the corpus of academic work under review. This paper tests the feasibility of the technique in the sub-field of Business and Management Studies Research, producing rankings which are highly correlated with those generated by the much more complex and expensive direct peer review approach. More generally, the analysis also gives a rare opportunity directly to compare the equivalence of peer review bibliometric analysis over a whole sub-field of academic activity in a non-experimental setting.     

Investigating risky,distracting, and protective peer passenger effects in a dual process framework

Prior studies indicated higher collision rates among young novice drivers with peer passengers. This driving simulator study provided a test for a dual process theory of risky driving by examining social rewards (peer passengers) and cognitive control (inhibitory control). The analyses included age (17–18 yrs, n = 30; 21–24 yrs, n = 20). Risky, distracting, and protective effects were classified by underlying driver error mechanisms. In the first drive, participants drove alone. In the second, participants drove with a peer passenger. Red-light running (violation) was more prevalent in the presence of peer passengers, which provided initial support for a dual process theory of risk driving. In a subgroup with low inhibitory control, speeding (violation) was more prevalent in the presence of peer passengers. Reduced lane-keeping variability reflected distracting effects. Nevertheless, possible protective effects for amber-light running and hazard handling (cognition and decision-making) were found in the drive with peer passengers. Avenues for further research and possible implications for targets of future driver training programs are discussed.     

The use of multiple indicators in the assessment of basic research

This paper argues that evaluations of basic research are best carried out using a range of indicators. After setting out the reasons why assessments of government-funded basic research are increasingly needed, we examine the multi-dimensional nature of basic research. This is followed by a conceptual analysis of what the different indicators of basic research actually measure. Having discussed the limitations of various indicators, we describe the method of converging partial indicators used in several SPRU evaluations. Yet although most of those who now use science indicators would agree that a combination of indicators is desirable, analysis of a sample ofScientometrics articles suggests that in practice many continue to use just one or two indicators. The paper also reports the results of a survey of academic researchers. They, too, are strongly in favour of research evaluations being based on multiple indicators combined with peer review. The paper ends with a discussion as to why multiple indicators are not used more frequently.     

Does the <Emphasis Type="Italic">h</Emphasis>-index for ranking of scientists really work?

Summary Hirsch (2005) has proposed the h-index as a single-number criterion to evaluate the scientific output of a researcher (Ball, 2005): A scientist has index h if h of his/her N_p papers have at least h citations each, and the other (N_p − h) papers have fewer than h citations each. In a study on committee peer review (Bornmann & Daniel, 2005) we found that on average the h-index for successful applicants for post-doctoral research fellowships was consistently higher than for non-successful applicants.     

National research assessment exercises: a comparison of peer review and bibliometrics rankings

Development of bibliometric techniques has reached such a level as to suggest their integration or total substitution for classic peer review in the national research assessment exercises, as far as the hard sciences are concerned. In this work we compare rankings lists of universities captured by the first Italian evaluation exercise, through peer review, with the results of bibliometric simulations. The comparison shows the great differences between peer review and bibliometric rankings for excellence and productivity.     

General performance indexes calculated for research institutes of the Hungarian Academy of Sciences based on scientometric indicators

Activities of research institutes of the Hungarian Academy of Sciences were assessed multidimensionally. Taking into accountgoals and tasks of the institutes,weighted scientometric indicators were suggested. The weights of the individual indexes weredistributed among the institutes by the values of their indicators. The sum of the individual weighted scores representing special aspects of the total activity yields aGeneral Performance Index (GPI) which, together with a thorough peer review, may be used for distributing grants.     

Analysis of publications on sturgeon research between 1996 and 2010

Sturgeon species are among the commercially most valuable and the most endangered groups of fish. To assess the existing literature published within the field of sturgeon research over the past 15 years (1996–2010) we applied a bibliometric approach, in order to identify patterns and trends of the published research in this field. The analysis was performed based upon articles obtained from the ISI Web of Knowledge online database. The results revealed that although all 27 sturgeon species have been objects of the research, species that are endangered or facing a high probability of extinction have received disproportionately less attention. White sturgeon (Acipenser transmontanus) was the most frequently studied species, but it was recently surpassed by Persian sturgeon (A. persicus). Early life phases have been among the central objects of the research, and genetics, especially the use of microsatellite DNA, is becoming increasingly popular and had the highest impact. Research related to aquaculture was prominent, while the research related to hybrids (as a commodity of aquaculture production) was decreasing in popularity. Papers dealing with conservation issues were most frequently focused on European sturgeon (A. sturio). A steady increase in the number of published articles over time was observed. However, the overall citation rate declined significantly over time. During the period reviewed, the sturgeon research published in peer reviewed journals dominantly originated from the USA and EU. Nevertheless, considering the current trend in output, it is very likely that the Asian countries, mainly Iran and China, will surpass them within the next 5–10 years. International and inter-institutional collaboration both tended to increase the impact of the research. Stimulation and improvement of the international cooperation should be considered as future priorities.     

On peer review in computer science: analysis of its effectiveness and suggestions for improvement

In this paper we focus on the analysis of peer reviews and reviewers behaviour in a number of different review processes. More specifically, we report on the development, definition and rationale of a theoretical model for peer review processes to support the identification of appropriate metrics to assess the processes main characteristics in order to render peer review more transparent and understandable. Together with known metrics and techniques we introduce new ones to assess the overall quality (i.e. ,reliability, fairness, validity) and efficiency of peer review processes e.g. the robustness of the process, the degree of agreement/disagreement among reviewers, or positive/negative bias in the reviewers’ decision making process. We also check the ability of peer review to assess the impact of papers in subsequent years. We apply the proposed model and analysis framework to a large reviews data set from ten different conferences in computer science for a total of ca. 9,000 reviews on ca. 2,800 submitted contributions. We discuss the implications of the results and their potential use toward improving the analysed peer review processes. A number of interesting results were found, in particular: (1) a low correlation between peer review outcome and impact in time of the accepted contributions; (2) the influence of the assessment scale on the way how reviewers gave marks; (3) the effect and impact of rating bias, i.e. reviewers who constantly give lower/higher marks w.r.t. all other reviewers; (4) the effectiveness of statistical approaches to optimize some process parameters (e.g. ,number of papers per reviewer) to improve the process overall quality while maintaining the overall effort under control. Based on the lessons learned, we suggest ways to improve the overall quality of peer-review through procedures that can be easily implemented in current editorial management systems.