Analysis of the evaluation process of the research performance: An empirical case

Summary  In this paper we analyze the objectivity of the peer review process of research performance by research groups in the scientific and technological Valencian system, over the period 1998-2002. For that purpose, we use qualitative and quantitative indicators to assess which of them are the most important to determine a research group as excellent one, based on peer review evaluation methodology. The results show that excellence appears to be driven only by publications in SCI/SSCI and the number of sexenios, and suggest that the peer review process is not as objective as we expected.     

On peer review in computer science: analysis of its effectiveness and suggestions for improvement

In this paper we focus on the analysis of peer reviews and reviewers behaviour in a number of different review processes. More specifically, we report on the development, definition and rationale of a theoretical model for peer review processes to support the identification of appropriate metrics to assess the processes main characteristics in order to render peer review more transparent and understandable. Together with known metrics and techniques we introduce new ones to assess the overall quality (i.e. ,reliability, fairness, validity) and efficiency of peer review processes e.g. the robustness of the process, the degree of agreement/disagreement among reviewers, or positive/negative bias in the reviewers’ decision making process. We also check the ability of peer review to assess the impact of papers in subsequent years. We apply the proposed model and analysis framework to a large reviews data set from ten different conferences in computer science for a total of ca. 9,000 reviews on ca. 2,800 submitted contributions. We discuss the implications of the results and their potential use toward improving the analysed peer review processes. A number of interesting results were found, in particular: (1) a low correlation between peer review outcome and impact in time of the accepted contributions; (2) the influence of the assessment scale on the way how reviewers gave marks; (3) the effect and impact of rating bias, i.e. reviewers who constantly give lower/higher marks w.r.t. all other reviewers; (4) the effectiveness of statistical approaches to optimize some process parameters (e.g. ,number of papers per reviewer) to improve the process overall quality while maintaining the overall effort under control. Based on the lessons learned, we suggest ways to improve the overall quality of peer-review through procedures that can be easily implemented in current editorial management systems.     

Does cumulative advantage affect collective learning in science? An agent-based simulation

Agent-based simulation can model simple micro-level mechanisms capable of generating macro-level patterns, such as frequency distributions and network structures found in bibliometric data. Agent-based simulations of organisational learning have provided analogies for collective problem solving by boundedly rational agents employing heuristics. This paper brings these two areas together in one model of knowledge seeking through scientific publication. It describes a computer simulation in which academic papers are generated with authors, references, contents, and an extrinsic value, and must pass through peer review to become published. We demonstrate that the model can fit bibliometric data for a token journal, Research Policy. Different practices for generating authors and references produce different distributions of papers per author and citations per paper, including the scale-free distributions typical of cumulative advantage processes. We also demonstrate the model’s ability to simulate collective learning or problem solving, for which we use Kauffman’s NK fitness landscape. The model provides evidence that those practices leading to cumulative advantage in citations, that is, papers with many citations becoming even more cited, do not improve scientists’ ability to find good solutions to scientific problems, compared to those practices that ignore past citations. By contrast, what does make a difference is referring only to publications that have successfully passed peer review. Citation practice is one of many issues that a simulation model of science can address when the data-rich literature on scientometrics is connected to the analogy-rich literature on organisations and heuristic search.     

Reorientation patterns in central-place foraging: internal clocks and klinokinesis

We study central-place foraging patterns of Aphaenogaster senilis ants at a population level by video framing individual ant trajectories in a circular arena with a nest connected to its centre. The ants naturally leave and enter the nest and forage generating non-trivial movement patterns around the nest. Our data analysis indicated that the trajectories observed can be classified into two strategies: the risk-averse strategy, which involves wandering around the nest without departing far from it and the risk-prone strategy, which involves long exploration paths with periodic returns to the central region, nearby the nest. We found that both risk-prone and risk-averse strategies exhibit qualitatively the same reorientation patterns, with the time between consecutive reorientations covering a wide range of scales, and fitting a stretched exponential function. Nevertheless, differences in the temporal scales and the time variability of such reorientation events differ, together with other aspects of motion, such as average speed and turns. Our results give experimental evidence that the internal mechanisms driving reorientations in ants tend to favour frequently long relocations, as theory predicts for efficient exploration in patchy landscapes, but ants engaged in central-place foraging can modulate such behaviour to control distances from the nest. Previous works on the species support the idea that risk-prone and risk-averse strategies may reflect actual differences between individuals age and experience; these factors (age and experience) should be then relevant in modulating the internal reorientation clocks. To support the validity of our findings, we develop a random-walk model combining stretched exponential reorientation clocks with klinokinesis that fits the time length and the travelled distance distributions of the observed trajectories.     

The ability of different peer review procedures to flag problematic publications

There is a mounting worry about erroneous and outright fraudulent research that gets published in the scientific literature. Although peer review’s ability to filter out such publications is contentious, several peer review innovations attempt to do just that. However, there is very little systematic evidence documenting the ability of different review procedures to flag problematic publications. In this article, we use survey data on peer review in a wide range of journals to compare the retraction rates of specific review procedures, using the Retraction Watch database. We were able to identify which peer review procedures were used since 2000 for 361 journals, publishing a total of 833,172 articles, of which 670 were retracted. After addressing the dual character of retractions, signalling both a failure to identify problems prior to publication, but also the willingness to correct mistakes, we empirically assess review procedures. With considerable conceptual caveats, we were able to identify peer review procedures that seem able to detect problematic research better than others. Results were verified for disciplinary differences and variation between reasons for retraction. This leads to informed recommendations for journal editors about strengths and weaknesses of specific peer review procedures, allowing them to select review procedures that address issues most relevant to their field.

     

The influence of the applicants’ gender on the modeling of a peer review process by using latent Markov models

In the grant peer review process we can distinguish various evaluation stages in which assessors judge applications on a rating scale. Bornmann & al. [2008] show that latent Markov models offer a fundamentally good opportunity to model statistically peer review processes. The main objective of this short communication is to test the influence of the applicants’ gender on the modeling of a peer review process by using latent Markov models. We found differences in transition probabilities from one stage to the other for applications for a doctoral fellowship submitted by male and female applicants.     

Interfacial mechanics of push-out tests: theory and experiments

The thermo-mechanical characterization of interfaces in composite systems (PMC/MMC/IMC/CMC) is one of the challenging problems in composite mechanics and engineering. Each system has its own distinguishing features; however, in MMCs and IMCs the study is rendered more complex due to the evolving chemical species (both temporally and spatially), and the multi-axial state of residual stresses. Before MMCs or IMCs can be used in actual applications, the role of interfaces in not only the strengthening but also toughening mechanisms needs to be clearly understood. For evaluating the interfacial mechanical properties of interfaces, thin slice push-out test has emerged as the de-facto standard. Though, conceptually the testing procedure is simple, interpretation of the test results is not. It is essential to conduct very careful experiments, make precise meso- and macroscopic chemical/structural/mechanical observations and perform a thorough theoretical/numerical simulation before the test data can be used in a quantitative manner. In this paper, a comprehensive analysis of the push-out test is presented based on the theoretical/numerical and experimental research work of the authors’ group during the past few years.In this work, thin slice push-out tests were conducted primarily on Titanium Matrix Composites at various test temperatures (room and elevated) with different processing conditions (temperature and time). Different composite systems with Titanium based matrices (Ti–6Al–4V, Timetal 21S, Ti–15Nb–3Al) uniaxially reinforced with Silicon Carbide fibers (SCS-6) were chosen for the study. Effect of the evolution of interfacial chemistry and architecture (in matrix, coating and reaction zone) on both shear strength τ_s and frictional strength τ_f were studied. A novel finite element analysis based on nonlinear finite element method was implemented, in which not only the initiation but propagation of interfacial cracks are simulated. In the analysis, both shear stress and fracture energy based criteria are used to model the initiation of (closed) cracks. Quantitative values of τ_s, τ_f, G_I and G_II are then extracted based on the experimental data and the numerical simulation. A critical review of stress and energy based interface-modeling approaches and their applicability to various boundary value problems are made.     

Transient dynamic phenotypes as criteria for model discrimination: fold-change detection in Rhodobacter sphaeroides chemotaxis

The chemotaxis pathway of the bacterium Rhodobacter sphaeroides shares many similarities with that of Escherichia coli. It exhibits robust adaptation and has several homologues of the latter''s chemotaxis proteins. Recent theoretical results have correctly predicted that the E. coli output behaviour is unchanged under scaling of its ligand input signal; this property is known as fold-change detection (FCD). In the light of recent experimental results suggesting that R. sphaeroides may also show FCD, we present theoretical assumptions on the R. sphaeroides chemosensory dynamics that can be shown to yield FCD behaviour. Furthermore, it is shown that these assumptions make FCD a property of this system that is robust to structural and parametric variations in the chemotaxis pathway, in agreement with experimental results. We construct and examine models of the full chemotaxis pathway that satisfy these assumptions and reproduce experimental time-series data from earlier studies. We then propose experiments in which models satisfying our theoretical assumptions predict robust FCD behaviour where earlier models do not. In this way, we illustrate how transient dynamic phenotypes such as FCD can be used for the purposes of discriminating between models that reproduce the same experimental time-series data.     

The reviewer in the mirror: examining gendered and ethnicized notions of reciprocity in peer review

Numerous studies have sought to uncover violations of objectivity and impartiality in peer review; however the notion of reciprocity has been absent in much of this discussion, particularly as it relates to gendered and ethnicized behaviors of peer review. The current study addresses this gap in research by investigating patterns of reciprocity (i.e., correspondences between patterns of recommendations received by authors and patterns of recommendations given by reviewers in the same social group) by perceived gender and ethnicity of reviewers and authors for submissions to the Journal of the American Society for Information Science and Technology from June 2009 to May 2011. The degree of reciprocity for each social group was examined by employing Monte Carlo resampling to extrapolate more robust patterns from the limited data available. We found that papers with female authors received more negative reviews than reviews for male authors. Reciprocity was suggested by the fact that female reviewers gave lower reviews than male reviewers. Reciprocity was also exhibited by ethnicity, although non-Western reviewers gave disproportionately more recommendations of major revision, while non-Western authors tended to receive more outright rejections. This study provides a novel theoretical and methodological basis for future studies on reciprocity in peer review.     

Integrating in situ TEM experiments and atomistic simulations for defect mechanics

With recent advances in computational modeling and in situ transmission electron microscopy (TEM) technologies, there have been increased efforts to apply these approaches to understand defect-based mechanisms dictating deformation mechanics. In situ TEM experiments and atomistic simulations each have their own unique limitations, including observable length and time scales and accessibility of information, motivating approaches that combine the two approaches. In this paper, we review recent studies that combine atomistic simulations and in situ TEM experiments to understand defect mechanisms associated with deformation of metals and alloys. In addition, we discuss ongoing developments in characterization and simulation capabilities that are expected to significantly advance the field of defect mechanics and allow greater integration between atomistic simulations and in situ TEM experiments.     

The objectivity of national research foundation peer review in South Africa assessed against bibliometric indexes

This paper examines the strength of association between the outcomes of National Research Foundation (NRF) peer review based rating mechanisms, and a range of objective measures of performance of researchers. The analysis is conducted on 1932 scholars that have received an NRF rating or an NRF research chair. We find that on average scholars with higher NRF ratings record higher performance against research output and impact metrics. However, we also record anomalies in the probabilities of different NRF ratings when assessed against bibliometric performance measures, and record a disproportionately large incidence of scholars with high peer-review based ratings with low levels of recorded research output and impact. Moreover, we find strong cross-disciplinary differences in terms of the impact that objective levels of performance have on the probability of achieving different NRF ratings. Finally, we report evidence that NRF peer review is less likely to reward multi-authored research output than single-authored output. Claims of a lack of bias in NRF peer review are thus difficult to sustain.     

Extending the scope of ‘in silico experiments’: Theoretical approaches for the investigation of reaction mechanisms, nucleation events and phase transitions

The investigation of the atomistic mechanisms of processes in complex systems constitutes a major challenge to both theory and experiment. While experimental studies offer a wide variety of insights at the macroscopic scale, the atomistic level of detail often remains elusive. On the other hand, molecular simulation approaches may easily achieve microscopic resolution and hence appear particularly suited for detailed mechanistic analyses. However, the computational effort is typically quite considerable and in many cases special simulation strategies are needed to make simulations possible. This review is dedicated to special approaches for tackling the time/length-scale problem inherent to molecular dynamics simulations. Employing these techniques opened a series of new perspectives. The latter are illustrated with the example of recent simulation studies of the atomistic mechanisms involved in complex processes like crystal nucleation, phase transitions and reactions in solution. Along this line, we discuss the reaction mechanisms for He insertion into C₆₀ fullerenes, nucleation events and domain morphogenesis in pressure-induced phase transitions in solids and ion aggregation from solution.