A study of operational and testing reliability in software reliability analysis

Software reliability is an important aspect of any complex equipment today. Software reliability is usually estimated based on reliability models such as nonhomogeneous Poisson process (NHPP) models. Software systems are improving in testing phase, while it normally does not change in operational phase. Depending on whether the reliability is to be predicted for testing phase or operation phase, different measure should be used. In this paper, two different reliability concepts, namely, the operational reliability and the testing reliability, are clarified and studied in detail. These concepts have been mixed up or even misused in some existing literature. Using different reliability concept will lead to different reliability values obtained and it will further lead to different reliability-based decisions made. The difference of the estimated reliabilities is studied and the effect on the optimal release time is investigated.     

Astrophysics publications on arXiv,Scopus and Mendeley: a case study

In this study we examined a sample of 100 European astrophysicists and their publications indexed by the citation database Scopus, submitted to the arXiv repository and bookmarked by readers in the reference manager Mendeley. Although it is believed that astrophysicists use arXiv widely and extensively, the results show that on average more items are indexed by Scopus than submitted to arXiv. A considerable proportion of the items indexed by Scopus appear also on Mendeley, but on average the number of readers who bookmarked the item on Mendeley is much lower than the number of citations reported in Scopus. The comparisons between the data sources were done based on the authors and the titles of the publications.     

Contribution of Chinese publications in computer science: A case study on LNCS

Conference proceedings are one of the key communication channels in computer science. This paper aims to analyze the Chinese outputs in the context of conference papers in computer science through an exploration of the conference proceedings series book-Lecture Notes in Computer Science (LNCS) in the period of 1997–2005. Results indicate that: 1. The number of Chinese papers in LNCS keeps growing in the studied period; the share of Chinese papers in LNCS in recent years is much higher than that of Chinese SCI papers in the world; In sharp contrast with remarkable growth of the share of Chinese papers in LNCS, the share of SCI articles in top journals of computer science published by the scientists of mainland China is negligible during the same period. 2. Chinese researchers are more likely to collaborate with domestic fellows; 3. In spite of the increasing amounts of Chinese papers in LNCS, they receive only a few citations; 4. The articles are strikingly more cited by authors themselves and international authors’ citations are more than Chinese authors’ non-self-citations in the first three years after publication; 5. Based on the new indicator Impact Index (II) the authors proposed, the relative impact of Chinese articles in LNCS is increasing although the average impact of Chinese papers in LNCS is obviously less than that of the publications in LNCS in each year during the studied period.     

Field and numerical analysis of in-situ air sparging: a case study

An in-situ air sparging operation was used to remediate the sandy subsurface soils and shallow groundwater under a drum storage site near Chicago, IL, where either periodic or random spillage of a light non-aqueous phase liquid (LNAPL) occurred between 1980 and 1987. Both field measurements and model simulations using commercially available computer software suggested that microbial degradation was the most significant contributor to the removal of contaminant mass. Toluene, ethylbenzene and total xylenes (TEX), which were of major concern with regards to reaching clean-up criteria at the site, were observed to decline by 88% in concentration. Furthermore, up to 97% of the total mass removed through microbial degradation consisted of TEX. Of the total contaminant spill, up to 23% of initial organic chemical mass was removed through microbial degradation compared to less than 6% by physical stripping. Greater loss to microbial degradation is most likely attributed to the relatively low air injection rate used during the course of the air sparging remediation. Evaluation of air sparging at the site using model simulations supported this analysis by estimating 140 and 620 kg of total contaminant mass being removed through volatilization and biodegradation, respectively. An evaluation of several system design parameters using model simulations suggested that only the type of sparging operation (i.e. pulsed or continuous) was significant in terms of total contaminant removal time, while both the sparging operation and air injection rate were significant in terms of removal of a critical species, total xylenes.     

Age-related maintenance versus reliability centred maintenance: a case study on aero-engines

Reliability Centred Maintenance (RCM) is a procedure carried out as part of the logistic support analysis (LSA) process and is described in the US Department of Defence Military Standards (Mil Std 2173). RCM allows logisticians the opportunity to determine the best maintenance policy for each component within a system. However, the only data that are available to carryout RCM using Mil Std 2173 are of MTBF. This implies that all the necessary mathematical models need to be based on the exponential distribution. This is a serious drawback to the whole concept of RCM as the exponential distribution cannot be used to model items that fail due to wear, or any other mode that is related to their age. In this paper, a new approach to RCM is proposed using the concepts of soft life and hard life to optimise the total maintenance cost. For simplicity, only one mode of failure is considered for each component. However, the model can be readily applied to multiple failure modes. The proposed model is applied to find the optimal maintenance policies in the case of military aero-engines using Monte Carlo simulation. The case study shows a potential benefit from setting soft lives on relatively cheap components that can cause expensive, unplanned engine rejections.     

Highly cited non-journal publications in political science,economics and psychology: a first exploration

In this study we show that it is possible to identify top-cited publications other than Web of Science (WoS) publications, particularly non-journal publications, within fields in the social and behavioral sciences. We analyzed references in publications that were themselves highly cited, with at least one European address. Books represent between 62 (psychology) and 81% (political science) of the non-WoS references, journal articles 15–24%. Books (economics, political science) and manuals (psychology) account for the most highly cited publications. Between 50 (psychology, political science) and 71% (economics) of the top-ranked most cited publications originated from the US versus between 18 (economics) and 38% (psychology) from Europe. Finally, it is discussed how the methods and procedures of the study can be optimized.     

Thermophysical properties of nanoobjects: Data classification and validity evaluation

The common problems of acquisition, classification, and certification of numerical data on the properties of nanoobjects have been considered. The peculiarity of their physical properties is shown to affect the preparation procedure of the informational stock which predates database creation. As an example, the data on the properties of carbon nanomodifications (nanotubes, graphene, etc.) are given. The key features of the data on nanostructures have been detected: frequent variations of the property nomenclature, size effect, and high level of data uncertainty. A procedure of the data certification is suggested, which includs information on the uncertainty and the quality indexes, estimating the completeness of the information on the subject of inquiry, the measurement/estimation method, and the reproducibility of results.     

A case study for quantifying system reliability and uncertainty

The ability to estimate system reliability with an appropriate measure of associated uncertainty is important for understanding its expected performance over time. Frequently, obtaining full-system data is prohibitively expensive, impractical, or not permissible. Hence, methodology which allows for the combination of different types of data at the component or subsystem levels can allow for improved estimation at the system level. We apply methodologies for aggregating uncertainty from component-level data to estimate system reliability and quantify its overall uncertainty. This paper provides a proof-of-concept that uncertainty quantification methods using Bayesian methodology can be constructed and applied to system reliability problems for a system with both series and parallel structures.     

Evidence for the reliability of measures and validity of measure interpretation: a Rasch measurement perspective

In an era of high stakes testing and evaluation in education, psychology, and health care, there is need for rigorous methods and standards for obtaining evidence of the reliability of measures and validity of inferences. Messick (1989, 1995), the Standard for Educational and Psychological Testing (American Psychological Association, American Educational Research Association, and National Council on Measurement in Education, 1999), and the Medical Outcomes Trust (1995), among others, have described methods that may be used to gather evidence for reliability and validity, but ignored the potential role Rasch measurement may contribute to this process. This article will outline methods in Rasch measurement that are used to gather evidence for reliability and validity and attempt to articulate how these methods may be linked with the current views of reliability and validity.     

Language trends in nanoscience and technology: The case of Chinese-language publications

Nanoscience and technology (NST) is a young scientific and technological field that has generated great worldwide interest in the past two decades. Previous bibliometric analyses have unmistakably demonstrated the remarkable growth of the global NST literature. While almost all published research articles in NST are in English, increasingly a larger share of NST publications is published in the Chinese language. Perplexingly, Chinese is the only language — apart from English — that displays an ascendant trend in the NST literature. In this brief note, we explore and evaluate three arguments that could explain this phenomenon: coverage bias, language preference, and community formation.     

Analysis of a number and type of publications that editors publish in their own journals: case study of scholarly journals in Croatia

To assess the publication practices of editors in their own journals, we analysed the number of articles that Croatian editors published in the journals they edit. From 2005 to 2008, 256 decision-making editors of 180 journals published a total of 887 publications in their own journals. Out of these, 332 were relevant for their academic promotion. Only 18 editors published 5 or more articles in their own journals. A single journal had regulations for self-publishing in the instructions for authors. Although the majority of editors did not misuse their own journals for scientific publishing and academic promotion, there is a need for greater transparency of the declaration and management of editorial conflict of interest in academic and scholarly journals.     

The Dula Dangerous Driving Index in China: An investigation of reliability and validity

The aim of this study was to translate the Dula Dangerous Driving Index (DDDI) into Chinese and to verify its reliability and validity. A total of 246 drivers completed the Chinese version of the DDDI and the Driver Behavior Questionnaire (DBQ). Specific sociodemographic variables and traffic violations were also measured. A confirmatory factor analysis confirmed the internal structure of the DDDI, and the four-factor model was supported in China. Measures of convergent and criterion validity demonstrated that the Chinese DDDI was valid. Its convergent validity was supported by its positive relationship with the DBQ, and its criterion validity was tested using its relationship with self-reported accident involvement and traffic violations. Finally, score comparisons between different demographic groups revealed significant differences, thereby linking age and driving years to dangerous driving.     

Examining reliability and validity of job analysis survey data

Historically, job analysis has played a fundamental role for developing and validating licensure and certification examinations. Still, research on what constitutes reliable and valid job analysis data is lacking. This paper illustrates several ways to examine the reliability and validity of job analysis survey results. Generalizability theory and the many-facets Rasch model are applied to investigate consistency and generalizability in task importance measures, to suggest reliable sample size, and to justify the number and use of rating scales. By using random samples from job analysis data for two professions with divergent job activities, this study finds that a representative sample as small as 400 respondents produces reliable estimates of task importance to the same degree of generalizability as obtained from a larger sample of job analysis respondents. Analyses of rating scales suggest that the effectiveness of using different numbers and types of rating scales depends on the nature of a profession.     

Methods for multidimensional event classification: a case study using images from a Cherenkov gamma-ray telescope

We present results from a case study comparing different multivariate classification methods. The input is a set of Monte Carlo data, generated and approximately triggered and pre-processed for an imaging gamma-ray Cherenkov telescope. Such data belong to two classes, originating either from incident gamma rays or caused by hadronic showers. There is only a weak discrimination between signal (gamma) and background (hadrons), making the data an excellent proving ground for classification techniques.The data and methods are described, and a comparison of the results is made. Several methods give results comparable in quality within small fluctuations, suggesting that they perform at or close to the Bayesian limit of achievable separation. Other methods give clearly inferior or inconclusive results. Some problems that this study can not address are also discussed.     

Quasi-quantitative measures of research performance: An assessment of construct validity and reliability

This paper argues that research performance is essentially a multidimensional concept which cannot be encapsulated into a single universal criterion. Various indicators used in quantitative studies on research performance at micro or meso-levels can be classified into two broad categories: (i) objective or quantitative indicators (e.g. counts of publications, patents, algorithms or other artifacts of research output) and (ii) subjective or qualitative indicators which represent evaluative judgement of peers, usually measured on Likert or semantic differential scales. Because of their weak measurement properties, subjective indicators can also be designated as quasi-quantitative measures. This paper is concerned with the factorial structure and construct validity of quasi-quantitative measures of research performance used in a large-scale empirical study carried out in India. In this study, a reflective measurement model incorporating four latent variables (R & D effectiveness, Recognition, User-oriented effectiveness and Administrative effectiveness) is assumed. The latent variables are operationalized through thirteen indicators measured on 5-point semantic differential scales. Convergent validity, discriminant validity and reliability of the measurement model are tested through LISREL procedure.     

Inter-rater reliability and validity of peer reviews in an interdisciplinary field

Peer review is an integral part of science. Devised to ensure and enhance the quality of scientific work, it is a crucial step that influences the publication of papers, the provision of grants and, as a consequence, the career of scientists. In order to meet the challenges of this responsibility, a certain shared understanding of scientific quality seems necessary. Yet previous studies have shown that inter-rater reliability in peer reviews is relatively low. However, most of these studies did not take ill-structured measurement design of the data into account. Moreover, no prior (quantitative) study has analyzed inter-rater reliability in an interdisciplinary field. And finally, issues of validity have hardly ever been addressed. Therefore, the three major research goals of this paper are (1) to analyze inter-rater agreement of different rating dimensions (e.g., relevance and soundness) in an interdisciplinary field, (2) to account for ill-structured designs by applying state-of-the-art methods, and (3) to examine the construct and criterion validity of reviewers’ evaluations. A total of 443 reviews were analyzed. These reviews were provided by m = 130 reviewers for n = 145 submissions to an interdisciplinary conference. Our findings demonstrate the urgent need for improvement of scientific peer review. Inter-rater reliability was rather poor and there were no significant differences between evaluations from reviewers of the same scientific discipline as the papers they were reviewing versus reviewer evaluations of papers from disciplines other than their own. These findings extend beyond those of prior research. Furthermore, convergent and discriminant construct validity of the rating dimensions were low as well. Nevertheless, a multidimensional model yielded a better fit than a unidimensional model. Our study also shows that the citation rate of accepted papers was positively associated with the relevance ratings made by reviewers from the same discipline as the paper they were reviewing. In addition, high novelty ratings from same-discipline reviewers were negatively associated with citation rate.     

On the validity of citation counting in science evaluation: Content analyses of references and citations in psychological publications

In reference to the increasing significance of citation counting in evaluations of scientists and science institutes as well as in science historiography, it is analyzed empirically what is cited in which frequency and what types of citations in scientific texts are used. Content analyses refer to numbers of references, self-references, publication language of references cited, publication types of references cited, and type of citation within the texts. Validity of citation counting is empirically analyzed with reference to random samples of English and German journal articles as well as German textbooks, encyclopedias, and test-manuals from psychology. Results show that 25% of all citations are perfunctory, more than 50% of references are journal articles and up to 40% are books and book-chapters, 10% are self-references. Differences between publications from various psychological sub-disciplines, publication languages, and types of publication are weak. Thus, validity of evaluative citation counting is limited because at least one quarter refers to perfunctory citations exhibiting a very low information utility level and by the fact that existing citation-databases refer to journal articles only.     

A note on the validity and reliability of ergonomics methods

The view that, by analogy, ergonomics methods should be subject to the same requirements of validity and reliability as are demanded of psychometric tests is shown to be an oversimplification. Two broad classes of method are distinguished. Analytic methods aim to produce a better understanding of the processes affecting complex human-machine systems, whilst evaluative methods are limited to the measurement of specific variables. The analogy with psychometrics applies to the latter type but is less appropriate to the former. Analytic methods depend heavily on the application of currently accepted theories of performance. Empirical data may be variable, but the expertise of the analyst may be more to blame than the method as such.