Modelling citation age data: Simple graphical methods from reliability theory

Certain similarities between the types of data reported in retrospective citation analyses and lifetime/survival/reliability models are noted. Graphical techniques much used in reliability analyses are exploited to throw further light on observed citation age distributions and these are then compared and contrasted with previously reported studies. These simple techniques allow systematic departures of empirical data from assumed theoretical models to be highlighted and the models to be compared.     

Examining reliability and validity of job analysis survey data

Historically, job analysis has played a fundamental role for developing and validating licensure and certification examinations. Still, research on what constitutes reliable and valid job analysis data is lacking. This paper illustrates several ways to examine the reliability and validity of job analysis survey results. Generalizability theory and the many-facets Rasch model are applied to investigate consistency and generalizability in task importance measures, to suggest reliable sample size, and to justify the number and use of rating scales. By using random samples from job analysis data for two professions with divergent job activities, this study finds that a representative sample as small as 400 respondents produces reliable estimates of task importance to the same degree of generalizability as obtained from a larger sample of job analysis respondents. Analyses of rating scales suggest that the effectiveness of using different numbers and types of rating scales depends on the nature of a profession.     

The validity of the Roy approximation for citation analysis

The validity of the Roy approximation for citation counting is critically evaluated.     

Mining citation information from CiteSeer data

The CiteSeer digital library is a useful source of bibliographic information. It allows for retrieving citations, co-authorships, addresses, and affiliations of authors and publications. In spite of this, it has been relatively rarely used for automated citation analyses. This article describes our findings after extensively mining from the CiteSeer data. We explored citations between authors and determined rankings of influential scientists using various evaluation methods including citation and in-degree counts, HITS, PageRank, and its variations based on both the citation and collaboration graphs. We compare the resulting rankings with lists of computer science award winners and find out that award recipients are almost always ranked high. We conclude that CiteSeer is a valuable, yet not fully appreciated, repository of citation data and is appropriate for testing novel bibliometric methods.     

Motivations for citation: A comparison of self citation and citation to others

The citation motivations among 51 self citing authors in several natural science disciplines were investigated. Results of a survey on reasons for both self citation and citation to others show that there are very few differences in motivation, and that there are plausible intellectual grounds for those differences which are substantial. Analysis of exposure in text reveals virtually no differences between self citations and citations to others. Analysis of individual disciplines also uncover no substantive differences in either motivation or exposure in text.     

On the possibility and reliability of predictions based on stochastic citation processes

A statistical model for citation processes, a particular version of a non-homogeneous birth process, is analysed in the context of predictions of future citation rates. Important properties of the process were already studied by the author in earlier papers. Although the applicability of the model was demonstrated by several examples, practical aspects of predictions and questions of statistical reliability were not tackled so far. The present study is focused on the demonstration of the possibility of true predictions and on the analysis of the statistical reliability of predictions based on the mean value functionE(X(t)−X(s)/X(s)=i) of citation processes. The citation rates for papers published in 1980 and 1991 were recorded in the period 1980 through 1995, and 1991 through 1995, respectively, in all science areas. It is shown that parameters of mean value functions estimated for earlier time periods can be applied to more recent years, too. As a by-product, the model may serve as a validation tool for the particular choice of citation windows in evaluation studies.     

Time series citation data: the Nobel Prize in economics

Citation time series are not easy to compile from the most popular databases. The Data for Research service of the JSTOR journal database is a large and high-quality sample of citations, weighted towards humanities and social sciences. It provides time series citation data over many decades, back to the origins of the constituent journals. The citation trajectories of Nobel Prize winners in economics are analyzed here from 1930 to 2005. They are described mathematically by means of the Bass model of the diffusion of innovations. A bell-shaped curve provides a good fit with most prize winner citation trajectories, and suggests that economic knowledge follows the typical innovation cycle of adoption, peak, and decline within scholarly careers and shortly afterwards. Several variant trajectories are described.     

Statistical reliability of citation frequency as an indicator of scientific impact

The article deals with the statistical problem of the difference between the mean citation frequencies of two sets of papers required to be significantly different. An analysis of citation data indicated that, as a first-order approximation, (1) The relative spread due to a short observation interval is independent of the long-term citation frequency and (2) the relative spread in long-term citation frequencies of different papers from the same author is independent of the mean citation score for the papers by that author. As a rule-of-thumb, these two sources of variance can be characterized by standard deviations of a ratio (factor) of 2 and 3, respectively. By applying these results to citation data published in the literature, it is shown that sometimes statistically unjustified conclusions have been drawn in the past.     

Modelling citation age data with right censoring

Summary Klaus Fuchs, during his years in England as an immigrant, has written 20 scientific papers. One of these papers, published in 1938, became a fundamental text in solid state physics and for the development of microelectronics in succeeding decades. It was cited more than 1200 times in the period from 1945 until 2003. It appears to be a typical case of delayed recognition in science. Pioneering papers simultaneously written by Hahn & Straßmann and by Meitner & Frisch on the discovery of nuclear fission are considered for comparison.     

Quasi-quantitative measures of research performance: An assessment of construct validity and reliability

This paper argues that research performance is essentially a multidimensional concept which cannot be encapsulated into a single universal criterion. Various indicators used in quantitative studies on research performance at micro or meso-levels can be classified into two broad categories: (i) objective or quantitative indicators (e.g. counts of publications, patents, algorithms or other artifacts of research output) and (ii) subjective or qualitative indicators which represent evaluative judgement of peers, usually measured on Likert or semantic differential scales. Because of their weak measurement properties, subjective indicators can also be designated as quasi-quantitative measures. This paper is concerned with the factorial structure and construct validity of quasi-quantitative measures of research performance used in a large-scale empirical study carried out in India. In this study, a reflective measurement model incorporating four latent variables (R & D effectiveness, Recognition, User-oriented effectiveness and Administrative effectiveness) is assumed. The latent variables are operationalized through thirteen indicators measured on 5-point semantic differential scales. Convergent validity, discriminant validity and reliability of the measurement model are tested through LISREL procedure.     

Exploring reliability data

Despite the recent revolution in statistical thinking and methodology. practical reliability analysis and assessment remains almost exclusively based on a black-box approach employing parametric statistical techniques and significance tests. Such practice, which is largely automatic for the industrial practitioner, implicity involves a large number of physically unreasonable assumptions that in practice are rarely met. Extensive investigation of reliability source data indicates a variety of differing data structures, which contradict the assumptions implicit in the usual methodology. As well as these, lack of homogeneity in the data, due, for instance, to multiple failure modes or misdefinition of environment, is commonly overlooked by the standard methodology. In this paper we argue the case for exploring reliability data. The pattern revealed by such exploration of a data set provides intrinsic information which helps to reinforce and reinterpret the engineering knowledge about the physical nature of the technological system to which the data refers. Employed in this way, the data analyst and the reliability engineer are partners in an iterative process aimed towards the greater understanding of the system and the process of failure. Despite current standard practice, the authors believe it to be critical that the structure of data analysis must reflect the structure in the failure data. Although standard methodology provides an easy and repeatable analysis, the authors' experience indicates that it is rarely an appropriate one. It is ironic that whereas methods to analyse the data structures commonly found in reliability data have been available for some time, the insistence about the standard black-box approach has prevented the identification of such ‘abnormal’ features in reliability data and the application of these approaches. We discuss simple graphical procedures to investigate the structure of reliability data, as well as more formal testing procedures which assist in decision-making methodology. Partial reviews of such methods have appeared previously and a more detailed development of the exploration approach and of the appropriate analysis it implies will be dealt with elsewhere. Here, our aim is to argue the case for the reliability analyst to LOOK AT THE DATA. and to analyse it accordingly.     

Statistical reliability of comparisons based on the citation impact of scientific publications

A method for estimating the standard error of mean citation rates per publication is proposed and examplified on journal impact factors. The use of the standard error values in statistical tests is also illustrated.     

A note on the validity and reliability of ergonomics methods

The view that, by analogy, ergonomics methods should be subject to the same requirements of validity and reliability as are demanded of psychometric tests is shown to be an oversimplification. Two broad classes of method are distinguished. Analytic methods aim to produce a better understanding of the processes affecting complex human-machine systems, whilst evaluative methods are limited to the measurement of specific variables. The analogy with psychometrics applies to the latter type but is less appropriate to the former. Analytic methods depend heavily on the application of currently accepted theories of performance. Empirical data may be variable, but the expertise of the analyst may be more to blame than the method as such.     

Certainty equivalent citation: generalized classes of citation indexes

Drawing from the existing literature on risk and inequality measurement, we implement the notion of “certainty equivalent citation” in order (i) to generalize most of the h-type citation indexes (h-, g-, $\tilde{g},$ t-, f-, w-index), and (ii) to highlight the centrality of the decision-maker’s preferences on distributive aspects (concentration aversion) for the ranking of citation profiles. In order to highlight the sensitivity of citation orderings with respect to concentration aversion, an application to both simulated and real citation profiles is presented.     

On the validity of citation counting in science evaluation: Content analyses of references and citations in psychological publications

In reference to the increasing significance of citation counting in evaluations of scientists and science institutes as well as in science historiography, it is analyzed empirically what is cited in which frequency and what types of citations in scientific texts are used. Content analyses refer to numbers of references, self-references, publication language of references cited, publication types of references cited, and type of citation within the texts. Validity of citation counting is empirically analyzed with reference to random samples of English and German journal articles as well as German textbooks, encyclopedias, and test-manuals from psychology. Results show that 25% of all citations are perfunctory, more than 50% of references are journal articles and up to 40% are books and book-chapters, 10% are self-references. Differences between publications from various psychological sub-disciplines, publication languages, and types of publication are weak. Thus, validity of evaluative citation counting is limited because at least one quarter refers to perfunctory citations exhibiting a very low information utility level and by the fact that existing citation-databases refer to journal articles only.     

The Dula Dangerous Driving Index: an investigation of reliability and validity across cultures

The aim of this study is to further establish the validity and reliability of the Dula Dangerous Driving Index (DDDI). The reliability and validity of the instrument was investigated by comparing data from a US university sample, a US community sample, and a sample of Belgian traffic offenders. Exploratory and confirmatory factor analysis supported the presence of a four-factor structure with items for Drunk Driving forming a separate scale apart from items for Risky Driving, Negative Cognitive/Emotional Driving and Aggressive Driving. A multi-group confirmatory factor analysis with model constraints supported the validity of the DDDI. Inter-correlations revealed that the DDDI subscales are closely interrelated and uni-dimensionality of the measure was found in all three samples. This suggests the DDDI Total score can be used as a composite measure for dangerous driving. However, the validity of the subscales was demonstrated in the Belgian sample, as specific traffic offender groups (convicted for drunk driving, aggressive driving, speeding) scored higher on corresponding scales (Drunk Driving, Aggressive Driving, and Risky Driving, respectively), indicating that it is clinically meaningful to differentiate the subscales.     

The Dula Dangerous Driving Index in China: An investigation of reliability and validity

The aim of this study was to translate the Dula Dangerous Driving Index (DDDI) into Chinese and to verify its reliability and validity. A total of 246 drivers completed the Chinese version of the DDDI and the Driver Behavior Questionnaire (DBQ). Specific sociodemographic variables and traffic violations were also measured. A confirmatory factor analysis confirmed the internal structure of the DDDI, and the four-factor model was supported in China. Measures of convergent and criterion validity demonstrated that the Chinese DDDI was valid. Its convergent validity was supported by its positive relationship with the DBQ, and its criterion validity was tested using its relationship with self-reported accident involvement and traffic violations. Finally, score comparisons between different demographic groups revealed significant differences, thereby linking age and driving years to dangerous driving.     

Practical reliability data analysis

The practical problems encountered in Reliability Data Analysis are considered, particularly when censoring is an important feature. Approaches to Statistical Data Analysis rather than specific techniques are described. Careful examination of the data is recommended using simple plotting methods. The importance of the objective of a Reliability Study is emphasized as this must always be kept in mind. Statistical methods should be a tool to achieve a given end. Three examples of Reliability Studies from three different areas of application are used for illustration. These examples consist of data collected from the field and do not cover laboratory-controlled testing.