Are international co-publications an indicator for quality of scientific research?

This article deals with the role of internationally co-authored papers (co-publications). Specifically, we compare, within a data-set of German research units, citation and co-publication indicators as a proxy for the unobserved quality dimension of scientific research. In that course we will also deal with the question whether both citations and co-publications are considerably related. Our results suggest that, although there is a strong partial correlation between citations and co-publications within a multivariate setting, we cannot use reasonably normalised co-publication indicators as an alternative proxy for quality. Thus, concerning quality assessment, there remains a primer on citation analysis.     

International comparisons of scientific performance revisited

This paper presents a methodological analysis of the latest update of the CHI/NSF Science Literature Indicators Data-Base. The data-base contains a range of publication and citation indicators borken down by country and field or subfield, and now convers the period from 1973 to 1984. It can be used to draw comparisons of the changing output and impact of basic research in different countries. Earlier applications of the data-base have been constrained by various technical limitations, and have been subject to certain criticism. In this article, after some conceptual analysis of what aspects of scientific performance the different indicators relate to, we show that much of the criticism is misplaced. We also describe subsequent methodological improvements to the indicators and the effect these have on the policy use that can be made. Finally, we examine what the latest statistics reveal about the relative international standing of seven leading scientific nations.     

Assessing insularity in global science

Most scientific research has some form of local geographical bias. This could be caused by researchers addressing a geographically localized issue, working within a nationally or regionally defined research network, or responding to research agendas that are influenced by national policy. These influences should be reflected in citation behavior, e.g., more citations than expected by chance of papers by scientists from institutions within the same country. Thus, assessing adjusted levels of national self-citation may give insights into the extent to which national research agendas and scientific cultures influence the behavior of scientists. Here we develop a simple metric of scientific insularism based on rates of national self citation corrected for total scientific output. Based on recent publications (1996?C2010), higher than average levels of insularism are associated with geographically large rapidly developing nations (Brazil, Russia, India, and China??the so-called BRIC nations), and countries with strongly ideological political regimes (Iran). Moreover, there is a significant negative correlation between insularism and the average number of citations at the national level. Based on these data we argue that insularism (higher than average levels of national self-citation) may reflect scientific cultures whose priorities and focus are less tightly linked to global scientific norms and agendas. We argue that reducing such insularity is an overlooked challenge that requires policy changes at multiple levels of science education and governance.     

Early citation counts correlate with accumulated impact

Summary The present paper addresses the objective of developing forward indicators of research performance using bibliometric information on the UK science base. Most research indicators rely primarily on historical time series relating to inputs to, activity within and outputs from the research system. Policy makers wish to be able to monitor changing research profiles in a more timely fashion, the better to determine where new investment is having the greatest effect. Initial (e.g. 12 months from publication) citation counts might be useful as a forward indicator of the long-term (e.g. 10 years from publication) quality of research publications, but - although there is literature on citation-time functions - no study to evaluate this specifically has been carried out by Thomson ISI or any other analysts. Here, I describe the outcomes of a preliminary study to explore these citation relationships, drawing on the UK National Citation Report held by Evidence Ltd under licence from Thomson ISI for OST policy use. Annual citation counts typically peak at around the third year after publication. I show that there is a statistically highly significant correlation between initial (years 1-2) and later (years 3-10) citations in six research categories across the life and physical sciences. The relationship holds over a wide range of initial citation counts. Papers that attract more than a definable but field dependent threshold of citations in the initial period after publication are usually among the top 1% (the most highly cited papers) for their field and year. Some papers may take off slowly but can later join the high impact group. It is important to recognise that the statistical relationship is applicable to groups of publications. The citation profiles of individual articles may be quite different. Nonetheless, it seems reasonable to conclude that leading indicators of research excellence could be developed. This initial study should now be extended across a wider range fields to test the initial outcomes: earlier papers suggest the model holds in economics. Additional statistical tests should be applied to explore and model the relationship between initial, later and total citation counts and thus to create a general tool for policy application.     

Validation of bibliometric indicators in the field of mathematics

Bibliometric analyses of scientific publications provide quantitative information that enables evaluators to obtain a useful picture of a team's research visibility. In combination with peer judgements and other qualitative background knowledge, these analyses can serve as a basis for discussions about research performance quality. However, many mathematicians are not convinced that citation counts do in fact provide useful information in the field of mathematics. According to these mathematicians, citation and publication habits differ completely from scholarly fields such as chemistry or physics. Therefore, it is impossible to derive valid information regarding research performance from citation counts. The aim of this study is to obtain more insight into the significance of citation-based indicators in the field of mathematics. To which extent do citation-scores mirror to the opinions of experts concerning the quality of a paper or a journal? A survey was conducted to answer this question.Top journals, as qualified by experts, receive significantly higher citation rates thangood journals. Thesegood journals, in turn, have significantly higher scores than journals with the qualificationless good. Top publications, recorded in the ISI database, receive on the average 15 times more citations than the mean score within the field of mathematics as a whole. In conclusion, the experts' views on top publications or top journals correspond very well to bibliometric indicators based on citation counts.     

Reasons for and developments in international scientific collaboration: does an Asia–Pacific research area exist from a bibliometric point of view?

This paper describes the different forms of and tries to give reasons for international scientific collaboration in general. It focuses on eleven countries in the Asia–Pacific region by evaluating their national research output with the help of bibliometric indicators in particular. Over two million journal articles published by these countries between 1998 and 2007 in ISI-listed periodicals are analyzed. Discipline-specific publication and citation profiles reveal national strengths and weaknesses in the different research domains. The exponential increase in publication output by China over the last few years is astonishing, but in terms of visibility, i.e. citation rates, China cannot keep up with leading science nations, remaining below the world average. A discipline-specific analysis shows that Chinese authors took an active part in more than a quarter of all articles and reviews published in the field of materials science in 2007, while their contribution to medical research is very low. Co-publication networks among the eleven countries are generated to observe the development of cooperation bonds in the region. Applying Salton’s measure of international collaboration strength, an above-average strengthening of scientific collaboration in the Asia–Pacific region can be observed.     

A study on the citation situation within the citing paper: citation distribution of references according to mention frequency

Citation impact indicators play a significant role in evaluating the scientific research activity. Most of citation impact indicators are based on the citation count that the publication is cited as a reference in the other publications, but the difference between each citation situation was not considered. Normally, the number of citations that a publication is cited in the other publications may represent the formal quality of the publication. Similarly, the number of times that a publication is really mentioned within the citing publication, it may also represent the formal quality of the citation. We have examined about how many times each reference was really mentioned within the citing publications and studied about the citation situation within the citing publications. We verified that the citation distribution of references according to the mention frequency follows the Generalized Pareto distribution. The results showed that about 20% of total references were mentioned three and more times, and the number of citation mentions for the about 50% of total references were from about 20% of the total references in the citing publications.     

Absolute and specific measures of research group excellence

A desirable goal of scientific management is to introduce, if it exists, a simple and reliable way to measure the scientific excellence of publicly funded research institutions and universities to serve as a basis for their ranking and financing. While citation-based indicators and metrics are easily accessible, they are far from being universally accepted as way to automate or inform evaluation processes or to replace evaluations based on peer review. Here we consider absolute measurements of research excellence at an amalgamated, institutional level and specific measures of research excellence as performance per head. Using biology research institutions in the UK as a test case, we examine the correlations between peer review-based and citation-based measures of research excellence on these two scales. We find that citation-based indicators are very highly correlated with peer-evaluated measures of group strength, but are poorly correlated with group quality. Thus, and almost paradoxically, our analysis indicates that citation counts could possibly form a basis for deciding on, how to fund research institutions, but they should not be used as a basis for ranking them in terms of quality.     

The global science base of information and communication technologiges: Bibliometric analysis of ICT research papers

The science and engineering base is a key source of knowledge for the development and use of Information and Communication Technologies (ICTs). In order to be able to effectively describe and monitor world-wide scientific activity related to ICTs, it is important to be able to provide reliable macro-level statistics of this knowledge base. International bibliographic databases and related bibliometric indicators together provide an analytical framework and appropriate measures to cover both the ‘supply side’—research capabilities and outputs—and ‘demand side’—collaboration, diffusion and citation impact—related to the ICT research. This paper presents results of such a bibliometric study describing macro-level features of this ICT knowledge base. The data were retrieved from a specially developedCWTS ICT Database which provides a broad-scope world-wide coverage of ICT-relevant research papers published in high-quality international scientific and technical journals. The cross-country comparison focuses on the level of scientific output and co-operation patterns of the most actively publishing nations with a focus on the three Triad zones—the European Union, the USA and Japan.     

A macro study of self-citation

This study investigates the role of self-citation in the scientific production of Norway (1981-1996). More than 45,000 publications have been analysed. Using a three-year citation window we find that 36% of all citations represent author self-citations. However, this percentage is decreasing when citations are traced for longer periods. We find the highest share of self-citation among the least cited papers. There is a strong positive correlation between the number of self-citations and the number of authors of the publications. Still, only a minor part of the overall increase in citation rates that can be found for multi-authored papers is due to self-citations. Also, the share of self-citation shows significant variations among different scientific disciplines. The results are relevant for the discussion concerning use of citation indicators in research assessments. This revised version was published online in August 2006 with corrections to the Cover Date.     

Evaluation of Moroccan research using a bibliometric-based approach: investigation of the validity of the h-index

This paper presents the results of an evaluation of the national research system in Morocco. The exercise focuses on the period 1997–2006 and includes a comparison with South Africa, Egypt, Nigeria, Tunisia, Algeria, Portugal and Greece. Ratings of highly ranked researchers are developed on the basis of their number of publications, number of citations and also their ‘h-index’ (or Hirsch index). Finally, we examine the empirical model set by Glänzel that related the h-index to the number of publications and the mean citation rate per paper for these ‘upper-class’ researchers. The use of this model confirms that the h-index is likely to reflect the importance and the quality of the scientific output of a given researcher.     

Profiling institutes: Identifying high research performance and social relevance in the social and behavioral sciences

This study focuses on the identification of high output research and high impact research in the social and behavioral sciences. A second objective is to monitor developments in research that is related to societal needs and problems. For each topic, we identify institutes and authors that have contributed a considerable number of SSCI articles and/or several (relatively) highly cited articles on a topic. To identify papers with a (relatively) high impact, the present study used two citation thresholds, each based upon a combination of a statistically determined minimum number of ‘external’ citations (by others than the authors of a paper), and a statistical comparison with world average citation levels. Topics were categorised according to social relevance. Detailed profiles of both large and small institutes are obtained, showing strengths and weaknesses in research performance that tend to be obscured by standard indicators focusing on ‘average’ research performance. Socially relevant topics tend to be concentrated in multidisciplinary clusters, whereas clusters dominated by one or two disciplines contain more basis research. The results provide a first insight in the extent to which institutes and authors address socially relevant topics. Use of the results by policy bodies seems dependent upon which organisational level of research they tend to address.     

Inflationary bibliometric values: The role of scientific collaboration and the need for relative indicators in evaluative studies

Several research studies and reports on national and European science and technology indicators have recently presented figures reflecting intensifying scientific collaboration and increasing citation impact in practically all science areas and at all levels of aggregation. The main objective of this paper is twofold, namely first to analyse if the number or weight of actors in scientific communication has increased, if patterns of documented scientific communication and collaboration have changed in the last two decades and if these tendencies have inflationary features. The second question is concerned with the role of scientific collaboration in this context. In particular, the question will be answered to what extent co-authorship and publication activity, on one hand, and co-authorship and citation impact, on the other hand, do interact. The answers found to these questions have strong implication for the application of bibliometric indicators in research evaluation, moreover, the construction of indicators applied to trend analyses and studies based on medium-term or long-term observations have to be reconsidered to guarantee the validity of conclusions drawn from bibliometric results. This revised version was published online in June 2006 with corrections to the Cover Date.     

Using Hypertext Metrics to Measure Research Output Levels

Two common ways to measure the "output" of a researcher (or research group) are to count numbers of publications and to count the citations (references to these publications in publications of others). These simple methods are flawed because they cannot easily take into account the differences in publication and citation habits in different scientific communities.An alternative approach is to view citations as hypertext links, and to use or adapt hypertext metrics to compare the scientific output of researchers, in comparison to that of others in areas with similar publication and citation patterns. We show how hypertext metrics, introduced by Botafogo, Rivlin and Shneiderman, can be modified in order to identify comparable research fields based on their publication and citation pattern. An author's performance can then be compared to that of others in research fields with a similar pattern.     

Korean journals in the Science Citation Index: What do they reveal about the intellectual structure of S&T in Korea?

During the last decade, we have witnessed a sustained growth of South Korea’s research output in terms of the world share of publications in the Science Citation Index database. However, Korea’s citation performance is not yet as competitive as publication performance. In this study, the authors examine the intellectual structure of Korean S&T field based on social network analysis of journal-journal citation data using the ten Korean SCI journals as seed journals. The results reveal that Korean SCI journals function more like publication places, neither research channels nor information sources among national scientists. Thus, these journals may provide Korean scholars with access to international scientific communities by facilitating the respective entry barriers. However, there are no citation relations based on their Korean background. Furthermore, we intend to draw some policy implications which may be helpful to increase Korea’s research potential.     

Scientometrics of laser research in India during 1970-1994

An analysis of 952 publications published by Indian scientists and abstracted by Journal of Current Laser Abstracts during 1970-1994 indicates that laser research in India picked up during 1978-1994 and reached its peak in 1980. The Indian output in the field of laser research forms an integral part of the mainstream science as reflected by the pattern of publications and their citations in the international literature. Laser research performed in India improved considerably during 1985-1994 as compared to 1970-1984 as seen by different impact indicators such as citation per paper, proportion of high quality papers, and publication effective index. The publication output is concentrated among few institutions and there is a similarity in the activity and attractively profile of the highly productive institutions. India"s citation rate per paper for highly productive authors is at par with the world citation rate per paper. The study indicates that the proportion of mega authored papers increased during 1990-1994 and the international collaboration is mainly with the USA.     

Citations versus journal impact factor as proxy of quality: could the latter ever be preferable?

In recent years bibliometricians have paid increasing attention to research evaluation methodological problems, among these being the choice of the most appropriate indicators for evaluating quality of scientific publications, and thus for evaluating the work of single scientists, research groups and entire organizations. Much literature has been devoted to analyzing the robustness of various indicators, and many works warn against the risks of using easily available and relatively simple proxies, such as journal impact factor. The present work continues this line of research, examining whether it is valid that the use of the impact factor should always be avoided in favour of citations, or whether the use of impact factor could be acceptable, even preferable, in certain circumstances. The evaluation was conducted by observing all scientific publications in the hard sciences by Italian universities, for the period 2004–2007. Performance sensitivity analyses were conducted with changing indicators of quality and years of observation.     

‘Triad’ or ‘tetrad’? On global changes in a dynamic world

The US-EU race for world leadership in science and technology has become the favourite subject of recent studies. Studies issued by the European Commission reported the increase of the European share in the world’s scientific production and announced world leadership of the EU in scientific output at the end of the last century. In order to be able to monitor those types of global changes, the present study is based on the 15-year period 1991–2005. A set of bibliometric and technometric indicators is used to analyse activity and impact patterns in science and technology output. This set comprises publication output indicators such as (1) the share in the world total, (2) subject-based publication profiles, (3) citation-based indicators like journal-and subject-normalised mean citation rates, (4) international co-publications and their impact as well as (5) patent indicators and publication-patent citation links (both directions). The evolution of national bibliometric profiles, ‘scientific weight’ and science-technology linkage patterns are discussed as well. The authors show, using the mirror of science and technology indicators, that the triad model does no longer hold in the 21^st century. China is challenging the leading sciento-economic powers and the time is approaching when this country will represent the world’s second largest potential in science and technology. China and other emerging scientific nations like South Korea, Taiwan, Brazil and Turkey are already changing the balance of power as measured by scientific production, as they are at least in part responsible for the relative decline of the former triad.     

The “meaning” of citation in the context of a scientifically peripheral country

This paper reports an investigation into the referencing pattern of Brazilian agricultural scientists. The study was based on the use of both quantitative data-citations appearing in a sizeable sample of articles published by these scientists—and qualitative data-interviews with a large number of scientists who authored the source, papers. The aim was to explore the extent to which citation counts may be taken as valid indicators of the quality, influence or impact of published scientific knowledge in the general context of a scientifically peripheral country. The findings presented confirm the view that in this context, citation patterns are significantly influenced by factors external to the scientific realm and, thus, reflect neither simply the quality, influence nor even the impact of the research work referred to.