Bibliometric performance measures

Three different types of bibliometrics — literature bibliometrics, patent bibliometrics, and linkage bibliometric can all be used to address various government performance and results questions. Applications of these three bibliometric types will be described within the framework of Weinberg's internal and external criteria, whether the work being done is good science, efficiently and effectively done, and whether it is important science from a technological viewpoint. Within all bibliometrics the fundamental assumption is that the frequency with which a set of papers or patents is cited is a measure of the impact or influence of the set of papers. The literature bibliometric indicators are counts of publications and citations received in the scientific literature and various derived indicators including such phenomena as cross-sectoral citation, coauthorship and concentration within influential journals. One basic observation of literature bibliometrics, which carries over to patent bibliometrics, is that of highly skewed distributions — with a relatively small number of high-impact patents and papers, and large numbers of patents and papers of minimal impact. The key measure is whether an agency is producing or supporting highly cited papers and patents. The final set of data are in the area of linkage bibliometrics, looking at citations from patents to scientific papers. These are particularly relevant to the external criteria, in that it is quite obvious that institutions and supporting agencies whose papers are highly cited in patents are making measurable contributions to a nation's technological progress.     

Intellectual activities and influences of Belver C. Griffith: A citation perspective

Based on an analysis of the 377 documents that cited Griffith's publications in the ISI citation databases, it has been found that Griffith made pioneer and significant contributions with his collaborators to the fields of bibliometrics and scholarly communication among scientists. His research work has also greatly influenced people from all over the world conducting research in psychology, bibliometric information science, and social studies of science in the past several decades. This revised version was published online in June 2006 with corrections to the Cover Date.     

Bibliometrics in Press. Representations and uses of bibliometric indicators in the Italian daily newspapers

Scholars in science and technology studies and bibliometricians are increasingly revealing the performative nature of bibliometric indicators. Far from being neutral technical measures, indicators such as the Impact Factor and the h-index are deeply transforming the social and epistemic structures of contemporary science. At the same time, scholars have highlighted how bibliometric indicators are endowed with social meanings that go beyond their purely technical definitions. These social representations of bibliometric indicators are constructed and negotiated between different groups of actors within several arenas. This study aims to investigate how bibliometric indicators are used in a context, which, so far, has not yet been covered by researchers, that of daily newspapers. By a content analysis of a corpus of 583 articles that appeared in four major Italian newspapers between 1990 and 2020, we chronicle the main functions that bibliometrics and bibliometric indicators played in the Italian press. Our material shows, among other things, that the public discourse developed in newspapers creates a favorable environment for bibliometrics-centered science policies, that bibliometric indicators contribute to the social construction of scientific facts in the press, especially in science news related to medicine, and that professional bibliometric expertise struggles to be represented in newspapers and hence reach the general public.

     

The rate of growth in scientific publication and the decline in coverage provided by Science Citation Index

The growth rate of scientific publication has been studied from 1907 to 2007 using available data from a number of literature databases, including Science Citation Index (SCI) and Social Sciences Citation Index (SSCI). Traditional scientific publishing, that is publication in peer-reviewed journals, is still increasing although there are big differences between fields. There are no indications that the growth rate has decreased in the last 50 years. At the same time publication using new channels, for example conference proceedings, open archives and home pages, is growing fast. The growth rate for SCI up to 2007 is smaller than for comparable databases. This means that SCI was covering a decreasing part of the traditional scientific literature. There are also clear indications that the coverage by SCI is especially low in some of the scientific areas with the highest growth rate, including computer science and engineering sciences. The role of conference proceedings, open access archives and publications published on the net is increasing, especially in scientific fields with high growth rates, but this has only partially been reflected in the databases. The new publication channels challenge the use of the big databases in measurements of scientific productivity or output and of the growth rate of science. Because of the declining coverage and this challenge it is problematic that SCI has been used and is used as the dominant source for science indicators based on publication and citation numbers. The limited data available for social sciences show that the growth rate in SSCI was remarkably low and indicate that the coverage by SSCI was declining over time. National Science Indicators from Thomson Reuters is based solely on SCI, SSCI and Arts and Humanities Citation Index (AHCI). Therefore the declining coverage of the citation databases problematizes the use of this source.     

A comparison of Scopus and Web of Science for a typical university

For many years, the ISI Web of Knowledge from Thomson Reuters was the sole publication and citation database covering all areas of science thus becoming an invaluable tool in bibliometric analysis. In 2004, Elsevier introduced Scopus and this is rapidly becoming a good alternative. Several attempts have been made at comparing these two instruments from the point of view of journal coverage for research or for bibliometric assessment of research output.This paper attempts to answer the question that all researchers ask, i.e., what is to be gained by searching both databases? Or, if you are forced to opt for one of them, which should you prefer? To answer this question, a detailed paper by paper study is presented of the coverage achieved by ISI Web of Science and by Scopus of the output of a typical university. After considering the set of Portuguese universities, the detailed analysis is made for two of them for 2006, the two being chosen for their comprehensiveness typical of most European universities. The general conclusion is that about 2/3 of the documents referenced in any of the two databases may be found in both databases while a fringe of 1/3 are only referenced in one or the other. The citation impact of the documents in the core present in both databases is higher, but the impact of the fringe that are present only in one of the databases should not be disregarded as some high impact documents may be found among them.     

Benchmarking scientific output in the social sciences and humanities: The limits of existing databases

Summary The goal of this paper is to examine the impact of linguistic coverage of databases used by bibliometricians on the capacity to effectively benchmark the work of researchers in social sciences and humanities. We examine the strong link between bibliometrics and the Thomson Scientific's database and review the differences in the production and diffusion of knowledge in the social sciences and humanities (SSH) and the natural sciences and engineering (NSE). This leads to a re-examination of the debate on the coverage of these databases, more specifically in the SSH. The methods section explains how we have compared the coverage of Thomson Scientific databases in the NSE and SSH to the Ulrich extensive database of journals. Our results show that there is a 20 to 25% overrepresentation of English-language journals in Thomson Scientific's databases compared to the list of journals presented in Ulrich. This paper concludes that because of this bias, Thomson Scientific databases cannot be used in isolation to benchmark the output of countries in the SSH.     

Little scientometrics,big scientometrics ... and beyond

Although the field of scientometrics/bibliometrics is rapidly growing, and the interest in scientometric indicators is constantly rising, the field is in a crisis: subfields are drifting apart, the field is lacking consensus in basic questions and of internal communication, the quality of scientometric research is questioned by other disciplines. Among the causes stated are: the loss of integrating personalities; shift from basic and methodological research to applied bibliometrics; domination of the interests of science policy and business in commissioning and funding research; vendor policies and failing quality-management on the side of database-producers; misuse of bibliometric research results and disregard for scientific standards. To overcome the situation, the authors plead for integrative and interdisciplinary research approaches, for reinforcing fundamental, methodological and experimental research programs in scientometrics, for independent funding of research, and for an enhancement of scientometric databases. The need for acknowledged technical and scientific standards in research and publication is stressed. Finally, the establishment of aCode of Ethics for the field of scientometrics is proposed.This paper has been presented at the Fourth International Conference on Bibliometrics, Informetrics and Scinetometrics in Berlin, Sept. 1993.     

Blockmodeling of co-authorship networks in library and information science in Argentina: a case study

The paper introduces the use of blockmodeling in the micro-level study of the internal structure of co-authorship networks over time. Variations in scientific productivity and researcher or research group visibility were determined by observing authors?? role in the core-periphery structure and crossing this information with bibliometric data. Three techniques were applied to represent the structure of collaborative science: (1) the blockmodeling; (2) the Kamada-Kawai algorithm based on the similarities in co-authorships present in the documents analysed; (3) bibliometrics to determine output volume, impact and degree of collaboration from the bibliographic data drawn from publications. The goal was to determine the extent to which the use of these two complementary approaches, in conjunction with bibliometric data, provides greater insight into the structure and characteristics of a given field of scientific endeavour. The paper describes certain features of Pajek software and how it can be used to study research group composition, structure and dynamics. The approach combines bibliometric and social network analysis to explore scientific collaboration networks and monitor individual and group careers from new perspectives. Its application on a small-scale case study is intended as an example and can be used in other disciplines. It may be very useful for the appraisal of scientific developments.     

An item-by-item subject classification of papers published in journals covered by the SSCI database using reference analysis

A serious shortcoming of bibliometric studies based on theSocial Sciences Citation Index is the lack of a universally applicable subject classification scheme as individual papers are concerned. Moreover, the selective coverage of more than thousand scientific journals per annum proved to be an insuperable obstacle in the delimitation of social science subject areas. Subject classification of papers on the basis of assigning journals to subject categories (like those found in the various supplements of ISI databases) works well in case of fully covered and highly specialised journals in the social sciences, too, but fails for multidisciplinary and selectively covered journals. This study presents the results of an item-by-item subject classification approach, where assignment is based on the analysis of the subject categories of reference literature. This analysis extends the results of an earlier study by the authors on the possibility of delimiting subfields in the hard and life sciences based on reference analysis. The assignment proved also reliable for a considerable share of literature in the social sciences. Due to the peculiarities of the database this share is lower in the SSCI than that in the SCI. Although an iterated application of the procedure is expected to increase the number of classifiable publications, it is suggested that in the sociated sciences the method should be used in combination with other means of subject assignment.     

Which role do non-source items play in the social sciences? A case study in political science in Germany

Publications that are not indexed by citation indices such as Web of Science (WoS) or Scopus are called “non-source items”. These have so far been neglected by most bibliometric analyses. The central issue of this study is to investigate the characteristics of non-source items and the effect of their inclusion in bibliometric evaluations in the social sciences, specifically German political science publications. The results of this study show that non-source items significantly increase the number of publications (+1,350 %) and to a lesser extent the number of citations from SCIE, SSCI, and A&HCI (+150 %) for evaluated political scientists. 42 % of non-source items are published as book chapters. Edited books and books are cited the most among non-source items. About 40 % of non-source items are in English, while 80 % of source items are in English. The citation rates of researchers taking non-source items into account are lower than those from source items, partially as a result of the limited coverage of WoS. In contrast, the H-indices of researchers taking only non-source items into account are higher than those from source items. In short, the results of this study show that non-source items should be included in bibliometric evaluations, regardless of their impact or the citations from them. The demand for a more comprehensive coverage of bibliometric database in the social sciences for a higher quality of evaluations is shown.     

Systematic analysis of coverage and usage of conference proceedings in web of science

The role of conference proceedings for scientific communication varies among the different research fields. It is thus difficult to determine how to use them in bibliometric studies that cover all or at least a variety of the research fields without favouring or penalizing observation subjects that are specialized in fields that rely heavily on conference proceedings. Also, the coverage of conference proceedings in bibliometric databases is often unclear. Not only have there been reports of misclassification but also of insufficient coverage. In this study, the Web of Science is used to give an overview of coverage of conference proceedings as well as advantages and pitfalls of their usage in bibliometric analyses. In particular, the focus lies on different citation behaviour of and for conference proceedings and the implications for bibliometric indicators. This is complemented by an investigation of the development of coverage and publication behaviour in conference proceedings which is compared to those of journal publications. Finally, the importance but also drawbacks and opportunities of acknowledging conference proceedings publications for bibliometric studies are summarized.     

Evaluating research: from informed peer review to bibliometrics

National research assessment exercises are becoming regular events in ever more countries. The present work contrasts the peer-review and bibliometrics approaches in the conduct of these exercises. The comparison is conducted in terms of the essential parameters of any measurement system: accuracy, robustness, validity, functionality, time and costs. Empirical evidence shows that for the natural and formal sciences, the bibliometric methodology is by far preferable to peer-review. Setting up national databases of publications by individual authors, derived from Web of Science or Scopus databases, would allow much better, cheaper and more frequent national research assessments.     

Correcting glasses help fair comparisons in international science landscape: Country indicators as a function of ISI database delineation

The increasing use of bibliometric indicators in science policy calls for a reassessment of their robustness and limits. The perimeter of journal inclusion within ISI databases will determine variations in the classic bibliometric indicators used for international comparison, such as world shares of publications or relative impacts. We show in this article that when this perimeter is adjusted using a natural criterion for inclusion of journals, the journal impact, the variation of the most common country indicators (publication and citation shares; relative impacts) with the perimeter chosen depends on two phenomena. The first one is a bibliometric regularity rooted in the main features of competition in the open space of science, that can be modeled by bibliometric laws, the parameters of which are “coverage-independent” indicators. But this regularity is obscured for many countries by a second phenomenon, the presence of a sub-population of journals that does not reflect the same international openness, the nationally-oriented journals. As a result indicators based on standard SCI or SCISearch perimeters are jeopardized to a certain extent by this sub-population which creates large irregularities. These irregularities often lead to an over-estimation of share and an under-estimation of the impact, for countries with national editorial tradition, while the impact of a few mainstream countries arguably benefits from the presence of this sub-population. This revised version was published online in August 2006 with corrections to the Cover Date.     

Evaluation of accounting journals by coverage of accounting topics in 2018–2019

The purpose of this article is to evaluate the coverage of accounting topics in 2018 and 2019 by the high-ranked and cited journals of popular international publishers. To achieve this, the study provides (1) research on the proportion of accounting journals in the international publishers; (2) identification of the largest publishers by the number of accounting journals; (3) a check of the high-ranked and cited accounting journals according to the journals’ websites, the SCImago database, and Web of Science database; (4) investigation of the trend in the number of accounting publications in 2018–2019 compared with 2016–2017 by SSRN, IDEAS/RePEc, and DOAJ; (5) a detailed overview of 1749 accounting publications for 2018 and 2019 using the reference databases of such international publishers as Blackwell, Elsevier, American Accounting Association, Kluwer Academic Publishers, Oxford University Press, Cambridge University Press, and Springer; (6) a self-developed classification of the accounting topics into 14 main directions; and (7) an assessment of the extent of coverage of accounting topics by each journal. The research methods are based on a cross-sectional survey between the website of each publisher and the SCImago database, and on a bibliometric analysis of the metrics and citation impact, the number and proportion of articles in the “major” journals in accounting, and the major bibliometrics databases. SCImago, as the main bibliometrics database, was used for the first time to identify the top-ranked and cited journals. This study highlights the importance of accounting in international publications in the high-ranked and cited journals, emphasizing the actual publications obtained in 2018–2019. The results indicate that a high quartile or high international citation level does not determine the full coverage of accounting topics in a journal. It was found that assets, control, management accounting, and financial reporting were the most frequently published topics in 2018 and 2019. The blockchain technology and IFRS research in accounting were published within 2018–2019 in only 125 articles (7.14%) by 26 accounting journals among the high-ranked and cited journals with an obtained high coverage of accounting topics. This study may help scientists to select a journal for publishing or researching an appropriate accounting topic. Moreover, to increase the coverage extent of accounting topics, the analysis and classification of publication topics conducted in this work are also recommended to editorial boards of accounting journals as information regarding the least disclosed accounting topics over the last 2 years.

     

Trends in Russian research output in post-Soviet era

Recently, the Russian government has ordered evaluation and reform of the basic research system. As a consequence, the number of research staff at the Russian Academy of Sciences will be reduced by 20% by 2007. The basis for research evaluation and institute budgeting will be bibliometric indicators. In view of these changes we look at the Russian publication output and argue that

The global science base of information and communication technologiges: Bibliometric analysis of ICT research papers

The science and engineering base is a key source of knowledge for the development and use of Information and Communication Technologies (ICTs). In order to be able to effectively describe and monitor world-wide scientific activity related to ICTs, it is important to be able to provide reliable macro-level statistics of this knowledge base. International bibliographic databases and related bibliometric indicators together provide an analytical framework and appropriate measures to cover both the ‘supply side’—research capabilities and outputs—and ‘demand side’—collaboration, diffusion and citation impact—related to the ICT research. This paper presents results of such a bibliometric study describing macro-level features of this ICT knowledge base. The data were retrieved from a specially developedCWTS ICT Database which provides a broad-scope world-wide coverage of ICT-relevant research papers published in high-quality international scientific and technical journals. The cross-country comparison focuses on the level of scientific output and co-operation patterns of the most actively publishing nations with a focus on the three Triad zones—the European Union, the USA and Japan.     

The Literature of Bibliometrics,Scientometrics, and Informetrics

Since Vassily V. Nalimov coined the term ‘scientometrics’ in the 1960s, this term has grown in popularity and is used to describe the study of science: growth, structure, interrelationships and productivity. Scientometrics is related to and has overlapping interests with bibliometrics and informetrics. The terms bibliometrics, scientometrics, and informetrics refer to component fields related to the study of the dynamics of disciplines as reflected in the production of their literature. Areas of study range from charting changes in the output of a scholarly field through time and across countries, to the library collection problem of maintaining control of the output, and to the low publication productivity of most researchers. These terms are used to describe similar and overlapping methodologies. The origins and historical survey of the development of each of these terms are presented. Profiles of the usage of each of these terms over time are presented, using an appropriate subject category of databases on the DIALOG information service. Various definitions of each of the terms are provided from an examination of the literature. The size of the overall literature of these fields is determined and the growth and stabilisation of both the dissertation and non-dissertation literature are shown. A listing of the top journals in the three fields are given, as well as a list of the major reviews and bibliographies that have been published over the years. This revised version was published online in June 2006 with corrections to the Cover Date. This revised version was published online in June 2006 with corrections to the Cover Date. This revised version was published online in June 2006 with corrections to the Cover Date.     

The distribution of bibliographic records in databases using different counting methods for duplicate records

Knowing how records on a particular topic are distributed over databases is useful for both practical and theoretical reasons; however little work in this area appears to have been done. This paper examines the distribution of records on the topic of “Fuzzy Set Theory” in over 100 bibliographic databases and determines whether the distribution of records over databases is similar to the traditional Bradford hyperbolic distribution of records over journals. Different methods for counting duplicate records between and within databases have been developed. A comparison of the various distributions based on these counting methods is presented; and the distributions are compared to results of earlier studies. The results also give an indication of the number of databases necessary to search for coverage of a literature to specified percentages using the different counting techniques developed in this study.     

A hybrid mapping of information science

Previous studies have shown that hybrid clustering methods that incorporate textual content and bibliometric information can outperform clustering methods that use only one of these components. In this paper we apply a hybrid clustering method based on Fisher’s inverse chisquare to integrate full-text with citations and to provide a mapping of the field of information science. We quantitatively and qualitatively asses the added value of such an integrated analysis and we investigate whether the clustering outcome is a better representation of the field by comparing with a text-only clustering and with another hybrid method based on linear combination of distance matrices. Our data set consists of almost 1000 articles and notes published in the period 2002–2004 in 5 representative journals. The optimal number of clusters for the field is 5, determined by using a combination of distance-based and stability-based methods. Term networks present the cognitive structure of the field and are complemented by the most representative publications. Three large traditional sub-disciplines, particularly, information retrieval, bibliometrics/scientometrics, and more social aspects, and two smaller clusters about patent analysis and webometrics, can be distinguished.