Towards understanding the relation between citations and research quality in software engineering studies

The importance of achieving high quality in research practice has been highlighted in different disciplines. At the same time, citations are utilized to measure the impact of academic researchers and institutions. One open question is whether the quality in the reporting of research is related to scientific impact, which would be desired. In this exploratory study we aim to: (1) Investigate how consistently a scoring rubric for rigor and relevance has been used to assess research quality of software engineering studies; (2) Explore the relationship between rigor, relevance and citation count. Through backward snowball sampling we identified 718 primary studies assessed through the scoring rubric. We utilized cluster analysis and conditional inference tree to explore the relationship between quality in the reporting of research (represented by rigor and relevance) and scientiometrics (represented by normalized citations). The results show that only rigor is related to studies’ normalized citations. Besides that, confounding factors are likely to influence the number of citations. The results also suggest that the scoring rubric is not applied the same way by all studies, and one of the likely reasons is because it was found to be too abstract and in need to be further refined. Our findings could be used as a basis to further understand the relation between the quality in the reporting of research and scientific impact, and foster new discussions on how to fairly acknowledge studies for performing well with respect to the emphasized research quality. Furthermore, we highlighted the need to further improve the scoring rubric.     

Bibliometrics in Press. Representations and uses of bibliometric indicators in the Italian daily newspapers

Scholars in science and technology studies and bibliometricians are increasingly revealing the performative nature of bibliometric indicators. Far from being neutral technical measures, indicators such as the Impact Factor and the h-index are deeply transforming the social and epistemic structures of contemporary science. At the same time, scholars have highlighted how bibliometric indicators are endowed with social meanings that go beyond their purely technical definitions. These social representations of bibliometric indicators are constructed and negotiated between different groups of actors within several arenas. This study aims to investigate how bibliometric indicators are used in a context, which, so far, has not yet been covered by researchers, that of daily newspapers. By a content analysis of a corpus of 583 articles that appeared in four major Italian newspapers between 1990 and 2020, we chronicle the main functions that bibliometrics and bibliometric indicators played in the Italian press. Our material shows, among other things, that the public discourse developed in newspapers creates a favorable environment for bibliometrics-centered science policies, that bibliometric indicators contribute to the social construction of scientific facts in the press, especially in science news related to medicine, and that professional bibliometric expertise struggles to be represented in newspapers and hence reach the general public.

     

The conundrum of research productivity: a study on sociologists in Italy

This paper aims to understand the influence of institutional and organisational embeddedness on research productivity of Italian sociologists. We looked at all records published by Italian sociologists in Scopus from 1973 to 2016 and reconstructed their co-authorship patterns. We built an individual productivity index by considering the number and type of records, the impact factor of journals in which these records were published and each record’s citations. We found that sociologists who co-authored more frequently with international authors were more productive and that having a stable group of co-authors had a positive effect on the number of publications but not on citations. We found that organisational embeddedness has a positive effect on productivity at the group level (i.e., sociologists working in the same institute), less at the individual level. We did not found any effect of the scientific disciplinary sectors, which are extremely influential administratively and politically for promotion and career in Italy. With all caveats due to several limitations of our analysis, our findings suggest that internationalisation and certain context-specific organisational settings could promote scientist productivity .     

Does it pay to cooperate? A bibliometric case study in molecular biology

Various bibliometric studies report that multiinstitutional or multinational authored papers are more frequently cited than papers that come from a single institute. The conclusion, however, that there is a systematic improvement of scientific success by cooperation on every level of scientific research in leading or mediocre research institutes might be misleading: In a citation analysis of 13 well-known research institutes in molecular biology there was no difference in the average citations per paper with regard to cooperations. In a subsample of 7 German institutes that difference found could be explained by selfcitations. In another case, all articles of a two year sample of an excellent journal in molecular biology, the EMBO-Journal, the same phenomenon could be observed: Differences in the average citations per article with regard to cooperations could be explained by selfcitations.     

What is Special about Patent Citations? Differences between Scientific and Patent Citations

The emergence of pattent bibliometrics as a new branch of scientometrics necessitates a deeper understanding of the relationship between patents and papers. As this connection is established through the linkage between patents and research papers, one must have a clear idea of similarities and differences between patent and paper citations. This paper will investigate to what extent one can not only apply bibliometric methods to patents but also extend the existing interpretative framework for citations in research papers to the field of patent citations. After pointing out some parallels in the debates about the nature of citations in patents and scientific articles, the paper outlines those parts of bibliometric theory covering scientific citations that could be relevant to patent citations too. Then it highlights the specialties and peculiarities of patent citations. One major conclusion is that the general nature of a common framework for both scientific and patent citations would severely limit its usefulness, but research on academic citations might still be a great source of inspiration to the study of patent citations.     

Measuring social media activity of scientific literature: an exhaustive comparison of scopus and novel altmetrics big data

This paper measures social media activities of 15 broad scientific disciplines indexed in Scopus database using Altmetric.com data. First, the presence of Altmetric.com data in Scopus database is investigated, overall and across disciplines. Second, a zero-truncated negative binomial model is used to determine the association of various factors with increasing or decreasing citations. Lastly, the effectiveness of altmetric indices to identify publications with high citation impact is comprehensively evaluated by deploying area under the curve (AUC)—an application of receiver operating characteristic. Results indicate a rapid increase in the presence of Altmetric.com data in Scopus database from 10.19% in 2011 to 20.46% in 2015. It was found that Blog count was the most important factor in the field of Health Professions and Nursing as it increased the number of citations by 38.6%, followed by Twitter count increasing the number of citations by 8% in the field of Physics and Astronomy. The results of receiver operating characteristic show that altmetric indices can be a good indicator to discriminate highly cited publications, with an encouragingly AUC = 0.725 between highly cited publications and total altmetric count. Overall, findings suggest that altmetrics can be used to distinguish highly cited publications. The implications of this research are significant in many different directions. Firstly, they set the basis for a further investigation of altmetrics efficiency to predict publications impact and most significantly promote new insights for the measurement of research outcome dissemination over social media.     

A bibliometric analysis of physics publications in Korea, 1994-1998

This study examined research performance of Korean physicists, comparing Korean-authoredpapers versus internationally co-authored papers, indexed in SCI, 1994-1998, and using thenumber of citations received by internationally co-authored papers covered by the SCI CD-ROM.For the study, 4,665 papers published from the researchers affiliated with the physics departmentsor physics-associated laboratories at Korean universities and indexed by SCI were analyzed.Korean authored papers tended to be published in Korean, Japanese, and UK journals, whileinternationally co-authored papers were more likely to appear in German, Dutch, and Swissjournals. Among the 18 authorship countries (on the basis of first author), 93 internationally co-authored papers by U.S. researchers had the highest citation rate, an average 15.9 citations perpaper. Of the eight countries that published over 5 papers, there was no correlation between theaverage number of citations per paper and the total number of citations. However, an ANOVAindicated a significant difference between the average number of citations per paper according tocountry (F = 5.84, p < 0.0005). In other words, papers by the U.S. and French researchers tendedto be cited more frequently than papers by the Italian, Japanese, Korean, Russian, and Germanresearchers.     

Patent-bibliometric analysis on the Chinese science — technology linkages

The purpose of this study is to explore the character and pattern of the linkage between science and technology in China, based on the database of United States Patent and Trademark Office (USPTO). The analysis is focused on the period 1995–2004, a rapid increasing period for Chinese US patents. Using the scientific non-patent references (NPRs) within patents, we investigate the science-technology connection in the context of Chinese regions as well as industrial sectors classified by International Patent Classification (IPC). 11 technological domains have been selected to describe the science intensity of the technology. The results suggest that the patents and the corresponding scientific citations are related in different ways. Finally, we match the scientific NPRs to the Science Citation Index (SCI) covered publications to identify the core journals and categories. It reveals that the scientific references covered by SCI show a skewed distribution not only in journals but also in categories.     

Research productivity in soil science in the Philippines

Understanding the direction and magnitude of soil science publication in the Philippines is crucial in formulating research priorities and funding allocation. There is no consensus on the current state of soil science publication in the Philippines, thus this study was conducted to elucidate the trend in the soil science publication. We conducted an in-depth analysis on the total number of publications and the total number of citations of soil science publications collected from Thomson ISI database. Results revealed an upsurge in soil science publication from 1970 to 2000 with no indication that this trend is slowing down. Increases in the number of citations with time are consistent with increases in the total number of publications (r = 0.93; p < 0.05). Results further revealed that the soil science publication in the Philippines is biased towards rice research particularly soil water with very few studies were published for plant nutrition and soil chemistry. The present study highlights the need for a paradigm shift in soil science research from mostly rice related research to environmental research. Ways to increase soil science publication among Filipino soil scientist’s particularly in academic institutions is proposed. Finally, since only a few government-funded research have been published, future studies should stress on identifying factors that influence scientific productivity of most soil scientists in the Philippines.     

Publication and citation patterns in the literature of liquid crystals with special reference to the contribution of India,Canada, Japan,United Kingdom and the Soviet Union

From an analysis of bibliographic data on 430 journal articles on liquid crystals covered inPhysics Abstracts 1976 and the 4729 citations to them up to the end of 1987, we have identified the geographic origin, the prominent institutions, language and journal-wise distribution of the papers, the citedness of these papers, and the distribution of citations as a time series for the highly cited papers. We have also analysed the 126 papers published by authors from India, Canada, Australia, Israel, Japan and the United Kingdom and covered inPhysics Abstracts 1978, and the 1154 citations to them up to 1987. Unlike in most other high tech areas of physics, in LC research the difference in performance between the USA and the other leading countries is not very pronounced. Publication data from 1976, 1978 and 1985 reveal that LC literature is on the rise and that the percentage share of the Soviet Union is rising fast and that of the USA is on the decline.Dedicated to the memory of Michael J. Moravcsik     

Crossing the hurdle: the determinants of individual scientific performance

An original cross-sectional dataset referring to a medium-sized Italian university is implemented in order to analyze the determinants of scientific research production at individual level. The dataset includes 942 permanent researchers of various scientific sectors for a 3-year time-span (2008–2010). Three different indicators—based on the number of publications and/or citations—are considered as response variables. The corresponding distributions are highly skewed and display an excess of zero-valued observations. In this setting, the goodness-of-fit of several Poisson mixture regression models are explored by assuming an extensive set of explanatory variables. As to the personal observable characteristics of the researchers, the results emphasize the age effect and the gender productivity gap—as previously documented by existing studies. Analogously, the analysis confirms that productivity is strongly affected by the publication and citation practices adopted in different scientific disciplines. The empirical evidence on the connection between teaching and research activities suggests that no univocal substitution or complementarity thesis can be claimed: a major teaching load does not affect the odds to be a non-active researcher and does not significantly reduce the number of publications for active researchers. In addition, new evidence emerges on the effect of researchers administrative tasks—which seem to be negatively related with researcher’s productivity—and on the composition of departments. Researchers’ productivity is apparently enhanced by operating in department filled with more administrative and technical staff, and it is not significantly affected by the composition of the department in terms of senior/junior researchers.     

Directing Public Interest: Danish Newspaper Science 1900-1903

Abstract.   The growing historical literature on science in the written media between 1789 and 1914 has mainly focused on magazines, journals and periodicals. By comparison studies of science in daily newspapers of the nineteenth century have received much less attention. This disparity has also characterised the Danish context where science in nineteenth-century periodicals and popular science journals, unlike science in newspapers, has attracted the attention of historians. In this article we aim to do away with this imbalance by exploring science in three Danish newspapers in the period 1900-1903. The article suggests that newspapers served as key mediators between users and producers of science in Copenhagen at that time. We argue that the significance of newspapers owed less to the scattered articles they carried on scientific topics, and more to the newspapers' important function of directing public interest. By this we mean that newspapers were directing readers' attention to specific science events such as lectures, excursions and celebrations; to scientific literature through adverts and reviews; to new scientific electrical wonder cures of ailments ranging from baldness to arthritis; and to the latest news about communication technologies such as trams and telephones which increasingly became part of everyday life in Copenhagen.     

Dentistry scientometric analysis: a comparative study between Brazil and other most productive countries in the area

This research aims at performing a comparative study between the Brazilian scientific production in Dentistry, from 2000 to 2009 and countries that contribute with at least 2 % of the world’s scientific production indexed in the Scopus database. More specifically, we intend to assess the annual Brazilian scientific production by comparing it to the other countries’, analyze the Brazilian and other countries’ publications in journals with higher impact factors, as well as to highlight the scientific production from these countries and its international visibility, measured by its total and by its average of citations and normalized citation index per year, by comparing the countries, and to compare the index h of such countries. As work procedure, the SCImago Journal and Country Rank was used as source, identifying the group of producing countries in the Dentistry area from 1996 to 2009. From a total of 136 countries, 13 were highlighted as the most productive, each one of them accounting for at least 2 % the worldwide scientific production in the area. The following indicators were raised for each country: number of produced documents, total of citations, self-citations, average of citations per document and index h. We verified that Brazil is the only country in Latin America that is pictured among the most productive ones in the Dentistry area. We observed that Brazil presents a growing visibility and impact in the international scenery, what suggests that its production is constantly consolidating, with Brazilian scientific recognition in the main vehicles of dissemination in the area.     

Mapping past,current and future energy research trend in Pakistan: a scientometric assessment

This work describes the contribution of researchers in the field of the energy from Pakistan in the period 1990–2016. A scientometric approach was applied to analyze the scientific publications in the field using the Scopus Elsevier database. Different aspects of the publications were analyzed, such as publication type, major research areas, journals, citations, authorship pattern, affiliations as well as the keyword occurrence frequency. The present research trends are analyzed and future research directions are outlined. The impact factor, h-index and number of citations were used to investigate the strength of active institutes, authors, and journals in the field of the energy in Pakistan. From 1990 to 2016, 991 articles have been published by 2139 authors from 213 research institutes. The total number of citations and impact factor are 10,287 and 2301 respectively, corresponding to 10 citations per paper and an impact factor of 2.32 per publication. The research articles originate primarily from COMSATS, NUST, PIEAS, and PINSTECH. Pakistan has published 60% of publication with the collaboration of the foreign institutes, mainly from the United States, the United Kingdom, China and Malaysia. The core research activities in the field are mainly focused on resource assessment, energy policy, energy efficiency, feasibility study, energy economics, and performance assessment. The most productive journal, author, institution, are renewable & sustainable energy review, Shahbaz M., and COMSATS, respectively.     

Citations to the “Introduction to informetrics” indexed by WOS,Scopus and Google Scholar

Google Scholar and Scopus are recent rivals to Web of Science. In this paper we examined these three citation databases through the citations of the book “Introduction to informetrics” by Leo Egghe and Ronald Rousseau. Scopus citations are comparable to Web of Science citations when limiting the citation period to 1996 and onwards (the citation coverage of Scopus)—each database covered about 90% of the citations located by the other. Google Scholar missed about 30% of the citations covered by Scopus and Web of Science (90 citations), but another 108 citations located by Google Scholar were not covered either by Scopus or by Web of Science. Google Scholar performed considerably better than reported in previous studies, however Google Scholar is not very “user-friendly” as a bibliometric data collection tool at this point in time. Such “microscopic” analysis of the citing documents retrieved by each of the citation databases allows us a deeper understanding of the similarities and the differences between the databases.     

From biomedical research to health improvement

Traditional means of analysis of research outputs have focussed on citations to papers in journals in other journal publications. But these only chronicle the early stages whereby research in biomedicine is converted into health improvement through better patient care and through preventive measures. New evaluation methods, still based on the concept of citation of research in other documents, are needed and are now being developed. These include the use of textbooks in medical education and the analysis of governmental regulations and health policies, which can influence both the availability of new drugs and the control of toxic substances in food and the environment. There is also an interest in the way that newspapers report biomedical research advances. Readers include politicians, healthcare professionals, the general public (who are increasingly becoming active consumers of healthcare products) and other researchers who may value the immediacy of the reporting. Newspaper articles tend to focus on fashionable topics and to offer premature hopes of cures to disease, but they can also provide a valuable service in showing the importance of animal experiments to biomedical progress. It would be useful to create an international database of newspaper citations through a consortium of partners in different countries who would agree a common protocol and exchange information on a regular basis. This revised version was published online in June 2006 with corrections to the Cover Date.     

Two simple new bibliometric indexes to better evaluate research in disciplines where publications typically receive less citations

The paper proposes two simple new indexes—k and w—to assess a scientist’s publications record based on citations. The two indexes are superior to the widely used h index (Hirsch, 2005), as they preserve all its valuable characteristics and try to overcome one of its shortcomings, i.e. that it uses only a fraction of the information contained in a scientist’s citations profile and, as a result, it is defined over the set of positive integers and does not show a sufficiently fine ‘granularity’ to allow a fully satisfactory ranking of scientists. This problem is particularly acute in many areas of Social Sciences and Humanities, where scientific productivity and citation practices typically yield fewer citations per paper and, as a consequence, are characterized by ‘structurally’ lower values of the h index. Both the indexes proposed are defined over R⁺, their integer part is equal to the scientist’s h index and they fall in the right-open interval [h, h+1). While the h index is influenced only by part of the citations received by a scientist’s most-cited publications, the k index takes into account all the citations received by her most-cited publications and the w index accounts for the citations received by the entire set of her publications. Variants of the k and w indexes are proposed which consider co-authorship. To show the extent to which the h index and the new indexes proposed may yield different results, they are calculated for 332 professors of economics in Italian universities and the results obtained used to rank Italian university departments.     

Do interdisciplinary research teams deliver higher gains to science?

The present paper takes its place in the stream of studies that analyze the effect of interdisciplinarity on the impact of research output. Unlike previous studies, in this study the interdisciplinarity of the publications is not inferred through their citing or cited references, but rather by identifying the authors’ designated fields of research. For this we draw on the scientific classification of Italian academics, and their publications as indexed in the WoS over a 5-year period (2004–2008). We divide the publications in three subsets on the basis the nature of co-authorship: those papers coauthored with academics from different fields, which show high intensity of inter-field collaboration (“specific” collaboration, occurring in 110 pairings of fields); those papers coauthored with academics who are simply from different “non-specific” fields; and finally co-authorships within a single field. We then compare the citations of the papers and the impact factor of the publishing journals between the three subsets. The results show significant differences, generally in favor of the interdisciplinary authorships, in only one third (or slightly more) of the cases. The analysis provides the value of the median differences for each pair of publication subsets.     

Specialization and size of scientific activities: A bibliometric analysis of advanced countries

The relationship between the size of national scientific activities of advanced countries and the degree of specialization by fields of science is examined using bibliometric indicators of the number of papers and of paper citations. A negative relation between the amount of scientific activity and the degree of scientific specialization has emerged, with Japan and, to a lesser extent Italy, showing a specialization degree higher than expected. Countries with established scientific traditions (such as the US, the UK, the Netherlands, and Switzerland) have a lower than expected specialization degree, suggesting a more diversified range of research activities. Over time, however, most countries have reduced their scientific specialization, a pattern which is in contrast with recent research on patents and technological specialization.Rotating first authorship. This paper is part of a research on the Scientific and Technological Specialization of advanced countries jointly financed by the Commission of European Communities, D.G. XII, Science, Research and Development, Service Research Evaluation, and the Italian National Research Council. We are grateful to PatriziaPrincipessa and RobertoSimonetti for research assistance.