Citations as a measure of technological impact: A review of forward citation-based measures

The number of forward citations a patent receives accumulates over time and appears to be correlated to the patent‘s (i.e. invention's) technological impact. A dominant theory suggests that highly cited patents contain an important technological advance. However, a variety of citation based measures have been proposed by different authors. This study, via a narrative literature review, identified nine forward citation-based measures that appear of particular relevance. We describe each measure and present them in a comparative format.The measures are divided into two broad categories: firstly the ones that are particularly relevant to the patent level (citation index, forward citation frequency, generality, influence), and secondly the ones that are relevant to the patent portfolio level (current impact index, herfindal-hirschman index, hindrance index, relative patent position, technology strength).We hope research scholars and industrial users find this review helpful for citation analysis and intellectual property analytics, especially when wanting to employ forward citation-based measures to assess technological impact.     

The impact of internet resources on scholarly communication: A citation analysis

The quality and credibility of Internet resources has been a concern in scholarly communication. This paper reports a quantitative analysis of the use of Internet resources in journal articles and addresses the concerns for the use of Internet resources scholarly journals articles. We collected the references listed in 35,698 articles from 14 journals published during 1996 to 2005, which resulted in 1,000,724 citations. The citation data was divided into two groups: traditional citations and Web citations, and examined based on frequencies of occurrences by domain and type of Web citation sources. The findings included: (1) The number of Web citations in the journals investigated had been increasing steadily, though the quantity was too small to draw an inclusive conclusion on the data about their impact on scientific research; (2) A great disparity existed among different disciplines in terms of using information on the Web. Applied disciplines and interdisciplinary sciences tended to cite more information on the Web, while classical and experimental disciplines cited little of Web information; (3) The frequency of citations was related to the reputation of the author or the institution issuing the information, and not to the domain or webpage types; and (4) The researchers seemed to lack confidence in Internet resources, and Web information was not as frequently cited as reported in some publications before. The paper also discusses the need for developing a guideline system to evaluate Web resources regarding their authority and quality that lies in the core of credibility of Web information.     

The effect of characteristics of title on citation rates of articles

This study aimed to assess the association between some features of articles title and number of citations in a volume of Addictive Behavior journal. All research articles published in the volume number 32 (2007) in the Addictive Behaviors journal (n = 302) were analyzed by two independent authors. For each article, the following information has been extracted: number of citations up to June 2013 in the Scopus citation database, type of and characteristics of titles, having different words in the keywords, reference to place and presence of an acronym. The summary statistics showed that mean number of citation was 16.36 ± 19.55 times. Articles with combinational title (use of a hyphen or a colon separating different ideas within a sentence) and articles with different words in the keywords (at least two different keywords) had higher number of citations. The number of citations was not correlated with the number of words in the title (r = 0.05, P = 0.325). Our results suggested that some features in the paper such as type of the title and articles with keywords different from words included in the title can help to predict the number of citation counts. These findings can be used by authors and reviewers in order to maximize the impact of articles. The length of title is not associated with citation counts. Therefore, the guide for authors of journals can be more flexible regarding the length of the title.     

Disciplinary determinants of bibliometric impact in Danish industrial research: Collaboration and visibility

Firms are increasingly dependent on networks and network visibility for innovation. Bibliometric impact can be regarded as a measure of a firm's visibility in knowledge-producing networks and may explain why companies publish their results. However, this visibility varies across disciplines. This paper examines publications produced by Danish companies in 1996, 1998 and 2000 to show how citation and collaboration patterns relate in different disciplines. The main findings are that for disciplines characterized by international collaboration and many authors per paper, international collaboration results in a greater number of citations. National collaboration does not, however, seem to make any difference to citation impact in industrial research. In disciplines where multinational collaboration and multi-authorship is uncommon, no clear picture of impact patterns can be obtained. By extension, this research may provide knowledge on how citations of papers in scientific journals can be used as a potential window to scientific networks for firms. This revised version was published online in June 2006 with corrections to the Cover Date.     

Computer models for identifying instrumental citations in the biomedical literature

The most popular method for evaluating the quality of a scientific publication is citation count. This metric assumes that a citation is a positive indicator of the quality of the cited work. This assumption is not always true since citations serve many purposes. As a result, citation count is an indirect and imprecise measure of impact. If instrumental citations could be reliably distinguished from non-instrumental ones, this would readily improve the performance of existing citation-based metrics by excluding the non-instrumental citations. A citation was operationally defined as instrumental if either of the following was true: the hypothesis of the citing work was motivated by the cited work, or the citing work could not have been executed without the cited work. This work investigated the feasibility of developing computer models for automatically classifying citations as instrumental or non-instrumental. Instrumental citations were manually labeled, and machine learning models were trained on a combination of content and bibliometric features. The experimental results indicate that models based on content and bibliometric features are able to automatically classify instrumental citations with high predictivity (AUC = 0.86). Additional experiments using independent hold out data and prospective validation show that the models are generalizeable and can handle unseen cases. This work demonstrates that it is feasible to train computer models to automatically identify instrumental citations.     

Towards understanding the relation between citations and research quality in software engineering studies

The importance of achieving high quality in research practice has been highlighted in different disciplines. At the same time, citations are utilized to measure the impact of academic researchers and institutions. One open question is whether the quality in the reporting of research is related to scientific impact, which would be desired. In this exploratory study we aim to: (1) Investigate how consistently a scoring rubric for rigor and relevance has been used to assess research quality of software engineering studies; (2) Explore the relationship between rigor, relevance and citation count. Through backward snowball sampling we identified 718 primary studies assessed through the scoring rubric. We utilized cluster analysis and conditional inference tree to explore the relationship between quality in the reporting of research (represented by rigor and relevance) and scientiometrics (represented by normalized citations). The results show that only rigor is related to studies’ normalized citations. Besides that, confounding factors are likely to influence the number of citations. The results also suggest that the scoring rubric is not applied the same way by all studies, and one of the likely reasons is because it was found to be too abstract and in need to be further refined. Our findings could be used as a basis to further understand the relation between the quality in the reporting of research and scientific impact, and foster new discussions on how to fairly acknowledge studies for performing well with respect to the emphasized research quality. Furthermore, we highlighted the need to further improve the scoring rubric.     

Early Mendeley readers correlate with later citation counts

Counts of the number of readers registered in the social reference manager Mendeley have been proposed as an early impact indicator for journal articles. Although previous research has shown that Mendeley reader counts for articles tend to have a strong positive correlation with synchronous citation counts after a few years, no previous studies have compared early Mendeley reader counts with later citation counts. In response, this first diachronic analysis compares reader counts within a month of publication with citation counts after 20 months for ten fields. There are moderate or strong correlations in eight out of ten fields, with the two exceptions being the smallest categories (n?=?18, 36) with wide confidence intervals. The correlations are higher than the correlations between later citations and early citations, showing that Mendeley reader counts are more useful early impact indicators than citation counts.     

Citation frequency: A biased measure of research impact significantly influenced by the geographical origin of research articles

Context. The use of citation frequency and impact factor as measures of research quality and journal prestige is being criticized. Citation frequency is augmented by self-citation and for most journals the majority of citations originate from a minority of papers. We hypothesized that citation frequency is also associated with the geographical origin of the research publication. Objective. We determined whether citations originate more frequently from institutes that are located in the same country as the authors of the cited publication than would be expected by chance. Design. We screened citations referring to 1200 cardiovascular publications in the 7 years following their publication. For the 1200 citation recipient publications we documented the country where the research originated (9 countries/regions) and the total number of received citations. For a selection of 8864 citation donor papers we registered the country/region where the citing paper originated. Results. Self-citation was common in cardiovascular journals (n = 1534, 17.8%). After exclusion of self-citation, however, the number of citations that originated from the same country as the author of the citation recipient was found to be on average 31.6% higher than would be expected by chance (p<0.01 for all countries/regions). In absolute numbers, nation oriented citation bias was most pronounced in the USA, the country with the largest research output (p<0.001). Conclusion. Citation frequency was significantly augmented by nation oriented citation bias. This nation oriented citation behaviour seems to mainly influence the cumulative citation number for papers originating from the countries with a larger research output.     

Interdisciplinarity and insularity in the diffusion of knowledge: an analysis of disciplinary boundaries between philosophy of science and the sciences

Two fundamentally different perspectives on knowledge diffusion dominate debates about academic disciplines. On the one hand, critics of disciplinary research and education have argued that disciplines are isolated silos, within which specialists pursue inward-looking and increasingly narrow research agendas. On the other hand, critics of the silo argument have demonstrated that researchers constantly import and export ideas across disciplinary boundaries. These perspectives have different implications for how knowledge diffuses, how intellectuals gain and lose status within their disciplines, and how intellectual reputations evolve within and across disciplines. We argue that highly general claims about the nature of disciplinary boundaries are counterproductive, and that research on the nature of specific disciplinary boundaries is more useful. To that end, this paper uses a novel publication and citation network dataset and statistical models of citation networks to test hypotheses about the boundaries between philosophy of science and 11 disciplinary clusters. Specifically, we test hypotheses about whether engaging with and being cited by scientific communities outside philosophy of science has an impact on one’s position within philosophy of science. Our results suggest that philosophers of science produce interdisciplinary scholarship, but they tend not to cite work by other philosophers when it is published in journals outside of their discipline. Furthermore, net of other factors, receiving citations from other disciplines has no meaningful impact—positive or negative—on citations within philosophy of science. We conclude by considering this evidence for simultaneous interdisciplinarity and insularity in terms of scientific trading theory and other work on disciplinary boundaries and communication.     

Scientometrics of laser research in India during 1970-1994

An analysis of 952 publications published by Indian scientists and abstracted by Journal of Current Laser Abstracts during 1970-1994 indicates that laser research in India picked up during 1978-1994 and reached its peak in 1980. The Indian output in the field of laser research forms an integral part of the mainstream science as reflected by the pattern of publications and their citations in the international literature. Laser research performed in India improved considerably during 1985-1994 as compared to 1970-1984 as seen by different impact indicators such as citation per paper, proportion of high quality papers, and publication effective index. The publication output is concentrated among few institutions and there is a similarity in the activity and attractively profile of the highly productive institutions. India"s citation rate per paper for highly productive authors is at par with the world citation rate per paper. The study indicates that the proportion of mega authored papers increased during 1990-1994 and the international collaboration is mainly with the USA.     

Some bibliometric correlates of quality in scientific research

The following kinds of data were collected on three samples of cancer research literature representing three levels of quality: (1) collaboration as measured by the number of authors per paper, (2) quantitative productivity of countries, (3) diachronous citations covering the first five years of publication, (4) total self-citations, (5) proportions of self-citations made by first-named authors, and (6) the extent of dispersion of articles among journals. Analyses showed that as the number of authors per paper increases, the proportion of high quality papers also increases and the Collaborative Index can be used to measure quality in the aggregate. It was found that the quantity and quality of cancer research done in a country are positively related. All analyses of the citation data confirmed the hypotheses that highly rated papers are significantly more highly cited than average papers and the rates of uncitedness decline with quality. The proportion of self-citations to total citations decreases with increasing quality and, on average, first-named authors of quality papers cite them proportionally fewer times than first-named authors of run-of-the mill papers do. This study also shows that, as quality increases, the extent of literature scatter or dispersion increases.     

A citation-analysis of economic research institutes

The citation analysis of the research output of the German economic research institutes presented here is based on publications in peer-reviewed journals listed in the Social Science Citation Index for the 2000–2009 period. The novel feature of the paper is that a count data model quantifies the determinants of citation success and simulates their citation potential. Among the determinants of the number of cites the quality of the publication outlet exhibits a strong positive effect. The same effect has the number of the published pages, but journals with size limits also yield more cites. Field journals get less citations in comparison to general journals. Controlling for journal quality, the number of co-authors of a paper has no effect, but it is positive when co-authors are located outside the own institution. We find that the potential citations predicted by our best model lead to different rankings across the institutes than current citations indicating structural change.     

Validation of bibliometric indicators in the field of mathematics

Bibliometric analyses of scientific publications provide quantitative information that enables evaluators to obtain a useful picture of a team's research visibility. In combination with peer judgements and other qualitative background knowledge, these analyses can serve as a basis for discussions about research performance quality. However, many mathematicians are not convinced that citation counts do in fact provide useful information in the field of mathematics. According to these mathematicians, citation and publication habits differ completely from scholarly fields such as chemistry or physics. Therefore, it is impossible to derive valid information regarding research performance from citation counts. The aim of this study is to obtain more insight into the significance of citation-based indicators in the field of mathematics. To which extent do citation-scores mirror to the opinions of experts concerning the quality of a paper or a journal? A survey was conducted to answer this question.Top journals, as qualified by experts, receive significantly higher citation rates thangood journals. Thesegood journals, in turn, have significantly higher scores than journals with the qualificationless good. Top publications, recorded in the ISI database, receive on the average 15 times more citations than the mean score within the field of mathematics as a whole. In conclusion, the experts' views on top publications or top journals correspond very well to bibliometric indicators based on citation counts.     

International and domestic co-publishing and their citation impact in different disciplines

This paper studies disciplinary differences in citation impacts of different types of co-publishing. The citation impacts of international, domestic inter-organizational and domestic intra-organizational co-publications, and single-authored publications, are compared. In particular, we examine the extent to which the number of authors explains the potential differences in citation impacts when compared to the influence of different types of international and domestic collaborations. The analysis is based on Finland’s publications in Thomson Reuters Web of Science database in 1990–2008. Finland is a small country, thus, it has fewer opportunities to find collaborators inside own country when compared to larger countries. Finland’s science policy has underlined internationalization and research collaboration as key means to increase the quality and impact of Finnish research. This study indicates that both international and domestic co-publishing have steadily increased during the past two decades in all disciplinary groups. International co-publications gain on average more citations than domestic co-publications. In natural sciences and engineering, co-authorship explains only a small proportion of variability in publications’ citation rates. When the effect of the number of authors is taken into account there are no big differences in citation impacts between international and domestic co-publications. However, international co-publications by ten authors or more gather significantly more citations than other publications. In humanities, the difference in citation impacts between co-authored publications in relation to single-authored publications is significant. However, international co-publications are not on average more highly cited in relation to domestic co-publications in humanities.     

Perceptions of author order versus contribution among researchers with different professional ranks and the potential of harmonic counts for encouraging ethical co-authorship practices

Current research performance assessment criteria contribute to some extent to author inflation per publication. Among various indicators for evaluating the quality of research with multiple authors, harmonic counting is relatively superior in terms of calculation, scientific ethics, and application. However, two important factors in harmonic counting are not yet clearly understood. These factors are the perceptions of scientists regarding the (1) corresponding author and (2) equally credited authors (ECAs). We carry out a survey investigation on different perceptions of author position versus contribution among medical researchers with different subfields and professional ranks in China, in order to provide several pieces of evidence on the aforementioned factors. We are surprised to find that researchers with different professional ranks tend to largely acknowledge their own contribution in collaborative research. Next, we conduct an empirical study to measure individual’s citation impact using inflated counts versus harmonic counts. The results indicate that harmonic h-index cannot reflect the high peak of harmonic citations. Therefore, we use (1) harmonic R-index to differentiate authors based on the harmonic citations of each paper belonging to their respective h-cores; and (2) Normalization harmonic (h, R) index as a meaningful indicator in ranking scientists. Using a sample of 40 Ph.D. mentors in the field of cardiac and cardiovascular diseases, harmonic counts can distinguish between scientists who are often listed as major contributors and those regularly listed as co-authors. This method may also discourage unethical publication practices such as ghost authorship and gift authorship.     

Early citation counts correlate with accumulated impact

Summary The present paper addresses the objective of developing forward indicators of research performance using bibliometric information on the UK science base. Most research indicators rely primarily on historical time series relating to inputs to, activity within and outputs from the research system. Policy makers wish to be able to monitor changing research profiles in a more timely fashion, the better to determine where new investment is having the greatest effect. Initial (e.g. 12 months from publication) citation counts might be useful as a forward indicator of the long-term (e.g. 10 years from publication) quality of research publications, but - although there is literature on citation-time functions - no study to evaluate this specifically has been carried out by Thomson ISI or any other analysts. Here, I describe the outcomes of a preliminary study to explore these citation relationships, drawing on the UK National Citation Report held by Evidence Ltd under licence from Thomson ISI for OST policy use. Annual citation counts typically peak at around the third year after publication. I show that there is a statistically highly significant correlation between initial (years 1-2) and later (years 3-10) citations in six research categories across the life and physical sciences. The relationship holds over a wide range of initial citation counts. Papers that attract more than a definable but field dependent threshold of citations in the initial period after publication are usually among the top 1&percnt; (the most highly cited papers) for their field and year. Some papers may take off slowly but can later join the high impact group. It is important to recognise that the statistical relationship is applicable to groups of publications. The citation profiles of individual articles may be quite different. Nonetheless, it seems reasonable to conclude that leading indicators of research excellence could be developed. This initial study should now be extended across a wider range fields to test the initial outcomes: earlier papers suggest the model holds in economics. Additional statistical tests should be applied to explore and model the relationship between initial, later and total citation counts and thus to create a general tool for policy application.     

Observations and hypotheses on the phenomenon of multiple citation to a research group's oeuvre

This paper analyses the phenomenon when a publication referring to the oeuvre of a research group (i.e. all the articles published by its members) cites several articles rather than one article from that oeuvre (multiple citations, MC). It is shown that significant differences exist between research groups with respect to the frequency at which MC to their respective oeuvres occur, and that these differences affect to some extent rankings of these groups based on citation counts. In order to find an explanation for our results, four factors are discussed: (1) the impact of a research group; (2) mutual multiple citing arrangements; (3) the size of a group's oeuvre and (4): the degree of common intellectual interest between the research activities in a group. No definite conclusions can be drawn yet on the extent to which these factors are responsible for the observed patterns in the MC frequency. We conclude however that attempts to identify ‘top’ or ‘sub-top’ groups in comparative evaluations based on citation analysis should be performed with the greatest care.     

Using Hypertext Metrics to Measure Research Output Levels

Two common ways to measure the "output" of a researcher (or research group) are to count numbers of publications and to count the citations (references to these publications in publications of others). These simple methods are flawed because they cannot easily take into account the differences in publication and citation habits in different scientific communities.An alternative approach is to view citations as hypertext links, and to use or adapt hypertext metrics to compare the scientific output of researchers, in comparison to that of others in areas with similar publication and citation patterns. We show how hypertext metrics, introduced by Botafogo, Rivlin and Shneiderman, can be modified in order to identify comparable research fields based on their publication and citation pattern. An author's performance can then be compared to that of others in research fields with a similar pattern.     

Citer analysis as a measure of research impact: library and information science as a case study

The investigators studied author research impact using the number of citers per publication an author’s research has been able to attract, as opposed to the more traditional measure of citations. A focus on citers provides a complementary measure of an author’s reach or influence in a field, whereas citations, although possibly numerous, may not reflect this reach, particularly if many citations are received from a small number of citers. In this exploratory study, Web of Science was used to tally citer and citation-based counts for 25 highly cited researchers in information studies in the United States and 26 highly cited researchers from the United Kingdom. Outcomes of the tallies based on several measures, including an introduced ch-index, were used to determine whether differences arise in author rankings when using citer-based versus citation-based counts. The findings indicate a strong correlation between some citation and citer-based measures, but not with others. The findings of the study have implications for the way authors’ research impact may be assessed.     

Patterns of annual citation of highly cited articles and the prediction of their citation ranking: A comparison across subjects

High citation is associated with research quality and consequently findings on highly cited articles are useful to increase understanding of the factors that produce high quality research. This study explores highly cited articles in six subjects, focusing on late citation and peak citation years. Longitudinal citation patterns were found to be highly varied and, on average, different from the remaining articles in each subject. For four of the six subjects, there is a correlation of over 0.42 between the percentage of early citations and total citation ranking but more highly ranked articles had a lower percentage of early citations. Surprisingly, for highly cited articles in all six subjects the prediction of citation ranking of from the sum of citations during their first six years was less accurate than prediction using the sum of the citations for only the fifth and sixth year.