Is Google Scholar useful for bibliometrics? A webometric analysis

Google Scholar, the academic bibliographic database provided free-of-charge by the search engine giant Google, has been suggested as an alternative or complementary resource to the commercial citation databases like Web of Knowledge (ISI/Thomson) or Scopus (Elsevier). In order to check the usefulness of this database for bibliometric analysis, and especially research evaluation, a novel approach is introduced. Instead of names of authors or institutions, a webometric analysis of academic web domains is performed. The bibliographic records for 225 top level web domains (TLD), 19,240 university and 6,380 research centres institutional web domains have been collected from the Google Scholar database. About 63.8% of the records are hosted in generic domains like .com or .org, confirming that most of the Scholar data come from large commercial or non-profit sources. Considering only institutions with at least one record, one-third of the other items (10.6% from the global) are hosted by the 10,442 universities, while 3,901 research centres amount for an additional 7.9% from the total. The individual analysis show that universities from China, Brazil, Spain, Taiwan or Indonesia are far better ranked than expected. In some cases, large international or national databases, or repositories are responsible for the high numbers found. However, in many others, the local contents, including papers in low impact journals, popular scientific literature, and unpublished reports or teaching supporting materials are clearly overrepresented. Google Scholar lacks the quality control needed for its use as a bibliometric tool; the larger coverage it provides consists in some cases of items not comparable with those provided by other similar databases.     

Impact of open access on citation of scholarly publications in the field of civil engineering

The development of science is accompanied by growth of scholarly publications, primarily in the form of articles in peer-reviewed journals. Scientific work is often evaluated through the number of scientific publications in international journals and their citations. This article discusses the impact of open access (OA) on the number of citations for an institution from the field of civil engineering. We analyzed articles, published in 2007 in 14 international journals with impact factor, which are included in the Journal Citation Reports subject category “Civil Engineering”. The influence of open access on the number of citations was analyzed. The aim of our research was to determine if open access articles from the field of civil engineering receive more citations than non-open access articles. Based on the value of impact factor and ranking in quartiles, we also looked at the influence of the rank of journals on the number of citations, separately for OA and Non OA articles, in databases Web of Science (WOS), Scopus and Google Scholar. For 2,026 studied articles we found out that 22 % of them were published as OA articles. They received 29 % of all citations in the observed period. We can conclude by the significance level 5 % or less that in the databases WOS and Scopus the articles from top ranked journals (first quartile) achieved more citations than Non OA articles. This argument can be confirmed for some other journals from second quartile as well, while for the journals ranked into the third quartile it can’t be confirmed. This could be confirmed only partly for journals from the second quartile, and would not be confirmed for journals ranked into the third quartile. This shows that open access is not a sufficient condition for citation, but increases the number of citations for articles published in journals with high impact.     

Do lead articles signal higher quality in the digital age? Evidence from finance journals

Citations are regarded as measures of quality yet citation rates vary widely within each of the top finance journals. Since article ordering is at the discretion of editors, lead articles can be interpreted as signals of quality that academics can use to allocate their attention and assert the value of their publications. Advances in electronic journal access allow researchers to directly access articles, suggesting article ordering may be less relevant today. We confirm the past importance of lead articles by examining citation rates from published papers as well as the wider source of papers that are listed in Google Scholar. Our findings also confirm using Google Scholar as a citation source provides congruent results to using citations from articles published in ISI-listed journals, with the additional benefit of it potentially being more timely since it includes wider citation sources, inclusive of working and conference papers.     

Comparing Google Scholar and ISI Web of Science for Earth Sciences

In order to measure the degree to which Google Scholar can compete with bibliographical databases, search results from this database is compared with Thomson’s ISI WoS (Institute for Scientific Information, Web of Science). For earth science literature 85% of documents indexed by ISI WoS were recalled by Google Scholar. The rank of records displayed in Google Scholar and ISI WoS, is compared by means of Spearman’s footrule. For impact measures the h-index is investigated. Similarities in measures were significant for the two sources.     

Web of Science with the Conference Proceedings Citation Indexes: the case of computer science

Google Scholar and Scopus are recent rivals to Web of Science. In this paper we examined these three citation databases through the citations of the book “Introduction to informetrics” by Leo Egghe and Ronald Rousseau. Scopus citations are comparable to Web of Science citations when limiting the citation period to 1996 and onwards (the citation coverage of Scopus)—each database covered about 90% of the citations located by the other. Google Scholar missed about 30% of the citations covered by Scopus and Web of Science (90 citations), but another 108 citations located by Google Scholar were not covered either by Scopus or by Web of Science. Google Scholar performed considerably better than reported in previous studies, however Google Scholar is not very “user-friendly” as a bibliometric data collection tool at this point in time. Such “microscopic” analysis of the citing documents retrieved by each of the citation databases allows us a deeper understanding of the similarities and the differences between the databases.     

Citations to the “Introduction to informetrics” indexed by WOS,Scopus and Google Scholar

Google Scholar and Scopus are recent rivals to Web of Science. In this paper we examined these three citation databases through the citations of the book “Introduction to informetrics” by Leo Egghe and Ronald Rousseau. Scopus citations are comparable to Web of Science citations when limiting the citation period to 1996 and onwards (the citation coverage of Scopus)—each database covered about 90% of the citations located by the other. Google Scholar missed about 30% of the citations covered by Scopus and Web of Science (90 citations), but another 108 citations located by Google Scholar were not covered either by Scopus or by Web of Science. Google Scholar performed considerably better than reported in previous studies, however Google Scholar is not very “user-friendly” as a bibliometric data collection tool at this point in time. Such “microscopic” analysis of the citing documents retrieved by each of the citation databases allows us a deeper understanding of the similarities and the differences between the databases.     

Motivations for URL citations to open access library and information science articles

Summary We define the URL citations of a Web page to be the mentions of its URL in the text of other Web pages, whether hyperlinked or not. The proportions of formal and informal scholarly motivations for creating URL citations to Library and Information Science open access journal articles were identified. Five characteristics for each source of URL citations equivalent to formal citations were manually extracted and the relationship between Web and conventional citation counts at the e-journal level was examined. Results of Google searches showed that 282 research articles published in the year 2000 in 15 peer-reviewed LIS open access journals were invoked by 3,045 URL citations. Of these URL citations, 43% were created for formal scholarly reasons equivalent to traditional citations and 18% for informal scholarly reasons. Of the sources of URL citations, 82% were in English, 88% were full text papers and 58% were non-HTML documents. Of the URL citations, 60% were text URLs only and 40% were hyperlinked. About 50% of URL citations were created within one year after the publication of the cited e-article. A slight correlation was found between average numbers of URL citations and average numbers of ISI citations for the journals in 2000. Separating out the citing HTML and non-HTML documents showed that formal scholarly communication trends on the Web were mainly influenced by text URL citations from non-HTML documents.     

Reception of integrative and complementary medicine (ICM) in scientific journals: a citation and co-word analysis

Even if integrative and complementary medicine (ICM) is a growing scientific field, it is also a highly contested area in terms of scientific legitimacy. The aim of this article is to analyze the reception of ICM research in scientific journals. Is this kind of research acknowledged outside the ICM context, for example, in general or specialized medicine? What is the impact of ICM research? and Is it possible to identify any shift in content, from the original ICM research to the documents where it is acknowledged? The material consisted of two sets: documents published in 12 ICM journals in 2007; and all documents citing these documents during the years 2007–2012. These sets were analyzed with help from citation and co-word analysis. When analyzing the citation pattern, it was clear that a majority of the cited documents were acknowledged in journals and documents that could be related to research areas outside the ICM context, such as pharmacology & pharmacy and plant science—even if the most frequent singular journals and subject categories were connected to ICM. However, after analyzing the content of cited and citing documents, it was striking how similar the content was. It was also evident that much of this research was related to basic preclinical research, in fields such as cell biology, plant pharmacology, and animal experiments.     

A comparison of bibliometric indicators for computer science scholars and journals on Web of Science and Google Scholar

Given the current availability of different bibliometric indicators and of production and citation data sources, the following two questions immediately arise: do the indicators’ scores differ when computed on different data sources? More importantly, do the indicator-based rankings significantly change when computed on different data sources? We provide a case study for computer science scholars and journals evaluated on Web of Science and Google Scholar databases. The study concludes that Google scholar computes significantly higher indicators’ scores than Web of Science. Nevertheless, citation-based rankings of both scholars and journals do not significantly change when compiled on the two data sources, while rankings based on the h index show a moderate degree of variation.     

Some aspects of citation indexes in Spain: A comparative analysis

This paper studies the main characteristics of the citation indexes currently developed in Spain. The paper compares the impact factors offered by Spanish citation indexes with the impact factor of Spanish journals also collected by the JCRs of the ISI (SCI and SSCI) over a five-year period (2001–2005). Spanish journals published in English have higher impact factor scores in the JCR databases of the ISI than in Spanish citation indexes.     

Reconsidering the gold open access citation advantage postulate in a multidisciplinary context: an analysis of the subject categories in the Web of Science database 2009–2014

Since Lawrence in 2001 proposed the open access (OA) citation advantage, the potential benefit of OA in relation to citation impact has been discussed in depth. The methodology to test this postulate ranges from comparing the impact factors of OA journals versus traditional ones, to comparing citations of OA versus non-OA articles published in the same non-OA journals. However, conclusions are not entirely consistent among fields, and two possible explications have been suggested in those fields where a citation advantage has been observed for OA: the early view and the selection bias postulates. In this study, a longitudinal and multidisciplinary analysis of gold OA citation advantage is developed. All research articles in all journals for all subject categories in the multidisciplinary database Web of Science are considered. A total of 1,138,392 articles—60,566 (5.3%) OA articles and 1,077,826 (94.7%) non-OA articles—published in 2009 are analysed. The citation window considered goes from 2009 to 2014, and data are aggregated for the 249 disciplines (subject categories). At journal level, we also study the evolution of journal impact factors for OA and non-OA journals in those disciplines whose OA prevalence is higher (top 36 subject categories). As the main conclusion, there is no generalizable gold OA citation advantage, neither at article nor at journal level.     

Journal self-citations that contribute to the impact factor: Documents labeled “editorial material” in journals covered by the Science Citation Index

Summary We investigated the distribution of citations included in documents labeled by the ISI as “editorial material” and how they contribute to the impact factor of journals in which the citing items were published. We studied all documents classified by the ISI as “editorial material” in the Science Citation Index between 1999 and 2004 (277,231 records corresponding to editorial material published in 6141 journals). The results show that most journals published only a few documents that included 1 or 2 citations that contributed to the impact factor, although a few journals published many such documents. The data suggest that manipulation of the impact factor by publishing large amounts of editorial material with many citations to the journal itself is not a widely used strategy to increase the impact factor.     

Benchmarking Google Scholar with the New Zealand PBRF research assessment exercise

Google Scholar was used to generate citation counts to the web-based research output of New Zealand Universities. Total citations and hits from Google Scholar correlated with the research output as measured by the official New Zealand Performance-Based Research Fund (PBRF) exercise. The article discusses the use of Google Scholar as a cybermetric tool and methodology issues in obtaining citation counts for institutions. Google Scholar is compared with other tools that provide web citation data: Web of Science, SCOPUS, and the Wolverhampton Cybermetric Crawler.     

Do journals flipping to gold open access show an OA citation or publication advantage?

The effects of Open Access (OA) upon journal performance are investigated. The key research question holds: How does the citation impact and publication output of journals switching (“flipping”) from non-OA to Gold-OA develop after their switch to Gold-OA? A review is given of the literature, with an emphasis on studies dealing with flipping journals. Two study sets with 119 and 100 flipping journals, derived from two different OA data sources (DOAJ and OAD), are compared with two control groups, one based on a standard bibliometric criterion, and a second controlling for a journal’s national orientation. Comparing post-switch indicators with pre-switch ones in paired T-tests, evidence was obtained of an OA Citation advantage but not of an OA Publication Advantage. Shifts in the affiliation countries of publishing and citing authors are characterized in terms of countries’ income class and geographical world region. Suggestions are made for qualitative follow-up studies to obtain more insight into OA flipping or reverse-flipping.

     

Citation analysis with microsoft academic

In this article, we compare publication and citation coverage of the new Microsoft Academic with all other major sources for bibliometric data: Google Scholar, Scopus, and the Web of Science, using a sample of 145 academics in five broad disciplinary areas: Life Sciences, Sciences, Engineering, Social Sciences, and Humanities. When using the more conservative linked citation counts for Microsoft Academic, this data-source provides higher citation counts than both Scopus and the Web of Science for Engineering, the Social Sciences, and the Humanities, whereas citation counts for the Life Sciences and the Sciences are fairly similar across these three databases. Google Scholar still reports the highest citation counts for all disciplines. When using the more liberal estimated citation counts for Microsoft Academic, its average citations counts are higher than both Scopus and the Web of Science for all disciplines. For the Life Sciences, Microsoft Academic estimated citation counts are higher even than Google Scholar counts, whereas for the Sciences they are almost identical. For Engineering, Microsoft Academic estimated citation counts are 14% lower than Google Scholar citation counts, whereas for the Social Sciences this is 23%. Only for the Humanities are they substantially (69%) lower than Google Scholar citations counts. Overall, this first large-scale comparative study suggests that the new incarnation of Microsoft Academic presents us with an excellent alternative for citation analysis. We therefore conclude that the Microsoft Academic Phoenix is undeniably growing wings; it might be ready to fly off and start its adult life in the field of research evaluation soon.

     

Betweenness and diversity in journal citation networks as measures of interdisciplinarity—A tribute to Eugene Garfield

Journals were central to Eugene Garfield’s research interests. Among other things, journals are considered as units of analysis for bibliographic databases such as the Web of Science and Scopus. In addition to providing a basis for disciplinary classifications of journals, journal citation patterns span networks across boundaries to variable extents. Using betweenness centrality (BC) and diversity, we elaborate on the question of how to distinguish and rank journals in terms of interdisciplinarity. Interdisciplinarity, however, is difficult to operationalize in the absence of an operational definition of disciplines; the diversity of a unit of analysis is sample-dependent. BC can be considered as a measure of multi-disciplinarity. Diversity of co-citation in a citing document has been considered as an indicator of knowledge integration, but an author can also generate trans-disciplinary—that is, non-disciplined—variation by citing sources from other disciplines. Diversity in the bibliographic coupling among citing documents can analogously be considered as diffusion  or differentiation of knowledge across disciplines. Because the citation networks in the cited direction reflect both structure and variation, diversity in this direction is perhaps the best available measure of interdisciplinarity at the journal level. Furthermore, diversity is based on a summation and can therefore be decomposed; differences among (sub)sets can be tested for statistical significance. In the appendix, a general-purpose routine for measuring diversity in networks is provided.     

An author co-citation analysis of information science in China with Chinese Google Scholar search engine, 2004–2006

Author co-citation analysis (ACA) is an important method for discovering the intellectual structure of a given scientific field. Since traditional ACA was confined to ISI Web of Knowledge (WoK), the co-citation counts of pairs of authors mainly depended on the data indexed in WoK. Fortunately, Google Scholar has integrated different academic databases from different publishers, providing an opportunity of conducting ACA in wider a range. In this paper, we conduct ACA of information science in China with the Chinese Google Scholar. Firstly, a brief introduction of Chinese Google Scholar is made, including retrieval principles and data formats. Secondly, the methods used in our paper are given. Thirdly, 31 most important authors of information science in China are selected as research objects. In the part of empirical study, factor analysis is used to find the main research directions of information science in China. Pajek, a powerful tool in social network analysis, is employed to visualize the author co-citation matrix as well. Finally, the resemblances and the differences between China and other countries in information science are pointed out.