Institutional research rankings via bibliometric analysis and direct peer review: A comparative case study with policy implications

Recent years have seen enormously increased interest in the comparative evaluation of research quality in the UK, with considerable resources devoted to ranking the output of academic institutions relative to one another at the sub-discipline level, and the disposition of even greater resources dependent on the outcome of this process. The preferred methodology has been that of traditional peer review, with expert groups of academics tasked to assess the relative worth of all research activity in ‘their’ field. Extension toinstitutional evaluation of a recently refined technique ofjournal ranking (Discipline Contribution Scoring) holds out the possibility of ‘automatic’ evaluation within a time-frame considerably less than would be required using methods based directly on citation counts within the corpus of academic work under review. This paper tests the feasibility of the technique in the sub-field of Business and Management Studies Research, producing rankings which are highly correlated with those generated by the much more complex and expensive direct peer review approach. More generally, the analysis also gives a rare opportunity directly to compare the equivalence of peer review bibliometric analysis over a whole sub-field of academic activity in a non-experimental setting.     

Patterns of annual citation of highly cited articles and the prediction of their citation ranking: A comparison across subjects

High citation is associated with research quality and consequently findings on highly cited articles are useful to increase understanding of the factors that produce high quality research. This study explores highly cited articles in six subjects, focusing on late citation and peak citation years. Longitudinal citation patterns were found to be highly varied and, on average, different from the remaining articles in each subject. For four of the six subjects, there is a correlation of over 0.42 between the percentage of early citations and total citation ranking but more highly ranked articles had a lower percentage of early citations. Surprisingly, for highly cited articles in all six subjects the prediction of citation ranking of from the sum of citations during their first six years was less accurate than prediction using the sum of the citations for only the fifth and sixth year.     

Empirical studies of negative political advertising: A quantitative review using a method of combined citation and content analysis

This study quantitatively reviews the empirical studies of negative political advertising. A method of the combination of citation analysis and content analysis is used. The citation analysis examines each cited work in 20 selected studies with respect to its citation information; and the content analysis investigates these 20 selected studies (citing sources) in terms of their hypotheses, research questions, and methodologies. The aggregated information from the individual cited works and the citing works show that scholars from communication and other disciplines have strong influence on the development of the empirical studies on negative political ads, but communication scholars remain as the driving force. Facing continuously increased literatures in the area, communication scholars need to develop a theory or theories to guide the research. The direction of the research has been moving toward focusing on the boarder and more general effects of negative political ads.     

Prediction and application of article potential citations based on nonlinear citation-forecasting combined model

As the number of academic articles rapidly increases, a reasonable evaluation method for the articles is highly required in the current academic research. Meanwhile, a faster access to the high-quality academic articles for the researchers is also of critical significance. This paper first improves the AVG model and presents a new Nonlinear Citation-Forecasting Combined Model (NCFCM) based on a neural network to predict the potential increase of citation counts. Then, the NCFCM is used to analyze and rank the academic articles in online databases. The results of NCFCM model are compared to the results from other existing methods. Empirical analysis and comparisons demonstrate that the NCFCM model is of high accuracy and robustness in forecasting potential citation counts and ranking academic articles. Ranking academic articles according to the potentional citation counts can help researchers retrieve the desired articles efficiently in a short time.

     

Exploring time factors in measuring the scientific impact of scholars

Taking advantage of the easy access to the rich and massive scholarly data, more and more researchers are focusing on the studies of analyzing and utilizing the scholarly big data. Among them, evaluating the scientific impact of scholars has significant importance. Measuring the scientific impact of scholars can not only provide basis for the applications of academic foundations and awards, but also shed light on the research directions for scholars. Currently, citation based methods and network based metrics are the most commonly used ways to evaluate the scientific impact. However, these approaches ignore several important facts, i.e. the dynamics of citations and the initial qualities of different articles. To alleviate the shortcomings of them, we propose a Time-aware Ranking algorithm (TRank) to evaluate the impact of scholars. Due to scholars’ sustainable supreme concerns of academic innovations, the TRank algorithm gives more credits to the newly published scholarly papers as well as their references according to the representative time functions. Our method also combines the merits of random walk algorithms and heterogeneous network topology, i.e. the mutual influences among different scholarly entities in heterogeneous academic networks. To validate the suitable time function for TRank algorithm and explore its performance, we construct the experiments on two real datasets: (1) Digitial Bibliography and Library Project, and (2) American Physical Society. The experimental results demonstrate that our algorithm outperforms other methods in selecting outstanding scholars and the evaluation results on the overall impact of scholars.     

基于互补判断矩阵排序的供应商绩效评价方法

提出了一种基于互补判断矩阵排序的供应商绩效评价方法.由多位专家对供应商的绩效进行两两比较并进行评分;评分结果以互补判断矩阵的形式构造出相应的评价矩阵;利用权的最小平方法计算互补判断矩阵的排序向量;将排序向量中的分量按照大小排列,得到供应商绩效的比较结果.算例的计算过程和结果表明,该方法具有很好的可操作性.     

营造联觉体验的儿童感统训练互动玩教具设计研究

目的 探索一种营造联觉体验的儿童感统训练互动玩教具设计方法,以科学指导儿童玩教具的功能与形态设计过程。方法 首先,通过访谈调研提取现有儿童感统训练玩教具的共性设计要素,结合用户联觉体验和Kano模型制定调查问卷,总结出目标用户即3～6岁儿童的设计需求,分析得出优先级排序;其次,根据问卷结果构建层次分析模型,分析设计原则,再根据专家组的评判结果,对各个指标的权值进行计算,从而得出其权重值和排序;最后对实例研究方案的各个指标进行模糊综合评价。结果 根据Kano模型和模糊层次分析法(FAHP),最终得出对儿童感统训练互动玩教具设计影响最大的几个因素与设计方案评分,以指导优化设计方案。结论 通过将Kano模型与模糊层次分析法(FAHP)相结合的研究模型,应用于儿童感统训练互动玩教具设计研究中,使设计方案更加科学合理,为相关儿童玩教具产品的设计研究提供了新的理论方法与思路。     

A relevance ranking method for citation-based search results

A growing number of researchers are exploring the use of citation relationships such as direct citation, bibliographic coupling, and co-citation for information retrieval in scientific databases and digital libraries. In this paper, I propose a method of ranking the relevance of citation-based search results to a set of key, or seed, papers by measuring the number of citation relationships they share with those key papers. I tested the method against 23 published systematic reviews and found that the method retrieved 87% of the studies included in these reviews. The relevance ranking approach identified a subset of the citation search results that comprised 27% of the total documents retrieved by the method, and 7% of the documents retrieved by these reviews, but that contained 75% of the studies included in these reviews. Additional testing suggested that the method may be less appropriate for reviews that combine literature in ways that are not reflected in the literature itself. These results suggest that this ranking method could be useful in a range of information retrieval contexts.     

Comparison of the Hirsch-index with standard bibliometric indicators and with peer judgment for 147 chemistry research groups

In this paper we present characteristics of the statistical correlation between the Hirsch (h-) index and several standard bibliometric indicators, as well as with the results of peer review judgment. We use the results of a large evaluation study of 147 university chemistry research groups in the Netherlands covering the work of about 700 senior researchers during the period 1991–2000. Thus, we deal with research groups rather than individual scientists, as we consider the research group as the most important work floor unit in research, particularly in the natural sciences. Furthermore, we restrict the citation period to a three-year window instead of ‘life time counts’ in order to focus on the impact of recent work and thus on current research performance. Results show that the h-index and our bibliometric ‘crown indicator’ both relate in a quite comparable way with peer judgments. But for smaller groups in fields with ‘less heavy citation traffic’ the crown indicator appears to be a more appropriate measure of research performance.     

An improved FMEA method based on the linguistic weighted geometric operator and fuzzy priority

Using the group linguistic information aggregation method, this research improved upon conventional failure mode and effect analysis (FMEA), which had the problems of the inevitable subjectivity of expert evaluations, and of the difficulty of determining the weights of experts' evaluations. First, this research treated the three risk factors of FMEA evaluation as linguistic variables and introduced a linguistic weighted geometric (LWG) operator to implement algebraic operations on the results of expert evaluations. This treatment overcame the disadvantage of fuzzy theory-based methods in which decision-making information could be lost due to the twice conversion process. Second, the weights of experts' evaluations were calculated based on the consistency of experts' ranking using a fuzzy priority method. This method placed a lower weight for the experts whose evaluation departed from group consensus, and therefore decreased the impact of the unfairness of experts on evaluation results. Finally, this article demonstrated the effectiveness and applicability of this method by an example of failure-mode analysis on the grinding wheel system of a numerical control machine.     

Graph-based algorithms for ranking researchers: not all swans are white!

Scientific importance ranking has long been an important research topic in scientometrics. Many indices based on citation counts have been proposed. In recent years, several graph-based ranking algorithms have been studied and claimed to be reasonable and effective. However, most current researches fall short of a concrete view of what these graph-based ranking algorithms bring to bibliometric analysis. In this paper, we make a comparative study of state-of-the-art graph-based algorithms using the APS (American Physical Society) dataset. We focus on ranking researchers. Some interesting findings are made. Firstly, simple citation-based indices like citation count can return surprisingly better results than many cutting-edge graph-based ranking algorithms. Secondly, how we define researcher importance may have tremendous impacts on ranking performance. Thirdly, some ranking methods which at the first glance are totally different have high rank correlations. Finally, the data of which time period are chosen for ranking greatly influence ranking performance but still remains open for further study. We also try to give explanations to a large part of the above findings. The results of this study open a third eye on the current research status of bibliometric analysis.     

Decision support system for rapid prototyping process selection through integration of fuzzy synthetic evaluation and an expert system

A new method is proposed for selecting the most appropriate rapid prototyping process according to user's specific requirements by using the expert system and fuzzy synthetic evaluation. The selection process is divided into two stages. First, it is necessary to generate feasible alternatives, which are executed under the expert system environment. Second, given those feasible alternatives, the fuzzy synthetic evaluation approach is employed to produce a ranking order of the alternatives and to finalize the most suirapid prototyping system. One distinctive characteristic of this method is that quantitative as well as qualitative measures are employed, providing more accurate results. The decision system developed based on the proposed method is composed of four modules: a database to store the specifications of various rapid prototyping processes; a knowledge-based expert system for determining the feasible alternatives; a fuzzy synthetic evaluation model to select the most suitable rapid prototyping process; and a user interface and an expert interface to interact with the system. The fuzzy synthetic evaluation approach used in the system is illustrated in detail by a numerical example. Furthermore, a Java-enabled solution, together with web techniques, is employed for developing such a networked decision support system. Finally, two examples of rapid prototyping process selection are designed to demonstrate the application of the system. The system has been implemented and can run at a rapid prototyping and manufacturing networked service platform that the authors have developed.     

Absolute and specific measures of research group excellence

A desirable goal of scientific management is to introduce, if it exists, a simple and reliable way to measure the scientific excellence of publicly funded research institutions and universities to serve as a basis for their ranking and financing. While citation-based indicators and metrics are easily accessible, they are far from being universally accepted as way to automate or inform evaluation processes or to replace evaluations based on peer review. Here we consider absolute measurements of research excellence at an amalgamated, institutional level and specific measures of research excellence as performance per head. Using biology research institutions in the UK as a test case, we examine the correlations between peer review-based and citation-based measures of research excellence on these two scales. We find that citation-based indicators are very highly correlated with peer-evaluated measures of group strength, but are poorly correlated with group quality. Thus, and almost paradoxically, our analysis indicates that citation counts could possibly form a basis for deciding on, how to fund research institutions, but they should not be used as a basis for ranking them in terms of quality.     

Google Scholar University Ranking Algorithm to Evaluate the Quality of Institutional Research

Education quality has undoubtedly become an important local and international benchmark for education, and an institute’s ranking is assessed based on the quality of education, research projects, theses, and dissertations, which has always been controversial. Hence, this research paper is influenced by the institutes ranking all over the world. The data of institutes are obtained through Google Scholar (GS), as input to investigate the United Kingdom’s Research Excellence Framework (UK-REF) process. For this purpose, the current research used a Bespoke Program to evaluate the institutes’ ranking based on their source. The bespoke program requires changes to improve the results by addressing these methodological issues: Firstly, Redundant profiles, which increased their citation and rank to produce false results. Secondly, the exclusion of theses and dissertation documents to retrieve the actual publications to count for citations. Thirdly, the elimination of falsely owned articles from scholars’ profiles. To accomplish this task, the experimental design referred to collecting data from 120 UK-REF institutes and GS for the present year to enhance its correlation analysis in this new evaluation. The data extracted from GS is processed into structured data, and afterward, it is utilized to generate statistical computations of citations’ analysis that contribute to the ranking based on their citations. The research promoted the predictive approach of correlational research. Furthermore, experimental evaluation reported encouraging results in comparison to the previous modification made by the proposed taxonomy. This paper discussed the limitations of the current evaluation and suggested the potential paths to improve the research impact algorithm.     

A comparative study on world university rankings: a bibliometric survey

Recently there are many organizations conducting projects on ranking world universities from different perspectives. These ranking activities have made impacts and caused controversy. This study does not favor using bibliometric indicators to evaluate universities?? performances, but not against the idea either. We regard these ranking activities as important phenomena and aim to investigate correlation of different ranking systems taking bibliometric approach. Four research questions are discussed: (1) the inter-correlation among different ranking systems; (2) the intra-correlation within ranking systems; (3) the correlation of indicators across ranking systems; and (4) the impact of different citation indexes on rankings. The preliminary results show that 55?% of top 200 universities are covered in all ranking systems. The rankings of ARWU and PRSPWU show stronger correlation. With inclusion of another ranking, WRWU (2009?C2010), these rankings tend to converge. In addition, intra-correlation is significant and this means that it is possible to find out some ranking indicators with high degree of discriminativeness or representativeness. Finally, it is found that there is no significant impact of using different citation indexes on the ranking results for top 200 universities.     

Impact of bibliometrics upon the science system: Inadvertent consequences?

Summary Ranking of research institutions by bibliometric methods is an improper tool for research performance evaluation, even at the level of large institutions. The problem, however, is not the ranking as such. The indicators used for ranking are often not advanced enough, and this situation is part of the broader problem of the application of insufficiently developed bibliometric indicators used by persons who do not have clear competence and experience in the field of quantitative studies of science. After a brief overview of the basic elements of bibliometric analysis, I discuss the major technical and methodological problems in the application of publication and citation data in the context of evaluation. Then I contend that the core of the problem lies not necessarily at the side of the data producer. Quite often persons responsible for research performance evaluation, for instance scientists themselves in their role as head of institutions and departments, science administrators at the government level and other policy makers show an attitude that encourages 'quick and dirty' bibliometric analyses whereas better quality is available. Finally, the necessary conditions for a successful application of advanced bibliometric indicators as support tool for peer review are discussed.     

Chinese communication studies: A citation analysis of Chinese communication research in English-language journals

The study employs citation analysis method to identify the disciplines and active research areas in communication studies on communication systems in China. Moreover, the study seeks to contribute to the methodological issues of citation analysis by including new variables in the analysis. Using Chinese communication research in 11 Chinese/Asian studies journals and 13 journalism/communication journals published in English since 1931, the study found that there were little exchanges between Chinese studies and communication scholars. Howerver, the study showed that by including two variables—theme of articles and academic affiliation of authors, the findings can more accurate demonstrate the relationship between the research activities and disciplines cited.     

Geographical knowledge diffusion and spatial diversity citation rank

This paper proposes a citation rank based on spatial diversity (SDCR) in terms of cities and countries, focusing on the measurement of the “spatial” aspect in citation networks. Our main goal is to solve the citation bias caused by different geographical locations of citations. We empirically investigate spatial properties of citing distances, citation patterns and spatial diversity to understand geographical knowledge diffusion, based on the data from “Transportation Science and Technology” subject category in the Web of Science (1966–2009). We also compare the proposed ranking method with other bibliometric measures, and conduct a case study to figure out the recent ranks of the well-established authors in Transportation research. It is found that the SDCR of a focal author is highly correlated with the sum of spatial diversity weights (“strength”) of all his in-links, and it is better to set the damping factors smaller than 0.75 when ranking authors with various initial academic years by SDCR. The cases show that Hong Kong is becoming a cluster in Transportation research.     

A study on power-law distribution of hostnames in the URL references

The power-law distribution and the Garfield’s Law of Concentration of journal citation have long been verified by empirical data. As a relatively new type of reference, the URL references are cited more and more frequently in the scientific papers and their distribution is proved to fit for the Garfield’s Law of Concentration too. In this article, we collect three URL references datasets extracted from papers written by researchers belonging to three big research groups : Chinese Academy of Sciences, Max Planck Institute, and the whole Chinese scientific researchers. Through the curve-fitting with SPSS and contrast the results with the judgment standard of power-law distribution, we verify that there also exists power-law distribution in the citation frequency of hostnames in these three URL references datasets. And our experimental results show that the range of power exponent in the journal references and the URL references are different. Started from the concrete empirical procedures and the final experimental results, we analyze four factors that may lead to this difference between journal references and URL references: the sample size, the sampling method, the concentration of citation and the type property of citation.