Research project evaluation and selection: an evidential reasoning rule-based method for aggregating peer review information with reliabilities

Wei-dong Zhu, Fang Liu (), Yu-wang Chen, Jian-bo Yang, Dong-ling Xu and Dong-peng Wang
Additional contact information
Wei-dong Zhu: Hefei University of Technology
Fang Liu: Hefei University of Technology
Yu-wang Chen: The University of Manchester
Jian-bo Yang: Hefei University of Technology
Dong-ling Xu: Hefei University of Technology
Dong-peng Wang: Hefei University of Technology

Scientometrics, 2015, vol. 105, issue 3, No 7, 1469-1490

Abstract: Abstract Research project evaluation and selection is mainly concerned with evaluating a number of research projects and then choosing some of them for implementation. It involves a complex multiple-experts multiple-criteria decision making process. Thus this paper presents an effective method for evaluating and selecting research projects by using the recently-developed evidential reasoning (ER) rule. The proposed ER rule based evaluation and selection method mainly includes (1) using belief structures to represent peer review information provided by multiple experts, (2) employing a confusion matrix for generating experts’ reliabilities, (3) implementing utility based information transformation to handle qualitative evaluation criteria with different evaluation grades, and (4) aggregating multiple experts’ evaluation information on multiple criteria using the ER rule. An experimental study on the evaluation and selection of research proposals submitted to the National Science Foundation of China demonstrates the applicability and effectiveness of the proposed method. The results show that (1) the ER rule based method can provide consistent and informative support to make informed decisions, and (2) the reliabilities of the review information provided by different experts should be taken into account in a rational research project evaluation and selection process, as they have a significant influence to the selection of eligible projects for panel review.

Keywords: Research project evaluation and selection; Evidential reasoning; Reliability; Confusion matrix (search for similar items in EconPapers)
Date: 2015
References: View references in EconPapers View complete reference list from CitEc
Citations: View citations in EconPapers (7)

Downloads: (external link)
http://link.springer.com/10.1007/s11192-015-1770-8 Abstract (text/html)
Access to the full text of the articles in this series is restricted.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:spr:scient:v:105:y:2015:i:3:d:10.1007_s11192-015-1770-8

Ordering information: This journal article can be ordered from
http://www.springer.com/economics/journal/11192

DOI: 10.1007/s11192-015-1770-8

Access Statistics for this article

Scientometrics is currently edited by Wolfgang Glänzel

More articles in Scientometrics from Springer, Akadémiai Kiadó
Bibliographic data for series maintained by Sonal Shukla () and Springer Nature Abstracting and Indexing ().