The study deals with research performance assessment issues as important aspects of research management and research quality. The case of Kazakhstan clearly demonstrates the impact of the prevailing bibliometric-centered approach. The aim of this study is to suggest an inclusive scale of individual research performance assessment. The method used is a quantitative study of the opinions of researchers and academics on the range of research related activities they traditionally carry out. The study expands the knowledge base on academic human resource management, and can be of high relevance for substantiating the criteria of performance assessment of researchers by HR managers of universities and public research institutions. The research results can be helpful for setting and complying with individual and institutional criteria for research performance evaluation. Policy highlights A Survey among 264 researchers in Kazakh universities and public research institutions (response rate: 63%) asked them to rate their activities in five groups: supervising activity, professional advancement, publications, public recognition, and scientific & organizational activities. The results demonstrate that their priorities correspond to national and international priorities: publishing papers in local and international peer-reviewed journals indexed in WoS and Scopus, and monographs. Findings revealed that “Supervising” and “Professional Advancement” activities have the highest importance among all criteria groups. Respondents gave the highest preference to participating at overseas and international conferences, seminars and workshops, thus expressing their desire to disseminate their research findings internationally and to build international links. Scientific & organizational activities are the core activities which correlate to all other activities. And the role of S&O criteria is definitely underestimated in the performance assessment of researchers. The current research performance evaluation system in Kazakhstan is dominantly based on bibliometrics and is one-sided and biased. An inclusive scale of individual research performance assessment needs to be developed, considering researchers’ ideas and preferences.
A critical discussion is presented for the Author Metrics Database (AMD) created by Ioannides et al. (2016, 2020) containing citation-based indicators for 165,000 authors publishing in journals indexed in Scopus. It is concluded that the AMD is a rich intermediary dataset open for further analysis to all interested users. However, its indicators suggest a false precision and lack transparency. The theoretical and statistical basis of the database’s key composite impact indicator is weak, and information on whether or not underlying author publication lists were validated is lacking. The paper aims to broaden the perspective on the further development of an AMD, highlighting its bottom-up, interactive use, aptness for self-assessment and educational function for a wide user community. POLICY HIGHLIGHTS Scopus diverges from Eugene Garfield’s original concept of the Science Citation Index, as citation impact plays a weaker role as journal selection criterion. The transparency of the Article Metrics Database (AMD) is seriously hampered by the lack of information on whether the data were verified by scientists themselves. A complex composite indicator in the AMD decides whether or not a particular author is included. Its components are strongly statistically dependent and are largely based on the position an author has in a paper’s author sequence but lack a sound theoretical foundation. An assessment of an individual researcher cannot be merely based on whether or not he or she is included in the AMD. The issue as to how to deal with multi-authored papers in research assessment of individuals can to some extent be enlightened by bibliometric indicators but cannot be solved bibliometrically. This is why the Composite Indicator suggests a false precision. The AMD focuses almost exclusively on senior scientists. Early career scientists and emerging research groups who will shape science and scholarship in the near future hardly appear in the AMD. Desktop bibliometrics using the AMD as a sole source of information must be rejected. Using the AMD as a starting point in a more extensive bibliometric data collection makes it de facto a promotion tool for other Elsevier products. An alternative approach is an interactive, bottom-up bibliometric tool designed for self-assessment and educational purposes, showing how bibliometric indicators depend upon the way in which initial publication lists, author benchmark sets, subject delimitations, thresholds and evaluative assumptions are chosen. Research assessment is much more than just bibliometrics. It requires an overarching evaluative framework based on normative views on what constitutes research performance and which policy objectives should be achieved.
Although large citation databases such as Web of Science and Scopus are widely used in bibliometric research, they have several disadvantages, including limited availability, poor coverage of books and conference proceedings, and inadequate mechanisms for distinguishing among authors. We discuss these issues, then examine the comparative advantages and disadvantages of other bibliographic databases, with emphasis on (a) discipline-centered article databases such as EconLit, MEDLINE, PsycINFO, and SocINDEX, and (b) book databases such as Amazon.com, Books in Print, Google Books, and OCLC WorldCat. Finally, we document the methods used to compile a freely available data set that includes five-year publication counts from SocINDEX and Amazon along with a range of individual and institutional characteristics for 2,132 faculty in 426 U.S. departments of sociology. Although our methods are time-consuming, they can be readily adopted in other subject areas by investigators without access to Web of Science or Scopus (i.e., by faculty at institutions other than the top research universities). Data sets that combine bibliographic, individual, and institutional information may be especially useful for bibliometric studies grounded in disciplines such as labor economics and the sociology of professions. Policy highlights While nearly all research universities provide access to Web of Science or Scopus, these databases are available at only a small minority of undergraduate colleges. Systematic restrictions on access may result in systematic biases in the literature of scholarly communication and assessment. The limitations of the largest citation databases influence the kinds of research that can be most readily pursued. In particular, research problems that use exclusively bibliometric data may be preferred over those that draw on a wider range of information sources. Because books, conference papers, and other research outputs remain important in many fields of study, journal databases cover just one component of scholarly accomplishment. Likewise, data on publications and citation impact cannot fully account for the influence of scholarly work on teaching, practice, and public knowledge. The automation of data compilation processes removes opportunities for investigators to gain first-hand, in-depth understanding of the patterns and relationships among variables. In contrast, manual processes may stimulate the kind of associative thinking that can lead to new insights and perspectives.
Describes a method to provide an independent, community-sourced set of best practice criteria with which to assess global university rankings and to identify the extent to which a sample of six rankings, Academic Ranking of World Universities (ARWU), CWTS Leiden, QS World University Rankings (QS WUR), Times Higher Education World University Rankings (THE WUR), U-Multirank, and US News & World Report Best Global Universities, met those criteria. The criteria fell into four categories: good governance, transparency, measure what matters, and rigour. The relative strengths and weaknesses of each ranking were compared. Overall, the rankings assessed fell short of all criteria, with greatest strengths in the area of transparency and greatest weaknesses in the area of measuring what matters to the communities they were ranking. The ranking that most closely met the criteria was CWTS Leiden. Scoring poorly across all the criteria were the THE WUR and US News rankings. Suggestions for developing the ranker rating method are described.
Research and innovation is one of Flanders’ priorities and over the last three decades its public funding has strongly increased. Universities are key actors in this strategy. They have a large autonomy and receive a substantial share of additional R&D expenditures as lump sum funding.The Flemish authorities use quantitative indicators to allocate these lump sums to the universities. The funding formulae take into account each institution’s size, its research performance and, if relevant, its valorization activities. Also significant is the realization of governmental priorities, such as mobility and diversity of the academic staff. This paper describes the development of the Flemish university funding model, analyses its weaknesses and its strengths, and compares it with nine national metrics-based research performance funding systems.Policy highlightsFlanders, like many other regions and nations, has adopted performance-based research funding systems (PRFSs) to improve and provide accountability for its science and innovation system. The Flemish PRFS criteria have evolved considerably over the last three decades, and due to competition between universities and a consensus model of political decision making, the funding formula is comparatively complex.Building on an historical background of lump sum payments to universities, supporting both education and research, special supplementary funds for blue-sky research (BOF) and for strategic applied research, innovation, and outreach activities (IOF) were introduced by the Flemish government in 1994 and 2004, respectively. The “three-legged stool” funding mechanism for research in Flanders is unique within Europe.The introduction of publication and citation metrics in 2003 changed the character of the Flemish PRFS considerably, previously focused on measures of a university’s size in terms of students, degrees granted, and previous funding. Humanities and social sciences (HSS) research metrics were considered only after first adoption of bibliometric methods traditionally deployed for assessment of the natural and life sciences, but this is not unique to Flanders. HSS research has been addressed in Flanders through the creation of a special database to supplement standard citation indexes, the VABB-SHW. This special resource is a strong point of the Flemish PRFS, and other nations should appreciate the value of increased coverage of the HSS literature.The PRFS in Flanders differs from that of nine European nations in several ways: analysis is undertaken annually, it includes a diversity and mobility measure (2006), and one of interdisciplinary will soon be added.Flanders, like other nations using PRFSs, has generally seen increased research output and intellectual property activities after its introduction; however, in some nations the increase began before a PRFS was implemented. In almost all cases, demonstrating a causal link between a PRFS and increased output is difficult since R&D investments and the number of researchers have also increased for a variety of reasons. This is a cautionary note in not over-interpreting the effects or effectiveness of a PRFS.Flanders now exhibits one of the highest levels among European nations of research funding allocations determined by a PRFS: 50% of BOF-funding and 75% of IOF-funding. By employing such a metrics-heavy scheme, Flanders is a good candidate for a detailed study of unintended consequences of a PRFS, at the national, institutional, and research group and individual researcher level. This remains a large gap in our understanding of the use of a PRFS.
