Background : Bibliometric analysis is a valuable approach for studying and evaluating scholarly communication, research output and impact, collaboration patterns, and knowledge flows by examining various aspects of scholarly works across different levels of actors and aggregations. Despite the strong growth in publications using bibliometrics, there is limited guidance on how to report bibliometric analyses. The lack of standardized reporting practices has led to the creation of the Guidance List for the repOrting of Bibliometric AnaLyses (GLOBAL) . This evidence-based guideline, developed via the EQUATOR Network’s framework, aims to provide a systematic approach to reporting bibliometric analyses in scholarly literature with the ultimate goal of improving reporting quality. This Explanation and Elaboration (E&E) document is a supplementary document intended to facilitate the use of the GLOBAL. Methods : This E&E document was modeled after similar documents prepared for other reporting guidelines, such as Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) and Consolidated Standards of Reporting Trials (CONSORT). It includes feedback from Delphi participants and steering committee members, as well as references from peer-reviewed research papers. The GLOBAL steering committee, which supervised and provided input to the GLOBAL’s development, consisted of six international researchers, four with expertise in bibliometrics (LW, MSabé, MSolmi, and SH) and two with expertise in reporting guidelines (DM and JYN). This document underwent pilot-testing and iterative processes of editing to enhance its clarity and applicability. Results: This document includes detailed explanations and examples that satisfy the checklist items. Each item has an expanded explanation where the necessary components and rationale of the checklist items are presented, alongside practical examples. These examples are extracted from published bibliometric literature and provide authors with a framework on how to approach the reporting of their analyses when following the GLOBAL. Conclusion: The GLOBAL is designed to guide the reporting of bibliometric analyses with a particular focus on manuscript-style publications (including journal and conference papers), encouraging more transparent reporting of studies using bibliometric methods. The supplementary document aims to assist in the adoption of the GLOBAL by providing explanations and examples on its implementation.

Over the past years, the phenomenon of 'predatory publishing' has undergone fundamental changes raising pressing methodological and conceptual challenges for its study, particularly in the context of research evaluation. The complex constellation of commercial, evaluative, and scholarly actors and logics now involved necessitates an interdisciplinary, geographically-diverse, and collaborative approach to studying scholarly - and especially 'predatory' - publishing. In this piece, we outline four key conceptual and methodological dimensions that, we argue, scholars must account for when studying this phenomenon. Firstly, the constantly changing dynamics of who and what constitutes predatory publishers and practices. Secondly, disentangling the complex relationships between evaluation and practice, accounting methodologically for the myriad factors that influence these ties, and recognising that scholarly practices are not a unidirectional effect of evaluations. Thirdly, scholars must recognise that evaluation regimes are embedded in distinct political economies of academia and that the notion of predatoriness is not universal but culturally, methodologically, and institutionally contingent. Finally, the common practice of using quantitative analyses alone to study questionable publishing practices risks reproducing existing biases and overlooking structural dynamics, and thus mixed approaches incorporating qualitative methods are necessary to ensure a nuanced understanding of the topic. We argue that scholars' approach to 'predatory publishing' crucially shapes what empirical dynamics are observed, and consequently call for scholars to take a holistic approach to studying this phenomenon.

Abstract The recent surge in bibliometric studies published has been accompanied by increasing diversity in the completeness of reporting these studies’ details, affecting reliability, reproducibility, and robustness. Our study systematizes the reporting of bibliometric research using open peer reviews. We examined 182 peer reviews of 85 bibliometric studies published in library and information science (LIS) journals and conference proceedings, and non-LIS journals. We extracted 968 reviewer comments and inductively classified them into 11 broad thematic categories and 68 sub-categories, determining that reviewers largely focus on the completeness and clarity of reporting data, methods, and results. We subsequently derived 49 recommendations for the details authors should report and compared them with the GLOBAL, PRIBA, and BIBLIO reporting guidelines to identify (dis)similarities in content. Our recommendations addressed 60–80% of the guidelines’ items, while the guidelines covered 45–65% of our recommendations. Our recommendations provided greater range and specificity, but did not incorporate the functions of guidelines beyond addressing academic content. We argue that peer reviews provide valuable information for the development of future guidelines. Further, our recommendations can be read as the implicit community standards for reporting bibliometric studies and could be used by authors to aid complete and accurate reporting of their manuscripts.

This ongoing study investigates the viability of distinguishing articles in questionable journals (QJs) from those in non-QJs on the basis of quantitative indicators typically associated with quality, and what can be deduced about the quality of articles in QJs based on the differences observed. I contrast the length of abstracts and full-texts, prevalence of spelling errors, text readability, number of references and citations, and other characteristics of 1,714 articles from 31 QJs, 1,691 articles from 16 journals indexed in Web of Science (WoS), and 1,900 articles from 45 non-WoS/non-QJs, all in the field of psychology. Initial results indicate that there are differences between QJs and non-QJ samples, however these are relatively small, perhaps indicating that QJs may not substantially differ from non-QJs on these quantitative indicators of quality. However, I intend to use additional analyses to further explore any potential differences.

Despite the growth in the number of bibliometric analyses published in the peer-reviewed literature, few articles provide guidance on methods and reporting to ensure reliability, robustness, and reproducibility. Consequently, the quality of reporting in existing bibliometric studies varies greatly. In response, we are developing a preliminary Guidance List for the repOrting of Bibliometric AnaLyses (GLOBAL), a reporting guideline for bibliometric analyses. This paper outlines a scoping review that aims to identify and categorize bibliometric recommendations from the literature to develop an initial list of candidate items for GLOBAL. Five bibliographic databases, three preprint servers, and gray literature were systematically searched. Twenty-three out of 48,750 records fulfilled the inclusion criteria. Six documents contained bibliometric reporting recommendations based on a complete or partial literature review; all other sources (n = 17) contained opinion-based recommendations. A 32-item recommendation list that will inform the development of GLOBAL was created. A paucity of evidence-based studies on bibliometric reporting exists in the literature, supporting the need to create a reporting guideline for bibliometric analyses. The next step in GLOBAL project will focus on conducting a two-round Delphi study to achieve consensus on which of the 32 items should be included in GLOBAL.

Large bibliographic databases highlight tangible and symbolic differences regarding the standards of quality attached to them, underlining diverging incentive structures for small and large academic publishers. To assess the academic differences associated with these publishers, we explore bibliometric data for small publishers' journals from the Web of Science and Scopus. We then discuss the visibility and impact of highly cited literature in small open access journals in relation to their cited references from indexed and non-indexed sources. We find that non-indexed references are consistently relevant for highly cited literature, yet the share of items that obtain high citation counts is rather small and uneven across disciplines. In general, we identify regional and linguistic specificities, whilst there are some observable thematic differences compared to more mainstream publications. In particular, we underline that a healthy bibliodiversity can, depending on language or regional contexts, shape epistemic and scientific practices and narratives.

The rapid growth in the number of bibliometric studies in recent years has been accompanied by increasing diversity in the quality of the reporting of these studies’ methodologies and results. This ongoing study explores and systematises the quality and completeness of reporting bibliometric research using a bottom-up approach based on open peer review. We first identified 89 bibliometric studies published in library and information science (LIS) journals and conference proceedings and non-LIS journals, and then retrieved the 194 corresponding first-round reviews. From these reviews we extracted 968 reviewer comments pertaining to aspects of reporting the details of the se studies, and inductively classified these comments into 11 broad thematic categories and 68 sub-categories. Our preliminary results find that 77% of comments overall and the majority in each broad category were critical, which could be expected given the purpose of peer review to identify opportunities for improvement. In contrast, comments relating to the provision of study data and to the overall assessment of articles were more likely to be positive. The most common themes of reviewers’ comments were critically appraising the details of the data, methods, visualisations and tables used, and the clarity of the research questions and text. The finalised results will provide a precise and practical outline of concrete items that should be reported in bibliometric research according to the implicit community standard. Our findings will highlight particular features of bibliometric reporting that could be strengthened, complementing existing initiatives to generate guidance for the complete and accurate reporting of bibliometric studies.

Leiden Madtrics readers might already be familiar with what some have called a "great project" but a "terrible acronym": GLOBAL, the Guidance List for the repOrting of Bibliometric AnaLyses. Last summer, we invited bibliometricians to join the GLOBAL Delphi study to co-develop a reporting guideline for bibliometric analyses.

This educational article explores the utility of bibliometrics and altmetrics in evaluating traditional, complementary, and integrative medicine (TCIM) research. We introduce the concept of bibliometrics and altmetrics, provide an overview of the opportunities and challenges in using these analytical techniques, and highlight their future potential in TCIM research. Bibliometrics, based on publication and citation information, offer insights into TCIM research activity, output, scholarly influence, field structure, and collaboration practices. Altmetrics, including social media mentions, downloads, and online bookmarks, can capture the broader societal engagement with TCIM research beyond traditional academic circles. We discuss challenges and opportunities in utilizing these metrics effectively, such as addressing biases, incorporating cultural nuances, and exploring emerging trends. Additionally, we issue a call to action for researchers, policymakers, and practitioners to collaborate and leverage bibliometric and altmetric data to advance evidence-based healthcare practices to promote integrative approaches to health and wellness. By embracing a multidimensional approach to research evaluation, stakeholders can harness the potential of bibliometrics and altmetrics to improve TCIM research and healthcare delivery worldwide.

Science of science using bibliometric databases and methods has identified a specific set of studies dealing with peer review as their main topic. A unique feature of this field is the unity of the subject and object of study. It means a reviewer may find themselves reviewing a manuscript that directly examines the peer review system, i.e., the process they are undertaking. Priming, a well-established concept in psychology, describes the process by which the exposure of a person to a stimulus unconsciously influences their subsequent behaviour. For example, individuals exposed to words typically associated with elderly people were subsequently more likely to walk slowly and perform poorly on memory tests4. Priming can have a powerful influence, both positively and negatively, on a range of behaviours, such as volunteering, health and financial decision-making, and habit formation. This priming effect has been proven to affect scientists’ behaviour in remembering the works they have cited and the impact they attribute to them after receiving additional quantitative information about their impact. In the context of peer review, this raises the question as to whether the exposure of the reviewer to a manuscript examining peer review influences their reviewing behaviour, such as by priming them to review more rigorously than they otherwise would have, had the manuscript not reminded them of the key features, criteria, or potential flaws in the peer review system. Potential evidence of a priming effect on the peer review process might be seen in higher non-publication rates of peer review studies than studies on other topics. For instance, 39% of 504 peer review-related biomedical studies presented at an international conference until 2009 were not published by 2013, compared to the 29% of 585 registered medical trials completed in 2009 that remained unpublished in 2013. The Peer Review Workbench presents an opportunity to more systematically investigate a potential priming effect of meta-research on peer review behaviour. As such, we propose to investigate whether studies of peer review undergo a more rigorous peer review process, represented by longer review times and less favourable reviewer recommendations and publication decisions, than studies of other topics. We propose a case-control study in which we compare the peer review process and outcomes of an observation group of manuscripts related to peer review, i.e. peer review research, and a control group of manuscripts dealing with other topics, i.e. other research. Our dependent variables are the reviewers’ accept/reject recommendations, the overall publication decision, the number of rounds of revision, and the length of the process from submission to decision using Peer Review Workbench data of ICSR. We will also control for author, reviewer and editor attributes in our statistical analyses to see if the difference between the peer review research and other research groups is still substantial after accounting for these attributes. This research potentially deals with the ICSR theme of “Research practices” and could help us understand whether a negative feedback loop, i.e., a vicious circle, is at work discouraging scholars from studying peer review due to a more lengthy and exacting review process that may dissuade them from making further contributions to our knowledge in this area. Both authors of this proposal have experience in quantitative science studies, especially peer review as a theme, and our expertise suits the proposed research. From the metadata provided through the Peer Review Workbench, we would identify the observation group via the inclusion of the term “peer review” in the keywords of both accepted and rejected submissions. Manuscripts that were desk-rejected will be excluded as this decision may reflect issues unrelated to the quality of the manuscript, such as poor alignment with the scope of the journal, and offers no insight into the peer review process. We would also examine the manuscripts’ abstracts, where available, to ensure the accurate allocation of publications to the observation group and check whether the results of the two sets using these more fine-grained metadata differ substantially from the group assignments done solely through keywords (i.e., rejected or accepted). We will select a control group based on a random selection of articles of similar characteristics that do not study topics related to peer review. These characteristics could include article type (empirical research vs. informational), subject area classification, publication year, number of authors, and journal impact factor conditional on availability. We will explore the possibility to link and complement the Peer Review Workbench data using a search for the title, DOI or similar in bibliometric databases (we already have access to the mostly used databases) in case this will not be against the Workbench procedures and will not compromise the anonymity of review data. We do understand if this is not possible and will follow Workbench’s guidelines precisely. Upon analysing the selected variables described above, we will provide a descriptive analysis for the two publication groups, including sample sizes, the distribution of the rounds of revision involved, the length of review and decision processes, and publication outcomes. In one set of models, we will use logistic regression with the decision (accept/reject) as the dependent variable and the group assignment (observation/control) as the main independent variable to compare our peer review and other research groups. We will also include control variables based on the author, editor and reviewers’ attributes reshaped into each manuscript level (e.g., the authorship team is mixed-gender versus single gender, and similarly, the group of reviewers were mixed- or single-gender). In a second set of models, we will use the length of submission to decision as the dependent variable with the same set of independent and control variables to see if differences in decision time are observable between groups. Further, we would compare the ratio of accepted to rejected peer review studies with the overall accept/reject ratio of the publishing journal, and the ratio for peer review studies versus all other articles in the journal. This would provide additional data for any evidence of differences in outcomes between peer review and other research topics. However, as we understand the journals will be anonymised, if the journal code is not sufficient for such an analysis, we would ask assistance from the Workbench staff to identify the journals’ ratios or all publications related to a journal to make this calculation while maintaining anonymity. The results of this research will be submitted to journals dealing with peer review as a topic, e.g., Science Advances, Nature Scientific Reports, Journal of Informetrics, Quantitative Science Studies, Scientometrics, and similar, and also will be presented at relevant international conferences e.g., ISSI, STI and IC2S2.

Since the Altmetric Manifesto was published in 2010, a large number of studies have quantitatively examined altmetric-citation associations to assess if and how altmetrics differ from citations. These efforts resulted in a diverse array of observations that varies within and between indicators. These studies also differ in the measurement environment, potentially introducing confounding variables. We sought here to clarify this diversity with a meta-analysis and examine if any of several factors moderated the altmetric-citation relationship. We identified relevant literature examining altmetric-citation correlations in the Web of Science and extracted from each study the correlation coefficient, sample size, and potential moderator variables, such as publication year, field, and citation source. We collated 914 correlation coefficients from 111 studies examining 13 altmetric indicators. We fitted multi-level random-effects meta-analysis models for each altmetric indicator individually and examined moderators for Mendeley, Twitter, Altmetric Attention Score, and usage metrics. Mendeley, usage metrics, ResearchGate, Twitter, and peer ratings showed the strongest association with citations, while Facebook, Wikipedia, Google + , blogs, news, and Reddit were only weakly associated with citations. No variable consistently moderated the altmetric-citation relationship, suggesting that, when associations were observed, they were structural and persistent. Altmetrics that did not demonstrate an association evidently differ from citations in the impact construct measured. Hence, our study highlights the diversity among altmetrics channels, as the characteristics of each channel partly define the potential mentions of research in these channels, suggesting a nuanced application of the diverse channels.

A key feature of questionable journals is a lack of adequate peer review of their articles. Content of thus unknown quality may be utilised by unsuspecting practitioners or incorporated into peer-reviewed research, becoming legitimised. It is therefore necessary to examine the citation patterns of articles in questionable journals to understand their impact and reach. Similar research has tended to focus on authors from low- and middle-income countries. As such, this study investigates the profile and impact of research in questionable journals by authors in Germany. Questionable journals were identified by matching journals with articles by authors at German institutions from Dimensions to Cabell’s Predatory Reports. Metadata for these articles and a comparative sample of articles in non-questionable journals were extracted from Dimensions and the 3 year citations, self-citations, uncited rate, profile of co-authoring and citing countries, and institution type of authors were compared between groups. Nearly 600 articles in 88 questionable journals were published by German authors in 2010–2020. Three-quarters were in the medical and health sciences. Medical articles in questionable journals received significantly fewer citations than similar articles in non-questionable journals. However, articles in questionable journals were still extensively cited in 1736 primarily non-questionable journals. Self-citations accounted for only 12% of these citations. Authors from non-university medical facilities were over-represented in articles in questionable journals. System-level changes are necessary to eliminate questionable journals and shift high-quality research into reputable venues.

The primary aims of peer review are to detect flaws and deficiencies in the design and interpretation of studies, and ensure the clarity and quality of their presentation. However, it has been questioned whether peer review fulfils this function. Studies have highlighted a stronger focus of reviewers on critiquing methodological aspects of studies and the quality of writing in biomedical sciences, with less focus on theoretical grounding. In contrast, reviewers in the social sciences appear more concerned with theoretical underpinnings. These studies also found the effect of peer review on manuscripts' content to be variable, but generally modest and positive. I qualitatively analysed 1430 peer reviewers' comments for a sample of 40 social science preprint-publication pairs to identify the key foci of reviewers' comments. I then quantified the effect of peer review on manuscripts by examining differences between the preprint and published versions using the normalised Levenshtein distance, cosine similarity, and word count ratios for titles, abstracts, document sections and full-texts. I also examined changes in references used between versions and linked changes to reviewers' comments. Reviewers' comments were nearly equally split between issues of methodology (30.7%), theory (30.0%), and writing quality (29.2%). Titles, abstracts, and the semantic content of documents remained similar, although publications were typically longer than preprints. Two-thirds of citations were unchanged, 20.9% were added during review and 13.1% were removed. These findings indicate reviewers equally attended to the theoretical and methodological details and communication style of manuscripts, although the effect on quantitative measures of the manuscripts was limited.

Manuscripts have a complex development process with multiple influencing factors. Reconstructing this process is difficult without large-scale, comparable data on different versions of manuscripts. Preprints are increasingly available and may provide access to the earliest manuscript versions. Here, we matched 6,024 preprint-publication pairs across multiple fields and examined changes in their reference lists between the manuscript versions as one aspect of manuscripts' development. We also qualitatively analysed the context of references to investigate the potential reasons for changes. We found that 90 percent of references were unchanged between versions and 8 percent were newly added. We found that manuscripts in the natural and medical sciences undergo more extensive reframing of the literature while changes in engineering mostly focused on methodological details. Our qualitative analysis suggests that peer review increases the methodological soundness of scientific claims, improves the communication of findings, and ensures appropriate credit for previous research.

Dimensions was introduced as an alternative bibliometric database to the well-established Web of Science (WoS) and Scopus, however all three databases have fundamental differences in coverage and content, resultant from their owners’ indexation philosophies. In light of these differences, we explore here, using a citation network analysis and assessment of normalized citation impact of overlapping publications indexed in all three databases, whether the three databases offer structurally different perspectives of the bibliometric landscape or if they are essentially homogenous substitutes. Our citation network analysis of core and exclusive 2016–2018 publications revealed a large set of core publications indexed in all three databases that are highly self-referential. In comparison, each database selected a set of exclusive publications that appeared to hold similarly low levels of relevance to the core set and to one another, with slightly more internal communication between exclusive publications in Scopus and Dimensions than WoS. Our comparison of normalized citations for 41,848 publications indexed in all three databases found that German sectors were valuated as more impactful in Scopus and Dimensions compared to WoS, particularly for sectors with an applied research focus. We conclude that the databases do present structurally different perspectives, although Scopus and Dimensions with their additional circle of applied research vary more from the more base research-focused WoS than they do from one another.

In 2018 Dimensions was introduced as an alternative bibliometric database to the well-established Web of Science (WoS) and Scopus, however all three databases have fundamental differences in coverage and content, resultant from their owners' philosophies. Considering these differences, we use a citation network analysis and assessment of normalised citation impact of "duplicate" publications to explore whether the three databases offer structurally different perspectives of the bibliometric landscape or if they are essentially homogenous substitutes. Our citation network analysis of core and exclusive 2016-2018 publications revealed a large set of core publications indexed in all three databases that are highly self-referential. In comparison, each database selected a set of exclusive publications that appeared to hold similarly low levels of relevance to the core set and to one another, with slightly more internal communication between exclusive publications in Scopus and Dimensions than WoS. Our comparison of normalised citations for 69,044 publications in all three databases found that German sectors were valuated as more impactful in Scopus and Dimensions compared to WoS, particularly for sectors with an applied focus. We conclude that the databases do present structurally different perspectives, although Scopus and Dimensions vary more from WoS than they do from one another.

This paper provides insight into the changes manuscripts undergo during peer review, the potential reasons for these changes, and the differences between scientific fields. A growing body of literature is assessing the effect of peer review on manuscripts, however much of this research currently focuses on the social and medical sciences. We matched more than 6,000 preprint-publication pairs across multiple fields and quantified the changes in their reference lists. We also quantified the change in references per full-text section for 565 pairs from PLOS journals. In addition, we conducted manual checks of a randomly chosen sample of 98 pairs to validate our results, and undertook a qualitative analysis based on the context of the reference to investigate the potential reasons for reference changes. We found 10 disciplines, mostly in the natural sciences with high levels of removed references. Methods sections undergo the most relative change in the natural sciences, while in the medical and health sciences, the results and discussion sections underwent the most changes. Our qualitative analysis identified issues with our results due to incomplete preprint reference lists. In addition, we deduced 10 themes for changing references during peer review. This analysis suggested that manuscripts in the natural and medical sciences undergo more extensive reframing of the literature used to situate and interpret the results of studies than the social and agricultural sciences, which are further embedded in the existing literature through peer review. Peer review in engineering tends to focus on methodological details. Our results are useful to the body of literature examining the effectiveness of peer review in fulfilling its intended purposes.