Abstract This article presents the results of a scoping review on the social impact of scientific institutions, mapping the main concepts, indicators, and evaluation approaches identified in recent literature. The systematic analysis of studies published between 2019 and 2025 reveals a field that is expanding but remains conceptually and methodologically fragmented. The findings show the predominance of altmetrics derived from social media, while qualitative, contextual, and participatory approaches remain comparatively underrepresented. The review also highlights the persistent tendency to equate societal impact with visibility, despite growing evidence that these dimensions should be regarded as complementary rather than interchangeable. Another key finding concerns the limited and weakly theorised consideration of news media within existing social impact assessment frameworks. Rather than being systematically incorporated as differentiated channels of communication, news media are generally subsumed within broader indicators of online attention, overlooking dimensions related to editorial mediation, credibility, and public trust. From this perspective, the review suggests that future research may benefit from developing more integrative and context-sensitive evaluation frameworks capable of combining quantitative indicators with qualitative forms of evidence and a more differentiated consideration of dissemination channels. These implications are presented as interpretative conclusions derived from the reviewed literature and as potential directions for future research, rather than as operational models for evaluation.
Abstract The literature on the JIF is abundant due to its frequent use as an indicator in evaluation processes. In this collection of publications referring to it, observations about its limitations or inadequacy for the purposes to which it is applied are perhaps the most frequent section. One of the reasons for this criticism is the concentration of citations in a small fraction of the articles, which, according to the authors of these observations, undermines the value of the indicator insofar as it applies a measure based on averages to distributions that are highly skewed by this concentration of citations. This study analyses whether the JIF is indeed as affected by this distortion as has been pointed out, which would reduce its value as an indicator. Alternatively, the effect of Highly Cited Papers (HCP) may not disrupt the overall structure of journals in their respective categories when ranked according to this criterion. In this case, the value of this indicator, without going into other limitations that may affect it, would remain valid from this point of view in its application in evaluation exercises and, with it, other associated indicators such as quartiles. The study also analyses the behaviour of HCPs and whether their weight in the total journal citations has such a significant effect that, beyond a certain threshold, it would advise against using JIFs for this reason.
Animal methods bias is a peer review bias characterized by a preference for animal-based biomedical research methods or lack of expertise to adequately evaluate nonanimal methods, like complex in vitro models. Preliminary evidence indicates that some manuscript reviewers request animal experiments, which authors often feel are not scientifically justified. We explored this through a cross-sectional survey of 189 biomedical researchers and bibliometric analyses. Fifty-two percent of survey respondents had been asked to add animal experiments to nonanimal studies during manuscript review. On average, authors refused 79% of these requests, citing reasons like lack of necessity or feasibility. Moreover, 46% felt their nonanimal studies received unfair reviews due to absence of animals. Most respondents (84%) reported negative consequences, like publishing in lower-impact journals (65%), rejection (49%), and publication delays (42%). To triangulate self-reported findings, we investigated over 1.1 million publications (2009-2023), identifying significantly higher journal impact factors for studies using animal versus in vitro methods, although differences varied by field. Additionally, we created a natural language processing pipeline to analyze 304,849 animal-based publications and compare species used in 3,159 corresponding preprints. We identified 17 studies where an animal experiment was introduced after the first preprint version was posted, suggesting that authors occasionally add them after original study completion. Animal methods bias presents challenges for researchers using animal-free methods and addressing it is essential as funding bodies increasingly invest in nonanimal approaches to improve translatability. These findings help elucidate animal methods bias and highlight a critical challenge for biomedical research evaluation.
When a public official proposes direct employment as the test of whether publicly funded research is worth its cost, the remark is easy to dismiss as political rhetoric. Read from inside the evaluation community, however, it is a proposal about evaluation: it nominates a single, legible indicator as the measure of public value. Writing as a research-institute director, principal investigator and evaluator in Chile's national funding system, I take that proposal seriously to show what an assessment system would do if it adopted it. I argue that the temptation toward single-proxy assessment is strongest, and most damaging, in low-investment systems, where the cost of misreading what research produces is highest. Drawing on field experience, I distinguish five kinds of return-scientific, formative, institutional-territorial, democratic, and slow-maturing-that an employment figure cannot capture, and that are especially characteristic of the social sciences and humanities. Read against the 'productive interactions' approach, the typology also counts the collective capacity built by institutional continuity as a return in itself. I then translate this critique into three practical design moves for assessment: eliciting a plural, structured value proposition at the application stage; crediting public presence and uptake as evidence of return; and timing evaluation to the return through periodic ex-post review. The aim is not to reject accountability but to make our instruments of accountability intelligent enough to register what research actually contributes. The piece offers situated, transferable guidance for funders and panels, particularly in lower-investment and Global South research systems.
Large-scale bibliometric databases generally provide a selective and geographically-biased representation of scientific output. Given that these indexing biases reduce the likelihood of retrieving certain types of scientific production, it has been historically hard to assess the extent and characteristics of those excluded scholarly communications. Based on the Lattes platform, a unique data source with comprehensive coverage of Brazilian scholarly output, this paper assesses the coverage of two large-scale bibliometric databases: OpenAlex and Web of Science. Our findings identify the specific research areas underrepresented in both databases and assess the relation between publication language and indexing. Additionally, we examine gender-based differences in topic coverage, language, and indexing rates. Our results suggest that indexing biases disproportionately affect researchers focusing on topics of national relevance for Brazil through articles written in Portuguese. Given women's overrepresentation in this group, our findings illustrate how indexing biases contribute to gender inequalities in science.
