The causes of the reproducibility crisis include lack of standardization and transparency in scientific reporting. Checklists such as ARRIVE and CONSORT seek to improve transparency, but they are not always followed by authors and peer review often fails to identify missing items. To address these issues, there are several automated tools that have been designed to check different rigor criteria. We have conducted a broad comparison of 11 automated tools across 9 different rigor criteria from the ScreenIT group. We found some criteria, including detecting open data, where the combination of tools showed a clear winner, a tool which performed much better than other tools. In other cases, including detection of inclusion and exclusion criteria, the combination of tools exceeded the performance of any one tool. We also identified key areas where tool developers should focus their effort to make their tool maximally useful. We conclude with a set of insights and recommendations for stakeholders in the development of rigor and transparency detection tools. The code and data for the study is available at https://github.com/PeterEckmann1/tool-comparison.

We report evidence of a new set of sneaked references discovered in the scientific literature. Sneaked references are references registered in the metadata of publications without being listed in reference section or in the full text of the actual publications where they ought to be found. We document here 80,205 references sneaked in metadata of the International Journal of Innovative Science and Research Technology (IJISRT). These sneaked references are registered with Crossref and all cite – thus benefit – this same journal. Using this dataset, we evaluate three different methods to automatically identify sneaked references. These methods compare reference lists registered with Crossref against the full text or the reference lists extracted from PDF files. In addition, we report attempts to scale the search for sneaked references to the scholarly literature.

Generative AI proposes several large language models (LLMs) to automatically generate a message in response to users' requests. Such scientific breakthroughs promote new writing assistants but with some fears. The main focus of this study is to analyze the written style of one LLM called ChatGPT by comparing its generated messages with those of the recent French presidents. To achieve this, we compare end-of-the-year addresses written by Chirac, Sarkozy, Hollande, and Macron with those automatically produced by ChatGPT. We found that ChatGPT tends to overuse nouns, possessive determiners, and numbers. On the other hand, the generated speeches employ less verbs, pronouns, and adverbs and include, in mean, too standardized sentences. Considering some words, one can observe that ChatGPT tends to overuse "to must" (devoir), "to continue" or the lemma "we" (nous). Moreover, GPT underuses the auxiliary verb "to be" (^etre), or the modal verbs "to will" (vouloir) or "to have to" (falloir). In addition, when a short text is provided as example to ChatGPT, the machine can generate a short message with a style closed to the original wording. Finally, we reveal that ChatGPT style exposes distinct features compared to real presidential speeches.

The growing number of scientific publications in digital formats makes citations important in scientific digital libraries (SDLs) as they are used to record scientific publications. However, unreliable citations, arising from unintentional errors or deliberate manipulation, such as paper mill activity, pose a growing threat to the integrity of scholarly records. These issues can distort query results, inflate or deflate scholarly metrics, and mislead decision-making systems that rely on citation data. To address this challenge, we present CiteScreener, an open-source pipeline designed to automatically assist in estimating citation relevance. Our pipeline integrates citation context extraction, cited literature retrieval, and multi-method citation evaluation (including text overlap techniques and state-of-the-art language models) to detect citations that are irrelevant to the research they reference. We develop and benchmark two datasets to evaluate these methods and select the most effective combination for integration. Finally, we validate the pipeline using real scientific papers to demonstrate its practical applicability. Our work provides SDLs, publishers, and research integrity professionals with a practical assisting tool to enhance citation verification efforts.

Modeling new applications with deep learning (DL) algorithms requires substantial knowledge. Some systems aim to simplify design choices by providing support for specific pre-defined use cases, like blurred image backgrounds or text summaries, making it easier by limiting certain options. There is a gap in addressing diverse use cases and efficiently gathering knowledge output from the deep learning community to find and reuse models and datasets from various sources if they help solve a use case. In this experience study, we are interested in how to suggest and manage DL design choices stemming from artifacts published by the DL community to help non-expert users. We detail a system for this end using a business process (BP) model, discussing the requirements for software components implementing each BP model task. We also analyzed agility in recomposing pipelines using an in-house tool against open-sourced orchestration tools, implementing deep learning model adaptation components in one highly modular BP model task.

Recent advances in Artificial Intelligence (AI) have enabled the adoption of Natural Language Processing (NLP) technologies in Digital Libraries (DL) for tasks such as citation analysis, metadata enrichment, or content classification. While the transformer-based models offer a strong performance, the concerns about their environmental impact are growing. Before deploying such services, it is essential to evaluate alternative implementation strategies and select the most efficient in terms of both performance and environmental impact. In this study, we propose a practical methodology for assessing the carbon footprint of AI-driven services, which we apply to the specific task of citation verification. We compare two NLP approaches: (1) Fine-tuning a pretrained model for binary classification, and (2) using pretrained sentence embeddings with cosine similarity. Based on real energy measurements from multiple transformerbased models, we estimate both operational energy usage (OPEX) and hardware manufacturing emissions (CAPEX) across key lifecycle stages. We also project long-term impact under realistic deployment scenarios, including shared high-memory servers and lightweight local setups. Our results indicate that although both NLP strategies lead to similar emission levels over time, infrastructure decisions such as hardware configuration and resource reuse still have a major influence. This work offers a replicable framework to support more sustainable integration of AI technologies in DL.

Here we present the training and evaluation of NanoNER, a Named Entity Recognition (NER) model for Nanobiology. NER consists in the identification of specific entities in spans of unstructured texts and is often a primary task in Natural Language Processing (NLP) and Information Extraction. The aim of our model is to recognise entities previously identified by domain experts as constituting the essential knowledge of the domain. Relying on ontologies, which provide us with a domain vocabulary and taxonomy, we implemented an iterative process enabling experts to determine the entities relevant to the domain at hand. We then delve into the potential of distant supervision learning in NER, supporting how this method can increase the quantity of annotated data with minimal additional manpower. On our full corpus of 728 full-text nanobiology articles, containing more than 120k entity occurrences, NanoNER obtained a F1-score of 0.98 on the recognition of previously known entities. Our model also demonstrated its ability to discover new entities in the text, with precision scores ranging from 0.77 to 0.81. Ablation experiments further confirmed this and allowed us to assess the dependency of our approach on the external resources. It highlighted the dependency of the approach to the resource, while also confirming its ability to rediscover up to 30 ablated terms. This paper details the methodology employed, experimental design, and key findings, providing valuable insights and directions for future related researches on NER in specialized domain. Furthermore, since our approach require minimal manpower , we believe that it can be generalized to other specialized fields.

Foundation Models shift the interest to adapting models instead of creating proprietary models from scratch. Despite this change, performing hyperparameter optimization (HPO) is still needed. Users adapting systems powered by those models on proprietary data should not considerably increase the overall resource footprint with extensive hyperparameter search. Given that this footprint is also proportional to the data used in HPO, we aim to investigate how a user can effectively reduce the amount of data used, leveraging the deep learning model’s empirical facility to output the expected correct result for an item in the dataset.In this work, we describe a methodology for accomplishing this data reduction through estimating a measure of an item’s difficulty. This method allows keeping only a portion of data that conserves the overall proportions of item difficulty throughout the dataset while helping order them meaningfully. The rationale is derived from results from curriculum learning research as we try to answer if the adapted models could help organize and select subsets of data representative of the whole. Preliminary results of evaluating the method are provided for image recognition and scientific name entity recognition (NER). We observe that the amount of data for HPO can be reduced as far as 60% and still point to the same choice of hyperparameters compared to using the whole training set.

Reproducible laboratory research relies on correctly identified reagents. We have previously described gene research papers with wrongly identified nucleotide sequence(s), including papers studying miR-145. Manually verifying reagent identities in 36 recent miR-145 papers found that 56% and 17% of papers described misidentified nucleotide sequences and cell lines, respectively. We also found 5 cell line identifiers in miR-145 papers with misidentified nucleotide sequences and cell lines, and 18 cell line identifiers published elsewhere, that did not represent indexed human cell lines. These 23 identifiers were described as non-verifiable (NV), as their identities were unclear. Studying 420 papers that mentioned 8 NV identifier(s) found 235 papers (56%) that referred to 7 identifiers (BGC-803, BSG-803, BSG-823, GSE-1, HGC-7901, HGC-803, and MGC-823) as independent cell lines. We could not find any publications describing how these cell lines were established. Six cell lines were sourced from cell line repositories with externally accessible online catalogs, but these cell lines were not indexed as claimed. Some papers also stated that short tandem repeat (STR) profiles had been generated for three cell lines, yet no STR profiles could be identified. In summary, as NV cell lines represent new challenges to research integrity and reproducibility, further investigations are required to clarify their status and identities.

This paper presents various automatic detection methods to extract so called tortured phrases from scientific papers. These tortured phrases, e.g. flag to clamor instead of signal to noise, are the results of paraphrasing tools used to escape plagiarism detection. We built a dataset and evaluated several strategies to flag previously undocumented tortured phrases. The proposed and tested methods are based on language models and either on embeddings similarities or on predictions of masked token. We found that an approach using token prediction and that propagates the scores to the chunk level gives the best results. With a recall value of .87 and a precision value of .61, it could retrieve new tortured phrases to be submitted to domain experts for validation.

We report evidence of an undocumented method to manipulate citation counts involving "sneaked" references. Sneaked references are registered as metadata for published scientific articles in which they do not appear. This manipulation exploits trusted relationships between various actors: publishers, the Crossref metadata registration agency, digital libraries, and bibliometric platforms. By collecting metadata from various sources, we show that extra undue references are actually sneaked in at Digital Object Identifier (DOI) registration time, resulting in artificially inflated citation counts. As a case study, focusing on three journals from a given publisher, we identified at least 9% sneaked references (5978/65,836) mainly benefiting two authors. Despite not being present in the published articles, these sneaked references exist in metadata registries and inappropriately propagate to bibliometric dashboards. Furthermore, we discovered "lost" references: the studied bibliometric platform failed to index at least 56% (36,939/65,836) of the references present in the HTML version of the publications. This research led to an investigation by Crossref (confirming our findings) and to subsequent corrective actions. The extent of the distortion-due to sneaked and lost references-in the global literature remains unknown and requires further investigations. Bibliometric platforms producing citation counts should identify, quantify, and correct these flaws to provide accurate data to their patrons and prevent further citation gaming.

Reproducible laboratory research relies on correctly identified reagents. We have previously described human gene research papers with wrongly identified nucleotide sequence reagent(s), including papers studying miR-145 . Manually verifying reagent identities in more recent miR-145 papers found 20/36 (56%) and 6/36 (17%) miR-145 papers with misidentified nucleotide sequence reagent(s) and human cell line(s), respectively. We also found 5 cell line identifiers in two miR-145 papers with wrongly identified nucleotide sequences and cell lines, and 18 identifiers published elsewhere that did not correspond to indexed cell lines. These cell line identifiers were described as non-verifiable, as their identities appeared uncertain. Studying 420 papers that mentioned 8 different non-verifiable cell line identifier(s) found 235 papers (56%) that appeared to refer to BGC-803, BSG-803, BSG-823, GSE-1, HGC-7901, HGC-803 and/or MGC-823 as independent cell lines. We could not find publications describing how these cell lines were established, and they were not indexed in claimed externally accessible cell line repositories. While some papers stated that STR profiles had been generated for BGC-803, GSE-1 and/or MGC-823 cells, no STR profiles were identified. In summary, non-verifiable human cell lines represent new challenges to research reproducibility and require further investigation to clarify their identities.Novelty and Impact Statement Through verifying reagent identities in research publications, our team found 23 non-verifiable human cell line identifiers, most of which could represent misspellings of contaminated cancer cell lines. Of 8 identifiers studied in detail, 7 non-verifiable identifiers were unexpectedly referred to as independent cell lines across 235 publications. We therefore describe a process “miscelling”, where published cell lines lack descriptions of how they were established, cannot be found in claimed external repositories and lack STR profiles.### Competing Interest StatementThe authors have declared no competing interest.

BACKGROUND:Research culture is strongly influenced by academic incentives and pressures such as the imperative to publish in academic journals, and can influence the nature and quality of the evidence we produce. OBJECTIVE:The purpose of this rapid scoping review is to capture the breadth of differential pressures and contributors to current research culture, drawing together content from empirical research specific to the health and biomedical sciences. STUDY DESIGN AND SETTING:PubMed and Web of Science were searched for empirical studies of influences and impacts on health and biomedical research culture, published between January 2012 and April 2024. Data charting extracted the key findings and relationships in research culture from included papers such as workforce composition; equitable access to research; academic journal trends, incentives, and reproducibility; erroneous research; questionable research practices; biases vested interests; and misconduct. A diverse author network was consulted to ensure content validity of the proposed framework of i) inclusivity, ii) transparency, iii) rigor, and iv) objectivity. RESULTS:A growing field of studies examining research culture exists ranging from the inclusivity of the scientific workforce, the transparency of the data generated, the rigor of the methods used and the objectivity of the researchers involved. Figurative diagrams are presented to storyboard the links between research culture content and findings. CONCLUSION:The wide range of research culture influences in the recent literature indicates the need for coordinated and sustained research culture conversations. Core principles in effective research environments should include inclusive collaboration and diverse research workforces, rigorous methodological approaches, transparency, data sharing, and reflection on scientific objectivity.

We report evidence of an undocumented method to manipulate citation counts involving 'sneaked' references. Sneaked references are registered as metadata for scientific articles in which they do not appear. This manipulation exploits trusted relationships between various actors: publishers, the Crossref metadata registration agency, digital libraries, and bibliometric platforms. By collecting metadata from various sources, we show that extra undue references are actually sneaked in at Digital Object Identifier (DOI) registration time, resulting in artificially inflated citation counts. As a case study, focusing on three journals from a given publisher, we identified at least 9% sneaked references (5,978/65,836) mainly benefiting two authors. Despite not existing in the articles, these sneaked references exist in metadata registries and inappropriately propagate to bibliometric dashboards. Furthermore, we discovered 'lost' references: the studied bibliometric platform failed to index at least 56% (36,939/65,836) of the references listed in the HTML version of the publications. The extent of the sneaked and lost references in the global literature remains unknown and requires further investigations. Bibliometric platforms producing citation counts should identify, quantify, and correct these flaws to provide accurate data to their patrons and prevent further citation gaming.

With the help of online tools, unscrupulous authors can today generate a pseudo-scientific article and attempt to publish it. Some of these tools work by replacing or paraphrasing existing texts to produce new content, but they have a tendency to generate nonsensical expressions. A recent study introduced the concept of “tortured phrase”, an unexpected odd phrase that appears instead of the fixed expression. E.g. counterfeit consciousness instead of artificial intelligence. The present study aims at investigating how tortured phrases, that are not yet listed, can be detected automatically. We conducted several experiments, including non-neural binary classification, neural binary classification and cosine similarity comparison of the phrase tokens, yielding noticeable results.

Nucleotide sequence reagents underpin molecular techniques that have been applied across hundreds of thousands of publications. We have previously reported wrongly identified nucleotide sequence reagents in human research publications and described a semi-automated screening tool Seek & Blastn to fact-check their claimed status. We applied Seek & Blastn to screen >11,700 publications across five literature corpora, including all original publications in Gene from 2007 to 2018 and all original open-access publications in Oncology Reports from 2014 to 2018. After manually checking Seek & Blastn outputs for >3,400 human research articles, we identified 712 articles across 78 journals that described at least one wrongly identified nucleotide sequence. Verifying the claimed identities of >13,700 sequences highlighted 1,535 wrongly identified sequences, most of which were claimed targeting reagents for the analysis of 365 human protein-coding genes and 120 non-coding RNAs. The 712 problematic articles have received >17,000 citations, including citations by human clinical trials. Given our estimate that approximately one-quarter of problematic articles may misinform the future development of human therapies, urgent measures are required to address unreliable gene research articles.