Large language models (LLMs) are increasingly employed in scientometric studies for scientific text analysis and knowledge discovery. A key factor influencing their in-context learning (ICL) performance is the organization of demonstrations, which guides how LLMs interpret complex scientific texts. However, existing organization strategies often involve high computational costs and lack interpretability, limiting their effectiveness and transferability across models. We propose In-Context Curriculum Learning (ICCL), a lightweight and explainable framework that organizes demonstrations in ascending order of complexity to enhance LLMs’ deep semantic understanding of scientific text. ICCL comprises three stages: demonstration retrieval, label recall, and curriculum-guided ordering based on a novel Instruction Alignment Score (IAS) to quantify contextual difficulty. We evaluate ICCL on three representative scientific text mining benchmarks, SciCite, SciNLI, and SciERC, covering citation intent classification, scientific language inference, and entity extraction tasks. Relative to few-shot prompting, ICCL yields consistent average F1-score gains of 4.65 https://github.com/61peng/sci-iccl .

The expeditious identification of papers with potentially disruptive and developmental contributions is a meaningful issue in science of science research. To achieve this aim, we proposed an effective method to automatically build an abstract format-balanced dataset based on the basic idea of disruption index. Subsequently, we respectively utilized a range of machine learning models (MLs), deep learning models (DLs), and large language models (LLMs) to build a content-based identification model based on the dataset. The optimal model, Mistral-7B fine-tuned via QLoRA, significantly outperforms MLs and DLs, and achieved an F1 score of 0.7735 on this extremely challenging task. Hence, the model can promptly and effectively distinguish between disruptive, developmental, and general research based purely on research content. Further, we developed three evaluation datasets to scrutinize the effectiveness of the model, demonstrating its remarkable ability to distinguish Nobel Prize papers from randomly paired counterparts, as well as to differentiate between randomly sampled papers in journals of varying impact factors (IF). Additionally, the papers’ scores derived from our model’s prediction exhibit a strong positive correlation with their future scientific impact. We also reveal that evaluating papers based solely on IF is insufficient, and papers boasting high view counts are more likely to represent potentially disruptive or developmental research. Thus, our work not only provides an effective method for timely identifying high-quality research, but also provides guidance for promoting fair scientific evaluation.

Understanding scientific figures is challenging due to their complexity and the need for domain expertise. Existing studies, limited by data and model capabilities, primarily focus on captioning and classifying simpler visualizations, such as bar and pie charts, while neglecting the fine-grained interpretation of knowledge-rich diagrams like frameworks and flowcharts. To address this gap, we introduce a novel task, Figure Integrity Verification, to assess models’ ability to align textual knowledge with detailed visual elements and identify visual elements that lack textual explanations. To support this task, we construct a large-scale text-figure alignment dataset, Figure-seg, containing 15,761 instances. Building upon this dataset, we propose an innovative framework, Every Part Matters (EPM), which employs Multimodal Large Language Models (MLLMs) and analogical reasoning to verify and enhance figure integrity. Experimental results demonstrate that the proposed framework improves the IoU metric for text-figure alignment by 33.83 https://github.com/shixiang1a/figure_understanding ).

Understanding how processing, structure, properties, and performance interact is essential for guiding materials design and discovery. Yet, causal mechanisms linking these elements are typically scattered across text, figures, and references in the literature, and efforts to systematically mine and organize such knowledge remain limited. In this work, we leverage an LLM-based mechanism deduction framework to construct a dataset of 207,200 fine-grained mechanisms with 1,113,940 multimodal evidences from 61,766 materials science research articles. Each mechanism is linked to a specific causal relation among the tetrahedral elements and is supported by evidence from experiment information, characterization results, and external knowledge, with its accuracy verified by materials science researchers. This dataset provides a large-scale, cross-validated collection of multimodal mechanism knowledge in materials science, serving as a resource for data-driven research and intelligent analysis.

As Large Language Models (LLMs) like ChatGPT and GPT-4 continue to evolve, their application in facilitating the academic writing process is increasing. Writing the "Related Work" section in a scientific manuscript requires scholars to integrate knowledge from multiple sources, making it labor-intensive and time-consuming. Current predominant methods typically format the "Related Work" section Generation (RWG) in isolated environments with limited knowledge, which is quite different from how humans write. Therefore, our research pivots RWG toward a more author-centered approach that simulates the human academic writing procedure. We introduce a novel methodology that synergizes the strengths of both small language models (SLMs) and large language models. Our method involves generating author-oriented cues via SLMs, drafting initial versions with LLMs, and designing a delta feedback mechanism for cue refinement, which reveals the information gap between drafts. In the generation process, our approach assembles multiple sources of academic knowledge. This includes the LLMs' tacit knowledge, the explicit knowledge of domain-specific academic publications, and the context derived from scholarly graphs. Comparative evaluations, both LLM-based and human-based, demonstrate that our method significantly surpasses existing baseline models like original GPT-4 and CoT-based GPT-4. Specifically, the LLM-based evaluation indicates that, when guided by enhanced cues, our model can produce high-quality content with an average of only 3.46 revision edits, compared to approximately 4 revision edits required by the baseline model. Additionally, the human evaluation demonstrates that our model obtained a higher score on the content, structure, and argumentation dimensions. In conclusion, our model has enhanced capability in synthesizing input papers comprehensively and aligning more closely with the authors' focal points.

The advent of Large Language Models (LLMs) has shown the potential to improve relevance and provide direct answers in web searches. However, challenges arise in validating the reliability of generated results and the credibility of contributing sources, due to the limitations of traditional information retrieval algorithms and the LLM hallucination problem. Aiming to create a "PageRank" for the LLM era, we strive to transform LLM into a relevant, responsible, and trustworthy searcher. We propose a novel generative retrieval framework leveraging the knowledge of LLMs to foster a direct link between queries and online sources. This framework consists of three core modules: Generator, Validator, and Optimizer, each focusing on generating trustworthy online sources, verifying source reliability, and refining unreliable sources, respectively. Extensive experiments and evaluations highlight our method's superior relevance, responsibility, and trustfulness against various SOTA methods.

Topic analysis aims to study topic evolution and trends in order to help researchers understand the process of knowledge evolution and creation. This paper develops a novel topic evolution analysis framework, which we use to demonstrate, forecast, and explain topic evolution from the perspective of the geometrical motion of topic embeddings generated by pretrained language models. Our data set comprises approximately 15 million papers in the computer science field, with 7,000 "fields of study" to represent the topics. First, we demonstrate that over 80% of topics have undergone obvious motion in the semantic vector space, based on the hyperplane and its normal vector generated by a support vector machine. Subsequently, we verified the predictability of the motion based on three vector regression models by predicting topic embeddings. Finally, we employed a decoder to explain the predicted motion, whose forecast embeddings can capture about 50% of unseen topics. Our research framework shows that topic evolution can be analyzed via the geometrical motion of topic embeddings, and the semantic motion of old topics nurtures new topics. The current study opens new research pathways in topic analysis and sheds light on the topic evolution mechanism from a novel geometric perspective.

Timely identification of potentially disruptive research is a significant research issue, since disruptive innovation in science transforms the existing paradigm and/or opens a new paradigm. This study proposes a comparative power-based large model that can promptly and accurately identify potentially disruptive research via comparative analysis of semantically-related papers. To this end, a self-constructed dataset was built by treating accumulated disruptive and consolidating citations as crowdsourced annotation data. We employed a range of machine learning models (MLs), deep learning models (DLs), and large language models (LLMs) to build classifiers. Our optimal model, Mistral-7B+*, attains an impressive F1 score of 0.8210 and outperforms the best-performing ML and DL models by approximately 27.05 % and 14.03 %, respectively. Testing on 275 recently published biomedical papers further verifies its effectiveness. Additionally, we conduct comprehensive experiments to scrutinize the comparative power of the large model as well as the impact of the number and quality of comparative papers and distinct functional paragraphs within abstracts on identification performance. Our findings show that an appropriate number and quality of comparative papers can promote identification performance. Moreover, result-based paragraphs are the most important for identifying disruptive research, while method-based paragraphs are least important.

Retrieval-Augmented Generation (RAG) is applied to solve hallucination problems and real-time constraints of large language models, but it also induces vulnerabilities against retrieval corruption attacks. Existing research mainly explores the unreliability of RAG in white-box and closed-domain QA tasks. In this paper, we aim to reveal the vulnerabilities of Retrieval-Enhanced Generative (RAG) models when faced with black-box attacks for opinion manipulation. We explore the impact of such attacks on user cognition and decision-making, providing new insight to enhance the reliability and security of RAG models. We manipulate the ranking results of the retrieval model in RAG with instruction and use these results as data to train a surrogate model. By employing adversarial retrieval attack methods to the surrogate model, black-box transfer attacks on RAG are further realized. Experiments conducted on opinion datasets across multiple topics show that the proposed attack strategy can significantly alter the opinion polarity of the content generated by RAG. This demonstrates the model's vulnerability and, more importantly, reveals the potential negative impact on user cognition and decision-making, making it easier to mislead users into accepting incorrect or biased information.

This study quantifies and analyzes the individual-level abilities of scientists utilizing either an exploration or an exploitation strategy. Specifically, we present a Research Strategy Q model, which untangles the coupling effect of scientists' research ability (Q(alpha)) and research strategy ability (E alpha pi) on research performance. Q(alpha) indicates scientists' fundamental ability to publish high-quality papers, while E alpha pi indicates scientists' proficiency in terms of exploration and exploitation strategies. Five research strategies proposed by our previous study are employed. We generate synthetic data and collect empirical data as our experimental data set. We show that these research strategies present different benefit and risk characteristics. Adopting some exploitation strategies tends to stifle research performance, while exploration strategies are high risk and high yield. Q(alpha) and E(alpha)(pi )have predictive power for research performance. Moreover, we find that, first, scholars who prefer to execute a research strategy, pi, may not necessarily be better at executing pi. Second, some scholars have differences in their abilities towards different strategies, while other scholars have differences in their abilities towards the same strategy. Third, exploration and exploitation are not contradictory but complementary from the perspective of proficiency, while they are mutually exclusive from the perspective of selection preference.

Evaluating large language models (LLMs) is fundamental, particularly in the context of practical applications. Conventional evaluation methods, typically designed primarily for LLM development, yield numerical scores that ignore the user experience. Therefore, our study shifts the focus from model-centered to human-centered evaluation in the context of AI-powered writing assistance applications. Our proposed metric, termed “Revision Distance,” utilizes LLMs to suggest revision edits that mimic the human writing process. It is determined by counting the revision edits generated by LLMs. Benefiting from the generated revision edit details, our metric can provide a self-explained text evaluation result in a human-understandable manner beyond the context-independent score. Our results show that for the easy-writing task, “Revision Distance” is consistent with established metrics (ROUGE, Bert-score, and GPT-score), but offers more insightful, detailed feedback and better distinguishes between texts. Moreover, in the context of challenging academic writing tasks, our metric still delivers reliable evaluations where other metrics tend to struggle. Furthermore, our metric also holds significant potential for scenarios lacking reference texts.

Scientific papers, as pivotal tools for academic communication, should be articulated with clarity and precision to ensure the effective conveyance of scholarly ideas and to prevent reader confusion. Yet, many such papers conspicuously lack in-depth research, and their core content is often ambiguously presented. This pattern poses a significant impediment to the progressive evolution of science and technology. While numerous researchers have recognized this widespread challenge, a holistic theoretical or methodological solution remains elusive in the academic realm. To bridge this gap, we introduce the INTEGrity vERification (INTEGER) task. This task aids researchers in assessing the integrity of their papers by verifying the clarity of each knowledge unit. To implement this task on text, we propose a multi-task learning model that utilizes the Tucker decomposition and span-level attention mechanism to identify terms and their integrity precisely. More specifically, to provide insights into the INTEGER task and validate the effectiveness of the proposed model, we collect 8076 sentences and construct three new datasets containing various types of terms and descriptions in different domains. Extensive experimental results show that our proposed model has an average performance improvement of 1.1% F1 over the three datasets compared to a series of state-of-the-art baseline methods.

Demonstration ordering, which is an important strategy for in-context learning (ICL), can significantly affects the performance of large language models (LLMs). However, most of the current approaches of ordering require high computational costs to introduce the priori knowledge. In this paper, inspired by the human learning process, we propose a simple but effective demonstration ordering method for ICL, named the few-shot In-Context Curriculum Learning (ICCL). The ICCL implies gradually increasing the complexity of prompt demonstrations during the inference process. The difficulty can be assessed by human experts or LLMs-driven metrics, such as perplexity. Then we design extensive experiments to discuss the effectiveness of the ICCL at both corpus-level and instance-level. Moreover, we also investigate the formation mechanism of LLM's ICCL capability. Experimental results demonstrate that ICCL, developed during the instruction-tuning stage, is effective for representative open-source LLMs. To facilitate further research and applications by other scholars, we make the code publicly available.

Topic evolution has been studied extensively in the field of the science of science. This study first analyzes topic evolution pattern from topics’ semantic consistency in the semantic vector space, and explore its possible causes. Specifically, we extract papers in the computer science field from Microsoft Academic Graph as our dataset. We propose a novel method for encoding a topic with numerous Contextualized Word Embeddings (CWE), in which the title and abstract fields of papers studying the topic is taken as its context. Subsequently, we employ three geometric metrics to analyze topics’ semantic consistency over time, from which the influence of the anisotropy of CWE is excluded. The K-Means clustering algorithm is employed to identify four general evolution patterns of semantic consistency, that is, semantic consistency increases (IM), decreases (DM), increases first and then decreases (Inverted U-shape), and decreases first and then increases (U-shape). We also find that research methods tend to show DM and U-shape, but research questions tend to be IM and Inverted U-shape. Finally, we further utilize the regression analysis to explore whether and, if so, how a series of key features of a topic affect its semantic consistency. Importantly, semantic consistency of a topic varies inversely with the semantic similarity between the topic and other topics. Overall, this study sheds light on the evolution law of topics, and helps researchers to understand these patterns from a geometric perspective.

PurposeFine-tuning pre-trained language models (PLMs), e.g. SciBERT, generally require large numbers of annotated data to achieve state-of-the-art performance on a range of NLP tasks in the scientific domain. However, obtaining fine-tuning data for scientific NLP tasks is still challenging and expensive. In this paper, the authors propose the mix prompt tuning (MPT), which is a semi-supervised method aiming to alleviate the dependence on annotated data and improve the performance of multi-granularity academic function recognition tasks.Design/methodology/approachSpecifically, the proposed method provides multi-perspective representations by combining manually designed prompt templates with automatically learned continuous prompt templates to help the given academic function recognition task take full advantage of knowledge in PLMs. Based on these prompt templates and the fine-tuned PLM, a large number of pseudo labels are assigned to the unlabelled examples. Finally, the authors further fine-tune the PLM using the pseudo training set. The authors evaluate the method on three academic function recognition tasks of different granularity including the citation function, the abstract sentence function and the keyword function, with data sets from the computer science domain and the biomedical domain.FindingsExtensive experiments demonstrate the effectiveness of the method and statistically significant improvements against strong baselines. In particular, it achieves an average increase of 5% in Macro-F1 score compared with fine-tuning, and 6% in Macro-F1 score compared with other semi-supervised methods under low-resource settings.Originality/valueIn addition, MPT is a general method that can be easily applied to other low-resource scientific classification tasks.

This paper tackles a key issue in the interpretation of scientific figures: the fine-grained alignment of text and figures. It advances beyond prior research that primarily dealt with straightforward, data-driven visualizations such as bar and pie charts and only offered a basic understanding of diagrams through captioning and classification. We introduce a novel task, Figure Integrity Verification, designed to evaluate the precision of technologies in aligning textual knowledge with visual elements in scientific figures. To support this, we develop a semi-automated method for constructing a large-scale dataset, Figure-seg, specifically designed for this task. Additionally, we propose an innovative framework, Every Part Matters (EPM), which leverages Multimodal Large Language Models (MLLMs) to not only incrementally improve the alignment and verification of text-figure integrity but also enhance integrity through analogical reasoning. Our comprehensive experiments show that these innovations substantially improve upon existing methods, allowing for more precise and thorough analysis of complex scientific figures. This progress not only enhances our understanding of multimodal technologies but also stimulates further research and practical applications across fields requiring the accurate interpretation of complex visual data.

This brief communication finds a clear and universal inequality of authors’ reference reuse behaviour. We observe that a few references are reused many times in an author’s oeuvre while most of his or her references only occur in the reference list for quite a limited number of times. A power law distribution depicts such an inequality. We particularly utilise the power value, [Formula: see text], to characterise the nuanced difference of such inequalities. A pilot study based upon Microsoft Academic Graph (MAG) shows that the [Formula: see text] tends to be normally distributed, regardless of whether it is from a citing or a cited perspective. Our empirical study also reveals that the [Formula: see text] of highly cited publications tends to be greater than that of lowly cited ones, yet we also observe a saturation when the number of citations increases.

OpenAI has released a new generation of conversational language model: ChatGPT, showing natural, fluid conversation capabilities, reviving the promise of artificial general intelligence that was previously thought impossible. Large language models represented by ChatGPT are typical technological and application innovations in the era of digital intelligence. ChatGPT’s powerful ability of information processing, collecting, integrating and generating brings challenges and opportunities to information resource management discipline.It has had a profound impact on the six perspectives of information resources management including supporting algorithms and technologies, information resources construction, information organization and information retrieval, information governance, content security and evaluation, and human-computer intelligent interaction and collaboration. In the era of digital intelligence, with the rapid development of large-scale AI models, it is necessary for us to keep close attention to this and promote the corresponding transformation of subject technology application paradigm, innovation of theories and methods, and governance reform, so as to further consolidate the foundation of information resources to support the "Four Orientations".

The unprecedented COVID-19 outbreak at the end of 2019 has produced a worldwide health crisis. Scientific research, especially international research collaboration, is crucial to deal successfully with the epidemic. This article aims to review the response modes, and especially the international collaboration characteristic, of the academic community to similar public health events in the past. Based on relevant studies of four major public health emergencies in the past, the major public health emergencies were regarded as ‘new knowledge’ in the academic field. By using knowledge diffusion indicators, such as the breadth and speed of diffusion, and combined with the development characteristics of the event, this article explores the diffusion characteristics of the four major public health emergencies in the academic exchange system and then identifies the academic community’s response mode to the outbreaks. In addition, the characteristics of international collaboration in response to the public health events and the impact of international collaboration on the academic community’s response are analysed. Through the analysis of the international collaboration network, the cooperative groups and core countries in the research collaboration network related to the major public health emergencies are obtained. In terms of COVID-19, it is found that the response speed and intensity of scientists have been significantly improved, but more focus should be given to international collaboration. Our findings could be beneficial to both decision-makers and researchers in policy formulation and conducting research, respectively, to optimally deal with COVID-19 and possible outbreaks in the future.