Summarizing long, domain-specific documents with large language models (LLMs) remains challenging due to context limitations, information loss, and hallucinations, particularly in clinical and legal settings. We propose a Discrete Wavelet Transform (DWT)-based multi-resolution framework that treats text as a semantic signal and decomposes it into global (approximation) and local (detail) components. Applied to sentence- or word-level embeddings, DWT yields compact representations that preserve overall structure and critical domain-specific details, which are used directly as summaries or to guide LLM generation. Experiments on clinical and legal benchmarks demonstrate comparable ROUGE-L scores. Compared to a GPT-4o baseline, the DWT based summarization consistently improve semantic similarity and grounding, achieving gains of over 2

Evaluating the similarity of generated text and reference text in natural language generation (NLG) remains a challenge. Current automatic evaluation methods have limitations and fall short of expectations. Therefore, we propose an iterative multi-agent analysis system to enhance the evaluation of similarity between ground-truth and generated answers. The system has a Task Performer (Stage 1) for initial evaluation, Multi-Agent Analysis (Stage 2), where specialized analysis agents iteratively re-evaluate outputs, and a Convergence Checker (Stage 3) that determines consensus, with an Arbiter invoked when convergence is not reached. We implement our method in three architectures: (1) Architecture One employs all three stages with an Arbiter; (2) Architecture Two is limited to stages 2 and 3; and (3) Architecture Three is like Architecture One but without the Arbiter – if no convergence, the Task Performer’s final answer is the final output. We evaluated our system on four datasets (Open-QA, SummEval, MedQA, and ChatLaw) under three different LLM implementation scenarios (uniform, closed-source, and small open-source). The results show that our system performs better than a standalone LLM baseline and is significantly closer to human evaluation levels.

Wavelets have emerged as a cutting edge technology in a number of fields. Concrete results of their application in Image and Signal processing suggest that wavelets can be effectively applied to Natural Language Processing (NLP) tasks that capture a variety of linguistic properties. In this paper, we leverage the power of applying Discrete Wavelet Transforms (DWT) to word and sentence embeddings. We first evaluate, intrinsically and extrinsically, how wavelets can effectively be used to consolidate important information in a word vector while reducing its dimensionality. We further combine DWT with Discrete Cosine Transform (DCT) to propose a non-parameterized model that compresses a sentence with a dense amount of information in a fixed size vector based on locally varying word features. We show the efficacy of the proposed paradigm on downstream applications models yielding comparable and even superior (in some tasks) results to original embeddings.

Large Language Models (LLMs) have shown impressive capabilities in solving complex mathematical problems, making them valuable tools for education, research, and automated tutoring. However, top-performing models on benchmarks like MATH500, such as GPT-4 and DeepSeek-R1, are often large, proprietary, and costly to use, limiting their accessibility. In contrast, smaller open-source models are more affordable and easier to deploy locally but typically underperform in mathematical reasoning tasks. In this work, we explore the math-problem-solving potential of six small-scale, open-source LLMs (all under 10 billion parameters): Arithmo-Mistral-7B, MAmmoTH-7B, MAmmoTH-8B, MetaMath-7B, MetaMath-Llemma-7B, and MetaMath-Mistral-7B, on the MATH500 benchmark. To enhance their accuracy, we apply two “test-time” ensemble strategies: (1) Intra-model ensemble, where each model generates five independent outputs and the most frequent prediction is selected; and (2) Inter-model ensemble, where 2-level majority voting is performed: first at the intra-level, then across all the models in the ensemble. Our results show that the Intra-model ensemble consistently improves performance over individual runs, and combining outputs across models yields further gains. An ensemble of all six models achieves 38

This paper explores the potential of leveraging Large Language Models (LLMs) for the tasks of automated annotation and Part-of-Math (POM) tagging of equations. Traditional methods for math term annotation and POM tagging rely heavily on manually crafted rules and limited datasets, which often result in scalability issues and insufficient adaptability to new domains. In contrast, LLMs, with their vast knowledge and advanced natural language understanding capabilities, present a promising alternative. Our methodology involves crafting prompts for LLMs to elicit answers that can be read as key-value pairs where the keys are math terms and the values are the corresponding annotations. We also investigate the effect on the performance of LLMs when we provide in the prompt different levels of context, such as the sentence or paragraph containing the input equation. The performance is evaluated by consistency between the ground truth and the output of LLMs. Consistency is assessed by a separate LLM session and with a different prompt. Our results show that when different levels of context are involved, the consistency rate of binary classification increased from 14.8

The evaluation process of detecting the similarity between reference text and Large Language Model (LLM)-generated text is a challenging problem. It is difficult to measure it automatically. Traditional metrics, such as ROUGE and BERTScore, have been shown to have some limitations. They showed a relatively low correlation with humans. Also, they penalize LLM-generated text, and have difficulty recognizing noise in text and qualitatively evaluate similarity in texts.In this paper, we introduce an approach to studying the use of LLM to evaluate the similarity between LLM-generated answers and reference answers. The approach framework includes classes, prompt, and classification schemes. The framework defines classes to measure similarity. The prompt will include the classes and their definitions. We use two types of prompts: instructed and uninstructed. The instructed prompt contains specific evaluation rules for conducting the LLM to evaluate the similarity between LLM-generated answers and reference answers. The uninstructed prompt, though it will have an evaluation request, will not contain evaluation rules. The classification schemes range from binary classification to multi-class classification; the latter is finer-grained and more informative. Also, the classification schemes group into two categories, where in one category the classes are viewed as mutually exclusive and the classification is unguided, while in the other category the classes are not all mutually exclusive but can be viewed as partly hierarchical, and the classification is guided, i.e., totally ordered or partially ordered (a hybrid). The approach will use a classification scheme to allow LLMs to invoke prompts and classes to perform the evaluation. We experimented with the similarity evaluation with GPT-4 and Gemini.Our results indicate that the binary classification scheme shows significant accuracy results. In the multi-class classification schemes, the unguided classification showed very poor evaluation performance, while the guided classification showed excellent performance in the 90-percentiles of accuracy. Furthermore, both Gemini and GPT-4 perform very well as evaluators, especially under totally ordered classification, while in the case of partially ordered classification, the winning combination turned out to be GPT-4 as evaluator and Gemini as answerer.

Wavelet transforms, a powerful mathematical tool, have been widely used in different domains, including Signal and Image processing, to unravel intricate patterns, enhance data representation, and extract meaningful features from data. Tangible results from their application suggest that Wavelet transforms can be applied to NLP capturing a variety of linguistic and semantic properties. In this paper, we empirically leverage the application of Discrete Wavelet Transforms (DWT) to word and sentence embeddings. We aim to showcase the capabilities of DWT in analyzing embedding representations at different levels of resolution and compressing them while maintaining their overall quality. We assess the effectiveness of DWT embeddings on semantic similarity tasks to show how DWT can be used to consolidate important semantic information in an embedding vector. We show the efficacy of the proposed paradigm using different embedding models, including large language models, on downstream tasks. Our results show that DWT can reduce the dimensionality of embeddings by 50-93% with almost no change in performance for semantic similarity tasks, while achieving superior accuracy in most downstream tasks. Our findings pave the way for applying DWT to improve NLP applications.

Summarizing medical conversations is one of the tasks proposed by MEDIQA-Chat to promote research on automatic clinical note generation from doctor-patient conversations. In this paper, we present our submission to this task using fine-tuned language models, including T5, BART and BioGPT models. The fine-tuned models are evaluated using ensemble metrics including ROUGE, BERTScore andBLEURT. Among the fine-tuned models, Flan-T5 achieved the highest aggregated score for dialogue summarization.

A Mathematical Function Recognition (MFR) is an important research direction for efficient downstream math tasks such as information retrieval, knowledge extraction, and question answering. The aim of this task is to identify and classify mathematical function into a predefined set of function. However, the lack of annotated data is the bottleneck in the development of an MFR automated model. We begin this paper by describing our approach to creating a labelled dataset for MFR. Then, to identify five categories of mathematical functions, we fine-tuned a set of common pre-trained models: BERT base-cased, BERT baseuncased, DistilBERT-cased, and DistilBERT-uncased. As a result, our contributions in this paper include: (1) an annotated MFR dataset that future researchers can use; and (2) SOTA results obtained by finetuning pre-trained models for the MFR task. Our experiments demonstrate that the proposed approach achieved a high-quality recognition, with an F1 score of 96.80% on a held-out test set provided by DistilBERT-cased model.

In machine learning applications, mapping math knowledge from the series of tokens in a formula or expression to their linguistic semantic meaning remains an open area of research. One fundamental task towards that end is the chunking of a math equation/expression into meaningful math entities. It is the equivalent of sentence segmentation or chunking in natural language processing. Math chunking is quite broad and in a nascent stage in math linguistics. In this paper, we begin an exploration into this task using deep learning on a focused part of chunking, namely, recognition of functions (along with their arguments and parameters), in input equations. Specifically, we propose math-chunking models to identify a list of standard functions. We further develop an annotated dataset to train and evaluate our models. Our experimental results show that one of our proposed deep learning models, namely BiLSTM-CRF, can achieve rather high state-of-the-art performance on the mathematical formula chunking task.

Digital mathematical libraries assemble the knowledge of years of mathematical research. Numerous disciplines (e.g., physics, engineering, pure and applied mathematics) rely heavily on compendia gathered findings. Likewise, modern research applications rely more and more on computational solutions, which are often calculated and verified by computer algebra systems. Hence, the correctness, accuracy, and reliability of both digital mathematical libraries and computer algebra systems is a crucial attribute for modern research. In this paper, we present a novel approach to verify a digital mathematical library and two computer algebra systems with one another by converting mathematical expressions from one system to the other. We use our previously eveloped conversion tool (referred to as LaCASt) to translate formulae from the NIST Digital Library of Mathematical Functions to the computer algebra systems Maple and Mathematica. The contributions of our presented work are as follows: (1) we present the most comprehensive verification of computer algebra systems and digital mathematical libraries with one another; (2) we significantly enhance the performance of the underlying translator in terms of coverage and accuracy; and (3) we provide open access to translations for Maple and Mathematica of the formulae in the NIST Digital Library of Mathematical Functions.

The need for explainable AI (XAI) is well established but relatively little has been published outside of the supervised learning paradigm. This paper focuses on a principled approach to applying explainability and interpretability to the task of unsupervised anomaly detection. We argue that explainability is principally an algorithmic task and interpretability is principally a cognitive task, and draw on insights from the cognitive sciences to propose a general-purpose method for practical diagnosis using explained anomalies. We define Attribution Error, and demonstrate, using real-world labeled datasets, that our method based on Integrated Gradients (IG) yields significantly lower attribution errors than alternative methods.

Word disambiguation has been an important task in natural language processing. However, the problem of disambiguation is still less explored in mathematical text. Similar to natural languages, some math terms are not assigned a unique interpretation. As math text is an important part of the scientific literature, an accurate and efficient way of performing disambiguation of math terms will be a significant contribution. In this paper, we present some investigations on math-term disambiguation using machine learning. All experimental data are selected from the DLMF dataset. Our experiments consist of 3 steps: (1) create a labeled dataset of math equations (from the DLMF) where the instances are (math token, token meaning) pairs, grouped by equation; (2) build machine learning models and train them using our labeled dataset, and (3) evaluate and compare the performance of our models using different evaluation metrics. Our results show that machine learning is an effective approach to math-term disambiguation. The accuracy of our models ranges from 70% to 85%. There is potential for considerable improvements once we have much larger labeled datasets with more balanced classes.

Clear and efficient communication requires more than grammatical correctness to ensure fluency and semantic correctness, especially for non-native speakers. Thus, we propose a new task – Sentence Denoising, to go beyond Grammatical Error Correction (GEC). We define a rich and linguistics-inspired noise taxonomy consisting of 13 types of noise, and categorize them into vagueness, redundancy, and incoherence. We then generate and study 4 types of noise out of the 13 because they serve as building blocks. Methods are proposed to inject targeted noise into sentences for building datasets. We publish them and give benchmarks for denoising both individual noise and compound noise. Finally, an efficient training approach is designed for denoising combinations of noise.

We propose an approach to automatically test for originality in generation tasks where no standard automatic measures exist. Our proposal addresses original uses of language, not necessarily original ideas. We provide an algorithm for our approach and a run-time analysis. The algorithm, which finds all of the original fragments in a ground-truth corpus and can reveal whether a generated fragment copies an original without attribution, has a run-time complexity of theta(n log n) where n is the number of sentences in the ground truth.

Misunderstanding happens all the time, especially when a non-native speaker is involved. To help recover the original meaning, we define categories of noise in an English sentence and differentiate our problem from Grammatical Error Correction (GEC). Methods are proposed to inject targeted noise into sentences for building training sets. Finally, a system comprised of two parts is designed for sentence correction (aka denoising) - One being fine-tuned BERT models for noise classification, and the other being Transformer models for "translating" noisy sentences into correct ones.

Document classification is a fundamental task for many applications, including document annotation, document understanding, and knowledge discovery. This is especially true in STEM fields where the growth rate of scientific publications is exponential, and where the need for document processing and understanding is essential to technological advancement. Classifying a new publication into a specific domain based on the content of the document is an expensive process in terms of cost and time. Therefore, there is a high demand for a reliable document classification system. In this paper, we focus on classification of mathematics documents, which consist of English text and mathematics formulas and symbols. The paper addresses two key questions. The first question is whether math-document classification performance is impacted by math expressions and symbols, either alone or in conjunction with the text contents of documents. Our investigations show that Text-Only embedding produces better classification results. The second question we address is the optimization of a deep learning (DL) model, the LSTM combined with one dimension CNN, for math document classification. We examine the model with several input representations, key design parameters and decision choices, and choices of the best input representation for math documents classification.

Machine Learning (ML) and Natural Language Processing (NLP) have started to be applied to math language processing and math knowledge discovery. To fully utilize ML in those areas, there is a pressing need for Math labeled datasets. This paper presents a new dataset that we have derived from the widely used Digital Library of Mathematical Functions (DLMF) of NIST. The dataset is structured and labeled in a specific way. For each math equation and expression in the DLMF, there is a record that provides annotational and contextual elements. An accompanying dataset is also generated from the DLMF. It consists of “Simple XML” files, each organized as marked-up sentences within a marked-up hierarchy of paragraphs/subsections/sections. The math in each sentence is marked up in a way that enables users to extract the actual context of math elements, at various levels of granularity, for contextualized processing. This context-rich, sentence-oriented, equation/expression-centered, symbol-labeled dataset is motivated by the fact that much of ML-based NLP algorithms are sentence oriented.

In this paper we present our results from the Second Shared Task on Metaphor Detection, hosted by the Second Workshop on Figurative Language Processing. We use an ensemble of RNN models with bidirectional LSTMs and bidirectional attention mechanisms. Some of the models were trained on all parts of speech. Each of the other models was trained on one of four categories for parts of speech: "nouns", "verbs", "adverbs/adjectives", or "other". The models were combined into voting pools and the voting pools were combined using the logical "OR" operator.

In this paper, we show how selecting and combining encodings of natural and mathematical language affect classification and clustering of documents with mathematical content. We demonstrate this by using sets of documents, sections, and abstracts from the arXiv preprint server that are labeled by their subject class (mathematics, computer science, physics, etc.) to compare different encodings of text and formulae and evaluate the performance and runtimes of selected classification and clustering algorithms. Our encodings achieve classification accuracies up to 82.8% and cluster purities up to 69.4% (number of clusters equals number of classes), and 99.9% (unspecified number of clusters) respectively. We observe a relatively low correlation between text and math similarity, which indicates the independence of text and formulae and motivates treating them as separate features of a document. The classification and clustering can be employed, e.g., for document search and recommendation. Furthermore, we show that the computer outperforms a human expert when classifying documents. Finally, we evaluate and discuss multi-label classification and formula semantification.