Document understanding systems are increasingly deployed in settings where model predictions alone are insufficient for effective human use. Applications such as enterprise communication triage, decision support, and document review require not only accurate predictions, but also faithful explanations that indicate why a document was flagged. Extractive rationales provide a practical form of explainability by highlighting decision-relevant text spans, yet learning rationales that are both faithful to the model’s reasoning and aligned with human annotations remains challenging. Existing approaches typically adopt either fully coupled end-to-end training, which risks shortcut-driven explanations, or fully decoupled pipelines, which weaken the connection between explanations and predictions. In this work, we identify gradient interference between task optimization and explanation supervision as a key factor underlying this trade-off. We propose GRACE, a gradient-regulated framework for extractive rationale learning that explicitly controls gradient interaction between rationale supervision and task prediction. GRACE restricts task gradients from updating the rationale selector, while ensuring that predictions are computed exclusively from rationale-weighted representations. This design preserves the structural involvement of explanations in the decision process while reducing distortion from predictive shortcuts. We evaluate GRACE on public rationale benchmarks (e-SNLI and CoS-E) and large-scale enterprise email datasets, including an out-of-distribution evaluation on the Enron corpus. Across settings, GRACE achieves competitive faithfulness while improving explanation plausibility, stability, and robustness under distribution shift, while maintaining competitive task performance. These results demonstrate that regulating optimization dynamics is a practical and effective approach for building reliable explanation-driven document understanding systems.
Timeline summarization aims to condense evolving information across multiple documents into coherent chronological narratives. In the sports domain, this task is particularly challenging due to the volume of news and the need to focus on user-preferred attributes such as specific players or teams. While prior work has explored generic timeline summarization, datasets supporting attribute-aware timeline summarization in sports remain limited. To address this, we introduce MulSports, a dataset designed to facilitate research on attribute-conditioned timeline summarization for sports news. MulSports comprises 2,000 news articles spanning football, cricket, kabaddi, hockey, and volleyball, paired with extracted player and team attributes and human-written timeline summaries. Each instance associates temporally ordered articles with a target attribute, enabling evaluation of both attribute focus and temporal coherence. To illustrate the utility of MulSports, we present an empirical analysis using a simple multi-modular pipeline that integrates temporal retrieval with large language models. Experimental results show that fine-tuned models outperform prompt-based baselines. MulSports provides a foundation for studying personalized and temporally grounded summarization in sports. The code, datasets, and Appendix material are made available through an anonymous GitHub repository ( https://anonymous.4open.science/r/TimeAgent-277B/ ).
While document layout analysis for Latin scripts has advanced significantly, driven by the advent of large multimodal models (LMMs), progress for the Khmer language remains constrained because of the scarcity of annotated, large-scale training data. This gap is particularly acute for camera-captured documents, where perspective distortions and complex backgrounds challenge traditional methods. Given the structural complexities of Khmer script, such as diacritics and multi-layer character stacking, existing Latin-based layout analysis models fail to accurately delineate semantic layout units, particularly for dense text regions (e.g., list items). In this paper, we present the first comprehensive study on Khmer camera-captured document layout detection. We contribute a novel framework comprising three key elements: (1) robust training and benchmarking datasets specifically for Khmer camera-captured layouts; (2) an open-source document augmentation tool capable of synthesizing realistic camera-captured documents to scale up training data; and (3) layout detection baselines utilizing YOLO-based architectures with oriented bounding boxes (OBB) to handle geometric distortions. To foster further research in the Khmer document analysis and recognition (DAR) community, we release our models, codes, and datasets in this repository ( https://github.com/rinabuoy/Towards-Khmer-Camera-Captured-Document-Layout-Detection ).
Vision Language Models (VLMs) offer strong surface-level visual understanding. However, current benchmarks do not assess whether these models truly grasp mathematical documents or multi-step derivations. We introduce VLM-Math, a model-agnostic diagnostic framework that breaks down mathematical document understanding into four measurable dimensions: structural comprehension, semantic preservation, reasoning fidelity, and educational viability. The proposed framework uses a five-stage interpretable pipeline: OCR extraction, logical ordering, step-by-step explanation, error detection, and pedagogical feedback. To bridge transcription and understanding, we propose a six-pass post-processing module. We also add an LLM-as-a-Judge semantic evaluation protocol with explicit hallucination-rate monitoring and an adaptive tutoring component. Order quality is evaluated using both absolute-position accuracy and Kendall’s τ rank correlation. Evaluations of VLM-Math-100 with domain-stratified and bootstrap-validated results show a recurring pattern: models achieve high surface fidelity but much weaker performance in semantic consistency and step-level coherence. This gap indicates a significant difference between confidence and actual accuracy. VLM-Math serves as a scaffold diagnostic benchmark that measures capability ceilings through staged prompting rather than monolithic end-to-end deployment and provides detailed, stage-wise diagnostics. It helps advance reliable and pedagogically useful mathematical reasoning in multi-modal systems. The code and related resources are at https://vlmmath.github.io/ .
While massive, closed-source Vision-Language Models (VLMs) set strong benchmarks for document understanding, their dependence on commercial Application Programming Interfaces (APIs) limits adoption in institutional archives due to data autonomy concerns, recurring costs, and the environmental footprint of hyperscale computing. This is especially acute in heritage digitization, where documents include historical handwriting, domain-specific terminology (e.g., jewelry, prehistory, architecture), and non-standard layouts requiring high-dimensional structured extraction. We present a comparative study of eight open-source lightweight VLMs ( ≤ 7 B parameters) for Optical Character Recognition (OCR)-to-structure across three university heritage collections. Given a document image, models must extract text and generate schema-compliant JavaScript Object Notation (JSON), enabling automatic validation and downstream use. We evaluate models under a constraint-aware protocol across zero-shot, few-shot, and fine-tuning settings, measuring extraction fidelity and structured-output quality using Character Error Rate (CER), Approximate Normalized Levenshtein Similarity (ANLS) ^* , and mean Average Precision F1 (mAP-F1). To assess practical improvements, we run controlled additional studies on a baseline fine-tuning pipeline, testing the independent impact of (i) hyperparameter optimization, (ii) classical image preprocessing (illumination flattening, denoising, and Contrast Limited Adaptive Histogram Equalization (CLAHE)), and (iii) multi-stage training, each compared directly to the finetune baseline. Finally, we analyze the trade-off between dataset-specific fine-tuning and a single multi-dataset checkpoint, where joint training enables one model to operate across collections but can shift performance between datasets. Overall, we show that carefully adapted ≤ 7 B VLMs can provide a sustainable, private, high-performing alternative to manual transcription or commercial black-box systems, and we offer actionable guidance for heritage institutions seeking institution-controlled OCR-to-JSON extraction. Code for this paper is available at our project repository ( https://github.com/uddipan77/Analysis-of-Lightweight-Vision-Language-Models-for-Document-OCR-and-Structured-Output-Generation ).
Multimodal encoders such as CLIP are increasingly used to search and organize historical photographic collections in galleries, libraries, archives, and museums (GLAM). However, web-scale pretraining introduces social biases into these models that, when deployed on heritage collections, risk distorting the historical narratives surfaced to their users. In this work, we audit how gender bias in CLIP interacts with historical imagery, tracing its behavior across a century of archival photographs ranging from controlled yearbook portraits to unconstrained real-world collections. Building on a taxonomy of stereotype-related concepts and complementary fairness metrics, we quantify how the gender associations encoded by CLIP evolve across decades and semantic domains, and align them with US Census labor statistics. Our audit reveals a clear dissociation: CLIP broadly captures which occupations are gender-skewed, yet the temporal trends of its biases are largely uncorrelated with documented historical change. These results caution against zero-shot use of vision-language models as demographic instruments on archival collections without domain-specific calibration. Code and data are publicly available ( https://github.com/cesc47/clip-fairness-in-historical-photo-collections ).
Historical handwritten fragments often contain limited amounts of text and substantial background noise, which may lead to unreliable writer retrieval. In this work, we introduce HIDRA, a transformer-based retrieval model that suppresses background noise without explicit ink segmentation by deriving a foreground weighting directly from a self-supervised ViT backbone. HIDRA then aggregates the foreground-weighted patch features at two complementary scales: a fine scale to capture stroke details, and a coarse scale to remain stable under severe fragmentation. The resulting global–local descriptor is fused into a compact embedding for retrieval. On HisFrag20, HIDRA achieves 68.4
Nôm is a Chinese-derived historical script used in Vietnam from the 10th to the 20th century. Although there is a substantial corpus of document images has been digitalized, the development of robust Optical Character Recognition (OCR) systems is hindered by a lack of character-level annotations and the incomplete standardization of Nôm Unicode. This paper presents an automated framework for constructing character-level annotations without manual intervention. The proposed pipeline integrates a single-class character detector adapted from Chinese character detection models and a radical-based recognizer that exploits Ideographic Description Sequences (IDS) to leverage structural commonalities between Nôm and Chinese scripts. A key contribution is an iterative self-updating fine-tuning loop in which correctly recognized characters are progressively incorporated into the annotation dataset, and the recognizer is subsequently refined on the expanded data. Experiments conducted on historical NomNaOCR pages demonstrate that the automatically verified annotation set grows to 83.95
With over 1.4 billion people and dozens of widely used scripts, India represents one of the world’s most linguistically and visually diverse document ecosystems. Yet document understanding benchmarks remain largely Western- and English-centric. Consequently, the robustness of modern vision–language models (VLMs) in multilingual and low-resource settings remains under-evaluated. We introduce the first large-scale multilingual benchmark for Indian document understanding, comprising 52607 documents across 23 languages, 13 domains, 71 document classes, and 52 tasks, with over 2.4MQA annotations across multiple tasks. The dataset captures diverse layouts, scripts, and visual conditions spanning governance, education, finance, archival, and informal content, among others. We additionally include synthetic multilingual charts, tables, and diagrams generated via Patram-Syn, an Indian persona-driven data generation pipeline with human validation. The benchmark measures perception-level capabilities such as OCR, parsing, key information extraction, and layout detection, as well as reasoning-level capabilities including Document VQA with abstractive, multi-hop, and ambiguous questions. We further introduce targeted settings for cross-lingual, transliterated, and code-mixed VQA. We also propose two new evaluation metrics, Document Understanding Cross-Lingual Index (DUCLI) and Script Robustness Score (SRS), to quantify cross-lingual reasoning degradation and script robustness. Our benchmark exposes consistent weaknesses in both open-source and closed-source VLMs, particularly in low-resource languages, degraded scans, and layout-intensive settings, underscoring fundamental challenges in scalable multilingual document intelligence.
We introduce FormWR, a challenging large-scale dataset for writer retrieval and a supervised end-to-end full-image retrieval method using a novel learnable feature aggregation module, X-VLAD. FormWR contains almost 400k pages of application-for-assistance forms attributed to almost 100k distinct writer proxies. Documents are dominated by printed templates, and further reflect real-world uncertainty through multi-hand contamination and pages with sparse handwriting. Our model jointly trains a feature extractor and aggregator. It is pretrained on sets of 32× 32 handwriting-centered patches and then fine-tuned on full-page images. We report a comprehensive large-scale evaluation on FormWR. Further, we demonstrate transfer to fragment-level retrieval on the HisFragIR20 competition benchmark. We achieve new state-of-the-art performance with 97.9 https://github.com/Traven16/writer-retrieval-at-scale .
Effective information retrieval is essential for accessing historical document collections, but remains challenging due to OCR noise, multilingual content, historical language variation, and the lack of annotated evaluation data. In this work, we present an empirical study of retrieval methods for historical documents using the SHADOW dataset, a large corpus of Swiss foreign policy records. To enable systematic evaluation in the absence of ground-truth annotations, we construct a synthetic question-answer benchmark using an LLM-based generation-judging pipeline, complemented by expert assessment from professional historians. In total, more than 40,000 document-grounded question-answer-document triplets are generated. Based on this benchmark, we compare sparse, dense, and hybrid retrieval approaches under realistic constraints, including fixed context budgets and a systematic analysis of performance degradation as the corpus size increases. Our experiments show that while dense retrieval provides strong baseline performance, classical keyword-based methods such as BM25 remain highly competitive and often outperform non-fine-tuned dense models. Fine-tuning dense embeddings improves retrieval quality, and combining sparse and dense methods via reciprocal rank fusion yields the best overall performance. We further observe that dense retrieval degrades more rapidly than BM25 as corpus size increases. The results provide practical guidance for retrieval design in OCR-derived historical collections and highlight the continued importance of classical methods as strong baselines.
We present a method for generating handwritten Japanese characters without requiring reference images during inference. We focus on Kanji, where Ideographic Description Sequences (IDS) encode radical identities and their spatial composition, and define reference-free generation as inference without rendered glyphs, font exemplars, or handwritten reference images. Conventional style transfer approaches depend on reference samples, which limits their applicability to characters without existing samples. To address this, we employ a conditional Denoising Diffusion Probabilistic Model (DDPM) that generates characters directly from noise, conditioned on structural information derived from character descriptions. Our approach encodes radical structures—components of CJK characters—using a domain-specific CLIP text encoder trained from scratch. By describing character compositions as textual prompts (e.g., radical identity and position via Ideographic Description Sequences), we extract structural features that guide the diffusion process. This allows the model to leverage compositional knowledge rather than relying on rote memorization of individual character appearances. We evaluate generation quality by feeding each method’s generated images as test data into recognition models (ResNet-50 and TrOCR) pre-trained on real ETL-9 handwriting; higher recognition performance indicates more realistic and structurally correct generation. Our CLIP-conditioned diffusion model achieves 94.10
Neural academic search systems achieve strong retrieval performance but provide limited transparency regarding ranking decisions. This opacity reduces interpretability and user trust, particularly in scholarly contexts where understanding relevance is essential. We propose a knowledge-guided multi-stage retrieval framework, formulated within a Retrieval-Augmented Generation (RAG) paradigm, that integrates structured document representations into both ranking and explanation. Each document is modeled through a document-centric knowledge graph capturing entities and topics extracted from scholarly content. Retrieval proceeds in three stages: (1) query extraction and expansion, (2) hybrid candidate retrieval using semantic similarity and lexical overlap, and (3) knowledge-aware LLM re-ranking. In the final stage, structured knowledge graph signals, such as matched entities and topics, are explicitly incorporated into a cross-encoder, enabling ranking decisions informed by both textual evidence and document structure. Explanations are derived directly from the structured matching signals used during ranking, reducing hallucination in post-hoc rationalization. We evaluate each stage of the framework using MAP, MRR, nDCG, precision, and recall. Results demonstrate improved retrieval effectiveness while providing interpretable, graph-grounded explanations for academic document search.
Historical document recognition is an important yet challenging task due to complex layouts, large character sets, diverse writing styles, and degraded image quality. Although vision–language models (VLMs) have recently achieved promising performance in OCR tasks, they still lack support and adaptation for the unique challenges of historical document recognition. Instead of using the common parameter-efficient fine-tuning (PEFT) techniques, this paper proposes to incorporate a lightweight side network into a VLM to assist in transfer learning for historical document recognition. An adaptive gating mechanism is introduced to fuse feature maps extracted by the encoder of the side network with feature representations extracted from the vision encoder in the VLM. Furthermore, an auxiliary image reconstruction loss is introduced to guide the side network training process. Experimental results on the AncientDoc dataset demonstrate that the proposed method has improved the performance of historical document recognition.
Scientific documents present a fundamental challenge for automated understanding: key findings are distributed across multiple figures, tables, and captions that must be interpreted collectively rather than independently. We propose a structured multimodal pipeline for section-level scientific content generation that jointly reasons over multiple figures within a document section, including Methodology, Exploratory Data Analysis, Results, and other visually grounded sections. Instead of processing figures individually, we introduce a deterministic visual canonicalization strategy that aggregates all figures in a section into a unified composite representation, enabling cross-figure reasoning while maintaining fixed computational complexity. A fixed-token compression mechanism transforms high-resolution visual inputs into a compact visual prefix, ensuring constant-length conditioning regardless of the number of figures. This design mitigates the quadratic computational growth typically associated with multi-figure inputs and allows scalable long-form generation using standard generative transformers. We evaluate the proposed framework on CoCoOne, a benchmark comprising 668 scientific papers across Agriculture, Environment, and Atmosphere domains with over 10,000 figure–text pairs. Across multiple vision–language backbones, the approach achieves consistent improvements in lexical overlap, semantic alignment, and structural completeness, demonstrating that structured visual compression provides a practical and scalable solution for automated understanding of multi-figure scientific documents.
Multi-modal document retrieval is a critical bottleneck for Retrieval-Augmented Generation (RAG), particularly for visually rich documents like financial reports and academic papers. Existing text-only retrievers fail to capture spatial and structural semantics, while vision-based approaches necessitate computationally expensive vision-language generation, shifting the bottleneck from offline indexing to online RAG inference. To address this, we propose LMS-Retrieval, a layout-aware, modality-aware, and structure-aware framework that augments a ColBERT backbone with fine-grained spatial feature fusion and optional structural path encodings. This approach enables effective multi-modal modeling while maintaining the efficiency of text-based inference. To stabilize training, we implement Conditional Ring Negative Sampling, which dynamically selects negatives within an optimal difficulty range to mitigate false negatives and prevent training instability. Experiments on the MMDocIR benchmark reveal a clear duality: the core Layout-Modality-aware variant achieves the highest Macro and Micro Average Recall across all baselines, demonstrating superior robustness and generalization. Meanwhile, the structural component acts as a specialized enhancement for complex, hierarchical documents, enabling the full LMS-Retrieval model to excel in rigid domains by achieving a Recall@1 of 56.8 on Laws and 48.4 on News, significantly outperforming both ColBERT and ColPali. These results establish LMS-Retrieval as a scalable solution that reconciles the precision of text with the structural awareness of vision. Our code and pre-trained models are publicly available on github( https://github.com/Lumanman9/LMS-Retrieval ).
Attribute-based document image retrieval (ABDIR) was recently proposed as an alternative to query-by-example (QBE) searches, a major document image retrieval (DIR) paradigm. One drawback of QBE searches is that they require sample query documents on hand that may not be available. ABDIR aims to offer users a flexible way to retrieve document images based on memorable visual features of document contents, describing document images with combinations of visual attributes determined via convolutional neural network (CNN)-based binary classifiers. One drawback of ABDIR is the need for training a classifier for each potential attribute. We present an exploratory study of the use of generative AI to bridge the gap between QBE and ABDIR and to address both drawbacks. We focus on historical documents as a use case for their diversity and uniqueness in visual features. We hypothesize that text-to-image (T2I) generation can be leveraged to create query document images using text prompts based on ABDIR-like attributes. We thus propose T2I-QBE, which makes use of a T2I generator with prompts that include a short description of the desired document type and a list of the desired ABDIR-style attributes. This creates query images that are then used within the traditional QBE paradigm, which compares extracted query image features to those of the document images in the dataset to retrieve the most relevant documents. Experiments on the HisIR19 dataset of historical documents suggest that T2I-QBE is a viable option for historical document image retrieval. To the authors’ knowledge, this is the first attempt at utilizing T2I generation for DIR.
Ancient manuscripts such as Oracle Bone Inscriptions (OBI) possess invaluable historical significance but are inherently characterized by severe data imbalance, with character sample sizes following a natural long-tailed distribution. While Parameter-Efficient foundation model Fine-Tuning (PEFT) methods have recently demonstrated promising performance on generic long-tailed visual recognition (LTR) tasks, their application to domain-specific ancient character recognition is largely unexplored. Moreover, existing PEFT methods allocate uniform learnable parameter capacity across all layers of foundation models. However, our empirical analysis reveals that shallow layers primarily capture generic low-level features with limited cross-class discriminability, while deeper layers progressively enhance semantic separability for tail classes. Motivated by this observation, we propose PeaFormer, a Parameter-Efficient and Adaptive foundation model fine-tuning mechanism that allocates layer-aware, learnable parameter capacity to enhance feature representation for tail classes. We also introduce an adaptive augmentation strategy based on class frequency to avoid or reduce negative or semantically altering cropping for tail classes. Additionally, we design a confidence-aware sample-specific margin to prevent over-correction of easy samples. Extensive experiments on two long-tailed OBI datasets and four LTR benchmark datasets demonstrate that our method achieves state-of-the-art performance. Code is available at https://github.com/colored32/PeaFormer .
Named entity recognition (NER) in historical newspapers is challenged by OCR noise, diachronic language variation, and dataset-specific annotation guidelines. We introduce GINNER, a Guideline-based INstruction fine-tuning for historical NER method that conditions a multilingual decoder-only model on natural-language annotation guidelines and applies it directly to OCR text, without sentence-level pre-segmentation or language-specific variants. We use an answer-only loss to improve training stability by avoiding backpropagation through long instructions. On multilingual historical NER datasets, hipe-2020 and NewsEye, we show that GINNER outperforms guideline-based prompting and other decoder-only baselines, and achieves competitive performance with specialized encoder-based historical NER systems. Our ablation studies further show that moderate fixed context windows and larger multilingual backbones yield consistent gains, supporting scalable multilingual NER under heterogeneous annotation conventions.
Character recognition is a challenging task, especially in low-resource environments such as historical languages and scripts, where there is little data and high variance. Hieratic, an ancient Egyptian cursive script comprising over 800 character classes, is one such example. Currently, there is little annotated data available for this script, and no standardized digital font exists. In this paper, we tackle the problem of a large character class, scarcely annotated script as an open-set recognition problem. We present a unified, open-set, template-based, single-shot learning framework for recognizing hieratic characters. Operating in a shared representation space, the framework can handle multiple scenarios simultaneously. It uses many-shot learning for characters with ample data, few-shot learning for characters with limited templates, and open-set recognition to identify and categorize unseen instances as “unknown.” The proposed method produces competitive results in the many-shot setting and enhances recognition in the few-shot setting. It also demonstrates novelty detection to some degree, enabling the rejection of unknown characters. These properties make the approach a valuable tool for assisted data annotation and exploration in settings with limited resources. The code can be found at https://github.com/lancercat/icdar26-egyptology .
