The usability of synthetic handwritten text to improve machine learning models is assessed for the domain of HTR. Synthetic handwritten text is generated using an existing model based on a GAN. The output of this model is then used to train a state-of-the-art HTR model, which is then applied to recognize real datasets. While this results in a CER of 28.3
This paper presents our motivation, design and two experiments for automatic scoring of handwritten descriptive answers. The first experiment is on scoring of handwritten short descriptive answers in Japanese language exams. We used a deep neural network (DNN)-based handwriting recognizer and a transformer-based automatic scorer without correcting misrecognized characters or adding rubric annotations for scoring. We achieved acceptable agreement between the automatic scoring and the human scoring, while using only 1.7
Old Church Slavonic (OCS) is an ancient language, and it has unique challenges and hurdles in natural language processing. Currently, there is a lack of Python libraries devised for the analysis of OCS texts. This research is not just filling the crucial gap in the computational treatment of OCS language but also producing valuable resources for scholars in historical linguistics, cultural studies, and humanities for the development of further research in the field of ancient language processing. The main contribution of this research work is the development of an algorithm for the lemmatization of OCS texts based on a learned dictionary. The approach can deal with ancient languages without the need for prior linguistic knowledge. Preparing a dataset of more than 330K words of OCS and their corresponding lemmas, this approach integrates the algorithm and dictionary efficiently to achieve accurate lemmatization on test data.
Handwriting analysis in historical documents, crucial for paleography, includes both macroscopic and microscopic examinations. It ranges from assessing the general visual patterns of pages to detailed studies of individual letters. Such analysis not only aids in dating documents and identifying scribes but also provides insights into the evolution of handwriting styles and offers a comprehensive view of historical writing practices. This work proposes a novel end-to-end paleographic system designed for automated handwriting analysis, consisting of three main components: automatic detection and recognition of letters, contrastive hierarchical clustering of the detected letters, and the creation of representative models for the samples in each cluster. These models provide a visual representation of the style variations for each letter. This system processes full manuscript page images to model the diverse shapes of each letter, capturing the inherent variability in historical handwriting styles. The performance of each of the three main system components has been evaluated on a dataset of Greek letters on papyri, achieving results comparable to the state of the art across all three components.
We propose the LD-DOC, a lightweight Document Layout Analysis (DLA) model specifically designed to address the challenge of accurately partitioning document regions under limited data conditions. The LD-DOC model effectively utilizes information from various scale visual features, enhancing its adaptability to feature distributions in scenarios with limited data and thereby improving the accuracy of document region partitioning. Specifically, our model incorporates a feature fusion module comprising a Shallow Feature Enhancement Path (SFEP) and a Cross-Fusion Path (CFP). The SFEP employs a 2D-Discrete Wavelet Transform (2D-DWT) to capture edge features at different scales, which enhances the model's ability to perceive subtle variations and structural information in visual features. This enhancement is crucial for adapting to the nuanced requirements of limited data environments. On the other hand, the CFP uses a Local-Fusion Attention mechanism(LFA) to capture Discrepancy information adaptively among different scales. This approach reduces the model's sensitivity to scale variations and significantly improves its generalization capabilities across diverse document layouts. Furthermore, we introduce the ISCAS-CLAD, a specialized small-scale Chinese Document Layout Analysis Dataset, to demonstrate the effectiveness of our model. Through rigorous testing on ISCAS-CLAD and the PubLayNet datasets, LD-DOC has shown a notable improvement in mean Average Precision (mAP) accuracy, outperforming baseline models by 2.2% and 1.5%, respectively. These results highlight LDDOC's state-of-the-art performance, particularly in challenging datalimited environments, and underscore its potential for practical applications in DLA.
Optical Music Recognition (OMR) is the research area focused on transcribing images of musical scores. In recent years, this field has seen great development thanks to the emergence of Deep Learning. However, these types of solutions require large volumes of labeled data. To alleviate this problem, Contrastive Self-Supervised Learning (SSL) has emerged as a paradigm that leverages large amounts of unlabeled data to train neural networks, yielding meaningful and robust representations. In this work, we explore its first application to the field of OMR. By utilizing three datasets that represent the heterogeneity of musical scores in notations and graphic styles, and through multiple evaluation protocols, we demonstrate that contrastive SSL delivers promising results, significantly reducing data scarcity challenges in OMR. To the best of our knowledge, this is the first study that integrates these two fields. We hope this research serves as a baseline and stimulates further exploration.
Recent advances in text summarization primarily target high resource languages. However, their performance on low resource and unstructured languages like Roman Urdu (RU) is not yet evaluated. This research evaluates abstractive summarization of Roman Urdu text commonly used while communicating via social media in Urdu speaking communities. Due to scarcity of relevant datasets, a corpus of Roman Urdu text is generated by transliterating samples collected from two benchmark Urdu abstractive text summarization datasets. Baseline summaries are then generated using two state-of-the-art (SOTA) transformer-based models Bidirectional Encoder Representations from Transformers (BERT) and Text-To-Text Transfer Transformer (T5). The summaries generated by both models are evaluated using different intrinsic and extrinsic methods. Results of the experiments show that T5 outperforms BERT in generating abstractive summaries of Roman Urdu text. Nonetheless, there is more research required in this direction.
Multi-page document Visual Question Answering (VQA) poses realistic challenges in the realm of document understanding due to its complexity and volume of information distributed across multiple pages. Current state-of-the-art methods often struggle to process lengthy documents, because they either exceed the model's input token limits when treated as single-page document VQA problems, or compress pages into vectors that may omit crucial information. To our knowledge, our proposed method is the first to integrate recurrent memory mechanisms with the transformer architecture specialized for multi-page document VQA. Extensive experiments demonstrate that our proposed method achieves state-of-the-art performance while maintaining a manageable model size.
We introduce Instruction Document Visual Question Answering (iDocVQA) dataset and Large Language Document (LLaDoc) model, for training Language-Vision (LV) models for document analysis and predictions on document images, respectively. Usually, deep neural networks for the DocVQA task are trained on datasets lacking instructions. We show that using instruction-following datasets improves performance. We compare performance across document-related datasets using the recent state-of-the-art (SotA) Large Language and Vision Assistant (LLaVA)1.5 as the base model. We also evaluate the performance of the derived models for object hallucination using the Polling-based Object Probing Evaluation (POPE) dataset. The results show that instruction-tuning performance ranges from 11X to 32X of zero-shot performance and from 0.1% to 4.2% over non-instruction (traditional task) finetuning. Despite the gains, these still fall short of human performance (94.36%), implying there's much room for improvement.
In the current digital era, organizations primarily interact with their clients and users online. However, accurately identifying these digital users in the physical realm raises significant challenges. Several entities, including financial institutions, insurance companies, and government services, require photos of documents sent through mobile applications to associate the physical and digital personas. This procedure entails significant computational challenges, mainly due to the need for adequate user guidance when capturing images and the variability of devices. User dependence often results in occlusions in images caused by various factors such as human fingers, shadows, and the spotlight effect. The latter is particularly common and complex due to using the device's flash. While previous research has focused on automatically identifying occlusions caused by human fingers, the present work focuses on occlusions caused by the spotlight effect. We propose a new algorithm, DocLightDetect, which uses image segmentation as a preprocessing step to improve the accuracy of classifying occlusions caused by the spotlight effect in identification documents. The effectiveness of DocLightDetect is demonstrated through the new SpotBID Set dataset. The proposed algorithm improves performance compared to state-of-the-art document occlusion classification techniques. It is also optimized for low computational cost, making it suitable for applications in mobile devices, robotics, and the Internet of Things (IoT).
Advancements in language understanding have helped researchers develop a verdict prediction system that can assist a court judge in verdict formulation. This technological intervention can help streamline and standardize the decision-making process across all levels of courts. One key benefit of developing such a system is that the junior judges can benefit from the collective knowledge stored in the knowledge base, improving their ability to make consistent and well-informed decisions. For any such system to be practically useful, predictions should be explainable too. This research proposes a hierarchical pipeline that aims to leverage domain-specific variants of BERT to enhance the process of informed decision-making. The research is mainly divided into two modules: ‘Legal Judgment Prediction (LJP)’ and ‘Legal Judgment Explanation Extraction (LJEE)’. The LJP task pertains to predicting the outcome of legal decisions concerning the appellant. In contrast, the LJEE refers to extracting out the phrases/clauses that led to the final decision. To promote research in developing such a system for Pakistani legal documents, this paper also introduces the VerdictVaultPK dataset. The dataset comprises around 11,943 rental-property case proceedings, each annotated with the court decisions indicating whether the appeal was allowed or dismissed. This research highlights how the use of domain-specific transformer models enriches semantic embeddings, contributing to a substantial accuracy improvement of 3–4
We present a text and imaging dataset of Byzantine-era Medieval Greek inscriptions, suitable as a challenging testbed for Computer Vision and Natural Language Processing tasks. The lack of sizable related training sets, as well as difficulties related to the historical character and content of the inscriptions (natural wear of characters, systematic misspellings, etc.) make for a context where modern resource-hungry techniques are not straightforward to apply. We describe the dataset contents – images, geometric and text annotation, metadata – and discuss baselines for three Computer Vision tasks (Inscription Detection, Text Recognition) and one Natural Language Processing task (Word Classification). The dataset is publicly available at https://github.com/Archaeocomputers/Bessarion .
Table reconstruction, a critical task in the field of table structure recognition (TSR), plays a vital role in various domains, such as data mining, machine learning, and information retrieval. While many existing TSR methods employ transformer-based models with generally impressive performance, a gap remains in transformer models specifically designed to handle the distinct attributes of table rows and columns. Moreover, there is a lack of robust table reconstruction strategies based on object detection models. To address these issues, we introduce the Row-Column Attention Mechanism (RCAM). When combined with a transformer model and integrated with partial global attention, it forms the RCAM-Transformer. This model is tailored to effectively process the unique properties of tabular data. In addition, we have developed a novel table reconstruction strategy that leverages object detection models, which improves the recognition and treatment of tabular data. Our experiments, conducted using the PubTables-1M and FinTabNet dataset, along with our self-constructed Annual Report TableSet, not only validated the effectiveness of the RCAM but also demonstrated the improved accuracy of table reconstruction with the use of our RCAM-Transformer. Such outcomes highlight the potential of the RCAM-Transformer to advance table extraction in various fields.
The extraction and analysis of large numbers of glyphs, and the associated opportunities for constructing a corpus of glyphs from types of the fifteenth century, offer significant research potential for scholars in book science. Such a corpus could be used in many ways, not least in assisting in the identification of fragments, charting the movements of type, and examining the impact of wear on type. Recognising this potential, we have developed fang (Code is available at https://github.com/Werck-der-buecher/FAnG.), a software that efficiently extracts and categorises glyphs from historical printed documents. Our approach involves several stages: (1) using Optical Character Recognition to extract glyphs in bulk, (2) employing a joint energy-based model for character classification and out-of-distribution pruning, and (3) providing a comprehensive toolset for manual review and editing, including deletions/reassignments and sorting by similarity. A significant strength of this design is the utilisation of existing text transcriptions and the context-awareness of trained language models, eliminating the need for explicit glyph location ground truth or glyph templates. By parallelising the extraction, we can quickly process entire digitised books with hundreds of pages, setting our system apart from existing glyph annotation tools. In experiments on digital reproductions of the Catholicon and 36-line Bible, the method demonstrates good spatial coverage of the detected glyphs, high character classification accuracy, and yields a low number of outliers. Our system represents a significant advancement in historical document analysis, providing researchers with an efficient tool for glyph extraction and categorisation.
This paper proposes a method for correcting character-recognition errors in Japanese handwritten answers to writing-type questions from exercise books. We created a model to correct character-recognition errors by fine-tuning the text-to-text-transfer-transformer (T5) using pairs of automatically recognized data from handwritten answers and their manual corrections. The data comprised handwritten Japanese answers from 185 junior high school students to writing-type questions in a Japanese language task. In addition, we augmented the training data using the five best results of the character-recognition model with confidence scores to learn additional patterns of recognition errors. The experimental results revealed that the answers corrected by the proposed method were closer to the actual answers than those before the correction and data augmentation was effective for the correction model.
Merging information from physical and digital documents is essential in an era when information is becoming even more relevant. Different strategies have been used to combine knowledge from these two data sources. One state-of-the-art data extraction approach for this problem is the Named Entity Recognition (NER) strategy. However, even for those advanced models, the performance is still highly dependent on the Optical Character Recognition (OCR) system used to read the text from the physical documents. This paper investigates this dependence and how altering OCR systems between the training and inference phases influences NER performance. We verified that changing the OCR system negatively impacts the performance of data extraction models. Furthermore, we also show that models trained on less accurate OCR are more robust to OCR changes in the inference phase. The most accurate one regarding OCR errors should be preferred in scenarios where the OCR system is the same in the training and inference stages. We also propose a solution to mitigate this problem by mixing OCRs during the training phase. This approach enhances the model's robustness while simultaneously preserving a high F1-score.
Text Recognition (TR) technology leverages a range of deep learning techniques to analyze and identify characters and words embedded in images. Its scope encompasses handwritten, printed, and scene text recognition. In this paper, we take a holistic approach, treating these categories as a unified challenge to delve into the complexities associated with TR comprehensively. The state-of-the-art models predominantly rely on vision encoder–decoder (VED) transformer architectures. However, these models tend to be bulky, housing a multitude of parameters, which not only engender significant memory consumption but also lead to sluggish inference times due to their autoregressive nature. It is essential to note that these issues primarily stem from the decoder component. Consequently, our study aims to introduce an efficient workflow that substitutes the language modeling capabilities of the decoder with lightweight Mixer layers trained using Connectionist Temporal Classification. By following this approach, we unveil three decoder-free architectures that reduce the number of parameters by a 74.3 https://github.com/dparres/Mixer-ViT-Text-Recognition .
With the accelerated pace of digitization, a vast collection of Ottoman documents has become accessible to researchers and the general public. However, most users interested in these documents are unable to read them, as the text is Turkish written in the Arabic-Persian script. Manual transcription of such a massive amount of documents is also beyond the capacity of human experts. With the advancements in deep learning, we have been able to provide a solution to the long-standing problem of automatic transcription of printed Ottoman documents. We evaluated three decoding strategies including Word Beam Search that allows to use a recognition lexicon and n-gram statistics during the decoding phase. Furthermore, the effect of lexicon size and coverage and language modelling via character or word n-grams are also evaluated. Using a general purpose large lexicon of the Ottoman era (260K words and 86 6.59% character error rate and 28.46% word error rate on a test set of 6, 828 text lines.
Optical Character Recognition (OCR) continues to face accuracy challenges that impact subsequent applications. To address these errors, we explore the utility of OCR confidence scores for enhancing post-OCR error detection. Our study involves analyzing the correlation between confidence scores and error rates across different OCR systems. We develop ConfBERT, a BERT-based model that incorporates OCR confidence scores into token embeddings and offers an optional pre-training phase for noise adjustment. Our experimental results demonstrate that integrating OCR confidence scores can enhance error detection capabilities. This work underscores the importance of OCR confidence scores in improving detection accuracy and reveals substantial disparities in performance between commercial and open-source OCR technologies.
This paper introduces Fetch-A-Set (FAS), a comprehensive benchmark tailored for legislative historical document analysis systems, addressing the challenges of large-scale document retrieval in historical contexts. The benchmark comprises a vast repository of documents dating back to the XVII century, serving both as a training resource and an evaluation benchmark for retrieval systems. It fills a critical gap in the literature by focusing on complex extractive tasks within the domain of cultural heritage. The proposed benchmark tackles the multifaceted problem of historical document analysis, including text-to-image retrieval for queries and image-to-text topic extraction from document fragments, all while accommodating varying levels of document legibility. This benchmark aims to spur advancements in the field by providing baselines and data for the development and evaluation of robust historical document retrieval systems, particularly in scenarios characterized by wide historical spectrum.
