Handwritten Text Recognition (htr) is more interesting and challenging than printed text due to uneven variations in the handwriting style of the writers, content, and time. htr becomes more challenging for the Indic languages because of (i) multiple characters combined to form conjuncts which increase the number of characters of respective languages, and (ii) near to 100 unique basic Unicode characters in each Indic script. Recently, many recognition methods based on the encoder-decoder framework have been proposed to handle such problems. They still face many challenges, such as image blur and incomplete characters due to varying writing styles and ink density. We argue that most encoder-decoder methods are based on local visual features without explicit global semantic information. In this work, we enhance the performance of Indic handwritten text recognizers using global semantic information. We use a semantic module in an encoder-decoder framework for extracting global semantic information to recognize the Indic handwritten texts. The semantic information is used in both the encoder for supervision and the decoder for initialization. The semantic information is predicted from the word embedding of a pre-trained language model. Extensive experiments demonstrate that the proposed framework achieves state-of-the-art results on handwritten texts of ten Indic languages.
Handwritten text recognition is still an unsolved problem in the field of machine learning. Nevertheless, this technology has improved considerably in the last decade in part thanks to advancements in recurrent neural networks. Unfortunately, due to their sequential nature, recurrent models cannot be effectively parallelised during training. Meanwhile, in natural language processing research, the transformer has recently become the dominant architecture; replacing the recurrent networks that were once popular. These new models are far more efficient to train than their predecessors because their primary building block, the self-attention network, can process sequences entirely non-recurrently. This work demonstrates that self-attention networks can replace the recurrent networks of state-of-the-art handwriting recognition models and achieve competitive error rates, while reducing the time required to train and the number of parameters significantly.
The variety of handwriting styles and the scarcity of training data often result in poor performance of character recognizer. Rather than tedious data collection and annotation, researchers prefer to use low-cost data augmentation to improve the robustness of the recognizer. However, most existing data augmentation methods treat handwritten text as ordinary images and generate new samples through holistic transformation, which greatly limits the diversity of generated samples. To solve the problem, this paper proposes a script-level handwritten text augmentation method, where each component is treated as a Bézier curve. Specifically, we first segment the character into components based on skeleton detection. Then, we move the control points of each component according to the prior knowledge of languages. Finally, we transform the component by Bézier curves and assemble them into new samples. Our method is simple, controllable, and friendly to few-shot handwritten text. Experiments on four datasets in different languages show that the proposed script-level augmentation method performs better than the holistic augmentation methods. Apart from it, we also modify Affine transformation, a commonly used augmentation method, from a holistic to script-level way. Experimental results demonstrate that script-level affine can achieve better performance than holistic affine in the character recognition task. Our code is available at https://github.com/IMU-MachineLearningSXD/script-level_aug_ICFHR2022 .
Handwritten Text Recognition (HTR) relies on deep learning to achieve high performances. Its success is substantially driven by large annotated training datasets resulting in powerful recognition models. Performances suffer considerably when applied to document collections with a distinctive style that is not well represented by training data. Applying a recognition model to a new data collection poses a tremendous annotation effort, which is often out of scope, for example considering historic collections. To overcome this limitation, we propose a training scheme that combines multiple data sources. Synthetically generated samples are used to train an initial model. Self-training offers the possibility to exploit unlabeled samples. We further investigate the question of how a small number of manually annotated samples can be integrated to achieve maximal performance with limited annotation effort. Therefore, we add labeled samples at different stages of self-training and propose two criteria, namely confidence and diversity, for the selection of samples to annotate. In our experiments, we show that the proposed training scheme is able to considerably close the gap to fully-supervised training on the designated training set with less than ten percent of the labeling demand.
Digitization of ancient palm leaf manuscripts is gaining momentum due to the limited datasets and complex features of text images of palm leaf manuscripts. Thus far, the previous studies did not deeply analyze the application of the trending techniques on the palm leaf manuscripts, considering how deep learning approaches require large datasets, while some isolated glyphs contain more than one character with complex grammatical components. Therefore, this paper explores the possibilities and practical methods for improving isolated glyph classification. In particular, we focus on both the front-end and the back-end processes involved in the image classification task. For the front-end analysis, we present multi-task preprocessing techniques, including data augmentation techniques, new datasets extraction, and image enhancement techniques to increase the quality and quantity of datasets. For the back-end side, we aim to study the visual backbones of deep learning techniques, especially CNNs (including VGG, ResNet, and EfficientNet) and attention-based models (including ViT, DeiT, and CvT). Furthermore, the analysis and evaluation examined how data augmentation techniques and preprocessing interact with the amount of data used in training. Evidently, we experimented on three palm leaf manuscripts, including Balinese, Sundanese, and Khmer scripts from the ICFHR contest 2018, SluekRith, AMDI LontarSet, and Sunda datasets. Regarding the quality of research, the experiment delivers an effective way of training palm leaf datasets for the document analysis community.
In a crime scene, document fragments with similar contents might lead to significant evidence. A criminalist when encounters such a scene with an enormous amount of torn document pieces, automated analysis becomes imperative in procuring potential evidence in a fast and reliable way. To analyze document fragments with similar contents, a processing module to segment the homogeneous zones based on the content type is a prerequisite. This paper proposes a deep learning-based module DAZeTD that can detect textual (printed/handwritten) and non-textual ragged zones. For classifying the content of the zones, we adopt the scheme of vision transformer; and to draw the zone boundaries, we employ outer isothetic cover. We created a dataset of 881 torn documents on which we performed rigorous experiments. We obtained an overall 87.71% mAP@0.5, which is quite promising.
As the sixth largest minority language in China, the Yi language is used by 8 million people and records the development of human civilization. Deep learning has been widely used and effective in mainstream characters' recognition, but there are few achievements in recognition of Yi characters, particularly online handwriting recognition. Most of the Yi strokes are curved, and the writing is irregular. Consequently, there are some problems, such as arbitrarily change for the writing order, multiple strokes concatenated or abbreviated, stroke position offset, and so on. Due to different sample collection equipment, there is also dimensional diversity and sampling frequency diversity between sample collection devices, which will bring more significant interference to the identification. In this paper, we construct a Yi online handwriting recognition database and propose two Yi online handwriting recognition models based on different usage scenarios: RnnNet-Yi (for high accuracy requirements) and ParallelRnnNet-Yi (for resource-constrained lightweight requirements), merging deep learning and feature extraction methods. The experimental results verify the effectiveness of the models proposed in this paper in upgrading the accuracy and training speed of Yi online handwriting recognition, which fills the gap in Yi online handwriting recognition research.
Super-resolution aims to increase the resolution and the clarity of the details in low-resolution images, and document images are no exception. Although significant improvements have been achieved in super-resolution for different domains, historical document images have not been addressed well. Most of the current works in the text domain deal with modern fonts and rely on extracting prior semantic information from a recognizer to super-resolve images. The absence of a reliable handwritten recognizer for Arabic documents, where historical documents have a complex structure and overlapping parts, makes these text-domain works inapplicable. This paper presents a Text-Attentioned Super Resolution GAN (TASR-GAN) to address this problem. The model deals with historical Arabic documents and does not rely on prior semantic information. Since our input domain documents, text edges are essential for quality and readability; thus, we introduce a new loss function called text edge loss. This loss function provides more attention and weight to text edge information and guides through optimization to super-resolve images with accurate small regions' details and fine edges to improve image quality. Experiments on six Arabic manuscripts show that the proposed TASR achieves state-of-the-art performance in terms of PSNR/SSIM metrics and significantly improves the visual image quality, mainly the edges of small regions details, and eliminates artifacts noises. Also, a grid search experiment has been conducted to tune the best hyperparameters values for our text edge loss function.
Lecture videos have become a great resource for students and teachers. These videos are a vast information source, but most search engines only index them by their audio. To make these videos searchable by handwritten content, it is important to develop accurate methods for analyzing such content at scale. However, training deep neural networks to their full potential requires large-scale lecture video datasets. In this paper, we use synthetic data generation to improve binarization of lecture videos. We also use it to semantically segment pixels into background, speaker, text, mathematical expressions, and graphics. Our method for synthetic data generation renders content from multiple handwritten and typeset datasets, and blends it into real images using random tight layouts and the location of the people. In addition, we also propose a mixed data approach that trains networks on two detection tasks at once: person and text. Both binarization and semantic segmentation are carried out using fully convolutional neural networks with a typical encoder-decoder architecture and residual connections. Our experiments show that pretraining on both synthetic and mixed data leads to better performance than training with real data alone. While final results are promising, more work will be needed to reduce the domain shift between synthetic and real data. Our code and data are publicly available.
Gurmukhi script is used to write the official `Punjabi' language of the people of the western part of Indian Punjab. The script is having approximately 160 million native speakers. Recognition of handwritten characters in the Gurmukhi script is still in its embryonic stage due to intricate character shapes and the scarcity of standard datasets. This paper introduces a new large-scale benchmark dataset "Gurmukhi HWdb1.0" which is an important development in the handwritten character recognition of this script. This dataset has a total of 137,700 handwritten samples of 41 basic Gurmukhi characters and 10 numeral classes. Out of these, 110,160 images are used for training,13,770 images are set aside for validation, and 13,770 images are used for testing. Here, 265 individuals have contributed to the development of the dataset. Recognition of the script is carried out using a CNN architecture based on transfer learning on the VGG16 network. We fine-tuned the model and added our own fully connected layers needed for Gurmukhi characters. The proposed model is executed on this collected "Gurmukhi HWdb1.0" dataset for evaluation. A detailed comparison with different batch sizes is performed to understand the functionality of the model. Experimental results show that the proposed model can be benchmarked against the concerned dataset with a test accuracy of 98.42% for Gurmukhi characters and 97.51% for Gurmukhi numerals.
Recently, the booming of electronic devices has revolutionized the way we sign in our daily life. The sudden surge of online Chinese signatures calls for need online Chinese signature segmentation as prerequisite for downstream tasks such as building up database of Chinese characters and verifying signatures based upon individual characters. However, common approaches deriving from over-segmentation do not apply well to Chinese signatures, which have little linguistic meanings but instead have flourish, artistic styles, composite structures or even word overlaps. To cope with those difficulties, this paper exploits the benefits of signature recognition to boost segmentation performance and proposes a progressive multitask learning network (PMLNet) for online Chinese signature segmentation and recognition. PMLNet consists of a dual channel stroke feature extraction block (DSF-Block), a stacked transformer encoder block (STE-Block) and a progressive multitask learning block (PML-Block). DSF-block is used to extract stroke-wise spatial features and semantic features through dual channels; STE-block is used to model long-range dependencies among different strokes and enhance their feature representations; PML-block is used to branch interactively and progressively the signature's final segmentation and recognition. Specifically, we introduce a progressive learning strategy in PML-block to fine-tune segmentation and recognition results by fully leveraging their reciprocal relationship. The experiment results on our private database show that PMLNet achieves 8.77% higher accurate rate (AR), 9.15% higher correct rate (CR) and 9.93% higher sample-level segmentation accurate rate (ACC(seg)) than SOTAs.
In this study, individuals were asked to draw a symbol by using their fingertips on a digital device screen. This study focused on finger-touch direction information that can be extracted from a smartphone screen. To suppress rapid changes in the detected direction data, preprocessing was introduced, and its effectiveness was confirmed by evaluating the verification performance. Finally, representing the direction data as a frequency distribution was introduced as a new feature, which was demonstrated to improve the verification performance.
Identifying crime or individuals is one of the key tasks toward smart and safe city development when different nationals are involved. In this regard, identifying Nationality/Ethnicity through handwriting has received special attention. But due to freestyle and unconstrained writing, identifying nationality is challenging. This work considers words written by people of 10 nationals namely, India, Malaysia, Myanmar, Bangladesh, Iran, Pakistan, Sri Lanka, Cambodia, Palestine, and China, for identification. To extract invariant features, such as the distribution of edge patterns despite of the adverse effect of different writing styles, paper, pen, and ink, we explore a new Edge-Attention based U-Net (EAU-Net), which generates edge points for each input word image written by different nationals. Inspired by the success of the Convolutional Neural Network for classification, we explore CNN for the classification of 10 classes by considering candidate points given by EAU-Net as input. The proposed method is tested on our newly developed dataset of 10 classes, a standard dataset of 5 classes to demonstrate the effectiveness in classifying different nationalities. Furthermore, the efficacy of the proposed method is shown by testing on IAM dataset for gender identification. The results of the proposed and existing methods show that the proposed method outperforms the existing methods for both nationality and gender identification.
Deep learning has significantly improved handwriting text recognition, esp. for Latin scripts. Arabic scripts including Urdu is a family of complex scripts and they pose difficult challenges for deep learning architectures. Data availability is a significant obstacle in developing Urdu handwriting recognition systems. Since gathering data is a costly and challenging task, there is a need to increase training data using novel approaches. One possible solution is to make a model that can generate similar yet different samples from the existing data samples. In this paper, we propose such models based on Generative Adversarial Networks (GANs) that have the ability to synthesize realistic samples similar to the original dataset. Our generator is class conditioned to produce Urdu samples of varying characters that differ in style. Visual and quantitative analysis convey that generated samples are of realistic nature and can be used to increase datasets. Synthesized samples integrated with the existing training set is shown to increase the performance of a handwriting recognition system.
We present our Offline Urdu Handwritten Text Dataset (UOHTD) in this paper by collecting 800 Urdu handwritten samples written by 800 native language writers. It consists of images in the form of a dataset containing written text samples scanned with multiple spatial resolutions. 8000 text lines and 40000 words as patches have been extracted from sample pages and checked manually and formally using a ground truth database. Machine Learning Tools have been utilized to extract sample pages and segment them into lines and words. Initial trials on demographic (gender and age group) classification of Urdu writers with samples of Offline Urdu Handwritten Text Dataset (UOHTD) has produced promising results (85% for gender and 79% for age group classification) using CNNs. The database would be made available to the researcher worldwide for study into various handwritten-related topics including text recognition, identification of the writer's age, ethnicity, demographics, gender, and handedness, as well as verification.
An Online signature is a multivariate time series, a commonly used biometric source for user verification. Deep learning (DL) is increasingly becoming ubiquitous as a paradigm for solving problems that come with a wealth of data. Convolution has been its main workhorse. Recently, DL had marked its entry in online signature verification (OSV), a standard bio-metric method that has been mostly dealt with in traditional settings. However, embracing a DL solution to a problem requires certain issues to be tackled, viz. (i) type of convolution, (ii) order of convolution, and (iii) input representation. In this work, we experimentally analyse each of the issues mentioned above regarding OSV, and subsequently present a superior model that reports state-of-the-art (SOTA) performance on three widely used data-sets namely MCYT-100, SVC, and Mobisig. Specifically, the proposed model reports an equal error rate (EER) of 9.72% and 3.1% in Skilled_01 categories of MCYT-100 and SVC data-sets, with gains of around 4% and 3% over the next best performing methods, respectively. The experimental outcome confirms that the interrelationship between the type and order of convolution operation and the input signature representation plays a significant role in the performance of OSV frameworks.
The attention-based encoder-decoder (AED) models are increasingly used in handwritten mathematical expression recognition (HMER) tasks. Given the recent success of Transformer in computer vision and a variety of attempts to combine Transformer with convolutional neural network (CNN), in this paper, we study 3 ways of leveraging Transformer and CNN designs to improve AED-based HMER models: 1) Tandem way, which feeds CNN-extracted features to a Transformer encoder to capture global dependencies; 2) Parallel way, which adds a Transformer encoder branch taking raw image patches as input and concatenates its output with CNN’s as final feature; 3) Mixing way, which replaces convolution layers of CNN’s last stage with multi-head self-attention (MHSA). We compared these 3 methods on the CROHME benchmark. On CROHME 2016 and 2019, Tandem way attained the ExpRate of 54.85% and 58.56%, respectively; Parallel way attained the ExpRate of 55.63% and 57.39%; and Mixing way achieved the ExpRate of 53.93% and 55.64%. This result indicates that Parallel and Tandem ways perform better than Mixing way, and have little difference between each other.
In Northern Europe, parish records provide centuries of lineage information, useful not only for settling inheritance disputes, but also for studying hereditary diseases, social mobility, etc. The key information to extract from scans of parish records to obtain lineage information is dates: birth dates (of children and their parents) and dates of baptisms. We present a new dataset of birth dates from Danish parish records and use it to benchmark different approaches to handwritten date recognition, some based on classification and some based on transduction. We evaluate these approaches across several experimental protocols and different segmentation strategies. A state-of-the-art transformer-based transduction model exhibits lower error rates than image classifiers in most scenarios. The image classifiers can nevertheless offer a compelling trade-off in terms of accuracy and computational resource requirements.
Prototype learning has achieved good performance in many fields, showing higher flexibility and generalization. In this paper, we propose an efficient text line recognition method based on prototype learning with feature-level sliding windows for classification. In this framework, we combine weakly supervised discrimination and generation loss for learning feature representations with intra-class compactness and inter-class separability. Then, dynamic weighting and pseudo-label filtering are also adopted to reduce the influence of unreliable pseudo-labels and improve training stability significantly. Furthermore, we introduce consistency regularization to obtain more reliable confidence distributions and pseudo-labels. Experimental results on digital and Chinese handwritten text datasets demonstrate the superiority of our method and justify advantages in transfer learning on small-size datasets.
Recently, vision Transformer (ViT) has attracted more and more attention, many works introduce the ViT into concrete vision tasks and achieve impressive performance. However, there are only a few works focused on the applications of the ViT for scene text recognition. This paper takes a further step and proposes a strong scene text recognizer with a fully ViT-based architecture. Specifically, we introduce multi-grained features into both the encoder and decoder. For the encoder, we adopt a two-stage ViT with different grained patches, where the first stage extracts extent visual features with 2D fine-grained patches and the second stage aims at the sequence of contextual features with 1D coarse-grained patches. The decoder integrates Connectionist Temporal Classification (CTC)-based and attention-based decoding, where the two decoding schemes introduce different grained features into the decoder and benefit from each other with a deep interaction. To improve the extraction of fine-grained features, we additionally explore self-supervised learning for text recognition with masked autoencoders. Furthermore, a focusing mechanism is proposed to let the model target the pixel reconstruction of the text area. Our proposed method achieves state-of-the-art or comparable accuracies on benchmarks of scene text recognition with a faster inference speed and nearly 50 % reduction of parameters compared with other recent works.
