Text recognition, especially for complex scripts like Chinese, faces unique challenges due to its intricate character structures and vast vocabulary. Traditional one-hot encoding methods struggle with the representation of hierarchical radicals, recognition of Out-Of-Vocabulary (OOV) characters, and on-device deployment due to their computational intensity. To address these challenges, we propose HierCode, a novel and lightweight codebook that exploits the innate hierarchical nature of Chinese characters. HierCode employs a multi-hot encoding strategy, leveraging hierarchical binary tree encoding and prototype learning to create distinctive, informative representations for each character. This approach not only facilitates zero-shot recognition of OOV characters by utilizing shared radicals and structures but also excels in line-level recognition tasks by computing similarity with visual features, a notable advantage over existing methods. Extensive experiments across diverse benchmarks, including handwritten, scene, document, web, and ancient text, have showcased HierCode's superiority for both conventional and zero-shot Chinese character or text recognition, exhibiting state-of-the-art performance with significantly fewer parameters and fast inference speed.

Automatic reading of traditional mechanical water meters presents significant economic advantages. To achieve this objective, a cost-effective and efficient approach is to develop a computer vision-based recognition system that utilizes meter images as input and predicts the corresponding readings. However, this is challenging for pointer-type water meters, with significant difficulty in accurately determining transitions. Transition errors often occur when the pointer precisely indicates the transitional position, making it difficult to determine whether a transition should occur. In addition, unavoidable parallax introduced during the meter image acquisition process further complicates the situation. To address these issues, we propose a novel approach that utilizes contextual information between pointers, mimicking how human readers leverage contextual cues. Specifically, we develop a simple yet effective model comprising three components: individual pointer encoder, contextual modeling encoder, and readings decoder. The individual pointer encoder initially extracts text and vision features from each individual pointer, which are then integrated by the contextual modeling encoder to incorporate multimodal information and facilitate contextual modeling. Subsequently, the readings decoder leverages the contextually enriched representations to decode the accurate overall readings autoregressively. To facilitate research in this field, we construct a new dataset called WMeter5K. This is a multitask dataset that can be used not only for overall reading recognition but also for the detection and recognition of each digital wheel and pointer. Experimental results on WMeter5K show that our method outperforms prior arts by a large margin, highlighting the significance of contextual information in pointer meter recognition. The dataset and model are publicly available at https://github.com/ZZZHANG-jx/WMeter-Reader.

Scene text image super-resolution (STISR) has obtained widespread attention in recent years due to its ability to enhance text recognition performance. Many previous methods proposed to incorporate text prior knowledge into the super-resolution architecture for reconstructing high-quality text images. However, these text priors are typically derived from pretrained text recognition models, and the inaccurate recognition feedback will hinder overall performance. In this paper, we propose a novel model, QT-TextSR, which promotes scene text image super-resolution by introducing efficient interaction with text recognition to release the inaccurate text feedback through a Query-aware Transformer. Specifically, QT-TextSR decomposes scene text image super- resolution and scene text recognition into different sets of queries within a Vision-Language Cooperation Module, explicitly modeling discriminative and interactive features between text recognition and text image super-resolution tasks. By employing two separate yet simultaneous projection heads on the corresponding features, QT-TextSR can recover the low-quality text image meanwhile obtain the recognition results. Additionally, to mitigate the limitations caused by recognition errors and enhance text structure preservation, we introduce a strong texture prior through self-supervised pre-training, leveraging visual cues more effectively. Experiments on public dataset, TextZoom demonstrate that our QT-TextSR significantly outperforms previous state-of-the-art methods in the metrics of Recognition Accuracy (68% vs. 65.5%), PSNR (22.51 vs. 22.10), and SSIM (0.7960 vs. 7930). The code for QT-TextSR is available at https://github.com/lcy0604/QT-TextSR.

Recent advances in scene text removal have attracted growing research interest due to its applications on privacy protection, document restoration, and text editing. While deep learning and generative adversarial network have shown significant progress, existing methods often struggle to generate consistent and plausible textures when erasing texts on complex backgrounds. To address this challenge, we propose a Contextual-guided Text Removal Network (CTRNet). CTRNet utilizes Low-level/High-level Contextual Guidance blocks (LCG, HCG) to explore both low-level structure and high-level discriminative context features from existing data to guide the text erasure and background restoration process. We further extend CTRNet to CTRNet++ by incorporate an Auto-Encoder architecture as a novel and effective HCG block, which serves as an additional image inpainting branch, providing more accurate texture and context clues with the assiatance of a large volume of natural images. Then we introduce a Context Embedding and Content Feature Modeling (CECFM) block that combines Depth-wise CNN and Transformer layers to capture local features and establish long-term relationships among pixels globally. In addition, an efficient Progressive Feature Fusion Module (PFFM) is proposed to fully utilize multi-scale features from different branches. Experiments on benchmark datasets, SCUT-EnsText and SCUT-Syn, demonstrate that CTRNet++ significantly outperforms existing state-of-the-art methods and exhibits a stronger ability for complex background reconstruction. The code is available at https://github.com/lcy0604/CTRNet-plus.

The development of Chinese civilization has produced a vast collection of historical documents. Recognizing and analyzing these documents hold significant value for the research of ancient culture. Recently, researchers have tried to utilize deep-learning techniques to automate recognition and analysis. However, existing Chinese historical document datasets, which are heavily relied upon by deep-learning models, suffer from limited data scale, insufficient character category, and lack of book-level annotation. To fill this gap, we introduce HisDoc1B, a large-scale dataset for Chinese historical document recognition and analysis. The HisDoc1B comprises 40,281 books, over 3 million document images, and over 1 billion characters across 30,615 character categories. To the best of our knowledge, HisDoc1B is the largest dataset in the field, surpassing existing datasets by more than 200 times in scale. Additionally, it is the only dataset with book-level annotations and punctuation annotations. Furthermore, extensive experiments demonstrate the high quality and practical utility of the proposed HisDoc1B. We believe that HisDoc1B could provide valuable resources to boost the advancement of research in this domain.

The language model (LM) plays a crucial role in post-processing handwritten text recognition (HTR) by capturing linguistic patterns. However, traditional rule-based LMs are inefficient, and recent end-to-end LMs require customized training for each HTR model. To address these limitations, we propose an E fficient, G eneric, and O ut-of-the-box L anguage M odel (EGO-LM) for HTR. To unlock the out-of-the-box capability of the end-to- end LM, we introduce a vision-limited proxy task that focuses on visual-pattern-agnostic linguistic dependencies during training, enhancing the robustness and generality of the LM. The enhanced capabilities also enable EGO-LM to iteratively refine its output fora further accuracy boost without additional tuning. Moreover, we introduce a D iverse-Corpus O nline H andwriting dataset (DCOH-120K) with more diverse corpus types and more samples than existing datasets, including 83,142 Chinese and 39,398 English text lines. Extensive experiments demonstrate that EGO-LM can attain state-of-the-art performance while achieving up to 613x acceleration. The DCOH-120K dataset is available at https://github.com/SCUT-DLVCLab/DCOH-120K.

End-to-end scene text spotting, which aims to read the text in natural images, has garnered significant attention in recent years. However, recent state-of-the-art methods usually incorporate detection and recognition simply by sharing the backbone, which does not directly take advantage of the feature interaction between the two tasks. In this paper, we propose a new end-to-end scene text spotting framework termed SwinTextSpotter v2, which seeks to find a better synergy between text detection and recognition. Specifically, we enhance the relationship between two tasks using novel Recognition Conversion and Recognition Alignment modules. Recognition Conversion explicitly guides text localization through recognition loss, while Recognition Alignment dynamically extracts text features for recognition through the detection predictions. This simple yet effective design results in a concise framework that requires neither an additional rectification module nor character-level annotations for the arbitrarily-shaped text. Furthermore, the parameters of the detector are greatly reduced without performance degradation by introducing a Box Selection Schedule. Qualitative and quantitative experiments demonstrate that SwinTextSpotter v2 achieves state-of-the-art performance on various multilingual (English, Chinese, and Vietnamese) benchmarks. The code will be available at https://github.com/mxin262/SwinTextSpotterv2 .

In recent years, significant progress has been made in table structure recognition, yet the recognition of structures within dense tables remains a challenge that has been largely overlooked. To address this gap, we introduce DenseTab, a new dataset consisting of 16,575 dense tables with comprehensive annotation information that includes physical position, logical position, and text content of each cell, along with the HTML sequence. To tackle the challenge of dense table structure recognition, we propose a new method called Dense Table Splitting andMerging Model (DTSM). DTSM includes a novel text query encoder to leverage layout information associated with the text's location, and an adjacent feature aggregator to enhance the prediction of cell merging information. Experimental results demonstrate that our proposed method achieves state-of-the-art performance in recognizing dense table structures. The dataset and code is available at https://github.com/ TenMilesLotus/DTSM.

Automatic font generation is an imitation task, which aims to create a font library that mimics the style of reference images while preserving the content from source images. Although existing font generation methods have achieved satisfactory performance, they still struggle with complex characters and large style variations. To address these issues, we propose FontDiffuser, a diffusion-based image-to-image one-shot font generation method, which innovatively models the font imitation task as a noise-to-denoise paradigm. In our method, we introduce a Multi-scale Content Aggregation (MCA) block, which effectively combines global and local content cues across different scales, leading to enhanced preservation of intricate strokes of complex characters. Moreover, to better manage the large variations in style transfer, we propose a Style Contrastive Refinement (SCR) module, which is a novel structure for style representation learning. It utilizes a style extractor to disentangle styles from images, subsequently supervising the diffusion model via a meticulously designed style contrastive loss. Extensive experiments demonstrate FontDiffuser's state-of-the-art performance in generating diverse characters and styles. It consistently excels on complex characters and large style changes compared to previous methods. The code is available at https://github.com/yeungchenwa/FontDiffuser.

Classical Chinese Understanding (CCU) holds significant value in preserving and exploration of the outstanding traditional Chinese culture. Recently, researchers have attempted to leverage the potential of Large Language Models (LLMs) for CCU by capitalizing on their remarkable comprehension and semantic capabilities. However, no comprehensive benchmark is available to assess the CCU capabilities of LLMs. To fill this gap, this paper introduces C^3bench, a Comprehensive Classical Chinese understanding benchmark, which comprises 50,000 text pairs for five primary CCU tasks, including classification, retrieval, named entity recognition, punctuation, and translation. Furthermore, the data in C^3bench originates from ten different domains, covering most of the categories in classical Chinese. Leveraging the proposed C^3bench, we extensively evaluate the quantitative performance of 15 representative LLMs on all five CCU tasks. Our results not only establish a public leaderboard of LLMs' CCU capabilities but also gain some findings. Specifically, existing LLMs are struggle with CCU tasks and still inferior to supervised models. Additionally, the results indicate that CCU is a task that requires special attention. We believe this study could provide a standard benchmark, comprehensive baselines, and valuable insights for the future advancement of LLM-based CCU research. The evaluation pipeline and dataset are available at .

Existing optical character recognition (OCR) methods rely on task-specific designs with divergent paradigms, architectures, and training strategies, which significantly increases the complexity of research and maintenance and hinders the fast deployment in applications. To this end, we propose UPOCR, a simple-yet-effective generalist model for Unified Pixel-level OCR interface. Specifically, the UPOCR unifies the paradigm of diverse OCR tasks as image-to-image transformation and the architecture as a vision Transformer (ViT)-based encoder-decoder with learnable task prompts. The prompts push the general feature representations extracted by the encoder towards task-specific spaces, endowing the decoder with task awareness. Moreover, the model training is uniformly aimed at minimizing the discrepancy between the predicted and ground-truth images regardless of the inhomogeneity among tasks. Experiments are conducted on three pixel-level OCR tasks including text removal, text segmentation, and tampered text detection. Without bells and whistles, the experimental results showcase that the proposed method can simultaneously achieve state-of-the-art performance on three tasks with a unified single model, which provides valuable strategies and insights for future research on generalist OCR models. Code is available at https://github.com/shannanyinxiang/UPOCR.

Existing methods for zero-shot Chinese character recognition usually exploit a single type of side information such as radicals, glyphs, or strokes to establish a mapping with the input characters for the recognition of unseen categories. However, these approaches have two limitations. Firstly, the mappings are inefficient owing to their complexity. Some existing methods design radical-level mappings using a non-differentiable dictionary matching strategy, whereas others construct sophisticated embeddings to map seen and unseen characters into a unified latent space. Although the latter approach is straightforward, it lacks a learnable scheme for explicit structure construction. Secondly, the complementarity within multiple types of side information has not been effectively explored. For example, the radicals provide structural knowledge at an abstract level, whereas glyphs offer detailed information on their figurative counterparts. To this end, we propose a new method called SideNet that jointly learns character-level representations assisted by two types of interactive side information: radicals and glyphs. SideNet contains a structural conversion module that extracts radical knowledge via dimensional decomposition, and a spatial conversion module that encodes the radical counting map to produce an interactive outcome between radicals and glyph. Finally, we propose a new classifier that integrates the converted features by a similarity-guided fusion mechanism. To the best of our knowledge, this study represents the first attempt to integrate these two types of side information and explore a joint representation for zero-shot learning. Experiments show that SideNet consistently outperforms existing methods by a significant margin in diverse scenarios, including handwriting, printed art, natural scenes, and ancient Chinese characters, which demonstrates the potential of joint learning with multiple types of side information.

Classical Chinese is a gateway to the rich heritage and wisdom of ancient China, yet its complexities pose formidable comprehension barriers for most modern people without specialized knowledge. While Large Language Models (LLMs) have shown remarkable capabilities in Natural Language Processing (NLP), they struggle with Classical Chinese Understanding (CCU), especially in data-demanding and knowledge-intensive tasks. In response to this dilemma, we propose TongGu (mean understanding ancient and modern), the first CCU-specific LLM, underpinned by three core contributions. First, we construct a two-stage instruction-tuning dataset ACCN-INS derived from rich classical Chinese corpora, aiming to unlock the full CCU potential of LLMs. Second, we propose Redundancy-Aware Tuning (RAT) to prevent catastrophic forgetting, enabling TongGu to acquire new capabilities while preserving its foundational knowledge. Third, we present a CCU Retrieval-Augmented Generation (CCU-RAG) technique to reduce hallucinations based on knowledge-grounding. Extensive experiments across 24 diverse CCU tasks validate TongGu’s superior ability, underscoring the effectiveness of RAT and CCU-RAG. The model and dataset are available at https://github.com/SCUT-DLVCLab/TongGu-LLM .

Scene text removal (STR) aims at replacing text strokes in natural scenes with visually coherent backgrounds. Recent STR approaches rely on iterative refinements or explicit text masks, resulting in high complexity and sensitivity to the accuracy of text localization. Moreover, most existing STR methods adopt convolutional architectures while the potential of vision Transformers (ViTs) remains largely unexplored. In this paper, we propose a simple-yet-effective ViT-based text eraser, dubbed ViTEraser. Following a concise encoder-decoder framework, ViTEraser can easily incorporate various ViTs to enhance long-range modeling. Specifically, the encoder hierarchically maps the input image into the hidden space through ViT blocks and patch embedding layers, while the decoder gradually upsamples the hidden features to the text-erased image with ViT blocks and patch splitting layers. As ViTEraser implicitly integrates text localization and inpainting, we propose a novel end-to-end pretraining method, termed SegMIM, which focuses the encoder and decoder on the text box segmentation and masked image modeling tasks, respectively. Experimental results demonstrate that ViTEraser with SegMIM achieves state-of-the-art performance on STR by a substantial margin and exhibits strong generalization ability when extended to other tasks, e.g., tampered scene text detection. Furthermore, we comprehensively explore the architecture, pretraining, and scalability of the ViT-based encoder-decoder for STR, which provides deep insights into the application of ViT to the STR field. Code is available at https://github.com/shannanyinxiang/ViTEraser.

In recent years, image manipulation localization has attracted increasing attention due to its pivotal role in guaranteeing social media security. However, how to accurately identify the forged regions remains an open challenge. One of the main bottlenecks lies in the severe scarcity of high-quality data, due to its costly creation process. To address this limitation, we propose a novel paradigm, termed as CAAA, to automatically and precisely annotate the numerous manually forged images from the web at the pixel level. We further propose a novel metric QES to facilitate the automatic filtering of unreliable annotations. With CAAA and QES, we construct a large-scale, diverse, and high-quality dataset comprising 123,150 manually forged images with mask annotations. Besides, we develop a new model APSC-Net for accurate image manipulation localization. According to extensive experiments, our dataset significantly improves the performance of various models on the widely-used benchmarks and such improvements are attributed to our proposed effective methods. The dataset and code are publicly available at https://github.com/qcf-568/MIML.

Document image restoration is a crucial aspect of Document AI systems, as the quality of document images significantly influences the overall performance. Prevailing methods address distinct restoration tasks independently, leading to intricate systems and the incapability to harness the potential synergies of multi-task learning. To overcome this challenge, we propose DocRes, a generalist model that unifies five document image restoration tasks including dewarping, deshadowing, appearance enhancement, deblurring, and binarization. To instruct DocRes to perform various restoration tasks, we propose a novel visual prompt approach called Dynamic Task-Specific Prompt (DTSPrompt). The DTSPrompt for different tasks comprises distinct prior features, which are additional characteristics extracted from the input image. Beyond its role as a cue for task-specific execution, DTSPrompt can also serve as supplementary information to enhance the model's performance. Moreover, DTSPrompt is more flexible than prior visual prompt approaches as it can be seamlessly applied and adapted to inputs with high and variable resolutions. Experimental results demonstrate that DocRes achieves competitive or superior performance compared to existing state-of-the-art task-specific models. This under-scores the potential of DocRes across a broader spectrum of document image restoration tasks. The source code is publicly available at https://github.com/ZZZHANGjx/DocRes.

Handwritten Chinese text recognition (HCTR) is still a challenging and unsolved problem. The existing recognition methods are mainly categorized into two: explicit vs implicit segmentation-based methods. Explicit segmentation recognition methods use explicit character location information to train the recognizers. However, the widely used weakly supervised training strategy based on pseudo-label makes it difficult to get effective supervised training for difficult character samples. In contrast, the implicit segmentation recognition method use all transcript annotations for supervised training, but it is prone to misalignment problem due to the lack of explicit supervised information of character positions. To take advantage of the complementary nature of explicit and implicit segmentation approaches, we propose a new method, SegCTC, which better integrates these two approaches into a unified to be a more powerful recognizer. Specifically, we designed a hybrid Segmentation-based and Segmentation-free Feature Fusion Module (S $$^2$$ FFM) to better fuse the features of both explicit and implicit segmentation-based recognition branches. Moreover, a co-transcription strategy is also proposed to better combine the predictions from different branches. Experiments on four widely used benchmarks including CASIA-HWDB, ICDAR2013, SCUT-HCCDoc and MTHv2 show that our method achieves state-of-the-art performance for the HCTR task under different scenarios.

This paper aims to re-assess scene text recognition (STR) from a data-oriented perspective. We begin by revisiting the six commonly used benchmarks in STR and observe a trend of performance saturation, whereby only 2.91% of the benchmark images cannot be accurately recognized by an ensemble of 13 representative models. While these results are impressive and suggest that STR could be considered solved, however, we argue that this is primarily due to the less challenging nature of the common benchmarks, thus concealing the underlying issues that STR faces. To this end, we consolidate a large-scale real STR dataset, namely Union14M, which comprises 4 million labeled images and 10 million unlabeled images, to assess the performance of STR models in more complex real-world scenarios. Our experiments demonstrate that the 13 models can only achieve an average accuracy of 66.53% on the 4 million labeled images, indicating that STR still faces numerous challenges in the real world. By analyzing the error patterns of the 13 models, we identify seven open challenges in STR and develop a challenge-driven benchmark consisting of eight distinct subsets to facilitate further progress in the field. Our exploration demonstrates that STR is far from being solved and leveraging data may be a promising solution. In this regard, we find that utilizing the 10 million unlabeled images through self-supervised pre-training can significantly improve the robustness of STR model in real-world scenarios and leads to state-of-the-art performance. Code and dataset is available at https://github.com/Mountchicken/Union14M.

Handwritten text erasure on examination papers is an important new research topic with high practical value due to its ability to restore examination papers and collect questions that are answered incorrectly for review, thereby improving educational efficiency. However, to the best of our knowledge, there is no publicly available dataset for handwritten text erasure on examination papers. To facilitate the development of this field, we build a real-world dataset called SCUT-EnsExam (short for EnsExam). The dataset consists of 545 examination paper images, each of which has been carefully annotated to provide a visually reasonable erasure target. With EnsExam, we propose an end-to-end model, which introduces a soft stroke mask to erase the handwritten text precisely. Furthermore, we propose a simple yet effective loss called stroke normalization (SN) loss to alleviate the imbalance between text and non-text regions. Extensive numerical experiments shows that our proposed method outperforms previous state-of-the-art methods on EnsExam. In addition, quantitative experiments on scene text removal benchmark, SCUT-EnsText, demonstrate the generalizability of our method. The EnsExam will be made available at https://github.com/SCUT-DLVCLab/SCUT-EnsExam .