What is OCR?

Optical Character Recognition (OCR) is a computer vision technique that transforms document images into machine-readable text.
Modern OCR systems don’t simply “read” pixels—they follow a carefully designed pipeline to clean, detect, interpret, and refine text.
Below we walk through the core stages that make this possible.

1. Document Image Preprocessing

Every OCR journey begins with raw scans or photos that may contain dust, skew, shadows, or inconsistent lighting.
Preprocessing ensures these imperfections do not mislead later stages.

• Noise removal – Filters (median, bilateral, or deep-learning denoisers) suppress speckles and background textures.

• Binarization – Converts the page to pure black and white (e.g., Otsu or adaptive thresholding), separating foreground text from background.

• Skew and orientation correction – Algorithms such as Hough line detection or deep angle predictors straighten tilted pages and fix upside-down scans.

2. Text Detection & Classification

Next, the system identifies where the words are and what kind of text each region contains.

• Text region detection – Finds lines, paragraphs, or word boxes. Popular choices include EAST, CRAFT, and YOLO-based text detectors.

• Layout analysis – Handles complex formats such as multi-column newspapers, invoices, or tables.

• Printed vs. handwritten differentiation – Directs each region to the correct recognition model.

• Orientation & script detection – Determines if text is vertical, rotated, or multilingual.

This stage defines where and how the text will be read.

3. Text Recognition

With text regions isolated, OCR converts visual patterns into actual letters and words.

• Character segmentation – Separates connected characters or overlapping strokes when needed.

• Deep learning recognition –

⚪︎CNN-LSTM/CRNN models combine convolutional feature extraction with sequence modeling.

⚪︎Transformer-based recognizers (e.g., TrOCR, Vision-Transformer hybrids) excel at irregular or curved text lines.

• Rule-based recognition – Still effective for barcodes, fixed fonts, or structured forms.

This is the heart of OCR—pixels finally become digital characters.

4. Post-Processing

Polishing the output for human and machine use

Even advanced recognition models make occasional errors.
Post-processing refines the raw output into clean, contextually correct text.

• Language modeling & spell-checking – Uses n-gram or transformer language models to correct misread words.

• Dictionary or domain constraints – Enforces specialized vocabularies (medical terms, product codes, etc.).

• Structure alignment – Reconstructs tables, forms, or document hierarchy so the digital version mirrors the original.

The final result is machine-readable text that is accurate, searchable, and ready for downstream analytics or storage.

While OCR remains a foundational technology for extracting machine-readable text, its linear pipeline and plain-text outputs limit the ability to capture document structure, layout relationships, and visual context. To overcome these constraints, we next examine LayoutLMv3, a multimodal model that fuses textual, spatial, and visual signals to achieve a deeper understanding of document content.

Overview of LayoutLMv3

LayoutLMv3 is a state-of-the-art multimodal Transformer designed for rich document understanding.
Unlike traditional OCR pipelines that simply output linear text, LayoutLMv3[3] integrates three complementary information streams within a single unified representation space:

• Text embeddings – token-level representations produced from OCR-extracted text, each encoded with contextual information and precisely aligned to its position in the document.

• Layout (2-D positional) embeddings – absolute and relative coordinates describing the position and size of each text block. These enable the model to capture reading order, multi-column layouts, table structures, and header–body relationships crucial for forms, invoices, and complex financial statements.

• Visual embeddings – page-image representations created by splitting the document into fixed-size patches, as in a Vision Transformer[4] (ViT). Each patch is linearly projected into a vector that captures fonts, logos, graphics, shading, and other cues missed by text alone.

All three modalities are fused in a single-stream Transformer encoder with cross-modal self-attention so that tokens, spatial regions, and visual patches can attend to one another early and deeply.

This eliminates the need for separate visual and textual backbones and allows joint reasoning across modalities.

To learn these representations, LayoutLMv3 is pre-trained end-to-end with two complementary objectives:

• Masked Language Modeling (MLM): randomly masks text tokens and predicts the missing words using textual, spatial, and visual context, fostering strong semantic understanding and spatial–text alignment.

• Masked Image Modeling (MIM): masks random image patches and reconstructs them from neighboring patches and text, forcing the network to model fine-grained document textures and cross-modal consistency.

Pretraining on large-scale document datasets such as IIT-CDIP or DocBank lets LayoutLMv3 leverage millions of unlabeled pages, giving it broad generalization.
Once pre-trained, the model can be fine-tuned with lightweight task-specific heads for form understanding and key–value pair extraction, document classification, table or chart structure recognition, or visual question answering (VQA).

In benchmark studies (e.g., FUNSD, CORD, DocVQA), LayoutLMv3 consistently outperforms earlier multimodal models (LayoutLMv1/v2) and traditional OCR-only pipelines, thanks to its unified architecture, early cross-modal fusion, and dual masked-pretraining strategy.
It effectively treats a document as a rich visual–textual object, aligning words, their geometric positions, and their visual context in one deep model.

Although LayoutLMv3 significantly advances document understanding by modeling text, layout, and visual features in a single transformer framework, practical deployments in high-stakes domains such as finance reveal new challenges in accuracy, privacy, and scalability.
These limitations motivate the following analysis of financial-document requirements and the design of an OCR-free solution.

Financial-Document Perspective and Remaining Challenges

• Mission-critical need – Accurate extraction of information from financial document images is essential for auditing, analytics, and regulatory compliance.

• Limitations of standard OCR – Standard OCR systems typically output only plain text, losing structural semantics such as tables, headers, footnotes, and nested layouts.

• Impact of complex layouts – Complex multi-column invoices or deeply nested tables are often misread or merged incorrectly, causing missing or misaligned fields that can lead to serious financial and legal errors.

• Privacy and security concerns – Reliance on cloud-hosted OCR services can expose sensitive financial data to third parties, raising confidentiality and compliance risks.

• Dependence on upstream OCR quality – Because OCR provides the text and coordinates, misrecognitions or reliance on external OCR providers increase the risk of inaccurate extraction and data exposure.

• Strength of LayoutLMv3 – LayoutLMv3 retains and reasons over layout and visual context, enabling structure-aware extraction that is far superior to pure OCR.

• Weaknesses of LayoutLMv3:

⚪︎Depends on high-quality OCR outputs for token text and bounding boxes; errors at this stage propagate downstream.

⚪︎Struggles with handwritten annotations, severely degraded scans, or highly irregular/nested tables, which can lead to misaligned key–value pairs or partial recognition.

⚪︎Large model size and heavy computation may limit deployment on resource-constrained systems.

Addressing these issues requires a paradigm shift from text-only pipelines to end-to-end vision–language models that can generate rich structural representations.
Building on these insights, we propose Doc2HTML, an OCR-free multimodal VLM specifically tailored for Japanese financial documents.

Multimodal OCR with Vision-Language Models

Motivation and Overview

Accurate extraction of structured information from financial document images is critical for auditing, analytics, and regulatory compliance. Conventional OCR-based pipelines often fail in this domain:
they output only plain text and lose structural semantics such as tables, headers, and footnotes;
complex multi-column invoices or deeply nested tables are misread or merged incorrectly; and reliance on cloud-hosted OCR can expose sensitive financial data, creating privacy and compliance risks. Recent advances in multimodal vision–language models (VLMs) provide a way to overcome these limitations by directly reasoning over both visual and textual information.

To fully exploit this capability for financial data, we curated a dedicated dataset of Japanese financial documents—including annual reports, detailed balance sheets, and invoice-like forms—and used it to fine-tune the model with Low-Rank Adaptation (LoRA).This enables domain-specific adaptation while keeping training efficient. The result is Doc2HTML, an OCR-free architecture that learns directly from document pixels to produce structured HTML representations, eliminating error propagation from separate OCR stages and capturing rich spatial and visual cues.

Network Architecture

The Doc2HTML model (Fig. 3) fuses visual encoding, modality alignment, and large language generation into a single pipeline:

• Vision Transformer (ViT) – Visual Backbone

⚪︎Splits each page into fixed-size patches and encodes them as visual tokens.

⚪︎Captures fine-grained layout features such as table borders, columns, fonts, and graphic elements while maintaining global document structure.

⚪︎Supplies rich spatial–visual embeddings as the sole primary input, removing the need for OCR text tokens.

• Projection Layer / Visual Embedding Adapter

⚪︎Maps the ViT[4] output to the token-embedding space of the large language model.

⚪︎Provides a trainable interface for stable cross-modal fusion and preservation of spatial relationships.

• Large Language Model Core (Qwen2.5-3B[2]) with LoRA

⚪︎Performs semantic reasoning and structured generation, translating visual embeddings into precise HTML markup.

⚪︎LoRA modules (⚡) are trainable while most parameters remain frozen (❄), enabling parameter-efficient fine-tuning on the Japanese financial document dataset.

⚪︎Cross-modal attention allows each visual token to interact globally, supporting complex layout interpretation.

Having described the architectural design and inference workflow of Doc2HTML, we next outline the training configuration and fine-tuning strategy that enable the model to achieve state-of-the-art page accuracy on complex financial reports.

Doc2HTML Inference: Settings and Prompt

During inference, Doc2HTML is guided by a structured instruction prompt specifically designed to preserve full document fidelity. The model is configured with a maximum sequence length of 8,192 tokens and a decoding temperature of 0.5 to balance accuracy with flexibility. For training, we adopt a LoRA-based parameter-efficient fine-tuning strategy with settings of r = 16 and lora_alpha = 32, enabling the model to better capture document complexity. To further enhance multimodal understanding, the fine-tuning targets not only the core attention layers (q_proj, k_proj, v_proj, o_proj) and MLP components (gate_proj, up_proj, down_proj), but also key vision-specific normalization layers (ln_1, ln_2) that improve convergence and page-level comprehension.

Input Prompt: “Convert this document image to clean, fully valid HTML regardless of language.

REQUIREMENTS:
- Automatically detect and correctly handle document language (Japanese, English, or mixed).
- Use semantic HTML tags to represent structure (e.g., <h1>, <h2>, <p>, <table>, <tr>, <td>, <ul>, <li>).
- Preserve the visual hierarchy, reading order, and overall page layout.
- Capture and reproduce all text exactly, including Japanese characters (kanji, hiragana, katakana) and any English text.
- Represent tables with proper <table>, <tr>, and <td> markup, supporting both vertical and horizontal cell alignment.
- Ensure that every visible element (headings, paragraphs, footnotes, annotations, headers, footers) is extracted — nothing omitted, summarized, or truncated.
- Output only valid, standards-compliant HTML code with no additional explanations, comments, or natural-language text outside of HTML.

Extract EVERYTHING exactly as shown in the document image."

Advantages of the OCR-Free, LoRA-Tuned Approach

• True structure preservation: Maintains multi-column layouts, nested tables, and typographic details.

• No OCR error propagation: All reasoning occurs directly on pixel-derived embeddings.

• Enhanced privacy and compliance: Eliminates third-party OCR, reducing data-exposure risks.

• Domain specialization: LoRA fine-tuning on the custom Japanese financial dataset ensures high accuracy for local document formats with minimal compute.

Structuring Financial Documents with Doc2HTML for RAG

To highlight how an OCR-free approach improves downstream performance, we replaced the JSON-based Chipper pipeline with our Doc2HTML VLM model, a vision-encoder–decoder architecture inspired by Donut [3].
Whereas Chipper emits a JSON list of page elements and their bounding boxes, Doc2HTML generates a clean, standards-compliant HTML file for every document page.
Each page is expressed with rich semantic tags—headings, paragraphs, tables—together with the natural reading order.
This HTML output is directly consumable by retrieval-augmented generation (RAG) systems, eliminating the need to first flatten the content into plain OCR text.
Because financial reports contain highly structured material (titles, descriptive text, and complex tables), we apply a structural chunking strategy to the HTML output before indexing:

• If an HTML text block is shorter than 2,048 characters, we merge it with the next block when possible, ensuring that chunks remain coherent and no semantic unit is broken.

• This merging is repeated until the desired chunk length is reached.

• When an <h1> or <h2> heading is encountered, a new chunk is started to preserve document hierarchy.

• When a <table> element appears, a new chunk is started and the entire table is preserved intact.

After chunking, we enrich each composite chunk with metadata to support fast and accurate retrieval.
Using predefined GPT-4 prompt templates, we generate:

1. Up to six representative keywords capturing the essence of the chunk.

2. A short summary paragraph describing the key information.

3. A naïve prefix consisting of the first two sentences of the chunk; for tables this is replaced with the table caption or an auto-generated description.

Table1 : Retrieval results. For each chunking strategy, we show the number of chunks
for all the documents

Conclusion:

The study demonstrates that moving beyond traditional OCR and even JSON-based document parsers unlocks a major accuracy gain for financial document understanding.
By training the Doc2HTML VLM model on a custom Japanese financial dataset and fine-tuning it with LoRA, the system generates clean, structured HTML directly from document images.
This HTML output, combined with structural chunking and GPT-4–based metadata, integrates seamlessly into RAG workflows and achieves 94.3 % page accuracy, far surpassing both fixed-size base chunking and Chipper’s JSON aggregation.
In practice, Doc2HTML proves that an OCR-free, multimodal approach preserves layout, improves retrieval quality, and strengthens privacy, setting a new benchmark for complex financial document processing.

Reference:

最新情報をメールで取得

© Couger Inc. All rights reserved.