Developed by IBM Research and Hugging Face, SmolDocling[1] brings a breath of fresh air to the ever-expanding world of vision-language models. Designed for end-to-end multi-modal document conversion, this ultra-compact model punches well above its weight. At just 256M parameters, SmolDocling competes with models over 10x larger, offering accurate, structured document parsing without the bloat of traditional LVLMs (Large Vision-Language Models).

What’s the Architecture Like?

SmolDocling builds upon Hugging Face’s SmolVLM-256M and introduces a novel pipeline to transform document page images into richly tagged textual output using a new markup language: DocTags.

Here's a simplified breakdown of the architecture:

1. Vision Encoder

• Based on SigLIP (patch-16/512) as the visual backbone.
• Each 512×512 patch is compressed into 64 visual tokens using pixel shuffle strategy.
• This keeps things lightweight and efficient, while preserving the essence of visual information.

2. Projection & Pooling

The visual embeddings from the encoder are projected and pooled to align with language input formats.

3. Prompt Integration

• Textual prompts from the user (or instructional data) are tokenized and embedded.
• These are then interleaved with visual embeddings, allowing multi-modal interaction.

4. Language Model Head

• A SmolLM-135M serves as the lightweight LLM decoder.
• It receives the concatenated embeddings and autoregressively generates output in the form of DocTags, which encode both content and structure.

What are DocTags?

Think of DocTags as an XML-inspired, structure-preserving format that captures:
- Content (text, code, formulas, etc.)
- Layout (bounding boxes for positioning)
- Hierarchy (nested structures like lists inside sections, or captions within figures)

DocTags ensure that the document is not only semantically rich but also visually aware.

Tasks Handled by the Model

SmolDocling is trained and evaluated on a variety of document intelligence tasks, making it a versatile tool for document understanding. These include:

• Document Layout Analysis: Identifying structures like headers, tables, figures, lists, and paragraphs.
• Table Recognition: Parsing rows, columns, cell content, and table boundaries.
• Mathematical Formula Transcription: Recognizing inline and block-level LaTeX-style math expressions.
• Code Block Segmentation: Detecting and reproducing syntax-highlighted code snippets.
• Captioned Figure Understanding: Linking visual elements (like charts/figures) with their accompanying captions.

Figure1. DocTags format describes key features of document elements: type of an element

Additional Details About SmolDocling

• Compactness and Efficiency: The entire model is optimized for deployment on devices with limited resources, with <0.5 GB VRAM required and inference time as low as 0.35s per page.
• Tokenizer and Instruction Tuning: SmolDocling uses an instruction-tuned decoder-only language model with 135M parameters, adapted for the DocTag markup language.
• Zero-Shot Capabilities: Thanks to its training on richly annotated datasets, SmolDocling generalizes well across unseen document layouts.
• Training Regime: Pretraining involves freezing the SigLIP vision encoder and gradually unfreezing during instruction tuning to retain stable image representation.
• Output Interpretability: Outputs are interpretable as they preserve semantic and layout information, which can be parsed programmatically.

SmolDocling Result Analysis:

We’ve built a simple interactive application on Google Colab to test out SmolDocling in action.

Please use the below Colab notebook and try the model with your own document images — no setup required!

Figure 2. Interface of the SmolDocling app

For our case study, we utilized various document types, including tables, receipts, and equations. These were processed and converted into structured formats such as LaTeX for mathematical content.

1. Receipt Recognition:

Prompt: “Convert this page to docling“

The model failed to extract meaningful structured data from the receipt. The presence of handwritten text overlapping printed content likely disrupted OCR and tagging. As a result, only generic tags were produced without identifying key fields like items or totals.

Figure 3.  Sample Receipt image testing result

The model was able to accurately extract the text from the receipt, including correctly recognizing the overlapping handwritten text. This demonstrates its capability to handle moderate visual noise.

Figure 4. Sample Receipt image with testing result

Although the model does not officially support Japanese, we tested it on the receipt shown below. Interestingly, it was able to extract the word "FamilyMart," but failed to recognize other key elements such as numbers and the text "NETFLI."

Figure 5. Sample Japanese Receipt

2. Equation to Latex

Prompt: “Convert formula to latex“

The model struggled to recognize the third equation when converting equations to LaTeX format. In cases where the image contains complex equation structures, as shown below, the model consistently failed to detect and extract the third equation, even when tested with a second image of similar complexity.

Figure 6. Sample equation with its converted LaTeX output.

Figure 7. Sample equation with its converted LaTeX output.

The input image contains a series of mathematical equations with complex notation and structure. While the model successfully extracted the initial part of the equation and converted it into LaTeX, it failed to capture the complete content—especially the lower portion involving conditional expressions. As a result, the output is truncated and filled with redundant backslashes, indicating parsing failure.

Figure 8. Complex mathematical equations with partially extracted LaTeX output.

When individual equations are cropped from the image and passed to the model, the output quality improves significantly. This suggests that the model performs better with isolated equation inputs rather than complex, multi-line structures.

Figure 9. Accurate equation conversion achieved through isolated input of cropped equations. 

The input image contains a complex integral equation involving nested fractions and powers. The model was able to extract partial LaTeX output but failed to capture the full structure accurately, resulting in a fragmented expression. This highlights the model's limitations in handling deeply nested mathematical components.

Figure 10. Complex integral equation with partially extracted LaTeX output. 

3. Table Recognition:

Prompt: “Convert this table to OTSL.“

In Figure 11, the model attempts to extract two financial tables—Table 1 (Cash Book) and Table 2 (Bank Statement)—with moderate success. While much of the content such as dates, lodgements, and cheque amounts is accurately captured, several formatting and structural challenges remain. Table 1, which lacks visible borders and spacing between columns, presents a significant challenge for automated extraction. As a result, certain data fields blend together, and the "Dr." (Debit) and "Cr." (Credit) sections are not visually separated. This ambiguity can lead to confusion in classifying transactions and interpreting the ledger correctly. Table 2 fares slightly better, with the model preserving its overall layout and capturing key column headers like "Date," "Details," "Debit (€)," "Credit (€)," and "Balance (€)." Despite this, alignment inconsistencies are noticeable—some debit and credit values are misaligned, and entries such as “Bank charges” and “Bank interest” may be prone to misinterpretation due to inconsistent formatting.

Overall, while the model demonstrates reasonable effectiveness in extracting tabular data, manual adjustments are essential to ensure clarity, improve column mapping, and enhance border visibility. These refinements are critical for accurate financial analysis and downstream processing in real-world applications.

Figure 11. Comparison of input document and model-generated output for cash book and bank statement tables.

The model successfully extracted most of the table content, including dates, descriptions, and values. However, there is column misalignment in Figure 12— the "Amount" and "Balance" values are partially shifted into the wrong columns. Additionally, parts of the description (e.g., card number and transaction date) were incorrectly placed under the "Balance" column. While the overall structure is preserved, errors in semantic grouping and cell splitting reduce clarity and data accuracy.

Figure 12. Extraction results showing column misalignment and misplaced data in a financial transaction table.

The extracted table in the result figure 13 shows misaligned columns and incorrect cell merging due to the absence of visible borders in the original document. model struggled to distinguish individual columns, especially where numeric values (e.g., withdrawals, deposits, balance) were closely spaced. As a result, structured table recognition failed, highlighting the model's reliance on visual separators for accurate parsing.

Figure 13. Output of the model on a chequing account statement without visible table borders.

Figure 14 highlights a common challenge encountered by table extraction models when processing structured data that includes empty cells. These empty cells introduce ambiguity, often resulting in row misalignment and incorrect data grouping. To enhance extraction accuracy, future systems should leverage structural cues such as layout heuristics, bounding box continuity, and explicit row and column alignment techniques.

Figure 14. Impact of Empty Cells on Table Structure Recognition by Extraction Models

The main reason for the accurate results in Figure 15 is that all table cells are consistently filled, and the rows and columns are clearly separated. Additionally, the table does not contain multi-level headers or split header rows, reducing structural complexity. This clean and uniform layout enables the model to parse and align data with high precision.

Figure 15. Accurate Table Extraction from Cleanly Structured Multi-Section Data

Demo Video:

Conclusion

SmolDocling exhibits a versatile yet nuanced performance across a range of document understanding tasks. In the case of receipt recognition, the model encountered difficulties when handwritten annotations overlapped printed text, leading to incomplete tagging and omission of essential details like itemized entries and totals. Although it handled moderate visual noise effectively, its performance dropped when tested on non-English documents such as Japanese receipts, capturing only partial information. For mathematical content, the model struggled with complex or multi-line equations, often generating truncated LaTeX outputs. Notably, accuracy improved when equations were cropped and submitted as isolated inputs, indicating a preference for simpler, well-defined structures. Table extraction yielded the most consistent results when the input featured clean layouts—clearly separated rows and columns, fully populated cells, and no multi-tiered headers. In contrast, tables lacking visible borders or containing empty cells introduced structural ambiguity, resulting in misaligned or incorrectly grouped data. Overall, SmolDocling delivers impressive results on clean, well-formatted documents and holds promise for lightweight deployment, though improvements are needed for more complex or noisy layouts.

References

[1] SmolDocling: An Ultra-Compact Vision-Language Model for End-to-End Multi-Modal Document Conversion, IBM Research & Hugging Face. arXiv:2503.11576v1
[2] Sample document images used for testing were sourced from Google Image Search.

最新情報をメールで取得

© Couger Inc. All rights reserved.