Document Parsing

This content was generated with the assistance of AI and should be verified against the official documentation of each tool and framework before being relied on in production.

This section’s bibliography lists the reference material consulted while preparing these pages.

This page was written on 2026-09-30 against pypdf 6.19.0, PyMuPDF 1.28.2, Docling 2.131.0 (with docling-core 2.99.0), Unstructured 0.27.10 and langchain-docling 3.0.0 on Python 3.12, Spring AI 2.0.1 (spring-ai-tika-document-reader, spring-ai-pdf-document-reader) and LangChain4j 1.20.2 with its Tika and PDFBox parsers 1.20.2-beta30 (see the version baseline). The pypdf ladder step and the quality check were run on generated PDFs; the Docling and Unstructured calls were checked against the released sources of those packages but not run, because their layout models were not downloaded; the Java classes were compiled against the released jars.

The loaders in Document Ingestion & Chunking assume the text is already there: Markdown, AsciiDoc, clean HTML. Real corpora are not like that. DocsAssistant’s second corpus is a folder of PDFs — release notes exported from a word processor, a scanned architecture review, slide decks saved as PDF, a specification with multi-page tables. Retrieval quality is capped by parsing quality: a chunk that interleaves two columns, drops a table or repeats a page footer fifty times cannot be rescued by a better embedding model. This page is about getting the text — and its structure — out correctly and cheaply.

What is actually inside a PDF

A PDF page is a drawing program, not a text document. Its content stream contains operators that select a font, move to coordinates and paint glyphs: BT /F1 11 Tf 72 700 Td (Revenue) Tj ET paints the glyphs for "Revenue" at (72, 700). Several consequences matter for parsing:

  • There are no words, lines or paragraphs, only positioned glyphs. A parser reconstructs words from gaps, lines from baselines and reading order from geometry. Two-column layouts, sidebars and footnotes are where naive extraction goes wrong.

  • Glyphs are not always characters. The font’s ToUnicode map translates glyph codes back to Unicode; if it is missing or wrong (common with subsetted or custom fonts), extraction produces garbage or private-use characters even though the page looks perfect.

  • Tables are just lines and text boxes. A table’s structure exists only visually; detecting cells requires layout analysis.

  • Scanned pages contain an image and no text at all, unless OCR was already run and an invisible text layer added.

  • Tagged PDF (PDF/UA) does carry a structure tree of headings, paragraphs and tables, but most PDFs in the wild are untagged.

The parser ladder

The practical strategy is a ladder: try the cheapest parser that can succeed, check the result, and climb only for the documents that fail. Most pages of most corpora are born-digital text that a fast parser handles in milliseconds; a vision-language model (VLM) costs orders of magnitude more per page and should see only what nothing else could read.

Rung Tools Good at Cost per page

1. Text layer

pypdf, PyMuPDF (pymupdf), pdfminer.six

Born-digital, single-column text.

Milliseconds, CPU.

2. Layout-aware

Docling, Unstructured (hi_res)

Reading order, headings, tables, figures, multi-column pages; OCR for scanned regions.

Tenths of a second to seconds; CPU, faster on GPU.

3. Cloud document AI

Amazon Textract, Azure AI Document Intelligence, Google Document AI, LlamaParse

Forms, key-value pairs, complex and handwritten tables at scale.

Priced per page; data leaves your boundary.

4. VLM fallback

A vision-language model prompted to transcribe the page to Markdown

Anything a human can read, including charts and odd layouts.

Highest; slow; can hallucinate text, so keep it last and verify.

A ladder needs a quality gate between rungs. Cheap signals catch most failures: too few characters per page (a scan), a high share of non-printable or private-use characters (a broken ToUnicode map), very long "words" (lost spaces) or a page that repeats the previous one. The gate below is the first rung of DocsAssistant’s parser; it returns None when the page should climb:

docsassistant/parsing/ladder.py — rung 1 with a quality gate (Python, pypdf 6.19.0)
import unicodedata

from pypdf import PdfReader


def text_quality(text: str) -> float:
    """Share of characters that are ordinary letters, digits, punctuation or whitespace."""
    if not text:
        return 0.0
    ok = sum(1 for ch in text if ch.isspace() or unicodedata.category(ch)[0] in "LNPS")
    return ok / len(text)


def looks_parsed(text: str, min_chars: int = 200, min_quality: float = 0.95, max_word: int = 40) -> bool:
    words = text.split()
    return (len(text) >= min_chars
            and text_quality(text) >= min_quality
            and not any(len(w) > max_word for w in words))


def rung1_pages(path: str) -> list[str | None]:
    """Extract each page's text layer; None marks a page that must climb the ladder."""
    reader = PdfReader(path)
    pages: list[str | None] = []
    for page in reader.pages:
        text = page.extract_text(extraction_mode="layout") or ""
        pages.append(text if looks_parsed(text) else None)
    return pages

Source: pypdf — Extract Text from a PDF.

PyMuPDF is an alternative first rung that is faster on large files and returns spans with font size and flags, which is enough to detect headings by font size without a layout model (page.get_text("dict")); see PyMuPDF — Text recipes.

Rung 2 with Docling

Docling converts PDF, DOCX, PPTX, HTML, AsciiDoc and images into a DoclingDocument: a structured model of headings, paragraphs, lists, tables and pictures with their page provenance and bounding boxes. Its PDF pipeline runs a layout model, a table-structure model and, when a page has no usable text layer, OCR. The pipeline is configured through PdfPipelineOptions:

docsassistant/parsing/docling_rung.py — layout-aware parsing with OCR (Python, Docling 2.131.0)
from docling.datamodel.base_models import InputFormat
from docling.datamodel.pipeline_options import PdfPipelineOptions, TesseractCliOcrOptions
from docling.document_converter import DocumentConverter, PdfFormatOption

options = PdfPipelineOptions(
    do_ocr=True,                                          # OCR only where the text layer is missing
    ocr_options=TesseractCliOcrOptions(lang=["eng", "spa"]),
    do_table_structure=True,                              # recover rows, columns and spans
    generate_picture_images=True,                         # keep figure crops for multimodal indexing
    images_scale=2.0,
)
converter = DocumentConverter(format_options={InputFormat.PDF: PdfFormatOption(pipeline_options=options)})


def parse_with_docling(path: str):
    result = converter.convert(path)
    return result.document                                # a DoclingDocument


doc = parse_with_docling("architecture-review.pdf")
markdown = doc.export_to_markdown()                       # headings as #, tables as Markdown tables

Source: Docling — Usage and Docling — Custom conversion.

Force OCR on every page, even where a text layer exists, when the text layer is known to be bad (a scanner that embedded junk text): set mode=OcrMode.FULL_PAGE on the OCR options. In 2.131.0 the older force_full_page_ocr=True keyword is still accepted but deprecated in favour of mode.

The LangChain integration wraps the converter and Docling’s structure-aware HybridChunker, so each LangChain Document is a chunk that respects headings and tables and carries its headings and page numbers in metadata["dl_meta"]:

docsassistant/parsing/docling_loader.py — Docling chunks as LangChain documents (Python, langchain-docling 3.0.0)
from docling.chunking import HybridChunker
from langchain_docling import DoclingLoader
from langchain_docling.loader import ExportType

loader = DoclingLoader(
    file_path=["architecture-review.pdf", "release-notes-4.2.pdf"],
    export_type=ExportType.DOC_CHUNKS,
    chunker=HybridChunker(max_tokens=512),
)
chunks = loader.load()
for chunk in chunks[:3]:
    meta = chunk.metadata["dl_meta"]
    print(meta["headings"], [item["prov"][0]["page_no"] for item in meta["doc_items"]])

Source: LangChain — Docling and Docling — Chunking.

Rung 2 with Unstructured

Unstructured partitions a document into typed elements (Title, NarrativeText, ListItem, Table, Image, Header, Footer) and chunks them by title. The strategy argument is the ladder in miniature: "fast" uses the text layer, "hi_res" runs a layout model (and OCR where needed), "ocr_only" ignores the text layer, and "auto" chooses per document:

docsassistant/parsing/unstructured_rung.py — partition, drop furniture, chunk by title (Python, Unstructured 0.27.10)
from unstructured.chunking.title import chunk_by_title
from unstructured.partition.pdf import partition_pdf

elements = partition_pdf(
    filename="release-notes-4.2.pdf",
    strategy="hi_res",
    infer_table_structure=True,          # tables keep an HTML rendering in metadata.text_as_html
    languages=["eng"],
)
body = [el for el in elements if el.category not in ("Header", "Footer", "PageNumber")]
chunks = chunk_by_title(body, max_characters=1500, combine_text_under_n_chars=200)
for chunk in chunks[:3]:
    print(chunk.metadata.page_number, chunk.text[:80])

Source: Unstructured — Partitioning and Unstructured — Chunking.

Rungs 3 and 4

Cloud document AI services (Amazon Textract, Azure AI Document Intelligence’s layout model, Google Document AI) return blocks, tables and key-value pairs with confidence scores and bounding boxes. They are the right rung for forms and invoices at volume, and the wrong one for documents that must not leave your data boundary; see Access Control & Privacy. The VLM rung renders the page to an image and asks a vision-capable model to transcribe it; because a model can "correct" or invent text, keep the page image and link it as a visual citation (see Multimodal RAG), and mark such chunks with parser="vlm" so that evaluation can track them separately. On Azure, the layout-to-Markdown path is covered on Document Intelligence and its RAG ingestion page, and LLM-based field extraction on Content Understanding.

OCR with Tesseract

Tesseract is the open-source OCR engine most pipelines fall back to, directly or through Docling and Unstructured. Three settings decide most of its quality: the language packs (-l eng+spa; install the matching tessdata), the page segmentation mode (--psm 3 for automatic layout, --psm 6 for a single uniform block) and the resolution of the rendered image (300 DPI is the usual floor; below 200 DPI accuracy drops sharply). Pre-processing — deskewing, removing noise, binarising — often helps more than switching engines. Tesseract returns word-level confidences; keep the page’s mean confidence in the chunk metadata and route low-confidence pages to the next rung or to human review. See Tesseract documentation and Improving the quality of the output.

Keep structure as metadata

The structure a parser recovers is worth more than the text alone. DocsAssistant stores it as chunk metadata so that it can be used for filtering, boosting and citations:

Metadata From Used for

source, page, bbox

Docling provenance, Unstructured page_number and coordinates

Citations that open the right page and highlight the passage (Citations & Grounding).

headings (breadcrumb)

Docling headings, Unstructured Title elements

Prepending Section: Security  Tokens to the chunk text before embedding, and showing it in results.

element_type

Table, ListItem, Title

Routing tables to table-aware answering; filtering out boilerplate.

html_tag, speaker, channel

HTML tags, chat exports

Keeping who said what in chat logs; filtering by channel.

parser, ocr_confidence

The ladder

Evaluating and re-parsing weak documents later.

acl, content_hash

Ingestion

Access control and incremental updates (Ingestion Pipelines & Freshness).

The filter syntax that uses this metadata at query time is on Metadata Filtering.

The Java side

Neither Spring AI nor LangChain4j ships a layout model. On the JVM the first two rungs are Apache Tika (which detects the format and extracts text from more than a thousand file types, including PDF through PDFBox and Office formats) and PDFBox directly; the higher rungs are usually a Docling or Unstructured service called over HTTP, or a cloud document AI API.

The RAG & LLM Ingestion page of the Apache Tika section covers Tika’s side of this rung: Markdown output, embedded documents, metadata and inference vectors.

Spring AI’s PagePdfDocumentReader produces one Document per page (or per group of pages) with the page numbers in metadata, and trims running headers and footers with an ExtractedTextFormatter. ParagraphPdfDocumentReader uses the PDF’s outline (bookmarks) to split by section and fails on PDFs without one, so the reader below falls back to pages. TikaDocumentReader handles everything else:

DocumentParsingService.java — PDF by outline or by page, everything else by Tika (Java, Spring AI 2.0.1)
package com.example.docsassistant.rag.parsing;

import java.util.List;

import org.springframework.ai.document.Document;
import org.springframework.ai.reader.ExtractedTextFormatter;
import org.springframework.ai.reader.pdf.PagePdfDocumentReader;
import org.springframework.ai.reader.pdf.ParagraphPdfDocumentReader;
import org.springframework.ai.reader.pdf.config.PdfDocumentReaderConfig;
import org.springframework.ai.reader.tika.TikaDocumentReader;
import org.springframework.core.io.Resource;
import org.springframework.stereotype.Service;

@Service
public class DocumentParsingService {

    private final PdfDocumentReaderConfig pdfConfig = PdfDocumentReaderConfig.builder()
            .withPageTopMargin(0)
            .withPageExtractedTextFormatter(new ExtractedTextFormatter.Builder()
                    .withNumberOfTopTextLinesToDelete(1)      // running header
                    .withNumberOfBottomTextLinesToDelete(1)   // page number footer
                    .build())
            .withPagesPerDocument(1)
            .build();

    public List<Document> parse(Resource resource) {
        String name = resource.getFilename() == null ? "" : resource.getFilename().toLowerCase();
        if (!name.endsWith(".pdf")) {
            return new TikaDocumentReader(resource).get();
        }
        try {
            return new ParagraphPdfDocumentReader(resource, pdfConfig).get();   // needs a PDF outline
        } catch (IllegalArgumentException noOutline) {
            return new PagePdfDocumentReader(resource, pdfConfig).get();
        }
    }
}

Source: Spring AI — ETL Pipeline (PDF Page, PDF Paragraph and Tika readers).

LangChain4j separates loading (from the file system, a URL, S3 or Azure Blob Storage) from parsing. The ApacheTikaDocumentParser detects the format; the ApachePdfBoxDocumentParser is lighter for PDF-only corpora. Passing true to either constructor copies the document metadata (title, author, dates) into the LangChain4j Metadata:

JvmDocumentParsing.java — Tika for mixed formats, PDFBox for PDFs (Java, LangChain4j 1.20.2)
package com.example.docsassistant.l4j.parsing;

import java.nio.file.FileSystems;
import java.nio.file.Path;
import java.nio.file.PathMatcher;
import java.util.List;

import dev.langchain4j.data.document.Document;
import dev.langchain4j.data.document.loader.FileSystemDocumentLoader;
import dev.langchain4j.data.document.parser.apache.pdfbox.ApachePdfBoxDocumentParser;
import dev.langchain4j.data.document.parser.apache.tika.ApacheTikaDocumentParser;

public final class JvmDocumentParsing {

    private JvmDocumentParsing() {
    }

    public static List<Document> loadMixed(Path folder) {
        return FileSystemDocumentLoader.loadDocumentsRecursively(folder, new ApacheTikaDocumentParser(true));
    }

    public static List<Document> loadPdfs(Path folder) {
        PathMatcher pdfs = FileSystems.getDefault().getPathMatcher("glob:**.pdf");
        return FileSystemDocumentLoader.loadDocumentsRecursively(folder, pdfs, new ApachePdfBoxDocumentParser(true));
    }
}

Source: LangChain4j — RAG (Document Loader, Document Parser) and LangChain4j — Apache Tika.

Tika and PDFBox recover the text layer and basic metadata but not reading order across columns, table structure or OCR (Tika can call Tesseract when it is installed and enabled in the Tika configuration). For documents that need rung 2 or above, run Docling as a service — the docling-serve project exposes the converter over HTTP — and have the Java ingestion call it, storing its Markdown or JSON output as the parsed document.

Choosing parsers per document, not per corpus

The ladder is applied per page, and the choice is recorded. That makes three operational practices possible:

  • Measure the ladder. The share of pages that reach each rung, and the cost per rung, go on the ingestion dashboard (Observability & Tracing); a sudden rise in rung-4 pages usually means a new document source with a broken export.

  • Re-parse selectively. When a better parser version lands, re-parse only the pages that reached rung 2 or higher, identified by parser metadata, and let content hashes skip re-embedding unchanged chunks.

  • Evaluate by parser. Break retrieval metrics down by parser; if VLM-parsed chunks are rarely retrieved or often wrong, the problem is upstream of retrieval.

The concepts of the parser ladder and "the cheapest parser that succeeds first" are credited to Mendelevitch and Bao, Hands-On RAG for Production (Chapter 2), and the loaders and OCR recipes to Polzer, RAG with Python Cookbook (Chapter 3), both paraphrased; see the bibliography.

References