How a PDF File Works

A PDF file is not a stream that is read from beginning to end. It is a collection of numbered objects, plus an index that says where each object starts and a pointer to the root of the document. A reader finds the index by reading the end of the file, which is why a PDF that has been truncated is usually reported as damaged even though most of its bytes are present.

A classic file has four parts, in this order:

The body is built from eight kinds of objects:

An indirect reference is written 5 0 R and stands for "the object numbered 5, generation 0". References are what turn a pile of objects into a graph: a page does not contain its content stream, it refers to it, and ten pages can refer to the same font. Only indirect objects can be referenced, and streams must always be indirect. The generation number is a relic of the original design, where a deleted object’s number could be reused with a higher generation. In practice it is 0 for every object, and 65535 for the head of the free list.

Comments start with % and run to the end of the line, which is why a hand-written file is easy to annotate.

The smallest useful file has a catalog, a one-page page tree, a page, a content stream and a font. Nothing else is required: the base 14 fonts, such as Helvetica, can be referenced by name without being embedded in files that do not need to conform to an archival standard.

Official documentation: the file syntax of clause 7 of ISO 32000-2, available free of charge from the PDF Association sponsored standards page.

Each xref entry is exactly 20 bytes: a ten-digit offset, a five-digit generation, the letter n (in use) or f (free) and a two-byte line end. 0 6 says that the subsection starts at object 0 and has six entries. Entry 0 is always the head of the free list. If any offset is wrong by a byte, strict readers reject the file and lenient readers fall back to scanning the file for obj markers and rebuilding the table. Both iText and PDFBox do this, which is why damaged files often open with a log message and not an exception.

Because the offsets are counted in bytes, a file like this is not written by hand in practice. The following program writes it and computes the offsets. It uses a text block for the page dictionary, formatted for the length and a list of objects to number them.

Official documentation: JEP 378, text blocks and the file structure of clause 7 of ISO 32000-2, available from the PDF Association sponsored standards page.

It has no dependencies. Compile and run it with javac MinimalPdf.java and java MinimalPdf minimal.pdf, then open the result in any viewer. The syntax it produces is the one defined in clause 7 of ISO 32000, see the references. The same file is used by the inspection programs below.

A reader never starts at the first byte. It does this:

This is why a viewer can show the first page of a ten-thousand-page file without reading the rest, and why a file served over HTTP with range requests only needs its last kilobyte and the objects of the page that is shown. It is also why the end of the file is the most important part of it.

Pages are not stored as a list. The catalog’s /Pages entry points to the root of a tree of /Pages nodes whose leaves are /Page dictionaries. Each node has /Kids (children) and /Count (the number of leaf pages below it), so a reader can find page 5000 by adding counts and descending, without visiting every page. A /Page points back with /Parent.

Some page attributes are inherited: /Resources, /MediaBox, /CropBox and /Rotate can be set on any /Pages node and apply to every page below it that does not override them. This is why a page that looks like it has no size or no fonts can still render correctly, and why code that reads /MediaBox directly from a page dictionary can get null. The libraries below resolve inheritance in their page classes, PdfPage.getMediaBox() in iText and PDPage.getMediaBox() in PDFBox.

/MediaBox is [llx lly urx ury] in default user space units of 1/72 inch: [0 0 595 842] is A4. The origin is the lower left corner and the y axis points up, which is the opposite of most screen coordinate systems. The iText knowledge base explains this origin and how rotation and crop boxes change what is visible, see the references.

What is drawn on a page is described by a content stream, a stream whose bytes are a sequence of operands followed by an operator, in postfix notation: 72 700 Td moves the text position to x = 72, y = 700. There are no variables and no control flow. The main families of operators are:

Content streams refer to fonts, images and graphics states by name, and the names are looked up in the page’s /Resources dictionary: /F1 24 Tf means "use the font that the resources call F1, at 24 points". The same indirection lets many pages share one font object. A page’s /Contents can be one stream or an array of streams that are concatenated before parsing, which is how iText and PDFBox add content to an existing page without rewriting what is already there.

Higher-level libraries do not expose the operators directly. iText’s PdfCanvas is a thin wrapper that emits them, and the layout module (Document, Paragraph, Table) produces calls on it. Text in a PDF is positioned glyph by glyph: there are no paragraphs, no line breaks and no reading order in the content stream, which is why text extraction is a heuristic and why accessible documents need the separate structure tree described on the PDF/A and PDF/UA pages of this section.

A /Resources dictionary groups the named things a page can use: /Font, /XObject (images and reusable forms), /ExtGState, /ColorSpace, /Pattern, /Shading and /Properties (optional content layers). Fonts are the largest of these in real files. A font dictionary names the font and its type (/Type1, /TrueType, or /Type0 for composite fonts used with large character sets) and, when the font is embedded, refers to a font descriptor whose /FontFile2 or /FontFile3 entry is a stream with the font program. An embedded subset has a name with a six-letter tag, such as ABCDEF+DejaVuSans, and a /ToUnicode CMap that tells extraction tools what each glyph means. The base 14 fonts used in the minimal example are the only ones a reader must supply itself, and archival standards forbid relying on them. Fonts and encodings have their own pages later in this section.

The kernel module of iText 9.8 exposes the object model almost one-to-one. The Maven coordinates are the itext-core aggregator POM, which brings the kernel, layout, forms, signing and other modules. The programs on this page neither sign nor encrypt, so they need no Bouncy Castle adapter. iText logs through SLF4J: without a binding on the classpath its log messages are discarded, so add one, such as slf4j-simple, to see the warnings that a lenient reader prints when it repairs a damaged file.

Official documentation: itext-core on Maven Central and the iText Knowledge Base.

The first program opens a file, prints the trailer, the catalog and the type of every indirect object, and then walks the page tree from the catalog down to the first page.

Official documentation: PdfDocument and PdfReader in the iText 9.8.0 API reference.

For minimal.pdf it reports six cross-reference entries (object 0 plus five objects), the trailer keys /Root and /Size, the catalog <</Pages 2 0 R /Type /Catalog >>, a PdfStream for object 4 and dictionaries for the others, and the page keys /Contents, /MediaBox, /Parent, /Resources and /Type. This program derives from the iText API documentation of PdfDocument, PdfReader and PdfDictionary, see the references.

The next program decodes the content stream of the first page and prints each operator with its operands, using PdfCanvasParser. It stores them in a record, a Java 16 feature that suits a read-only pair of values.

Official documentation: PdfCanvasParser in the iText 9.8.0 API reference.

For minimal.pdf it prints BT, Tf with /F1 and 24, Td with 72 and 700, Tj with the string, and ET. The PdfCanvasParser class is documented in the iText API reference. For rendering events, such as text chunks with their positions, the PdfCanvasProcessor class and its event listeners are the higher-level route, described on the text extraction page of this section.

Apache PDFBox 3.0.8 has two layers. The cos package is the object model that mirrors the file, with COSDictionary, COSArray, COSStream, COSName and COSObject, and the pdmodel package wraps it in classes such as PDDocument, PDPage and PDFont. Files are opened with the static Loader.loadPDF since 3.0, which replaced the PDDocument.load of PDFBox 2.

Official documentation: getting started with PDFBox 3.0.

Official documentation: PDDocument, PDFStreamParser and COSDocument in the PDFBox 3.0.8 Javadoc.

It prints the trailer keys, every cross-reference entry with its offset, the media box of every page, the operators and operands of the first page’s content stream (PDFStreamParser returns operands as separate tokens before each Operator) and the font dictionary of object 5. PDFBox keeps the xref table in COSDocument and flattens the page tree for you through PDDocument.getPages(). The PDFStreamParser and COSDocument classes are documented in the PDFBox Javadoc, linked in the references, and the PDFBox examples directory shows the higher-level uses.

When a file has to be read by eye, tools can rewrite it in a form that is easy to read and edit: the qpdf documentation describes QDF mode, which decompresses streams and annotates each object with its number. It is a debugging aid and not a conformance mechanism.

Raw content streams are text-like and compress well. A stream’s /Filter entry names the encoding applied to its bytes, or an array of names applied in order, with optional /DecodeParms. The ones that matter in practice are:

Stream compression leaves the structure alone: every object is still written in the clear, with its n g obj header and an entry in a plain-text xref table. PDF 1.5 added a second layer of compression:

Object streams (/Type /ObjStm) pack many small objects, such as dictionaries and arrays, into one compressed stream. Their offsets are listed at the start of the stream, with /N giving the object count and /First the offset of the first object. Stream objects and the encryption dictionary cannot be placed in one. Cross-reference streams (/Type /XRef) replace the plain-text table with a compressed stream of binary rows whose field widths are given by /W, and whose entries can say that an object lives at index i of object stream n. The stream dictionary also takes over the role of the trailer, with /Size, /Root, /Info and /ID. Together they typically reduce the size of a file with many small objects, such as a tagged or form-heavy document, by a good fraction, at the cost of files that cannot be read by PDF 1.4 readers.

iText controls this through WriterProperties. The program below writes the same twenty pages three ways and prints the sizes.

Official documentation: WriterProperties and CompressionConstants in the iText 9.8.0 API reference.

In one run with iText 9.8.0 the files were 31522 bytes with no compression, 11221 bytes with Flate on streams (the default) and 8282 bytes with full compression mode, which writes object streams and a cross-reference stream. The exact numbers depend on the content. The page derives from the WriterProperties and CompressionConstants API documentation.

PDFBox 3 writes object streams by default: save without a second argument uses CompressParameters.DEFAULT_COMPRESSION, and a classic xref table needs CompressParameters.NO_COMPRESSION. The program below saves the same document both ways:

Official documentation: CompressParameters in the PDFBox 3.0.8 Javadoc and the migration guide to PDFBox 3.0.

On the small minimal.pdf the output went from 844 bytes with a classic table to 758 bytes with object and cross-reference streams. For a one-page file the gain is small, and the gain grows with the number of small objects. The CompressParameters constants are documented in the PDFBox Javadoc.

A PDF can be changed without rewriting it. An incremental update appends to the end of the file:

Nothing before the appended bytes changes. A reader starts from the last startxref, finds the newest version of an object in the newest section, and follows /Prev for the objects that were not changed. Deleting an object marks its entry free in a new section.

Running the iText program below on minimal.pdf writes an updated file whose first 601 bytes are identical to the original. useAppendMode on StampingProperties makes PdfDocument write only what changed.

Official documentation: StampingProperties in the iText 9.8.0 API reference.

The appended section it produced was the following. It contains the one new object, an xref subsection with a single entry, and a trailer with /Prev 418, the offset of the original xref section. The identifier strings and the date and producer values are abbreviated here.

Official documentation: incremental updates are described in clause 7.5.6 of ISO 32000-2, available from the PDF Association sponsored standards page.

PDFBox does the same with saveIncremental, and requires you to mark the objects that have changed, because it only writes objects flagged as needing an update:

Official documentation: PDDocument.saveIncremental in the PDFBox 3.0.8 Javadoc and the PDFBox signing example.

The iText stamping classes and StampingProperties are described in the API documentation. PDFBox documents saveIncremental in the Javadoc and its signing examples use it.

A linearized file, also called Fast Web View, is a PDF whose objects are arranged so that the first page can be displayed before the rest of the file has been downloaded. It starts with a linearization parameter dictionary as the first object after the header, with entries such as /Linearized (the version), /L (the file length), /O (the number of the first page’s object), /N (the number of pages) and /H (the hint stream location), followed by a first-page cross-reference section and the first page’s objects, with the rest of the file after them. A viewer reading from a server issues a range request for the start, learns the file length and the first page’s extent, and renders the page while the rest streams. The mechanism is described in the standard’s annex on linearized PDF.

Most documents produced on a server are not linearized, and for generated files that are mailed or downloaded as a whole it usually does not matter. It matters for large files that people read in a browser from a remote server: long reports, catalogues and scanned books. Neither iText nor PDFBox writes linearized files. The usual practice is to post-process with qpdf (qpdf --linearize in.pdf out.pdf), which is also able to check the result. Linearization uses a layout in which incremental updates do not fit well: appending changes the end of the file, so a file edited that way is no longer linearized in the strict sense, although most viewers keep working with it.

You can tell whether a file is linearized by looking at its start:

Official documentation: qpdf, linearization.

All the examples of this page, and the examples of every page of this section, are written in current Java. The books consulted for this section are about Java 1.3 to 5 and iText 1.x, and predate generics, try-with-resources, var, text blocks, records and pattern matching. In their code, streams are closed in finally blocks, collections are raw, and loops use index variables. The examples here use the Java 21 language instead:

The language changes are summarised in the Oracle documentation for Java 21 and specified in JEP 378 (text blocks), JEP 395 (records) and JEP 394 (pattern matching for instanceof).