A PDF Is Not a Document

And why getting information back out of one is harder than it looks

Almost everyone who works with documents has, at some point, been surprised by a PDF. Text that cannot be selected. A copied paragraph that pastes as gibberish. A table that arrives in a spreadsheet as a single column. An extraction tool that handles four hundred files perfectly and mangles the four hundred and first.

Enough people have quietly relabelled the initials Pretty Damn Frustrating for this to count as a shared experience rather than a personal failing.

These are not unrelated annoyances. They are all the same fact, showing up in different places.

This post is about that fact. It covers what a PDF contains, what it does not contain, and the specific mechanisms by which information that is obvious to a human reader fails to survive the trip into software, including into a model. It stops there deliberately. What to do about it is a different subject.

Two levels throughout: the main text needs no prior knowledge of the format, and the boxed notes are for people who will end up reading the specification.

1. It is a page-description language

A PDF is not a document format in the sense that a word processor file or a database record is a document format. It is a set of instructions for painting a page.

Put this glyph here. Set this font. Move to these coordinates. Draw a line from here to there. Place this image in this rectangle.

That is close to the whole of it.

The format was designed in 1993 to solve a problem that had nothing to do with information retrieval: making a document print identically on any device, anywhere, forever. It is extremely good at that. Fidelity of appearance was the entire brief, and the design succeeded so completely that we now use it for things it was never asked to do.

Everything a reader recognises as structure — this is a heading, these are numbered steps, this caption belongs to that diagram, this number is a torque value and not a page number — is a visual convention. The reader decodes it from size, weight, position and whitespace. The file itself is usually silent on all of it.

Under the hood
Page content lives in a content stream: a sequence of operators in a PostScript-derived syntax. Text is bracketed by BT/ET, the font set with Tf, the text matrix with Tm or Td, and glyphs painted with Tj or TJ. Coordinates are in points, origin at the bottom left of the page, modified by whatever transformation matrix is in effect. MediaBox and CropBox may disagree, so the visible page is not necessarily the page the coordinates refer to.

2. There are no words

The first thing people assume a PDF contains is text. It contains something weaker: positioned glyphs.

A visible line like Torque the cap to 37.5 Nm. is very often not stored as that string. It is stored as several separate painting operations, each with its own coordinates, and the spaces between words may not exist as characters at all. The space is a gap. A measurement.

So software has to decide where words begin and end by comparing the horizontal gap between fragments against the current font size. Set the threshold slightly too wide and words fuse together. Slightly too narrow and words split apart. Justified text, condensed table columns and generous letter-spacing all sit close to that threshold, which is why extraction quality varies within a single document.

The same applies vertically. Whether two fragments are on the same line, or on consecutive lines, or in different columns, is decided by comparing baselines and gaps.

Under the hood
TJ takes an array mixing strings with kerning adjustments in thousandths of an em, so a single word routinely arrives as several fragments. Horizontal scaling (Tz), character spacing (Tc) and word spacing (Tw) all change what a given gap means, and a generator may use Tw on a real space character or may just move the cursor. Both produce the same page.

There is a second-order version of this worth knowing about. Converters that successfully identify a run of text as a list will often normalise it into list markup, and normalising a list discards the marker, because in a structured document the marker is presentation rather than content. That is correct behaviour for a converter and destructive for a document where the numbers are the content: figure legends keyed to callouts on a diagram, or steps referenced by number elsewhere in the manual. The tool that understood the page best is sometimes the one that dropped the detail you needed.

3. There is no reading order

Nothing in an ordinary PDF states the order in which a human should read the page. What exists is the order in which things were drawn, and that is entirely up to the producing software.

Generators often draw a footer first, or a running header last, or one column and then the other, or all the body text and then all the captions. None of this affects the printed result. All of it affects extraction.

Extraction tools recover reading order by sorting fragments by position, which works well for a single column of body text and progressively worse for anything else: multi-column layouts, sidebars, rotated text, tables interleaved with prose, running headers and export furniture drawn wherever it suited the generator.

The failure is not usually dramatic. It is a paragraph appearing before the heading that introduces it, or a page footer landing in the middle of a procedure. Nothing errors. The output is fluent.

Under the hood
Tagged PDFs can carry an explicit logical order in a structure tree, and PDF/UA requires it. In practice, tagging is common in documents produced for accessibility compliance and rare in engineering exports, scanned material and anything produced by an older toolchain. When it is present it can still be wrong, because it is generated by the same software that laid the page out. It is evidence, not authority.

There is a nastier consequence hiding behind this. Because reading order is reconstructed rather than read, everything downstream inherits it as though it were a fact, including whatever page each piece of text is recorded as coming from. Get that wrong and a single section can absorb thousands of characters belonging to the headings after it. The section then reports itself as spanning one page, which is exactly what a well-behaved short section looks like, so a sanity check on section length or page span sees nothing unusual. A check derived from positional data cannot catch a fault in the positional data.

4. There are no tables

A table on a printed page is a visual arrangement. In the file it is, at best, a set of line segments and a set of independently positioned text runs. Nothing associates a run with a cell, a cell with a column, or a column with its header.

Reconstruction has to infer the grid, usually by clustering text runs on shared x and y coordinates and, where ruling lines exist, using them as boundaries. It works impressively often. Modern table-structure models do it well.

The hard cases are predictable once you know what the tool is working from:

Unruled tables. With no lines drawn, the only evidence of a table is alignment, which is also produced by indented lists, two-column layouts and captions beside figures.
Merged and spanning cells. A header spanning three columns has no more record of that fact than any other text run at that position.
Multi-page tables. The header row exists on page one. Rows on page four have no link to it.
Cells containing wrapped text. Now line grouping and cell grouping are the same problem, and getting one wrong corrupts the other.
Layout tables. Plenty of documents use invisible table structure purely to position content that is not tabular at all.

Two consequences follow that are worth stating plainly.

The first is that a table can be located correctly and still yield nothing. Structure detection finds the region, cell recovery produces no usable content, and the extracted table is empty while the same region remains perfectly legible to anyone looking at the page. Both facts are true at once.

The second is that when a header row goes missing, the values do not. A table whose headers were lost still produces a row of numbers, now unlabelled. N/A in three columns tells you nothing about which N/A belonged to which specification, and nothing in that output looks damaged.

5. There are not necessarily even characters

This is the one that surprises people who already know everything above.

The glyphs painted on a page are identified by codes that mean something to the embedded font program. Turning those codes back into characters requires a separate mapping, and the producing software may not have written it.

When that mapping is missing or wrong, the page prints perfectly, has always printed perfectly, and will print perfectly for another twenty years. It simply contains no recoverable text. Copy a paragraph and you get symbol soup. This is not a bug in the extraction tool, and no extraction tool can fully fix it, because the information required was never written into the file.

Related cases in the same family:

Subset fonts shipping only the glyphs used, with codes assigned arbitrarily.
Ligatures where one glyph stands for fi or ffl and may map back to one character or none.
Scanned pages, which contain an image and nothing else. Any text at all has to come from OCR, with its own error profile, which is a different problem wearing the same clothes.
Mixed documents, where the front matter is digital and the appendices are scans, so both problems appear in one file.

When this happens, the extracted text tends to arrive as strings of decorative symbols, and the natural assumption is that the extraction tool has failed. Usually it has not. It is faithfully reporting what the file contains. The page is legible on screen and unrecoverable as text, and the only two honest responses are to run OCR over the rendered page or to keep the page image itself.

6. Everything above compounds

Each of these problems would be manageable alone. They do not occur alone, and they do not stay in their own layer.

Only the bottom layer is actually in the file. Every layer above it is reconstructed from the output of the layer below. A word break misjudged by two points becomes a merged line. A merged line becomes a merged block. A merged block becomes a heading that has quietly absorbed the section beneath it. By the time the error is visible it has been laundered into confident structure, and the confidence is indistinguishable from the confidence attached to everything that went right.

This is also why extraction quality is not a property of a tool. It is a property of a tool meeting a particular generator’s output. The same library, same version, same settings will handle one publisher’s exports cleanly and stumble on another’s, because the second one draws its headers in a different order or prints its column labels on the section line.

Which means that in any large document set, extraction faults cluster by template rather than by document. Two variants of the same organisation’s own template can behave differently. A few hundred third-party manuals will span an unknown number of layouts, produced by an unknown number of tools over an unknown number of years. A rule tuned against one variant is, by definition, untuned against the rest, and there is no way to know which variant a new document belongs to without looking at it.

Under the hood
Second-order fragility worth knowing about: behaviour can differ across versions of the underlying text-extraction library, so a pinned version is part of the extraction contract, not an implementation detail. Bumping it casually is a change to your output.

Where that leaves things

A PDF preserves appearance with extraordinary fidelity and preserves meaning almost by accident. Everything a reader takes for granted about a page — that this is a heading, that these four things are steps, that this number belongs to that part, that the section continues over the page break — is reconstructed downstream from position, spacing and typeface, by software making a chain of individually reasonable guesses.

Most of the time the guesses are right. When they are wrong, the output does not look wrong. That is the whole difficulty in one sentence, and it is the reason this is an engineering problem rather than a procurement one.

What to do about it is another post.