Why the text in a PDF is not really text

SpecISO 32000-2 §9

At a glance

When you read a PDF, you see words. The file does not contain words. It contains instructions to paint shapes at coordinates, and the shapes happen to look like letters. The gap between what a PDF draws and what it means is the reason a page can look flawless and still copy out as nonsense. This page is about that gap, and about the small, separate map that closes it.

Why this matters

Everything you do to a PDF as text — selecting it, copying it, searching it, indexing it for retrieval, reading it aloud with a screen reader — depends on recovering characters the file never directly stored. If that recovery fails, the failure is invisible. The page still renders. Nobody notices until someone copies a paragraph into an email and gets □□□□, or searches a 400-page contract for a clause that is plainly there and finds nothing.

That is an expensive class of bug precisely because it survives every visual review. There is nothing to see. The document looks authoritative and is, for machine purposes, mute.

The short version

• A PDF draws glyphs — visual shapes chosen by numeric codes into a font — not Unicode characters.
• The same shape can mean different characters, and the same character can be drawn by different glyphs. Appearance and meaning are decoupled by design.
• To get text back out, a reader runs the showing path in reverse: code to character. The font’s /ToUnicode CMap is the lookup table for that reverse step (SpecSpec: ISO 32000-2, §9.10ISO 32000-2 §9.10).
• No /ToUnicode, or a wrong one, means garbled copy-paste and failed search — with the page still pixel-perfect.
• This is the meaning problem. Whether the glyph looks right is the separate appearance problem covered in Fonts: the hard part. NextPDF emits a correct /ToUnicode so the round-trip is faithful.

How NextPDF approaches it

Start with how text gets onto the page at all. A content stream does not say “write the word file.” It selects a font, then hands a string of codes to a text-showing operator (SpecSpec: ISO 32000-2, §9ISO 32000-2 §9). Each code is an index — a position in a font program — and the font turns that index into a glyph outline and paints it. The code 70 is not the character F. It is “slot 70 in this font,” and slot 70 happens to hold an F-shaped curve. Pick a different font and slot 70 could be a snowflake.

So a code only means something relative to its font’s encoding. For a simple font that encoding is roughly one byte per glyph. For the scripts that need thousands of glyphs — CJK, or any full-Unicode document — that is not enough room, and the PDF reaches for a composite (Type 0) font (SpecSpec: ISO 32000-2, §9.7ISO 32000-2 §9.7). A composite font reads multi-byte codes through a CMap, turns them into CIDs (character identifiers), and maps CIDs to glyphs. It is an elegant indirection that lets one font address an enormous glyph set. It is also one more place where the trail from “what is shown” to “what it means” can go cold.

Now run it backwards. To extract text, a reader takes the codes it finds in the content stream and has to recover characters (SpecSpec: ISO 32000-2, §9.10ISO 32000-2 §9.10). The encoding that drew the glyph went character to glyph; extraction needs glyph code to character, and that direction is not guaranteed to be invertible. A subset font may have renumbered its glyphs. A composite font’s CIDs may be private. The shape on the page carries no inherent Unicode meaning.

The fix is to ship the reverse map alongside the font. That map is the /ToUnicode CMap: for each code the document uses, it records the Unicode value (or values) that code stands for. With it, extraction is a clean lookup. Without it, a reader is left to guess from font encodings and heuristics — and guessing is exactly where fi becomes a question mark.

1. Your charactersThe Unicode text you set, for example the word 'file'.
2. Encoding → codesThe font's encoding turns characters into numeric codes; a composite font routes them through a CMap to CIDs.
3. Codes → glyphsEach code selects a glyph outline, which is painted to the page. This is the part you see.
4. Extraction reverses itA reader reads the codes back and looks each one up in /ToUnicode to recover Unicode.
5. Text againWith a correct /ToUnicode, copy, search, indexing, and screen readers all get the original characters back.

The two directions of PDF text. Rendering goes character → glyph and is what you see. Extraction goes code → character and is what you can copy, search, and read aloud — and it only works when the /ToUnicode map closes the loop.

NextPDF writes the reverse map as a matter of course. When it embeds a composite font it emits the CIDFontType2 descendant, the Type0 parent, and a /ToUnicode CMap so codes resolve back to Unicode — the encoding work described in Fonts: the hard part. The point worth separating here is why: the /ToUnicode stream is not about making the page look right. The glyphs already look right without it. It is the single artifact that keeps the text extractable.

Practical example

The most famous place this surfaces is the fi ligature. Many fonts draw f and i as one combined glyph, because the dot of the i collides with the hook of the f. On the page it is one shape, drawn by one code. The question is what happens when you copy it.

% A /ToUnicode entry that maps the single ligature code to TWO characters,
% so selecting the 'fi' glyph copies out as 'f' then 'i' — not one mystery box.
1 beginbfchar
<0085> <00660069>          % code 0x85  ->  U+0066 'f'  U+0069 'i'
endbfchar

That one entry is the difference between a search for “file” matching and a search for “file” silently missing every occurrence rendered with the ligature. Map the ligature code to the two-character sequence f + i, and the word is searchable and copyable. Leave it unmapped, and the page still shows file perfectly while the text quietly says nothing a machine can read.

This is why “fi” can extract as one character or two, and why the difference is not random — it is whatever the /ToUnicode map was told to say. A correct map decomposes the ligature back into its characters. NextPDF emits exactly that kind of entry, so a ligature on the page is two ordinary letters on the clipboard.

Common misconception

The trap, named: “the text renders, so the text is fine.” Rendering and extraction are different mechanisms reading different data. Rendering follows the code-to-glyph path and is what your eyes check. Extraction follows the code-to-character path through /ToUnicode and is what every machine consumer checks. A document can ace the first and fail the second completely, because by construction there is nothing visual to reveal it.

A second, subtler trap: assuming a glyph “obviously” knows which character it is. It does not. A glyph is a shape with an index. The mapping back to a character is additional information the producer has to provide. If the producer never wrote it, no reader can faithfully recover it — it can only guess.

Limits and boundaries

Faithful text extraction from documents NextPDF writes — edition availability

Edition	Availability
Core	○NextPDF emits a correct /ToUnicode CMap for the fonts it embeds, including the ligature and composite-font cases, so the documents it produces are searchable and copyable.
Pro	○Not in this edition
Enterprise	○Not in this edition

A /ToUnicode map can only carry information the source font actually exposes. Where a glyph has no determinable character — a purely decorative mark, a private-use symbol with no Unicode assignment — no map can invent meaning that was never there. NextPDF emits the truthful mapping it can derive; it does not fabricate characters to fill a gap.

This page is about the documents NextPDF writes. It is not a repair tool for arbitrary inbound PDFs whose /ToUnicode is already absent or wrong; recovering text from those is a separate, heuristic problem. And faithful extraction is necessary for accessibility but not the whole of it — reading order, tags, and alternative text are covered in What makes a PDF accessible.

Human-factors note:Human-factors note

The test that catches this is two seconds long and no visual review will ever substitute for it: select a sentence, copy it, and paste it somewhere plain. If what comes out is not what you see, the encoding is wrong — no matter how perfect the page looks. Then search the document for a word you can read on the screen. If the search misses it, the text is a picture of text.

Mini-FAQ

Why does the same PDF copy fine in one reader and badly in another? Different readers fall back differently when /ToUnicode is missing. Some guess from the font’s built-in encoding and get lucky on common Latin text; others do not. A correct /ToUnicode removes the lottery — every conformant reader gets the same characters.

Is this the same thing as the font not being embedded? No. Embedding is about appearance — whether the glyph paints without the font installed. /ToUnicode is about meaning — whether the code maps back to a character. A font can be perfectly embedded and still have no usable /ToUnicode, which is the unsearchable-but-pretty failure.

Does extraction ever need more than one character per code? Yes — the ligature case is exactly that: one code maps to two characters. A /ToUnicode entry can map a single code to a short sequence, which is how ligatures, and some composed forms, come back as their constituent letters.

Related docs

• Fonts: the hard part — the companion piece on the appearance side: embedding, subsetting, and how the encoding is built. This page is the meaning side of the same coin.
• What makes a PDF accessible — faithful text is one ingredient; reading order, tags, and alt text are the rest.
• What a PDF actually is — the object model the fonts, encodings, and /ToUnicode streams live inside.

Glossary

• Glyph — a visual shape in a font (an outline). What a PDF actually paints. A glyph has an index but no inherent character meaning.
• Character — a unit of written language, identified by a Unicode code point. What you mean, and what extraction tries to recover.
• Code — the numeric value in a content stream that selects a glyph from the current font. Not a Unicode character; meaningful only relative to the font.
• CID — a character identifier used by a composite font to address a glyph within a large set; an intermediate step between code and glyph.
• Composite (Type 0) font — a font that reads multi-byte codes through a CMap to reach CIDs and glyphs, used when one font must address thousands of glyphs.
• /ToUnicode — the CMap stream that maps the codes a document uses back to Unicode values; what makes PDF text searchable, copyable, and accessible.
• Ligature — a single glyph that draws two or more letters as one shape (for example fi); its /ToUnicode entry decomposes it back into its characters.