How to extract text from PDFs at scale

Two kinds of PDF, and only one is easy

Before you pick a tool, work out which kind you have. It decides everything else.

A digital PDF was made by software. The text is in there as text, with positions and fonts. Pulling it out is a solved problem and costs almost nothing.

A scanned PDF is a photograph of paper wrapped in a PDF container. There is no text in it at all, only pixels. Getting words out means optical character recognition, which is slower, costs more and is never quite perfect.

Most collections are a mix, and the mix is what ruins naive scripts. A library that handles digital files beautifully returns empty strings for the scanned ones, silently, and you find out three weeks later when someone asks why a supplier is missing from the report.

The free route

For a handful of files, open them and copy the text. For a few hundred digital ones, a Python library does the job in twenty lines and costs nothing but the afternoon you spend on edge cases. Genuinely: if that is your situation, take it.

The edge cases, in the order you will meet them. Multi-column layouts come out interleaved, because reading order is a guess the library has to make. Tables lose their structure and become a soup of numbers. Headers and footers repeat on every page and pollute the text. Ligatures and hyphenation split words in half. And encrypted or password-protected files stop the run dead unless you catch that specifically.

Where it stops being worth your time

Three situations flip the maths. When the volume goes past what you want to babysit, because a thousand files with a handful of failures is a triage job rather than a script. When the sources are mixed and you need OCR only for the files that need it, without paying for it on the ones that do not. And when the job has to run again next month on new files, which makes it software you maintain rather than a script you ran once.

What good output looks like

• Per page, not per file. One row per page keeps the reference intact, which is what you want when a model later has to cite where something came from.
• A record for failures. An encrypted file or a page with no extractable text should come back as a row with a reason, not vanish.
• Page count and detected type, so you can see at a glance how much of your collection was scanned rather than digital.
• Text you can feed straight on. No page furniture, no repeated headers, no stray form feeds.

Feeding a model with it

If this is going into a retrieval pipeline, the PDF is usually only half the corpus. The rest is web pages, and those need the same treatment: the main content without menus, cookie bars and footers. Keeping both in the same row shape, with a source and a page reference on every row, is what makes the retrieval step simple later.

What we charge for it

PDF text is billed per page extracted at $1.00 per 1,000, so a tenth of a cent a page. Pages that are encrypted, empty or unreadable come back as records and are not charged. Web pages converted to clean markdown are billed the same way, per page converted at $1.00 per 1,000.

For a thousand mixed documents you are looking at a couple of dollars and no afternoon. For twenty files, use the library and keep your money.