Public PDFs to clean data in any language

• 5 stages
• a fallback at every step
• JSON → CSV → many languages

The problem

The data existed, but only in the worst shape possible: locked inside PDFs published across the web, laid out for people to read and not for programs. Someone needed it as a clean table, and in more than one language.

Copying it by hand doesn't scale, and a single script breaks the first time a site or a document looks a little different. What was needed was a pipeline where every stage expects to fail sometimes and knows what to try next.

The pipeline

download PDFs ──▶ page images ──▶ blocks ──▶ JSON ──▶ clean ──▶ CSV ──▶ translated CSVs
      │                │            │         │
      └────────────────┴────────────┴─────────┘ fallbacks at each of these

Each stage writes its output before the next one starts. If a later stage fails, I fix it and re-run from there, without fetching or converting anything again.

1. Downloading the PDFs

The sources were PDFs, published publicly on the web. Everything downloaded was public, and the downloads followed each site's rules. Sites don't all serve their files the same way, so one way of fetching was never enough. The downloader had several automation methods and moved on to the next one when a method failed on a site. The goal at this stage was simple: get every source PDF onto disk.

2. From PDF pages to image blocks

Reading text straight out of a PDF works until it doesn't: scanned pages, odd encodings and multi-column layouts all break it. So every page became an image, and every image was cut into blocks, the separate regions of a page such as a heading, a paragraph or a table.

No single way of finding blocks worked on every layout, so this stage had fallbacks too. When the first method gave a bad split, the next one tried. A block is small and has one job, which makes the next step far more reliable than reading a whole page at once.

3. Blocks to JSON

Each block was then read into a fixed JSON shape. Every record has the same fields no matter what the source page looked like. Reading block by block also means an error stays inside one block instead of spoiling the whole page.

This step had fallbacks as well. No single extraction method read every block correctly, so when the first method failed or returned something that didn't fit the shape, the next method took the same block and tried again. Only a block that every method failed on was flagged for a person to look at.

4. Cleaning

Raw extraction is never final. A post-processing pass tidied every record and made sure it matched the expected format before anything moved on. Data that didn't pass was caught here, not by the person opening the spreadsheet.

5. CSV, in more than one language

The clean JSON was converted into CSV, the format the people using it actually work in. A translation step then produced the same CSV in multiple languages, so every reader got the same table in their own language.

Takeaway

When the input is messy, don't look for one perfect method. Split the job into small stages, save the output of each one, and give every fragile step a plan B. The pipeline gets reliable not because each method is perfect, but because no single method has to be.