Problem
Maritime spare-parts manuals arrive as PDFs, often scanned, and their tables must land in a fixed 15-column Excel schema. The documents cannot leave the company network, so every stage has to run on-premises. What began as a single-operator desktop tool now serves a whole team: employees upload folders of PDFs and get back one Excel file per document.
Constraints
- Nothing leaves the network. No cloud API, no API key, no telemetry. Even the LLM is self-hosted.
- Scanned pages are routine, so OCR had to ship inside the app.
- Multiple employees share the system, so it needed accounts, roles, and an admin view of every job.
- New table layouts keep appearing. Resolving them had to be automatic, but every automatic resolution had to stay auditable before other users' documents could trust it.
Architecture
The deterministic core is unchanged from day one: pdf_extractor → semantic_mapper → transformer → qc_validator → Excel. PyMuPDF reconstructs
tables from text blocks scored by header quality; pages without a text layer
route through bundled RapidOCR (ONNX), which rebuilds rows from word-level
bounding boxes. Headers map onto the schema by exact alias lookup first,
edit-distance fuzzy matching second. A failure on one page never aborts the
rest of the file.
Around that core now sits a multi-user web dashboard: a FastAPI backend with
SQLite, and a React + Vite frontend. Before the engine's mapping is accepted
as final, a fingerprint check asks whether this column layout has been
resolved before. A hit reuses the stored mapping instantly. A miss with
unmapped columns hands the document to a bounded, self-hosted LLM resolver
(Ollama, default qwen2.5:7b): it proposes a mapping, few-shot prompted
with the most similar already-approved templates, then verifies that
proposal against real sample rows before trusting it. The result is saved as
a pending template for an admin to approve.
Decisions
| Choice | Why |
|---|---|
| PyMuPDF only, no Camelot, no pdfplumber | Camelot needs a separate Ghostscript install and is GPL. pdfplumber took ~13 minutes to open a 492-page manual; PyMuPDF took 0.2 seconds, with no loss in extraction quality. |
| No ML for column mapping | A semantic-embedding tier (sentence-transformers + PyTorch) produced byte-identical results to exact + fuzzy matching on real manuals, while adding ~1 GB, ~18 s of startup, and a DLL conflict. Removed. |
| Fingerprint templates, not a trained model | A lightweight, explainable layout signature plus the mapping that resolved it. No training, no inference cost, fully auditable by an admin. |
| Self-hosted Ollama with propose-then-verify, not a cloud API | Runs entirely on-premises. Rather than one zero-shot call, the resolver retrieves approved templates as few-shot examples and self-verifies against real sample rows. Bounded, so it fails safe instead of crashing a run. |
| FastAPI + SQLite + a thread pool, not Postgres/Redis/Celery | A small internal team on a VPN doesn't justify the operational cost of a heavier stack. |
Outcomes
Employees upload a folder of PDFs and download validated Excel, with QC failures flagged on their own sheet. Admins manage accounts and curate the layout-template store, reviewing LLM-resolved templates before other users' documents reuse them. The dashboard's landing page is a self-hosted chat over your own uploaded documents: answers cite the source filename and page, and a parts manual routed through the same deterministic engine links straight to its generated Excel. The original desktop app still works, but the dashboard is now the supported way to run Parsel.