A case study
Archive/RT 2021
Making forty million scanned pages findable by their words, not their box number.
- In one line
- Searchable archive for 40M scanned records
- My role
- Engineer — OCR pipeline, search API, relevance tuning
- Timeline
- 7 months · 2020–2021
- Team
- 3 engineers with the records office archivists
- Built with
- Elasticsearch, OCR pipeline, S3, FastAPI
Archive/RT built full-text search over forty million scanned documents that had only ever been retrievable by physical location. An OCR pipeline turned images into searchable text, and a tuned search API made a corpus that took weeks to query answer in milliseconds.
Where it started
The situation
A records office held forty million scanned pages. Finding one meant knowing which physical box it lived in, so a retrieval request routinely took weeks and a poorly-remembered detail could make a document effectively unfindable.
The constraint
Scanned images vary wildly — typewriter carbon, faded stamps, handwritten margins. OCR would never be perfect, so search had to stay useful in the presence of recognition errors rather than assuming clean text.
How it was built
A resumable OCR pipeline
Images stream from S3 through an OCR stage that writes text plus per-word confidence. The pipeline is checkpointed and idempotent, so forty million pages could be processed over weeks with failures retried in place rather than restarting the batch.
Search tuned for messy text
Elasticsearch indexes the extracted text with fuzzy matching and phonetic fallbacks, so a query still finds a document when OCR misread a character. Relevance was tuned against real archivist searches, not synthetic queries.
A clean retrieval API
A FastAPI service exposes search with highlighting that shows the matched passage in context, so an archivist can confirm a hit without opening the full scan — the difference between a promising result and the right document.
Confidence surfaced, not hidden
Low-confidence OCR regions are flagged in results, so users know when to trust the text and when to look at the original image. Honesty about the pipeline's limits made people trust the parts that were solid.
What it changed
- Full-text query latency
- <200 ms
- Documents indexed
- 40M
- Retrieval time
- weeks → min
- OCR over the full corpus
- resumable
Looking back
Surfacing OCR confidence instead of pretending the text was clean was the call that made the system credible — archivists trusted it precisely because it was honest about what it couldn't read. Given more time I'd have added structured extraction (dates, names, case numbers) on top of free-text search; the words were findable, but the archive wanted to ask questions the plain-text index couldn't answer.