From Transcription Pipeline to Agentic Archive Researcher
Published:
In Spring 2026, I worked with Professor Edda Fields Black and the USCT Pension Files Project through Carnegie Mellon’s Information Systems Consulting Project course. Our work is documented in the official project summary published by CMU IS Projects, which can also be found through the CMU IS Projects site. The archive contains handwritten pension applications, affidavits, medical records, family relationships, addresses, and personal testimony from United States Colored Troops veterans and their communities.
These files preserve connections that rarely appear in a single structured record. One page may identify a spouse, another may use a different spelling of the same name, and a witness statement may connect a veteran to a place or family member mentioned in another file. The problem is therefore larger than transcription. It requires document processing, entity resolution, evidence retrieval, and human verification to operate as one system.
My team built a hybrid workflow around that problem. AI produced initial transcriptions and structured extractions; entity grouping scripts reconciled repeated names, dates, and places; Zooniverse gave volunteers a way to correct uncertain pages; and a public interface made the resulting material explorable. My part focused on the Zooniverse setup pipeline and post processing support. The work made maintainability and volunteer usability explicit engineering requirements.
The consulting project produced a working transcription validation workflow. I continued with the question that interested me most: how could the system turn thousands of extracted facts into research evidence without losing the uncertainty of the original documents?
The question that stayed with me
The pipeline could turn page images into data, but a historian still faced a different question: what should I look at first, and why should I trust it?
A pension file can be dozens of pages long. The same person may appear as “Abbs Wilkins,” “Wilkins Abbs,” or an OCR-like corruption. A date may be a birth date, a filing date, or simply the date stamped on an administrative cover sheet. Returning the most semantically similar paragraph is not enough. The system has to preserve provenance, expose uncertainty, and know when to stop before it turns a weak transcription into a confident historical claim.
That became the motivation for my own agentic document intelligence platform for archival research.
Why this is not a typical RAG project
Most RAG demos optimize for answering a question. This system optimizes for deciding whether an answer is ready to exist.
That distinction matters for historical archives. An incorrect product answer can be revised. An incorrect family relationship can enter a genealogy, be copied into another database, and become difficult to unwind. The most valuable system behavior is sometimes not generation. It is identifying the three pages that deserve human attention, preserving competing spellings instead of collapsing them, or refusing to draft a story until a key transcription is verified.
The unit of value is therefore an auditable evidence dossier, not a chat response. Each dossier records what the system found, where it found it, how an entity was resolved, what remains uncertain, and which human decision can improve the record next. This turns RAG from a question answering layer into infrastructure for allocating scarce historical review time.
System workflow
The platform separates expensive document processing from interactive research. The upper lane indexes source material once. The lower lane answers a research question with hybrid retrieval, an auditable review agent, and a human validation loop. Every derived record retains a link to its source page.
| Component | Technology | Why it fits this problem |
|---|---|---|
| Operational source of truth | SQLite | Preserves relational links among files, pages, entities, validation states, and stories without requiring infrastructure for a local research deployment |
| Artifact storage | S3 and CloudFront | Keeps scans and transcripts immutable, addressable, and separate from application records |
| Exact retrieval | BM25 | Handles uncommon names, military units, dates, and quoted phrases well |
| Semantic retrieval | Weaviate embeddings | Finds conceptually related evidence when historical language differs from the researcher’s query |
| Workflow control | Deterministic Python tools | Produces repeatable decisions and inspectable tool traces instead of hidden agent behavior |
| Research interface | Next.js and React | Supports dossier navigation, source comparison, validation state, and page level citations in one workspace |
Why I did not build a chatbot
My first mental model was conventional RAG: chunk the transcripts, embed them, retrieve the nearest passages, and ask a large language model to answer a question. It made for a convincing demo, but it hid too much. When an answer looked wrong, I could not immediately tell whether retrieval had missed the right page, entity resolution had merged the wrong person, the prompt had omitted a caveat, or the model had simply filled a gap.
I eventually narrowed the agent’s job. It does not “research” freely. It follows a deterministic review workflow.
| Stage | Required output |
|---|---|
| File resolution | One pension file with an explicit match reason |
| Coverage inspection | Transcription status, validation status, and missing pages |
| Entity collection | People, dates, locations, military units, and unresolved aliases |
| Evidence ranking | Story relevant pages with retrieval reasons and source links |
| Review routing | Human tasks for ambiguous or unverified evidence |
| Readiness decision | A typed decision to draft, revise retrieval, or abstain |
Each step calls a named tool and records its observation. The result contains the plan, tool-call trace, evidence packet, page URLs, warnings, review tasks, and a final policy decision such as needs_human_review. I found this much more useful than a fluent paragraph with invisible reasoning. It gives me something I can debug, and it gives a historian something they can challenge.
When I ran the agent on the Abbs Wilkins file, it found 77 transcribed pages, 289 person mentions, 333 extracted dates, and 225 location mentions. More importantly, it found that none of those pages had a human-verified transcription and that 42 person mentions still contained placeholder markers. The agent selected three high-value pages, attached the original page-image and transcript URLs, and refused to mark the dossier ready. That refusal is one of my favorite outputs from the system.
Retrieval is a budget, not a dump
The archive also changed how I think about context windows. A larger context window is tempting because it seems to remove the need to choose. In practice, putting an entire 77-page file into a prompt is expensive, slow, and surprisingly hard to audit. More context is not automatically better context; irrelevant administrative pages can drown out the few lines that establish a family relationship.
I started treating context as a deliberately assembled evidence packet. SQLite remains the source of truth for transcription rows, extracted entities, validation state, and generated dossiers. Retrieval selects a small set of page anchored chunks rather than sending a complete pension file to the model.
Chunking and embedding design
Page transcripts are split at paragraph boundaries with a maximum of 5,000 characters and 400 characters of overlap. This stays comfortably below the 2,048 input token limit of gemini-embedding-001 while preserving more structure than sentence level chunks. Pension evidence often depends on nearby form labels, witness questions, and answers. Smaller chunks improved lexical precision but separated names from the relationship or date that gave them meaning. Whole page chunks preserved context but reduced retrieval precision on dense administrative pages.
The 400 character overlap keeps evidence that crosses a paragraph boundary visible in both adjacent chunks. The embedding pipeline uses the default 3,072 dimensional output from gemini-embedding-001. I kept the full representation because the corpus is small enough that vector storage is not the bottleneck, while names, relationships, occupations, military language, and administrative terminology create a semantically varied retrieval space. Each vector stores the file name, page number, chunk index, total chunk count, source image URL, transcription URL, and validation status. Embeddings retrieve candidates but never replace provenance metadata.
| Context control | Configuration | Reason |
|---|---|---|
| Chunk size | Up to 5,000 characters | Retains local document structure and related facts |
| Chunk overlap | 400 characters | Protects evidence near paragraph boundaries |
| Initial candidate pool | 40 chunks per retriever | Preserves recall before fusion |
| Page deduplication | Best 2 chunks per page | Prevents one long page from dominating context |
| Final evidence packet | 5 to 9 pages | Keeps the prompt focused and auditable |
| Model context envelope | 128,000 tokens | Provides one consistent limit across the generation backends |
| Operational prompt budget | 32,000 tokens | Prevents cost and latency from scaling with the maximum window |
| Retrieved evidence budget | 12,000 tokens | Leaves room for instructions, tools, history, and a complete structured response |
The generation backends are normalized to a 128,000 token application window even when an individual model supports more. A normal review is capped at 32,000 tokens: 12,000 for retrieved evidence, 6,000 for system instructions and tool schemas, 4,000 for research history, and 10,000 reserved for reasoning and structured output. The unused portion of the model window is intentional headroom, not capacity that retrieval must fill.
The retrieval layer uses both signals. BM25 handles exact names, dates, military units, and distinctive phrases. Vector retrieval captures semantic similarity when the query and source use different language. A hybrid ranker normalizes and fuses both candidate sets, deduplicates at the page level, and reranks with entity matches, validation status, and source quality.
| Retrieval stage | Operation |
|---|---|
| Query analysis | Separates exact entities, temporal constraints, locations, units, and semantic intent |
| Parallel search | Sends exact fields to BM25 and conceptual intent to vector search |
| Score fusion | Normalizes incomparable lexical and vector scores before combining candidates |
| Page consolidation | Limits repeated chunks from the same scan and preserves source diversity |
| Evidence reranking | Promotes entity matches and verified transcripts while retaining uncertainty flags |
| Packet assembly | Selects the smallest cited set that covers the research question |
Query refinement happens before a second retrieval pass. The first pass identifies high confidence aliases, spelling variants, dates, locations, and military units. The controller expands only entities supported by retrieved evidence. It does not let the language model invent synonyms or relatives. The refined query keeps the original terms, adds verified aliases with lower weights, and applies file or page filters when the research question identifies a specific veteran.
The harness around the model
The more of the project I built, the less the model itself felt like the product. The product was the harness around it.
That harness includes typed response contracts, explicit tools, dry-run defaults for paid or state-changing operations, citation objects, confidence and caveat fields, and a human-review route. Transcription can spend API credits, write database rows, and upload artifacts to S3, so the CLI plans it by default and requires an explicit --execute flag to perform the work. The same principle applies to bulk validation manifests: an agent can recommend an action, but recommendation and execution are separate states.
This structure also makes evaluation less vague. I can test whether the agent chose the correct file, whether every evidence claim points to a real page, whether it surfaced unverified text, and whether the same inputs produce the same decision. “The response sounds good” is not an acceptance criterion.
Verification workflow
| Check | Pass condition | Failure action |
|---|---|---|
| Provenance | File, page, transcript, and source URL all resolve | Reject the evidence item |
| File identity | Retrieved page belongs to the resolved pension file | Rerun file resolution |
| Validation | Transcription has a recorded review state | Attach a warning and create a review task |
| Claim alignment | The cited passage directly supports the generated claim | Remove the claim or abstain |
| Dossier readiness | Required claims have sufficient verified evidence | Publish the dossier or route it to human review |
The system validates structure before generation. Every evidence item must include a file identifier, page number, transcription identifier, source URL, and retrieval reason. The response schema requires claim level citations. A post generation verifier checks that every citation belongs to the supplied packet and that quoted text appears in the referenced transcript.
Failures that changed the design
| Failure | What caused it | Engineering response |
|---|---|---|
| Exact names disappeared from results | Vector similarity favored topically similar pages | Added BM25 retrieval and weighted exact entity matches during reranking |
| One administrative page filled most of the prompt | Several overlapping chunks from the same page ranked highly | Added page level deduplication and a two chunk limit per page |
| Related people were merged incorrectly | Common surnames and OCR spelling errors inflated similarity | Required supporting date, location, relationship, or cross file evidence before automatic linkage |
| The model cited a real page for the wrong claim | Citation generation was unconstrained free text | Restricted citations to packet identifiers and added claim to source verification |
| Long files produced confident but incomplete summaries | The context window silently omitted low ranked evidence | Added explicit token accounting, evidence coverage warnings, and abstention rules |
| Repeated runs were difficult to compare | Prompt and retrieval settings changed independently | Versioned prompts, ranker configuration, schemas, and evidence packets in each trace |
The system as it exists now
The local corpus contains 4,515 transcribed pages, 17,156 person mentions, 14,646 dates, 12,450 locations, and 150 story records. A Next.js and React workspace presents file level dossiers with evidence pages, citation trails, aliases, validation status, and links back to the source scans. S3 and CloudFront hold page and transcription artifacts; SQLite organizes the operational data; BM25 and Weaviate provide two retrieval paths; and the review agent decides what needs a person before a narrative is drafted.
Reliability and evaluation
The platform uses a labeled evaluation set of 120 research questions, relevant pages, known aliases, and cases where the correct action is to abstain. BM25, vector, and hybrid retrieval runs are compared with recall at k and page level precision. Each run records token use, latency, retrieval configuration, prompt version, tool schema version, and the evidence packet shown to the model.
Retrieval benchmark
| Retrieval strategy | Recall at 5 | Recall at 10 | Page precision at 5 | Median latency |
|---|---|---|---|---|
| BM25 | 0.76 | 0.84 | 0.71 | 42 ms |
| Vector | 0.82 | 0.89 | 0.74 | 96 ms |
| Hybrid with reranking | 0.91 | 0.95 | 0.86 | 148 ms |
Hybrid retrieval improves exact name search without losing semantically related evidence. Reranking contributes the largest precision gain by incorporating entity overlap, validation state, and page type.
End to end review benchmark
| Measure | Reference result |
|---|---|
| Citation precision | 97.2% |
| Correct abstention rate | 94.1% |
| File resolution accuracy | 98.3% |
| Median evidence packet | 7 pages |
| Median context size | 9,400 tokens |
| Median review latency | 2.6 s |
| P95 review latency | 5.8 s |
The context assembler enforces both token and evidence count budgets. Duplicate chunks are collapsed at the page level, administrative forms receive lower priority unless they contain a relevant entity or date, and every generated claim must reference a retrieved page identifier.
Human corrections remain part of the data lineage. Consensus scoring combines multiple volunteer reviews, while corrected transcriptions flow back through extraction, entity resolution, and dossier generation without overwriting the original record. This keeps the pipeline reproducible and makes every historical claim traceable to both its source page and its review history.
Paid model calls and state changing tools remain approval gated. Retrieval and model failures produce explicit warnings instead of partial narratives. Structured traces make it possible to inspect each tool decision, reproduce a run, and distinguish a retrieval failure from an extraction or generation failure.
The personal shift for me was from asking, “How can I use an LLM on this archive?” to asking, “What is the smallest reliable decision I can let this system make?” Consulting taught me to begin with the client’s actual workflow. Building the system taught me that the same discipline applies to AI: constrain the task, preserve the evidence, and make uncertainty visible enough that a person can act on it.
Dataset note. The source material comes from United States Colored Troops pension files, federal Civil War pension records preserved by the U.S. National Archives and assembled for this work through the USCT Pension Files Project at the International African American Museum’s Center for Family History. The working page corpus was transcribed by Dietrich Computing at Carnegie Mellon University. I hope to continue collaborating with Dietrich Computing to strengthen the retrieval, validation, and research workflows around this collection.