Document intelligence at scale

A security and risk consultancy monitors people and entities on behalf of its clients - checking for sanctions, fraud allegations, political exposure, adverse media. The traditional way to do that is a point-in-time report: thorough, expert, and stale the day it’s delivered.

The starting point was five separate third-party systems, each with its own query language, its own data structures and its own result format. Analysts ran the same search five times and reconciled the answers by hand. That was slow and easy to get wrong, but the real cost was subtler: departments had each built their own workarounds, so the same underlying data could support different conclusions depending on who went looking.

The brief was to make monitoring continuous. Search many sources - sanctions and compliance databases, court records, news media - over and over, automatically, and surface only the findings that deserve a human’s attention. At the volumes involved, no team of analysts could read everything. The system had to do the reading, so the analysts could do the judging.

Working out what to build

We initially focused on the analysts rather than the pipeline. We conducted structured interviews with staff across different departments and seniority levels to understand their investigation methods and the factors that build their trust in a finding. We also asked about the tools they had created independently to bypass existing systems. The senior analysts had developed effective workarounds that we decided to retain. The less experienced analysts had specific needs that our project aimed to address.

Design sprints then put analysts, department leads and technical stakeholders in the same room, which is where the useful tensions surfaced: what an individual analyst wants is not always what their department wants, and neither is always what the business needs. Better to have that argument in a workshop than to discover it in production.

Alongside that we audited the vendor ecosystem - what each provider could actually do, where they overlapped, where they disagreed, and where they simply had nothing to offer. That audit shaped the interface and the architecture at the same time, which was the point: designing them together, rather than one and then the other, is what made a single query model possible at all. None of that was model work. In a system like this, the models are rarely the part that decides whether it succeeds.

What the work involved

The hard problem wasn’t any single source - it was that every source is different. Each database and media feed has its own API, its own formats, its own quirks. The architecture answered that with a microservice per source, each one responsible for exactly one thing: turning that source’s output into a single, normalised document structure. Once everything speaks the same shape, everything downstream gets simpler. Keeping each client small and isolated was deliberate: providers change their APIs without warning, and when one does, the blast radius is a single service rather than the platform.

The same assumption runs into the interface. Providers are slow, they rate-limit, and sometimes they simply don’t answer. Rather than hide that behind a spinner, the UI reports what each provider is doing while a query runs, and what has come back so far, so an analyst can see that four sources of five have returned and decide whether the fifth is worth waiting for. Treating provider uncertainty as a normal operating condition rather than an error state is a large part of why analysts trusted what came back.

From there, documents flow through a streaming pipeline of queues and workers. Natural-language models score each document against risk categories - bribery, fraud, political exposure and others - and when scores cross a threshold, the item is flagged by priority into an analyst’s review queue, where it can be assessed and, where warranted, reported onward.

On top of the pipeline sits the tool analysts actually touch: a UI for creating new monitoring searches, choosing sources, and expressing the query once - in a single unified structure - rather than once per source. The pipeline does the fan-out.

The system uses Go microservices and AWS SQS with a React front end. It processed over 20 million data points per month by the time I left. The architecture is designed for streaming, which ensures that increases in data volume require scaling existing resources rather than redesigning the system.

The pipeline’s job was never to replace the analysts. It was to make sure the next document they opened was worth opening.

Where it stands

The effect I didn’t predict was who benefited most. Not the senior analysts - they already knew which source to reach for and what its answers were worth. It was the newer ones, who got that judgement from the tool instead of from five years of accumulating it themselves. Good tooling turned out to be a training lever as much as a productivity one.

This was production document intelligence years before the current AI wave - purpose-built models, scoring real documents, with real consequences riding on the output, and a human decision at the end of every flag. The architecture decisions it forced - normalise at the edge, stream everything, keep humans on the judgement calls - are the same ones I bring to document and AI pipeline work today.