Document intelligence at scale

A security and risk consultancy monitors people and companies for its clients, checking for sanctions, fraud allegations, political exposure and bad press. The traditional way to do that is a report: thorough, expert, and out of date the day it’s delivered.

They were using many separate third-party systems, and the list grew over the years. Each had its own query language, data structure and result format, so analysts ran the same search in each one and reconciled the answers by hand. That was slow and easy to get wrong, and because each department had built its own workarounds, the same data could lead to different conclusions depending on who looked.

The brief was to make monitoring continuous: search sanctions lists, court records, news and the rest automatically, again and again, and only show analysts what needs a person. Nobody could read that volume. The system had to do the reading so the analysts could do the judging.

Working out what to build

We started with the analysts, not the pipeline. Interviews with staff across departments and seniority levels covered how they ran an investigation, what made them trust a finding, and the tools they’d built to get around the systems they had. The senior analysts’ workarounds were good, so we kept them.

Design sprints then put analysts, department leads and the technical people in the same room. What an analyst wants isn’t always what their department wants, and neither is always what the business needs. Better to find that out in a workshop than in production.

We also audited the providers: what each could do, where they overlapped, where they disagreed, and where they had nothing to offer. That shaped the interface and the architecture together, which is what made a single query possible.

What the work involved

Every source is different, with its own API, formats and quirks. So each source got its own small service with one job: turn that source’s output into a standard document. Once everything is the same shape, everything downstream is simpler, and when a provider changes its API without warning (they do), only one service breaks.

Providers are also slow, rate-limited, and sometimes don’t answer at all. Rather than hide that behind a spinner, the interface shows what each provider is doing and what has come back, so an analyst can see that most sources have returned and decide whether to wait for the rest. That’s a big part of why analysts trusted it.

Documents then flow through a pipeline of queues and workers. Language models score each one against risk categories such as bribery, fraud and political exposure, and anything over a threshold lands in an analyst’s review queue by priority.

Analysts write a search once and choose their sources. The pipeline does the rest.

It’s built on Go microservices and AWS SQS with a React front end, and was processing over 20 million data points a month when I left. Because it streams, more data means more of the same resources rather than a redesign.

The pipeline was never meant to replace the analysts. It was meant to make sure the next document they opened was worth opening.

Where it stands

The surprise was who benefited most. Not the senior analysts, who already knew which source to trust. It was the newer ones, who got that judgement from the tool instead of from five years on the job.

This was production document intelligence years before the current AI wave: purpose-built models, real documents, real consequences, and a person making the call on every flag. I still build AI pipelines the same way: standardise the data at the edge, stream everything, and keep people on the judgement calls.