Document intelligence at scale

A security and risk consultancy monitors people and entities on behalf of its clients - checking for sanctions, fraud allegations, political exposure, adverse media. The traditional way to do that is a point-in-time report: thorough, expert, and stale the day it's delivered.

The brief was to make monitoring continuous. Search many sources - sanctions and compliance databases, court records, news media - over and over, automatically, and surface only the findings that deserve a human's attention. At the volumes involved, no team of analysts could read everything. The system had to do the reading, so the analysts could do the judging.

What the work involved

The hard problem wasn't any single source - it was that every source is different. Each database and media feed has its own API, its own formats, its own quirks. The architecture answered that with a microservice per source, each one responsible for exactly one thing: turning that source's output into a single, normalised document structure. Once everything speaks the same shape, everything downstream gets simpler.

From there, documents flow through a streaming pipeline of queues and workers. Natural-language models score each document against risk categories - bribery, fraud, political exposure and others - and when scores cross a threshold, the item is flagged by priority into an analyst's review queue, where it can be assessed and, where warranted, reported onward.

On top of the pipeline sits the tool analysts actually touch: a UI for creating new monitoring searches, choosing sources, and expressing the query once - in a single unified structure - rather than once per source. The pipeline does the fan-out.

Built with Go microservices and AWS SQS behind a React front end, the system was handling more than 20 million data points a month when I moved on - and it earns the word "streaming": designed so that volume is a scaling problem, not a redesign.

The pipeline's job was never to replace the analysts. It was to make sure the next document they opened was worth opening.

Where it stands

This was production document intelligence years before the current AI wave - purpose-built models, scoring real documents, with real consequences riding on the output, and a human decision at the end of every flag. The architecture decisions it forced - normalise at the edge, stream everything, keep humans on the judgement calls - are the same ones I bring to document and AI pipeline work today.