Building autonomous evaluation pipelines for retrieval

Observability and evaluation are both critical to production systems using AI. In a previous blog post, we wrote about building an LLM observability platform in-house. This works great for evaluating LLM outputs, which are probabilistic models with non-deterministic outcomes. But a growing portion of our system relies on retrieval. When categorising transactions, we rely on finding historically similar transactions to direct our agent to categorise each line item to the correct account. If retrieval returns the wrong context, the downstream agent will produce the wrong answer, and no amount of prompt engineering will fix it.

At Briefcase, this is one of our most critical pipelines. Retrieval quality directly determines categorisation accuracy, which in turn determines whether transactions can be auto-published or need human review.

Evaluating retrieval has its own set of problems: Firstly, there's no single correct answer — there's a list of results ranked by relevance to the query, and that ranking depends on nuanced judgement. Secondly, retrieval is a deterministic process — running the same query against the same corpus will produce the same results time and time again.

We needed a way to evaluate retrieval quality at scale, and the standard approach of hand-labelling a static dataset wasn't going to cut it.

The simplest way to evaluate retrieval is to create a labelled dataset by hand. You take a set of queries and manually label which documents in the corpus are relevant to each query. Then you can run your retrieval system to measure how well it surfaces the relevant documents using standard IR metrics like NDCG (Normalised Discounted Cumulative Gain), precision@k, and recall@k.

We started here. It works, and it gives you a baseline. But it breaks down quickly:

• Labelling is a slow and manual process. For each query, you need to assess relevance against potentially hundreds of documents. To build a meaningful dataset, you're looking at days of manual work.
• The corpus grows continuously. A static labelled dataset doesn't reflect a production environment, where new documents are constantly added to the corpus.
• Coverage is sparse. You can only label a fraction of the query-document pairs that exist, and you won't catch regressions in areas that you didn't label.
• Subjectivity creeps in. Relevance isn't binary. Even between human labellers, there can be contention around how relevant two documents are to each other.

We needed a scalable approach to generate relevance labels, keep up with a changing corpus, and apply consistent judgement criteria.

The insight was straightforward: use an LLM to label relevance between document pairs. If we could build an agent that reliably judges whether a document is relevant to a given query, we could generate labelled datasets at whatever scale we needed.

We decided to use a graded relevance score from 1 (no relevance) to 5 (high relevance) rather than a binary judgement. Retrieval quality is about ranking – a highly relevant document should rank above one that's only tangentially relevant, and the scale needs enough granularity to capture this nuance without exceeding what an LLM can reliably distinguish.

The labelling agent receives the full context of both the query and corpus item. For our transaction categorisation use case, this means the transaction being categorised (query) and a historical transaction from the client's ledger (corpus item).

We started by manually labelling around a hundred pairs with justifications for each score. From these, we derived a rubric for the agent – a set of instructions covering the dimensions it should evaluate: supplier, line item descriptions, amounts, VAT treatment, and category. The agent produces a relevance score and justification for every pair.

The justification is important. It forces the agent to explicitly reason about why a corpus item is or isn't relevant to the query, which makes the labels more consistent and gives us an audit trail when we disagree with a label.

There's a subtlety here. When labelling offline, you often have access to ground truth data that isn't available at query time. In our case, retrieval helps decide how to categorise a transaction – so category isn't embedded, nor is it available during retrieval. But when labelling, we're working with published transactions where the category is already known. This means the labelling agent can use category as a primary signal for relevance, producing higher-quality labels than the retrieval system itself could. This principle applies broadly – any information that exists post-retrieval but is unavailable at query time becomes a powerful labelling signal.

Building an LLM to label data introduces an obvious question: how can we trust the labeller? If the labelling agent produces bad labels, every downstream eval built on these labels is worthless.

This is where the static dataset we built earlier becomes valuable again. Instead of using it to evaluate retrieval directly, we can repurpose it as a dataset to evaluate the labelling agent. Using our existing LLM evaluation infrastructure, we can measure agreement between human and agent labels.

This is a much better use of manual labelling effort. Instead of needing hundreds of labelled pairs to evaluate retrieval, we need a smaller but carefully curated set to validate that our labeller is reliable. The manual work is a one-time investment that unlocks automated labelling at scale.

We track standard agreement metrics and look specifically at cases where the agent and human disagree. These disagreements are informative. Sometimes they reveal genuine edge cases, and sometimes they reveal inconsistencies in our own human labels. We iteratively improve the labelling agent's prompts using the same eval-driven loop we use for all our agents: find disagreements, update prompts, rerun eval, check if agreement improves.

With a trusted labelling agent in hand, we can now generate labelled datasets at whatever scale needed.

The pipeline looks something like this: we take a set of queries, run the labelling agent across query-corpus pairs to produce relevance scores, then run our retrieval system on the same queries and evaluate the ranked results against those scores. We compute standard IR metrics – NDCG for ranking quality, recall@k for coverage, and MRR for how quickly the first relevant result appears.

The key advantage is that this pipeline is fully automated and repeatable. We can change our embedding model, modify retrieval parameters, or update the features embedded to get a quantitative comparison.

More importantly, the pipeline integrates into Briefcase's existing internal observability platform. Retrieval quality is now a metric we track. When something regresses in production, we can pinpoint whether the issue is in retrieval, in the downstream agent, or both. And with retrieval traces built into our production observability, we can inspect exactly which documents were retrieved for any given query.

This approach is not without its trade-offs.

The most obvious is that we've replaced human subjectivity with LLM subjectivity. The labelling agent applies consistent criteria, but those criteria are only as good as the rubric we derived from our initial manual labels. It will require us to continuously monitor and update our human-labelled cases when we encounter production cases that feel misjudged.

Cost is another consideration. Labelling thousands of pairs requires a significant number of LLM calls. We manage this by caching labels (since relevance between two documents doesn't change) and ensuring that we do not relabel a pair that has already been labelled.

Finally, this pipeline evaluates retrieval in isolation. In production, what matters is whether the retrieved documents lead to a correct downstream outcome. A retrieval result with an NDCG of 0.8 might still produce a perfect categorisation if the top result is sufficient, while a perfect NDCG of 1.0 is wasted if the downstream agent ignores the context. We're actively working on end-to-end evaluation that connects retrieval quality to downstream task accuracy.

When AI agents are writing the code that powers other AI agents, evaluation infrastructure becomes the most important thing you can build. It compounds: every agent we add, every retrieval system we tune, benefits from the same underlying eval platform. We're continuing to push on this, building more sophisticated pipelines across our systems.

If this is the kind of work that excites you (and not the manual data labelling), we're hiring.