AI Engineering with Low Volume Data

How understanding my data led me away from a standard RAG architecture

During my time working in the AI Lab at a financial broker, I was scoped to automate back-office workflows using GenAI, primarily for Internal Audit. One thing I quickly noticed was that the data volumes involved were surprisingly small. Because of that, I found myself thinking beyond the standard “RAG + vector database” solution that seems to get applied to everything nowadays.

This article walks through one of those cases.

I was tasked with automating the creation of a Risk & Control Matrix (RCM), which had an estimated ROI of around £200K for the business. The first stage of the workflow was retrieving the most relevant risks based on an audit description.

When I first spoke to my stakeholder, they estimated there were around 50 risks in the database. The data was also glacial - new risks were rarely added or modified.

My first thought was: why introduce a vector database at all?

With only 50 risks, it felt like unnecessary complexity. Instead, I considered simply passing every risk into a single LLM call and asking it to:

• Score each risk out of 100

• Provide a short rationale

• Return the top 10

At roughly 5,000 input tokens, the cost was negligible, latency would stay low, and the implementation was incredibly simple.

Sometimes the simplest solution really is the best one.

Then came the plot twist.

Once I actually profiled the data, I realised the stakeholder estimate wasn’t correct. There weren’t 50 risks.

There were 173.

That completely changed my approach.

Technically, I probably could have still fit everything into a single prompt, but that wasn’t really the issue. Asking an LLM to compare nearly 200 candidates in one go is asking a lot. Even if it fits inside the context window, the model has far more information to reason over, and I expected the ranking quality to suffer from context dilution.

So I went back to the drawing board.

One option was making one LLM call per risk. That would probably produce good rankings, but it would also mean significantly higher latency, dealing with rate limits, retries, orchestration, and generally a lot more moving parts than I wanted.

I also considered batching multiple risks into each request. While this reduced the number of API calls, it still ended up using a large number of tokens overall and wasn’t a particularly elegant solution.

Instead, I decided to reduce the search space before involving the LLM.

Rather than introducing a dedicated vector database, I used pgvector by adding an embedding column directly to the existing PostgreSQL Risks table.

For a dataset of only a few hundred records, the advantages of running a separate vector database were pretty negligible. Dedicated vector databases really start to shine at much larger scales where Approximate Nearest Neighbour (ANN) indexes become important. For my use case, pgvector was more than enough.

I generated embeddings using Cohere rather than Amazon Titan, as I found Cohere generally produced higher quality embeddings for shorter pieces of text.

I also didn’t want to rely solely on semantic similarity. Financial data often contains exact terminology, abbreviations and regulatory language that embedding models don’t always capture well. Because of that, I combined pgvector with PostgreSQL’s tsvector, giving me both semantic search and keyword search.

Rather than choosing one over the other, I fused the two ranked result sets using Reciprocal Rank Fusion (RRF). The nice thing about RRF is that it doesn’t require manually tuning weights between semantic and keyword search. Instead, it rewards documents that rank highly in either list, while naturally pushing candidates that consistently appear near the top towards the final ranking.

The result was a hybrid retrieval stage that returned the top 30 candidate risks.

From there, I performed an LLM reranking step to produce the final top 10 risks.

This meant the LLM wasn’t trying to search across all 173 risks itself. Instead, it was spending its reasoning capacity comparing only the most promising candidates, where it adds the most value.

For evaluation, I measured both Precision@10 and Recall@10 against an evaluation dataset.

The final system achieved:

• Precision@10: 95%

• Recall@10: 88%

The biggest lesson I took away from this project wasn’t that pgvector is better than a vector database.

It was that architecture should follow the data.

If there really had only been 50 risks, I genuinely think an LLM-only solution would have been the right engineering decision.

As the dataset grew, introducing retrieval became worthwhile, but only enough to justify pgvector rather than introducing another piece of infrastructure.

AI engineering isn’t about using the most sophisticated architecture possible.

It’s about understanding the characteristics of your data and choosing the simplest solution that solves the problem well.