At 9fin, we build AI-powered intelligence for debt markets. Our AI uses agentic RAG to retrieve passages from millions of financial and legal documents — earnings decks, offering memoranda, SEC filings — and generate answers. Our users are analysts and investors making high-stakes decisions, where the answer often depends on a single issuer, instrument or number. When the top passage is wrong, so is the answer. "Mostly relevant" isn't enough.
Precision has to be high, and latency has to stay low enough for interactive use.
To do this well, we needed to know what actually works in retrieval for finance. This is the story of what we tested, why the top open reranker on public benchmarks made our precision worse, and what actually improved it without blowing up latency.
What is Retrieval Augmented Generation (RAG)?
Retrieval Augmented Generation (RAG) is a pattern that pairs a language model with a search step. Instead of relying on what the model memorised during training, the system fetches relevant passages from an external corpus at query time and passes them to the model as context, then generates an answer grounded in those passages.
What we measured: Precision@1
Our goal was to surface the most relevant passage at position 1, with minimal added latency. We reported two metrics:
- Precision@1: the percentage of queries where the #1 result is actually relevant (contains the answer-bearing evidence).
- Latency: total retrieval time per query, reported as p50 and p95.
We define a result as relevant if it contains the answer-bearing evidence needed to correctly answer the query (binary labels: relevant / not relevant).
Architecture: bi-encoder and reranker
The pipeline has two stages: a first-stage retriever quickly narrows the corpus to a candidate set, then a reranker re-scores those candidates to surface the most relevant result at the top.
Two-stage retrieval — a fast first-stage retriever returns a candidate set, then a reranker reorders the top-*k to optimise rank #1.
First-stage retrievers:
- BM25 (lexical): ranks by exact term overlap — strong when queries hinge on names, dates, or line items, but can miss paraphrases.
- MP-Net (semantic bi-encoder): ranks by embedding similarity — good for paraphrases, but can miss fine-grained constraints (entities, dates, numbers).
The first-stage retriever surfaces a top-k candidate set (our evals focus on top-20); the reranker refines that ordering to promote the truly relevant passage.
Rerankers: re-score the top-k candidates from stage one, reading queries and chunks together for more granular relevance judgments.
- Point-wise cross-encoder rerankers (encoder-only): BERT-style encoder models that concatenate query and chunk and run a single forward pass with bidirectional attention across both texts, producing a scalar relevance score from a lightweight classification head (fast, stable latency per pair).
- IBM Granite Embedding Reranker (English R2): ModernBERT-based, trained on public, internal, and synthetic data.
- BAAI BGE-reranker-v2-m3: multilingual, combines semantic and lexical signals.
- Point-wise decoder-only rerankers: use a decoder-only transformer to score (query, chunk) pairs by running a short prompt over the concatenated query and passage; typically slower per pair at a given GPU budget.
- Qwen3-Reranker-0.6B: multilingual, trained heavily on synthetic data; top-ranked on MTEB at the time of writing.
- List-wise LLM reranker — GPT-4o: ranks the full candidate list in a single prompt, enabling list-level reasoning — it can down-rank redundancy, identify the most answer-bearing passage, and prefer chunks with direct evidence. Because of its high latency, we treat it as a practical upper bound rather than a production default.
All point-wise rerankers were hosted on the same SageMaker instance type (ml.g4dn) to ensure a fair latency comparison.
The dataset
The evaluation set mirrors a realistic finance RAG setting: a small number of questions over a large candidate pool.
- Corpus: ~27k chunks from documents for five companies. The documents came from our corpus of documents, representing the distribution of document types in a production setting.
- Queries: 60 questions per company, independently sampled from the most common user queries on 9fin's platform.
- Labels: binary (relevant / not relevant), produced with a two-stage AI annotation process and verified by human annotators on a sample. Fewer than 170 unique chunks (0.6% of the corpus) are relevant to at least one question.
While using only five companies, this evaluation reflects the real retrieval regime we care about: sparse relevance (few truly relevant chunks) and high penalty for near-misses in top-1 ranking.
What the data tells us
First, it’s worth noting the challenge of reliably promoting the relevant candidate to position 1. At least one of the top-20 candidates is relevant for 84% of queries. However, the results for Precision@1 reveals that reranking helps — but not always, and not equally across approaches.
Precision@1 improves most with list-wise LLM reranking (ceiling) and improves meaningfully with cross-encoders at much lower latency; benchmark-leading models may not transfer to finance constraints.
List-wise reranking: setting the ceiling
Our list-wise reranking experiments produced the largest gains: up to +24 percentage points (pp) in Precision@1 compared to the first-stage retriever (see Table 1). The downside is latency: it adds substantial overhead — ~2.2s p50 and ~4s p95 in our setup — so we treat it as a practical upper bound rather than a production default.
Once reranking is strong, the choice of first-stage retriever matters much less. 'BM25 + GPT‑4o' and 'MP-Net + GPT‑4o' achieve nearly identical Precision@1 (49–50%). This suggests that using an expensive but high-quality reranker, Stage 1 mainly needs to surface a reasonable candidate set, rather than perfectly ranking candidates by itself.
Table 1: List-wise reranking results
Point-wise reranking: lexical signals win in finance
The results show that BGE and Granite consistently improve Precision@1 over the baseline retriever for both pipelines (~+6–9pp), with the largest gain coming from 'BM25 + BGE' (24% → 36%). By contrast, Qwen is notably much slower and does not improve Precision@1 in either setup (MP-Net 26% → 24%, BM25 24% → 24%). Overall, this indicates that in finance, lexically anchored candidate sets (BM25) plus cross-encoder reranking is a particularly strong combination.
Table 2: Point-wise reranking results
Why Qwen underperformed: In our tests (prompting plus hosting setup), Qwen appeared more willing to reward broad topical relevance, even when a passage missed one important constraint. That matters in debt capital markets, where the difference between the right and wrong result can be a single issuer, quarter, debt instrument or tranche or number. These near misses can appear highly relevant but are still incorrect — and Precision@1 is particularly unforgiving of them.
Why BGE and Granite performed better: BGE and Granite were more effective at distinguishing between closely related passages and promoting the result that matched the full set of query details. The gains were slightly larger over BM25, likely because lexical retrieval produced a stronger pool of candidates containing the relevant entities, dates and financial terms.
These findings are based on one of our evaluation setups and remain an active area of research at 9fin. Model behaviour may vary with prompting, training data and candidate-set composition.
We also discovered that Qwen3-Reranker-0.6B is ~5× slower than Granite and BGE on the same instance (~3.2s vs ~0.7s p50). A major contributor is architectural: Qwen3 is built on a decoder-only transformer, whereas Granite and BGE are BERT-style encoder models with a much cheaper forward pass.
A larger GPU instance (e.g. ml.g5 with an A10G GPU) would likely bring Qwen's latency down to ~1s p50 — potentially within an acceptable budget. But the latency argument is secondary here: Qwen also reduced precision in our setup, so better hardware wouldn’t change the accuracy result we observed.
What does this mean for financial AI?
Reranking is an effective way to improve retrieval quality in agentic RAG, especially when the system can only include a small amount of context and the top result matters most.
In our finance focused evaluation, list-wise LLM reranking achieved the highest Precision@1, but its latency and variability make it a poor fit for interactive chat. Point-wise cross-encoder reranking delivered a better production trade-off, improving Precision@1 while keeping added latency much lower.
More importantly, this research highlights that off-the-shelf retrieval solutions don't transfer cleanly to finance. Unlike general domains where semantic similarity is often enough, correctness here depends on lexical precision — exact entities, dates, and numbers. As a result, strong benchmark performance does not guarantee real-world performance.
Building effective systems in this space requires combining lexical and semantic signals, alongside domain-specific evaluation and design. This remains an underexplored problem, with limited precedent in public datasets.
Coming next: extraction and chunking
Good retrieval also depends on extracting content reliably and chunking documents in a way that makes hard-constraint retrieval easier. We’ll reveal our extraction plus chunking strategy in future technical blogs — stay tuned.
If you enjoy measuring failure modes, making systems fast, and earning user trust — we’re hiring, and want to speak with you!
