Cohere's Rerank 5 leads its own scoreboard. The token savings are the test

By
CTOL Staff Reporter
1 min read

Rerank 5 Pro ranked first on 39 of 46 datasets using a metric Cohere defined three weeks ago. In its agent test, ranking added 3 points of accuracy, while Context Focus added up to 9 and cut token use.

Retrieval decides which text an AI model reads, and the model bills for every token. The search layer therefore affects both accuracy and cost.

Cohere's Rerank 5 saves money by removing text from the model's context. On 21 October Cohere released Rerank 5 at $0.05 per million tokens for the Pro tier and $0.02 for Fast. Its benchmark lead is measured on Cohere's yardstick and has not been tested on anyone else's.

Buyers traditionally use a reranker to put search results in a better order. In Cohere's agent test, ordering added 3 points of accuracy, while removing text added up to 9. That changes what the buyer is paying for.

Cohere's broad benchmark claim is that Rerank 5 Pro ranked first on 39 of the 46 datasets it evaluated. It placed at least second on 43 and outperformed every competing model tested.

The biggest gap is over Cohere's own older model. On the BRIGHT benchmark, Pro scores 82.2, which is 12.4 points above Rerank 4 Pro and 5.8 above Voyage AI Rerank 3.

Gaps over outside rivals are narrower on public sets. Cohere puts Pro 1.6 points ahead of the strongest external competitor on ViDoRe V3, 1.8 on BEIR and 3.4 on nanoMTEB.

The widest lead over an outside rival is in finance, at 9.6 points over Voyage AI Rerank 3. The multilingual claim rests on Cohere's own suite, where it says Pro ranks first on 14 of 19 non-English languages.

On parsed PDFs, Pro's lead is thin over every rival, including Cohere's older model. Pro scores 89.9, against 88.4 for Voyage AI Rerank 3 and 88.3 for Rerank 4 Pro.

Those scores use RCP-nDCG@10, which Cohere calls "a Cohere-developed methodology". Cohere published it on 30 September and says it is the metric against which it has "optimized" its next-generation search models. The scoreboard is three weeks old, and the model was tuned toward it.

Cohere has a reason to want a new ruler. Its RCP-nDCG post says human reviewers judged 28% of documents labelled irrelevant by the old benchmark to be useful.

The new score agrees with human reviewers more closely than the old labels. The same post reports an AUC of 0.91 against 0.65 for the original labels, in a study with 46 annotators. The post offers no replication by anyone outside Cohere.

The agent test is the evidence that bears on a budget. Cohere ran an agent called glm-5-3-flash across 17 datasets with 50 questions each. It started from BM25 keyword search and reranked the top 100 results.

Ranking alone was worth 3 points, from 46% with BM25 to 49% after adding Rerank 5 Pro. That is the ordering gain a reranker is traditionally bought to provide.

Context Focus did the heavier work. It returns scored text spans with document scores, so the model reads passages instead of whole documents. At the default threshold of 0.1, tokens per task fell by 19% and accuracy rose to 57%.

A stricter setting saved more tokens without losing accuracy in this test. At 0.4, tokens fell by 34% and accuracy reached 58%, the highest of the four stacks.

The gain is larger on hard questions. On the six deep-search benchmarks, the full stack used 41% fewer tokens, and accuracy rose from 39% to 52%.

Cohere's token count includes cached tokens, which cost less, so the bill should fall by less than the token count. That is our reading of Cohere's definition.

Compression delivered three quarters of the gain. The full stack gained 12 points over BM25 alone, and 9 of them, or 75%, came from compression.

Across 850 questions, that is about 102 more correct answers, of which ranking supplied about 26. Cohere also says the full stack used 35% fewer tokens than BM25 alone at higher accuracy. This remains one vendor test on one agent.

Removing more context costs accuracy. Across 30 question-answering datasets, removing half the context cost 2 points of answer quality, so 96% was retained.

Removing three quarters of the context costs 7 points, leaving 87%, and Cohere claims up to 10 times less context. Cohere's curve shows the loss growing as more text is removed.

Long documents carry the most risk from that loss. An extractive system can drop the paragraph that a contract clause or filing footnote depends on. The threshold trades the token bill against the answer.

Calibration means that a score matches the real chance that a document is useful. Cohere says a score of 0.7 now means a 70% chance, so the score can be read as a probability.

Rerank 5 Pro's calibration error is 0.058, against 0.384 for Rerank 4 Pro, about 6.6 times lower. Cohere measured expected calibration error against blind human graders on 2,262 documents.

A calibrated score lets one threshold work across document sets. Cohere calls a Pro score of 0.50 "more likely useful than not". It says the default stays between 0.47 and 0.53 across BRIGHT, NanoBEIR and ViDoRe V3.

An agent can rewrite its query if no document clears the threshold.

By the same measure, a rival's scores look less reliable. Voyage AI Rerank 3 scores 0.234 on the human set. Cohere says documents Voyage scored around 0.55 were useful to human graders only 24% of the time.

Pro's error is about the same on a second set from NIST: 0.057 across 20,646 documents. The low error is not tied to one panel of graders.

The reranking charge is small beside the text it removes. Scoring 100 documents of 1,000 tokens each means 100,000 tokens at the posted prices.

That costs $0.005 on Pro and $0.002 on Fast, on our arithmetic. The post does not price the language-model tokens that Rerank 5 removes, so the buyer has to do that comparison.

A buyer should test this on its own documents. Neither compression nor calibration can be read from a leaderboard. A search or AI platform team running retrieval over contracts, filings or manuals can answer the question itself. Rerank 5 works with keyword, vector or hybrid search with no reindexing.

The cleanest run is a head-to-head test on the same 200 questions. Put the current reranker against Rerank 5 with Context Focus at 0.4, then compare accuracy and cost per answered question.

Rerank 5 is already on Microsoft Foundry and Amazon SageMaker, so a buyer can run that test now. A public run on BRIGHT or BEIR under the original nDCG would test the 39-of-46 claim. Cohere did not build that ruler.

The compression gain needs no outside run. Any buyer can count tokens and correct answers on its own documents.

You May Also Like

This article is submitted by our user under the News Submission Rules and Guidelines. The cover photo is computer generated art for illustrative purposes only; not indicative of factual content. If you believe this article infringes upon copyright rights, please do not hesitate to report it by sending an email to us. Your vigilance and cooperation are invaluable in helping us maintain a respectful and legally compliant community.

Subscribe to our Newsletter

Get the latest in enterprise business and tech with exclusive peeks at our new offerings

We use cookies on our website to enable certain functions, to provide more relevant information to you and to optimize your experience on our website. Further information can be found in our Privacy Policy and our Terms of Service . Mandatory information can be found in the legal notice