Feasibility Discussion of Decision-based Rerankers
Problem
In the technical exploration of using generated questions for document relevance reranking, users discussed performance differences between LLM-only and LLM+Jev hybrid approaches, query latency and caching efficiency, differentiated weighting of question sensitivity, and the choice between summation and multiplication of scores. Despite localized performance data (e.g., processing 50,000 queries per hour), scalability and practical utility across diverse models and hardware remain to be validated.
Pain points
- LLM-only approach efficiency concerns regarding query cost and latency
- Caching and prefill token impact on overall latency in hybrid LLM+Jev setups
- Lack of differentiated question sensitivity weighting leading to reduced scoring accuracy
- Uncertainty in choosing between summation and multiplication of scores for relevance scoring
Questions to verify
- In hybrid LLM+Jev setups, does the latency improvement over pure LLM approaches show significant practical benefit?
- How can the sensitivity weighting of individual questions be quantified and optimized in real-world scenarios?
- What are the effect differences between summation and multiplication scoring methods across document types?
Evidence
He compared techniques that used: - BM25 only - Jev only - LLM + Jev The latter worked best.
A possible improvement could be generating a sensitivity score for each question. Some are more important than others. For example not mentioning kids at all should cause a huge decrease in relevance, but not mentioning a specific feature should only cause a small decline in score.
I think I was able to process 50,000 email questions per hour or so with Clef flash on Strix Halo with a custom engine. A modern gaming GPU would likely be much faster
Disclaimer: this card is distilled by AI from community discussions as “pain point evidence”. It is a starting point for need analysis and validation, not a “validated market need”. AI has analytical limits; please open the original discussion above to manually verify before key decisions.