QontoFAQ: A better Information Retrieval Benchmark [R]

QontoFAQ introduces a new information retrieval benchmark designed to better align with the objective of finding relevant articles for product questions. The creators developed a new metric to measure document relevance more proportionally and built a benchmarking dataset for evaluating embedding models. Further details on this approach and the associated code are available in an article and on GitHub.

This benchmark introduces a new metric for document relevance, unlike previous benchmarks that often felt "benchmaxxed" by models, and provides a dataset for evaluating embedding models.