Publishing a fair number for a model is harder than it looks, and most of the difficulty is invisible in the final figure.
Metering has to be fair
A model called through an agent wrapper is not the same as the model called directly. The wrapper adds its own system prompt, retries, and tool scaffolding, and it reports cost and latency for the whole apparatus rather than the model. If you grade that, you are grading the harness. So we call each model on the same footing and record the per-call cost, token counts, and latency ourselves, through a budget guard that gates every paid request.
Third-party numbers are context, not grades
Provider-claimed scores and community rankings are useful, but they are not ours and they are not neutral. We carry them in a clearly marked context box on each scorecard, every one tagged with its source, retrieval date, and whether it is self-reported or independent. None of them are ever folded into the grade we assign.
Append-only, on purpose
A benchmark you can silently edit is a benchmark you cannot trust. Our records are append-only: a regrade is a new entry that supersedes an old one, with the history intact. If we get something wrong, you will be able to see both what we said and what we corrected it to.
This is slower than scraping a leaderboard. We think that is the right trade.