What a leaderboard actually tells you
Public benchmarks are useful for tracking the field. They are close to useless for deciding whether a model will work on your task.
A benchmark measures performance on curated academic problems, in general language, with no knowledge of your terminology, document formats, edge cases or definition of correct. A model that scores well may still fail on your invoices, and one that scores lower may handle them fine.
The only evaluation that answers your question is one built from your own material.
Building an evaluation set that is worth trusting
The set does not need to be large. Fifty to two hundred well-chosen cases will tell you more than a public benchmark, provided they are chosen honestly.
- Sample from real traffic. Actual inputs, not invented examples that happen to be well-formed.
- Include the hard cases. Ambiguity, missing fields, unusual formats, and the ones a human had to think about.
- Write the answers independently. A knowledgeable person supplies the expected output without seeing the model's.
- Check agreement. If two experts disagree on the correct answer, the task definition is ambiguous; fix that before blaming the model.
- Keep a holdout. Reserve a portion never used during prompt iteration, or you will tune to the test.
Choosing what to measure
Accuracy is rarely the right single metric, because the cost of the two error types is usually asymmetric.
For extraction, field-level precision and recall separate 'got the wrong value' from 'missed it entirely'. For classification, a confusion matrix shows which categories the model conflates, which is actionable in a way an aggregate score is not. For generation, faithfulness to source matters more than fluency.
Define which error is worse before measuring. A system that never misses but occasionally invents is appropriate in some contexts and unacceptable in others.
Comparing against the baseline that matters
The comparison is not against another model. It is against how the work is done today.
If the manual process is 94% accurate at four minutes per case, an AI system at 91% is not an improvement, however impressive the demo. If manual is 78% because people are rushed and inconsistent, then 91% at three seconds is transformative. Neither conclusion is available without measuring the baseline first.
Evaluation as regression testing
The set earns its keep after launch. Prompts get edited, model versions are deprecated, retrieval sources change, and any of those can silently degrade quality.
Running the evaluation set in the deployment pipeline turns that invisible drift into a visible failure. It is the same discipline as unit testing, applied to a component whose behaviour is probabilistic rather than deterministic.
Key takeaways
- Public benchmarks measure public benchmarks, not your task.
- Fifty to two hundred real cases beat any leaderboard.
- Expert disagreement means the task is ambiguous, not the model wrong.
- Decide which error type is worse before choosing a metric.
- Compare against the current process, then keep the set as a regression suite.
Frequently asked questions
How large should an LLM evaluation set be?
Usually fifty to two hundred cases sampled from real traffic, provided they include the ambiguous and malformed inputs rather than only well-formed examples. Quality and coverage of edge cases matter far more than volume.
Should we use public benchmarks to choose a model?
They are useful for tracking the field but not for deciding whether a model suits your task. A benchmark knows nothing about your terminology, document formats or definition of correct, so build an evaluation set from your own material.
Evaluate AI on Your Data?
Evaluate AI on Your Data →