We study how to evaluate AI systems and what we can reasonably conclude from the results. This includes measuring capabilities, examining how models represent people, and testing the assumptions behind widely used evaluation methods.
Benchmarks are useful, but evaluation is a larger field. Understanding how people use AI, or how that use changes their behavior over time, requires studying people and the conditions in which they use it. The choice of method should follow the research question. There is no reason to expect every important question to fit a benchmark.
Researchers should be able to use one another's evaluation work. That requires shared implementations and results recorded in compatible formats, with the metadata needed to decide which comparisons are valid. We build infrastructure that lets teams contribute to a common body of evaluation data rather than start from scratch.
The harness gives researchers a shared set of task implementations and a common way to configure and run evaluations. A team can evaluate a new model using the same task, prompt, and scoring setup as earlier work, or contribute a new evaluation for others to use. These shared implementations let researchers extend one another's work without separately rebuilding each benchmark.
Explore the libraryEvery Eval Ever brings evaluation results from different tools into a common format and public database. A researcher can retrieve earlier runs and the metadata describing their setup, select results that are comparable to a new experiment, and run only the models they need to add. Reusing compatible results avoids the cost of evaluating every baseline again and makes studies across many models and benchmarks practical.
Read the introduction
Benchmarks need to change as the systems we build, and what we expect of them, change. Keeping a test useful means revisiting its tasks, evaluation methods, and the conclusions we draw from the results. A test designed for a model fine-tuned on labeled examples may still be useful for a conversational assistant, but changes in prompting, interaction, and scoring can change what a score means. Updating a benchmark also raises questions about which comparisons with earlier results remain valid.
Saturation is one example of this broader problem. In a study of 60 benchmarks, we examined whether scores could still reliably distinguish leading models. Scores can remain below perfect accuracy while the gaps between models become too small relative to uncertainty to support a ranking.
In that study, we found no statistically significant difference in saturation between public and private test sets.
Keeping test data private did not show the expected advantage in this study. Only 4 of the 60 benchmarks were private, however, so the comparison is limited: failing to detect a difference does not establish equivalence.
A benchmark turns selected observations into a number. Whether that number tells us anything about someone’s experience with a system is a separate question. A higher score might reflect better performance on the test without reducing the failures that matter in use. We want evaluations to establish that connection, rather than assume it.
More of the Same offers one example. Models frequently generated descriptions of women, yet stereotyped portrayals persisted. Increased representation would look like progress under a simple counting metric while missing how those people were described.
To understand consequences, we may need studies of people using systems, not only tests of models in isolation. In work on social harms, we draw on social-science research to decide which patterns to investigate and how to interpret them. A reproducible statistic still needs an argument connecting it to the harm or benefit it is meant to measure.
We co-founded the EvalEval Coalition with Hugging Face and the University of Edinburgh. EvalEval connects researchers working on evaluation across institutions, with joint projects including Every Eval Ever and our benchmark-lifecycle research.
Shared reporting standards allow results collected by different groups to be used together. The coalition develops these standards alongside the tools and research that use them, so contributions from one project can support work elsewhere.
Visit EvalEval