WHITEBOXX / GLOSSARY

LLM evaluation: testing AI
before you rely on it.

LLM evaluation is the testing of a language model, or a system built on one, against defined cases and criteria. It measures whether the system gives correct, safe and useful results for your task, and whether that stays true when the model, prompts or data change.

By Gautam Mehra · Published · Updated

What is LLM evaluation?

Public benchmarks compare models on general tasks. They say little about how a model will perform on your documents, your data and your definitions. Enterprise evaluation uses a test set built from your own work: real inputs, with the results an expert would expect.

For agents, evaluation covers the whole run, not only the final answer. A correct answer reached by calling the wrong tool, or at ten times the expected cost, is still a failure.

METHODS

Evaluation methods.

METHODS
MethodHow it worksBest for
Reference answersCompare output with a known correct answerExtraction, calculations and classification
Rubric scoringA person or a model scores output against written criteriaSummaries, reports and other open-ended work
Expert reviewExperts judge a sample of resultsSetting the standard and checking automated scores
Production monitoringScore a sample of live runs over timeCatching changes after release

TEST SET

How do you build a test set?

  1. 01

    Collect real cases

    Use past inputs from the task, including awkward ones.

  2. 02

    Write the expected result

    Ask the people who do the work today what a correct answer looks like.

  3. 03

    Add cases that should fail

    Include requests the system should refuse or pass to a person.

  4. 04

    Run it on every change

    Test again when you change the model, prompts, tools or data.

  5. 05

    Track cost and time too

    Record what each case costs and how long it takes.

Why does evaluation matter for risk?

The NIST AI Risk Management Framework, published in January 2023, makes measurement one of its four core functions, alongside govern, map and manage. If you have not tested a system against your own cases, you do not know its error rate.

How we test Evidence OS extensions.

Each extension project is tested against the user’s task before release: the data path, the analytical result and what happens when inputs are missing.

See delivery and operations