Evaluating generated output without fooling yourself

#The evaluation that is not one

A team changes a prompt, runs twenty examples they wrote themselves, reads the outputs, and decides it is better. That is a smoke test. It cannot detect a regression on the cases nobody thought to write, and the twenty examples were chosen by the person who wrote the prompt.

#What to build first

A baseline. Run the suite, save every output and every score. The next run compares against it. Without this you can tell that a change is good and not that it is better, and those are different claims.

Scenario coverage. Factual recall, summarisation, extraction, classification, reasoning, creative generation, safety refusal and multi-turn conversation stress different behaviours. A change that improves extraction can degrade refusal, and a suite that only tests one will not see it.

Scoring you can defend. Exact match works for extraction and not for summarisation. A model as judge works better for open-ended output and introduces the judge's own biases, including a preference for longer answers. Calibrate the judge against human ratings on a sample before trusting it.

#Adversarial cases are not optional

A suite made of reasonable inputs measures behaviour on reasonable inputs. Users are not uniformly reasonable and some of them are hostile.

Generating attacks against your own configuration, rather than testing against a fixed public corpus, is the part that finds real problems: prompt injection, jailbreak attempts, extraction of information the system should not disclose, and inputs designed to elicit confident fabrication. Hallucination is a persistent behaviour of these systems rather than a bug to be fixed once [1], so it needs continuous measurement.

#The uncomfortable finding

Running adversarial suites against my own layered content safety, the boundary layer of plain deny lists consistently accounts for more true positives than the classifier layers.

That is an unwelcome result for a design that leads with cognitive framing, and it has replicated every time I have measured it. Written-down rules work. Semantic detection is the interesting part and it is not carrying the weight I expected.

#Treat it as a production system

Evaluation infrastructure is a machine learning system and it accumulates the same debt [4]. A production readiness rubric is a useful discipline here, particularly the parts about data validation and monitoring [3].

The risk management framing is also worth adopting early, because it asks who is harmed and how the system is measured rather than only whether it scores well [5].

#What the numbers do not tell you

An aggregate pass rate hides the distribution. A change that improves the average while making the worst case much worse is usually a bad change, and only per-case comparison against the baseline shows it.

Alignment work has repeatedly shown that human preference and automated scores diverge [2]. Automated evaluation is necessary for iteration speed and it is not a substitute for looking at outputs.