Evaluating LLM Outputs

You can run an LLM in a demo and feel impressed. You cannot ship it without measuring what it produces. Evaluation is how you find out whether a prompt change made things better or just different, whether a model upgrade is worth the cost, and whether a regression slipped in overnight. This guide covers the practical stack: metrics, LLM judges, pairwise comparison, human rubrics, and regression sets.

Why exact-match metrics break

Traditional NLP metrics compare model output against a reference answer. BLEU counts overlapping n-grams, ROUGE counts overlapping sequences (common in summarization), and exact match demands a character-perfect answer. They work for short, constrained tasks like translation of a single sentence.

They fail for open-ended generation. A chatbot can answer the same question five correct ways and match none of the reference strings. BLEU punishes paraphrase even when the paraphrase is better. These metrics measure surface similarity, not correctness or usefulness, so treat them as smoke tests. If ROUGE drops 30 points after a prompt change, something changed. If it drops 2 points, that number tells you nothing about quality.

Semantic metrics

A second generation of metrics compares meaning instead of tokens. BERTScore computes similarity using contextual embeddings from a model like BERT, so a correct paraphrase scores high even with different wording. It correlates better with human judgment than BLEU on many tasks, according to its original paper.

Semantic metrics still need reference answers, which you may not have for creative or open tasks. They also inherit the biases of their underlying embedding model. Use BERTScore as a complement to exact-match metrics on tasks with known-good references, not as a standalone quality score. For anything subjective, tone sensitive, or instruction following, you need a judge.

LLM-as-judge setups

The most practical evaluation method today is asking a strong model to grade outputs. Zheng et al. showed that GPT-4 as a judge agrees with human preferences over 80% of the time on open-ended chat tasks, comparable to human to human agreement. Liu et al. improved the technique in G-Eval by having the judge first write out evaluation steps, then fill in scores in a structured form.

A minimal judge prompt has three parts: the rubric, the output, and the verdict format. Keep the rubric specific. "Rate this answer from 1 to 5" produces noise. "Rate factual accuracy: does the answer state only claims supported by the provided context, 1 for multiple unsupported claims, 5 for all claims supported" produces signal.

```python

JUDGE_TEMPLATE = """You are evaluating an assistant response.

Rubric (factual accuracy, 1-5):

1: multiple claims not supported by the context

3: mostly supported, one unsupported claim

5: every claim supported by the context

Context: {context}

Response: {response}

First write 2-3 sentences of reasoning, then output: SCORE=<1-5>

"""

```

Ask for reasoning before the score. Models that explain first score more consistently. Use temperature 0 and run each judgment at least twice if the decision matters; score variance across runs tells you how much to trust the number.

The biases you must control

LLM judges are not neutral. The documented failure modes from the MT-Bench paper and follow-up work:

Position bias: in pairwise comparison, judges favor whichever answer is listed first or second. Fix by swapping the order and averaging, or running both orders and requiring agreement.

Verbosity bias: judges reward longer answers even when the extra text adds nothing. Counter it by instructing the judge to penalize fluff, or by capping output length before judging.

Self-enhancement bias: a model tends to prefer outputs from its own family. Do not judge a model with itself when the result decides a vendor contract. Use a different judge family than the candidates, or calibrate against human scores.

Calibration matters. Before trusting a judge pipeline, score 50 to 100 outputs by hand and compute agreement with the judge. If agreement is below what you need, refine the rubric before scaling up. A judge that agrees with you 70% of the time is a productivity tool, not a replacement for review.

Pairwise comparison

Instead of scoring one answer in isolation, show the judge two answers and ask which is better. Pairwise comparison is more reliable than absolute scoring because models are better at ranking than at grading. It is also how MT-Bench and the LMSYS Chatbot Arena collect judgments.

Feed many pairs into an Elo or Bradley-Terry rating system and you get a ranked leaderboard of prompts or models. The cost is quadratic in the number of candidates if you compare everything to everything, so in practice you compare each candidate against a fixed baseline and a few neighbors. This is the standard workflow for deciding which prompt version wins.

Human evaluation rubrics

Automated evaluation scales; human evaluation tells you what good means. Build a rubric with 3 to 5 criteria, each with 3 to 5 levels described in concrete terms. Bad level description: "Tone: poor to excellent." Good level description: "Tone, 1: uses slang or sarcasm in a customer-facing answer. 2: neutral but stiff. 3: professional and natural."

Sample outputs from production traffic, not cherry-picked ones. Have two raters score each item and measure inter-rater agreement; low agreement means the rubric is ambiguous, not that the raters are wrong. Use human eval to define the rubric and calibrate the LLM judge, then let the judge handle volume. Re-run human spot checks weekly on live traffic, because production distributions drift.

Regression test sets

The highest ROI evaluation practice is a regression set: 50 to 200 input/output pairs covering your product's critical behaviors, with expected answers or judge rubrics pinned for each. Run it on every prompt, model, or configuration change before deploying.

Keep the set versioned in your repo. Include failure cases you have actually seen in production, not hypothetical ones. Tag each case with the behavior it guards (formatting, refusal, tone, factual accuracy) so a failure tells you what broke. When a new bug appears in production, add a case for it before fixing the prompt. Over a few months this becomes the contract your system keeps.

Putting it together

A workable evaluation stack for a production team: exact-match and semantic metrics for tasks with references, an LLM judge with a written rubric and measured calibration for open tasks, pairwise comparison for prompt and model selection, a regression set gating every deploy, and human raters defining the rubric and spot-checking live traffic. Start with the regression set. It is the cheapest thing here and the one that prevents the most damage.