How to Measure AI Output Quality When There Is No Single Right Answer

An AI answer can be accurate, useful, and well written without matching the answer you expected. It can also sound excellent while missing the user's goal. Those two facts make open-ended evaluation uncomfortable.

The uncomfortable part isn't that quality is subjective. The uncomfortable part is that teams often treat a subjective judgment as if it were an objective measurement, then build a release gate on top of it.

I have taught product managers who went from writing specs to building and shipping the product, and automating work around it. I kept seeing the same missing piece: nobody could say what “done” meant for the user. The model was not the first problem. The measurement contract was.

Result: For open-ended AI output, release against four separate evidence types: an observable outcome, criterion checks, a calibrated judgment signal, and veto conditions. Use pointwise rubrics for absolute quality, pairwise comparisons to choose between candidates, and human or user outcomes to test the real job. Never average a critical failure away.

What does AI output quality mean when several answers can be valid?

AI output quality is not one property that exists independently of a task. It is a judgment about whether an output satisfies the criteria and consequences that matter for a particular use.

That sounds obvious, but it changes the entire evaluation design. If the task is “classify this request as billing, technical, or other,” exact labels may be appropriate. If the task is “write a tactful response to a frustrated customer,” several responses can work. If the task is “summarize this policy for a new employee,” the language can vary while the required facts and safety limits stay fixed.

The first task is close to a reference-answer problem. The second and third are not. Define what must be true, preferred, and forbidden.

Separate four things people call quality

Use four separate constructs before you choose a metric:

Construct	Question it answers	Typical evidence	What it cannot prove
Outcome quality	Did the user achieve the intended result?	A completed task, correct state change, accepted deliverable, or successful next step	Why the result happened or whether the wording was good
Criterion quality	How well did this output satisfy the agreed dimensions?	A rubric score for accuracy, relevance, completeness, clarity, tone, or groundedness	Whether the user actually used it
Comparative preference	Which candidate is better for this task?	A blinded A versus B choice, including ties	How good either answer is in absolute terms
Constraint integrity	Did the output avoid unacceptable behavior?	Deterministic checks, policy review, grounding checks, or an escalation rule	Whether a safe answer was useful enough

These constructs can disagree. A customer may prefer a concise response that omits a necessary warning. A model judge may give a polished answer a high coherence score even though its main claim is not supported by the retrieved context. A user may complete a task despite a confusing explanation because they already knew what to do.

That disagreement is not noise to erase. It tells you that you are measuring different parts of the job.

Why one score is tempting

A single score is easy to put on a dashboard. It makes model A and model B look comparable. It gives a team something to celebrate after a prompt change. It also makes an unresolved product question look settled.

The temptation becomes stronger when the output has no single right answer. Without an exact-match target, a number feels like a substitute for certainty. But a number only becomes meaningful after you specify the construct, the scale, the reference frame, the reviewer, and the decision it will drive.

Suppose a response receives these results:

Signal	Result	What the result actually says
Relevance rubric	4 of 5	A judge saw the answer as relevant under the stated rubric.
Groundedness check	Pass	The required claims were supported by the supplied context, if the check itself is valid.
Pairwise preference	Candidate wins	A judge or person preferred this candidate to a baseline for this item.
Task completion	Fail	The user did not reach the intended next state.

The answer did not receive a quality score of “3.5.” It received evidence about four different questions. The failed task outcome should matter more than the pleasant average if the product's promise is task completion.

The AI agent evaluation guide makes a related distinction for systems that act through tools: inspect the result, actions, integrity, limits, and stability instead of grading only the final message. This article focuses on open-ended output, but the same discipline applies. A response is part of a system's behavior, not the whole system.

The practical definition

For an open-ended output, define quality as:

the degree to which the output satisfies the intended user outcome, meets explicit criteria, and stays inside non-negotiable constraints for the deployment context.

This is a working definition, not a universal theory. Its value is that it gives you something to measure and revise when the measurement fails.

What is a measurement contract for an open-ended AI output?

A measurement contract is a short agreement that states what quality means, how it will be observed, who or what will judge it, how disagreement will be handled, and what blocks release.

The contract is deliberately smaller than a full evaluation platform. A small team should be able to put it next to a prompt, workflow, or product requirement before writing a judge prompt.

The four required parts

Every contract needs four parts:

1. Observable outcome. What should be different for the user or system after the output is used?
2. Criterion checks. Which properties can be judged separately, and what does pass or fail mean for each one?
3. Calibrated judgment. Which human, code-based, or model-based grader will supply the signal, and what evidence says the grader is fit for purpose?
4. Veto conditions. Which failures must block the output or release even if other scores look good?

The first part protects you from grading language instead of work. The second protects you from a single vague “helpfulness” score. The third protects you from trusting an untested judge. The fourth protects you from averaging away the failure that matters most.

A copyable contract

This is a template, not a tested benchmark. Fill it with the requirements of the actual workflow.

evaluation_contract:
  name: support-answer-quality
  version: 1
  owner: support-product

  intended_use: "Help a support agent answer a customer question from approved sources"
  user_outcome: "The customer understands the next safe action or is routed to a person"

  output_construct:
    unit: one_response
    context_required: approved_support_sources
    valid_variation: "Different wording is acceptable when meaning and policy are preserved"

  criteria:
    - id: groundedness
      question: "Are factual claims supported by the supplied sources?"
      scale: "pass_fail"
      evidence: source_claim_pairs
    - id: completeness
      question: "Does the answer cover the required facts and next step?"
      scale: "0_to_3"
      anchors:
        0: "Misses the request or provides no usable next step"
        1: "Addresses the topic but omits a required element"
        2: "Covers the required elements with minor gaps"
        3: "Covers the required elements clearly and accurately"
    - id: clarity
      question: "Can the intended reader follow the answer without reconstructing it?"
      scale: "0_to_3"
    - id: tone
      question: "Does the response fit the interaction and avoid avoidable friction?"
      scale: "0_to_3"

  graders:
    deterministic:
      - forbidden_claim_check
      - citation_or_source_check
      - required_field_check
    model_judge:
      model: "record exact model and version at run time"
      mode: pointwise
      rubric_version: 1
    human_sample:
      audience: support_domain_reviewer
      blind_to_system_variant: true

  calibration:
    labeled_examples: "record the approved set and its date"
    compare_judge_to_human: true
    review_disagreements: true
    order_swap_pairwise_cases: true

  aggregation:
    absolute_quality: "report criterion distributions, not only a mean"
    candidate_choice: "report pairwise wins, losses, ties, and order sensitivity"
    deployment_signal: "report task completion and correction or escalation behavior"

  vetoes:
    - unsupported_high_risk_claim
    - privacy_or_policy_violation
    - unsafe_or_missing_escalation
    - failed_required_user_outcome

  decision:
    release: "all vetoes pass and approved criteria meet their thresholds"
    hold: "any veto fails or the judge is not calibrated for this use"
    investigate: "human disagreement or outcome drift exceeds the review rule"

The contract forces a useful distinction. “Valid variation” says you will not reject a response because it uses different words. The criteria say what cannot vary, such as a required warning or a supported claim. The vetoes say what the average cannot overrule.

NIST's AI Risk Management Framework describes measurement as quantitative, qualitative, or mixed and calls for documented methods, uncertainty, comparisons, and limitations that depend on context. A measurement contract is a small implementation of that idea for a product team. It does not replace risk management or domain review. It makes the assumptions visible. NIST's Measure guidance is the primary reference for that context-sensitive approach.

When to refuse a single contract

Some products contain several distinct jobs. A writing assistant may draft an email, summarize a legal document, propose code, and answer a customer question. One universal rubric would hide differences in risk and user expectation.

Create separate contracts when any of these change:

• the intended user outcome;
• the evidence or context the answer must use;
• the cost of an incorrect answer;
• the required escalation or approval path;
• the population judging the output;
• the product behavior after the answer is accepted.

The contracts can share a field vocabulary. They should not share thresholds simply because the dashboard wants one line.

Which measurement should you use for each AI quality decision?

Choose the measurement from the decision you need to make. Do not choose it from the metric that is easiest to calculate.

Decision you need to make	Primary measurement	Supporting measurement	Main trap	Release use
Did the output obey a hard rule?	Deterministic check or expert policy review	Model explanation only as a debugging aid	A fluent answer can still violate the rule	Veto or block
How good is this answer against agreed criteria?	Pointwise rubric score by criterion	Human sample and judge calibration	A high score can hide weak criteria or a judge's style preference	Threshold plus distribution review
Is version B better than version A?	Blind pairwise comparison	Order swap, tie rate, and human sample	Pairwise choice does not reveal absolute quality	Candidate selection
Did the output help the user finish the job?	Task outcome or behavioral signal	Response rubric and user feedback	Completion may be caused by prior knowledge or an easier task	Product decision
Is the system stable across conditions?	Regression cases and repeated trials	Slice-level quality and variance	An average can hide a failing user segment	Release and monitoring
Is the evaluation instrument trustworthy?	Agreement with blinded human labels and disagreement review	Position swap and known-case tests	The judge can be consistent and still consistently wrong	Calibrate or replace judge

Use deterministic checks for invariants

Deterministic checks are the right first tool when a requirement is actually deterministic. Check whether a structured field exists, whether a citation points to an allowed source, whether a forbidden phrase appears, whether a required step is present, whether the answer exceeds a limit, or whether an action was authorized.

These checks are cheap, reproducible, and easy to debug. Anthropic describes code-based graders in similar terms and also points out their weakness: they can be brittle when valid outputs vary. That means you should use them for invariants, not for a vague question such as “is this answer helpful?” Anthropic's grader comparison explains the trade-off.

If a requirement can be stated as a clear predicate, encode it. If that would reject an acceptable alternative, rewrite it or move it to a rubric.

Use a pointwise rubric for absolute quality

Pointwise evaluation shows one output to a grader and asks for a score or label against defined criteria. It is appropriate when you want to track quality over time, inspect regressions, or decide whether a single output clears a minimum bar.

A pointwise score must be decomposed enough to be useful. “Overall quality, one to five” produces a number, but it rarely tells you what to fix. Separate groundedness, completeness, relevance, clarity, tone, and domain-specific criteria when each could change the product decision.

More criteria do not always improve the instrument. Similar criteria can cause a judge to repeat the same impression. Keep a criterion only when it changes a decision or exposes a distinct failure.

Google's evaluation documentation shows pointwise results as rubric-based scores and includes explanations for why a response received a particular rating. That is useful for interpreting an item, but the explanation is not independent proof. It is another output from the judge. Google's pointwise evaluation example makes the mechanics clear.

Use pairwise comparison for a choice between candidates

Pairwise evaluation asks which of two responses is better for the same input and criteria. It is useful when you are choosing a prompt, model, retrieval strategy, or product variant and the practical decision is comparative.

Pairwise evaluation removes one difficult demand from the reviewer. They do not have to invent an absolute score for every response. They only have to decide which candidate better serves the specified job, or whether the difference is too small to call.

But pairwise comparison has a hard limit. If candidate B wins 58 percent of its comparisons, that does not mean B is 58 percent good. It means B was preferred in that comparison design. Pairwise results also depend on the baseline, the prompt order, the criteria, the judge, and the treatment of ties.

Run an order swap. Show the same pair with A first and then B first. Record whether the verdict changes. A 2025 study of LLM judges reports that position bias varies across judges, candidates, and tasks, and it evaluates position consistency and preference fairness as explicit concerns. The order swap is a simple local diagnostic, not a guarantee of unbiased judgment. Judging the Judges provides the research context.

Use outcome evidence for the product promise

If the product promises a completed task, measure the task. If it promises a draft, measure whether the user accepts or meaningfully edits the draft. If it promises a safe answer, measure correct abstention and escalation as well as useful answers.

Outcome evidence can be a state check, a test result, a user-confirmed completion, a correction rate, a successful next action, or an expert-approved deliverable. The right signal depends on the workflow.

Outcome evidence is not automatically better than judgment. A user can click “done” because the interface is confusing. A task can complete while the answer contains a dangerous unsupported claim. Treat outcomes and criteria as complementary, then decide which is the primary release condition.

How do you write a rubric for subjective AI output?

Write the rubric as a set of observable decisions, not as a list of adjectives.

“Helpful, accurate, and professional” sounds reasonable. It is too vague for repeatable judgment. A useful rubric tells the reviewer what evidence would justify each label and what would make the output fail.

Start with the user action

Before you name a criterion, complete this sentence:

After reading or using this output, the user should be able to ____.

Examples:

• choose the correct next step without opening the internal policy;
• identify which facts are confirmed and which are uncertain;
• revise the generated draft without hunting for missing requirements;
• understand why the recommendation fits the stated constraints;
• escalate a case when the system lacks enough evidence.

The blank gives you a job. The rubric can then ask whether the output made that job possible.

Keep criteria independent enough to diagnose

Use criteria that point to different fixes. For a grounded answer, these might be:

Criterion	What a high score requires	What a low score looks like	Likely fix
Groundedness	Claims are supported by the allowed context, with uncertainty where support is incomplete	The response adds claims the context does not support	Retrieval, citation, abstention, or prompt change
Completeness	The answer covers the required facts, constraints, and next action	A central requirement is missing	Context selection, output schema, or rubric change
Relevance	The answer spends attention on the user's actual request	It answers a nearby or broader question	Intent handling or prompt change
Clarity	The reader can follow the answer and act without reconstruction	Important information is buried or ambiguous	Structure, length, or editing policy
Tone	The response fits the situation without sounding dismissive or falsely certain	It creates avoidable friction or misstates confidence	Style instruction, examples, or human review

Microsoft's evaluation guidance distinguishes reference-free constructs such as fluency, coherence, relevance, and groundedness rather than treating them as one metric. The set should fit your product; separate criteria expose distinct evidence and fixes. Microsoft's metric guidance documents these distinctions.

Define anchors with examples

A scale is not a rubric until the labels mean something. Give each important score an anchor and include examples of acceptable variation.

For a three-level completeness criterion:

0 - Fails: misses the user's request or gives no usable next step.
1 - Partial: addresses the topic but omits one required element.
2 - Meets: covers every required element and gives an actionable next step.

If you need a four- or five-level scale, add meaningful distinctions rather than decorative numbers. For example, separate “meets” from “exceeds” only if the team can explain what extra value changes the decision. Otherwise, the extra level will create disagreement without creating information.

Examples should include boundary cases:

• a concise answer that contains every required fact;
• a long answer that contains every fact but adds an unsupported claim;
• a polite answer that refuses when it should answer;
• a useful answer that correctly escalates an ambiguous case;
• two differently worded responses that are both acceptable.

The point is not to teach the judge to imitate the examples. The point is to make the construct visible to the people who own the outcome.

Make uncertainty a valid result

Open-ended output often fails because the evaluator rewards a confident answer even when the evidence is incomplete. Add an explicit option for “not enough information,” “should abstain,” or “needs escalation” when that is part of the correct behavior.

Abstention is not a general quality virtue. An assistant that refuses every difficult question is safe but useless. The rubric should reward abstention only when the stated evidence or policy requires it, and it should penalize unnecessary refusal when the answer was supportable.

Avoid criterion leakage

Criterion leakage happens when a rubric accidentally reveals the preferred answer or rewards a proxy that is easier to see than the intended quality. A judge may reward length because longer answers look thorough. It may reward a confident tone because confidence is easier to recognize than evidence. It may reward the presence of citations even when the citations do not support the claim.

Write criteria in terms of the user job and inspect the evidence. “Includes three citations” is a format check. “Every material factual claim is supported by the allowed sources” is a grounding criterion. The format check can support the grounding check, but it cannot replace it.

When should you use a model as the judge?

Use a model judge when the criterion is open-ended, the output volume makes full human review impractical, and you can validate the judge against a human-labeled sample.

Do not use a model judge because the team has no agreement about what quality means. A model cannot repair an undefined construct. It will turn vague instructions into a repeatable-looking guess.

What model judges are good at

Model-based grading can handle free-form answers, natural language criteria, comparisons, and explanations. Anthropic describes model graders as flexible and scalable while also noting non-determinism and the need for human calibration. That combination makes them useful for screening, regression detection, and triage. It does not make them an authority. Anthropic's evaluation guidance states the trade-off directly.

G-Eval is a useful research example. Its authors use a rubric-guided GPT-4 evaluator for summarization and dialogue, and report a Spearman correlation of 0.514 with human judgments on the studied summarization task. They also identify a possible bias toward LLM-generated text. The result supports a narrow claim: a rubric-guided model judge can be a useful signal in a studied setting. It does not support a universal threshold or a claim that a judge replaces human review. The G-Eval paper contains both the result and the caveat.

What model judges are bad at by default

A model judge may:

• prefer a longer answer even when brevity is part of the job;
• reward smooth wording over source support;
• use its own knowledge when the task requires judging only supplied context;
• accept a persuasive explanation of a wrong answer;
• choose the first option in a pairwise prompt;
• apply a rubric inconsistently across domains or languages;
• become lenient when the generated answer sounds like its own style;
• hide uncertainty behind a neat score.

These are failure modes to test, not reasons to reject every model judge. The judge is an instrument, so it needs calibration, drift checks, and a stated domain of use.

Use separate judge calls when criteria conflict

One prompt that asks for relevance, correctness, completeness, tone, safety, and overall quality may produce a pleasing report with correlated scores. The judge forms one overall impression, then distributes that impression across the columns.

Separate calls are not always necessary, especially when cost matters. But separate criteria when they have different evidence requirements or different release consequences. A groundedness judge should inspect the allowed context and claim pairs. A tone judge should not be allowed to override a grounding failure. A policy judge should operate as a veto or a separate review path, not as one more point in an average.

How do you calibrate an LLM judge against human judgment?

Calibration means testing whether the judge's labels are useful for your decision, on your data, with your rubric, before you trust it at scale.

The process is not “ask two people whether the AI judge seems reasonable.” It is a small measurement study with a clear dataset, blinded labels, and disagreement review.

Build a calibration slice

Start with examples from the real output distribution, not only the easy demos. Include:

1. clear passes;
2. clear failures;
3. acceptable alternatives with different wording;
4. boundary cases close to the release threshold;
5. known failures from users, reviewers, or incident logs;
6. cases where the evidence is incomplete and abstention may be correct;
7. cases from important user or language slices.

Do not describe this as a statistically powered benchmark unless you designed one. It is a calibration slice. Its purpose is to find whether the judge understands the contract well enough for a stated use.

Blind the relevant information

If you are comparing model variants, remove the system name and randomize the order. If the judge can see which answer was generated by the newer model, its preference may be influenced by the label rather than the content. If human reviewers know the candidate's origin, they may also bring expectations into the judgment.

Blinding cannot remove every bias. It can remove an avoidable source of bias and make disagreements easier to interpret.

Compare labels, not explanations

The judge's written explanation can help you debug the rubric. It is not the same thing as agreement with a human label. Compare the actual score, pass/fail label, or pairwise choice. Then inspect the explanation for evidence that the judge followed the criterion.

For a pointwise judge, record:

• human criterion label;
• judge criterion label;
• exact output and context shown to both;
• judge version and rubric version;
• the type of disagreement;
• whether a domain reviewer resolved the case;
• the action taken on the rubric or judge.

For a pairwise judge, also record:

• candidate order;
• the swapped-order verdict;
• tie availability;
• human pairwise choice;
• whether the pair was too close to call;
• whether the preferred candidate passed all veto checks.

Google's judge-evaluation documentation recommends preparing human ratings for pointwise metrics and human pairwise choices for pairwise metrics, then comparing the model-based metric with those labels. That is the right shape of validation even if you do not use Google's SDK. Google's judge evaluation guidance is explicit about the ground-truth columns.

Look at the disagreement matrix

Do not reduce calibration to one agreement percentage. Build a table that shows where the judge and humans disagree.

Human label	Judge pass	Judge partial	Judge fail	What to inspect
Pass	Agreement	Over-strict judge	Hidden judge failure	Boundary examples and valid variation
Partial	Over-lenient judge	Agreement	Over-strict judge	Anchor definitions
Fail	Hidden judge failure	Over-lenient judge	Agreement	Veto cases, evidence checks, and failure severity

For safety or privacy criteria, a false pass may matter far more than a false fail. For tone, a false fail may cost time but not create a serious incident. Calibration quality is therefore decision-specific. A judge that is acceptable for triaging style issues may be unfit for approving a customer-facing compliance answer.

Check slices, not just the aggregate

A judge can look accurate overall while failing for one important group, document type, language, or interaction mode. Slice the calibration set by whatever changes the expected quality or risk:

• short versus long inputs;
• known versus ambiguous requests;
• novice versus expert users;
• supported versus missing context;
• different product areas;
• different languages or dialects;
• high-impact versus low-impact actions;
• answers that should abstain versus answers that should proceed.

NIST advises documenting the limits of generalization beyond the conditions in which an AI system was developed and evaluated. The same logic applies to a judge. A calibration claim is only as broad as its test conditions. NIST's Measure function provides the broader validity and deployment-context guidance.

Set a recalibration trigger

Do not calibrate once and forget the instrument. Recalibrate when:

• the model judge changes;
• the rubric changes;
• the output model or prompt changes;
• the retrieval source changes;
• the product starts serving a new user group;
• human disagreement clusters around a new failure;
• production corrections rise without a corresponding offline change;
• the judge's score distribution shifts sharply.

The trigger can be simple. “Review the calibration slice after every judge or rubric change, and monthly after launch” is better than “we will monitor quality.” The exact cadence should follow the risk and the speed of change.

How should humans participate in subjective AI evaluation?

Humans should define the construct, calibrate the automated judge, review ambiguous or high-risk cases, and check whether offline scores still connect to the real job. They do not need to read every low-risk output forever.

Human review is not a ceremonial final step. It is the reference process that tells you whether your automated measurement is still pointed at the intended quality.

Assign the right reviewer

Use a domain reviewer when the criterion requires domain judgment. Use a representative user when the criterion is about usefulness or preference in context. Use a safety or policy reviewer when the failure could harm a person, expose data, or create a non-reversible action.

Do not ask a general reviewer to decide a specialized question just because they are available. A reviewer can judge clarity without being able to confirm a medical, legal, financial, or operational claim.

Reduce reviewer ambiguity

Give reviewers:

• the original user request;
• the context the system was allowed to use;
• the response under review;
• the criterion and its anchors;
• instructions about acceptable variation;
• a clear abstain or uncertain option;
• a way to record the evidence for a failure;
• no unnecessary information about the system variant.

Ask one judgment at a time. “Is the answer grounded?” is easier to calibrate than “How good is this answer?” If the output fails groundedness, let the reviewer say which claim is unsupported. That creates a repair path instead of a low number.

Treat disagreement as data

Disagreement can mean at least four things:

1. the rubric is ambiguous;
2. the case is genuinely close;
3. reviewers have different domain assumptions;
4. the output sits between two valid product choices.

Do not force a consensus too early. Record the disagreement, discuss a sample, and decide whether the contract needs a new anchor, a separate user slice, a tie label, or a human-only review rule.

If reviewers disagree because the product has not chosen a preference, the solution is not a smarter judge. The product owner must choose what matters, accept several valid variants, or state that the decision is out of scope.

Use human review to improve the instrument

Every reviewed case should have a possible destination:

• add a deterministic check;
• add or revise a rubric anchor;
• add a regression case;
• change the prompt or retrieval behavior;
• route the category to a human;
• lower the scope of the product promise;
• accept the variation as valid.

That is how a qualitative review becomes engineering input. It also keeps the review from turning into a recurring meeting where everyone notices the same issue and nobody changes the system.

How do you connect AI output quality to real user outcomes?

Connect each output criterion to the next observable step in the user's workflow. The link does not need to be a perfect causal experiment. It does need to be explicit enough that you can tell when the metric is drifting away from usefulness.

Build an outcome chain

For each use case, write:

Input context
  -> AI output
  -> user interpretation or action
  -> intended workflow state
  -> correction, escalation, abandonment, or completion

Then map measurements to the chain:

Stage	Example signal	Question
Input context	Missing-field or ambiguity rate	Did the system receive enough information to answer?
AI output	Groundedness, completeness, clarity	Did the response meet the contract?
User interpretation	Follow-up question, correction, acceptance	Did the reader understand or trust it enough to continue?
Workflow state	Completed task, correct record, approved draft	Did the intended result happen?
Recovery	Escalation quality, safe refusal, human correction	Did the system fail in a controlled way?

This chain prevents two common mistakes. First, it stops you from treating a good-looking answer as a completed task. Second, it stops you from treating a user click as proof that the answer was correct.

Choose behavioral signals carefully

Useful product signals can include:

• whether the user accepts, edits, or rejects the output;
• whether the user asks the same question again;
• whether a human corrects the output;
• whether the workflow completes after the output;
• whether the user escalates to a person;
• whether the user abandons the flow;
• whether the downstream record passes a domain check.

These are evidence, not truth. A user may accept an answer because they did not notice the error. A repeat question may mean the answer was bad, or that the user is exploring. An escalation may be a healthy safety behavior rather than a failure.

Label the signal with its interpretation and collect a sample for review. Do not call it “quality” until you have checked what it means in the workflow.

Treat production as a new source of cases

Production feedback should not replace offline evaluation. It should expand it. When a user corrects an answer, sanitize the case, record the intended outcome, and add the failure to the relevant contract. When users disagree with the output but the system behaved within the contract, decide whether the contract missed a preference or whether the variation is acceptable.

Anthropic describes the value of combining automated evaluation, production monitoring, and periodic human review. Its agent example focuses on traces and outcomes, but the loop applies to open-ended answers as well. Anthropic's evaluation lifecycle guidance supports keeping these signals distinct and complementary.

Do not optimize for the easiest user signal

If the dashboard rewards acceptance, the system may become agreeable. If it rewards short sessions, the system may omit explanations. If it rewards low escalation, the system may answer questions it should route. If it rewards positive pairwise judgments, the system may become longer and more confident.

Tie the metric to the promise and keep a counter-metric for the most likely shortcut. A safe assistant might track completion and unsupported-claim rate together. A drafting assistant might track acceptance and substantive edit rate. A research assistant might track useful citations and expert correction rate.

There is no universal counter-metric. The team must identify how the system could appear better while doing the job worse.

What failure modes make subjective AI evaluation misleading?

Subjective evaluation fails in predictable ways. The fix is usually to narrow the construct, change the evidence, or add a control.

Failure mode 1: grading against a reference that is only one valid answer

You write one approved response and compare every generation to it with string similarity. The system rejects a concise answer that contains the same meaning and accepts a fluent answer that copies the reference's structure while missing the user's context.

Use reference-based checks for requirements that truly have a reference. Use criterion-level checks for semantic and contextual quality. If a reference contains required facts, extract those facts into a coverage check rather than treating the whole text as gold.

G-Eval's motivation is relevant here. The paper notes that conventional reference-based metrics have problems on open-ended tasks where creativity and diversity matter. That does not make reference answers useless. It means you should measure what the reference represents, not its wording alone. G-Eval provides the research basis.

Failure mode 2: replacing an undefined rubric with a bigger judge

The team cannot agree on what good means, so it uses a more capable model to grade outputs. The judge sounds persuasive, and the dashboard now has numbers. The product question is still unresolved.

Write the user outcome, the criteria, and the vetoes before choosing the judge. If the team cannot settle those, the next step is product definition, user research, or domain review.

Failure mode 3: averaging veto conditions

An answer scores 4.6 for clarity and relevance but makes a privacy-sensitive claim that the system is not allowed to make. If you average the criteria, the failure disappears inside a positive number.

Make hard constraints separate columns and separate release rules. A veto may be binary, human-reviewed, or a high-confidence classifier. The key is that it must not be traded away for tone or fluency.

Failure mode 4: trusting a model judge without calibration

The judge is tested on a few obvious examples, agrees with the team, and is then used for every future output. It later misses a subtle unsupported claim because the response style changed.

Keep a labeled calibration slice, test the judge on boundary cases, and sample human review after material changes. Google explicitly describes human ratings and pairwise choices as the ground truth used to evaluate model-based metrics. That validation step is part of the method, not an optional research exercise. Google's judge-model documentation supports this practice.

Failure mode 5: pairwise order decides the winner

The judge prefers whichever response appears first. You conclude that the prompt or model is better when the result was caused by presentation order.

Randomize the order, run the swap, allow ties, and report order-sensitive cases. If order changes the decision often enough to affect the release call, the judge is not ready for that use.

Failure mode 6: verbosity wins

The longer response sounds more complete. The judge rewards it even when the user asked for a short decision or when the extra words create more opportunities for unsupported claims.

Include an explicit length or usefulness criterion only when it matters, and anchor it to the user job. Add paired cases where the concise answer is the better answer. Review whether the judge is treating polish as evidence.

Research on LLM judges has reported biases related to position and superficial quality. The practical response is not to assume every judge is broken. It is to test the biases that could change your decision. The 2025 position-bias study is a useful reminder that the judge is part of the system under evaluation.

Failure mode 7: production behavior is treated as a pure quality label

Users rephrase, abandon, accept, or escalate for many reasons. You map one behavior directly to “bad answer” and optimize toward it.

Sample the behavior, inspect the output and context, and record alternate explanations. Use product signals to discover cases and test hypotheses, not to replace the contract.

Failure mode 8: the benchmark is too clean

The evaluation set contains clear prompts, complete context, and no competing requirements. The model looks good because the test removed the conditions that make the job difficult.

Add ambiguity, missing context, conflicting instructions, long inputs, common user mistakes, and known production failures. NIST recommends evaluating in conditions similar to deployment and documenting generalization limits. A test set that is easier than real use measures a different system.

Failure mode 9: rubric drift

Reviewers gradually change what “good” means. New examples enter the set without changing the rubric. The score improves because the evaluation got easier.

Version the rubric, retain old cases, record changes, and rerun a fixed regression slice. A quality number without a stable contract is a moving target.

Failure mode 10: optimizing the judge instead of the user

The team rewrites prompts until the judge score rises. The answers become longer, safer-sounding, or more aligned with the judge's preferred style while users do not improve.

Keep a human or outcome checkpoint outside the optimized loop. If the product outcome does not move, investigate whether the judge is measuring a proxy.

How do you build an evaluation dataset without a gold answer?

Build the dataset around tasks and decisions, not around one ideal response per prompt.

Each row should preserve enough context to recreate the evaluation and enough metadata to understand where the result applies.

Minimum fields

case_id: answer-0142
created_from: production_correction
date_added: 2026-08-21
workflow: support_answer
risk: medium
user_request: "Can I change the plan after the invoice was issued?"
allowed_context:
  - source_id: billing-policy-12
    version: 2026-07-01
system_version: prompt-24
model_version: record-at-run-time
expected_outcome: "Explain the allowed next step or escalate if the policy does not cover the case"
required_facts:
  - "Invoice status affects available options"
acceptable_variation:
  - "Wording and ordering may differ"
veto_conditions:
  - "Do not promise an unsupported refund"
human_labels:
  groundedness: pass
  completeness: 2
  clarity: 2
  pairwise_choice: tie
judge_labels:
  status: pending
notes: "Boundary case because the policy has an exception"

The dataset stores requirements and acceptable variation separately. That is the crucial difference from a collection of prompt and reference-response pairs. A reference answer may be useful as one example. It should not become the definition of the only acceptable wording.

Add cases from five sources

Use a mix of:

1. representative historical tasks;
2. real corrections or escalations, sanitized;
3. known policy and safety cases;
4. boundary cases created with domain experts;
5. adversarial or malformed requests that test abstention and scope.

Avoid generating the entire dataset from the same model you will evaluate. Synthetic cases can improve coverage, but they may carry the model's assumptions and miss the awkward language real users use.

Partition the dataset by purpose

Keep separate groups for:

• calibration, where humans label cases and the judge is tested;
• regression, where known behavior must not break;
• capability, where you explore new tasks;
• monitoring, where recent production cases reveal drift;
• holdout, where you make final comparisons without tuning against the answers.

The separation prevents a prompt from looking good because it was tuned on every case it later reports.

Use repeated trials when output varies

One output per prompt can hide variability. Repeat cases when the model, sampling setting, context retrieval, or external tool can change the answer. Record the full run configuration and avoid calling the result a stability estimate if the trial design is too small or narrow to support that conclusion.

The number of trials should follow risk, cost, and observed variability. There is no honest universal count. High-risk decisions deserve stronger evidence than a low-risk brainstorming assistant.

OpenAI's eval documentation emphasizes representative test data and explicit testing criteria. That is a useful baseline even for an open-ended task: first define the row, the behavior, and the criteria; then run the model. OpenAI's evals guide documents that workflow.

How do you turn subjective measurements into a release gate?

Write the gate before you inspect the candidate's results. Otherwise the team will change the rule to fit the outcome it wants.

Separate vetoes from thresholds

Use vetoes for unacceptable behavior:

• privacy or security breach;
• unsupported high-risk claim;
• unsafe answer where escalation was required;
• forbidden action or policy violation;
• failed required task outcome;
• missing evidence for a decision that requires grounding.

Use thresholds for tradeable or gradated qualities:

• relevance;
• completeness;
• clarity;
• tone;
• user preference;
• latency or cost where the product can accept a range.

The two rule types should appear separately in the report.

Use a release matrix

Result pattern	Decision	Why
All vetoes pass, criteria improve, outcomes stable or better	Release	The candidate improved the measured job without a critical regression.
Any veto fails	Hold or reduce scope	A critical failure cannot be exchanged for a better average.
Criteria improve, task outcomes do not	Investigate	The judge may be measuring a proxy or the product handoff may be the problem.
Pairwise preference improves, absolute rubric falls	Hold	The candidate may be better than a weak baseline but below the minimum bar.
Offline criteria improve, production corrections rise	Roll back or narrow	The evaluation set does not represent current use or the output is causing a downstream problem.
Human disagreement rises after a change	Calibrate	The rubric, judge, data, or product expectation may have drifted.
Results are tied within uncertainty or reviewer disagreement	Keep the safer option	Do not pay complexity or risk for a difference the measurement cannot establish.

The last row matters. A measurement system is allowed to say “we do not know.” If two versions are effectively tied and one has a simpler failure mode, lower cost, or safer fallback, keep the safer option until more evidence changes the decision.

Report distributions and examples

Report the mean only when it helps, and never alone. Include:

• criterion-level pass rates or score distributions;
• veto failures and their severity;
• pairwise wins, losses, ties, and order-sensitive cases;
• human-judge disagreement;
• representative passes and failures;
• slices where results change;
• task outcomes and corrections;
• the exact system, rubric, judge, and dataset versions.

HELM's multi-metric approach is a useful research model for keeping trade-offs visible. The paper measures several metrics across scenarios rather than allowing one capability to stand in for the whole system. Your product report can be much smaller, but it should preserve the same principle: show the dimensions that change the decision. Holistic Evaluation of Language Models describes that multi-metric approach.

Define the release decision in plain language

If the report cannot produce a sentence such as this, the gate is incomplete:

Release candidate B for the support-answer workflow only if all critical grounding and escalation cases pass, the calibrated completeness and clarity thresholds hold on the regression and holdout slices, and no rise in human correction is observed in the staged sample.

It names scope, vetoes, thresholds, slices, and a production check. It does not pretend quality is timeless.

Worked example: measuring a support answer with no single gold response

Imagine a support assistant answering this request:

“I was charged after I cancelled. Can you fix it?”

The approved policy says the assistant must inspect the account status, explain what it can verify, avoid promising a refund it cannot authorize, and escalate when the evidence is incomplete. There are many acceptable phrasings.

Two candidates

Candidate A:

“I’m sorry this happened. I can see why the charge is frustrating. I’ve issued a refund and it should appear soon. Let me know if you need anything else.”

Candidate B:

“I’m sorry about the unexpected charge. I can check whether the cancellation was recorded before the billing date, but I can’t promise a refund until that is verified. If the account record is unclear, I’ll send the case to the billing team for review.”

Candidate A is smoother and more reassuring. Candidate B is more conditional. If the system has no verified refund tool or account evidence, A fails a veto by promising an action it did not take. B may be less satisfying in the moment, but it preserves the policy and names the safe next step.

What the contract measures

Criterion	Candidate A	Candidate B	Evidence
Groundedness	Fail if no refund was issued or the record was not checked	Pass if it accurately states what can be checked and what happens next	Account and billing records
Completeness	Fail if cancellation timing and escalation are omitted	Pass if verification and escalation are covered	Required-facts checklist
Clarity	Pass on sentence flow	Pass if the conditional path is easy to follow	Pointwise rubric
Tone	Likely acceptable	Likely acceptable	Tone rubric and sample review
Policy integrity	Veto failure if it claims an unauthorized refund	Pass if it avoids an unsupported promise	Deterministic and policy review
User outcome	Unknown until the record is checked and the case moves	Unknown until the record is checked and the case moves	Workflow state

The example shows why “the best answer” is an incomplete question. Best for what? Immediate reassurance, policy compliance, successful resolution, or reduced support effort? The contract makes the product choice explicit.

How a pairwise result could mislead

Suppose reviewers prefer A in a blind comparison because it is warmer. That preference is useful feedback about tone, but it cannot override the policy veto. The pairwise result should be recorded as a trade-off, not as a release verdict.

Suppose the judge prefers B because it is longer and more cautious. That may be correct for policy integrity, but you still need to check whether B creates unnecessary friction. A third candidate could preserve B's conditions in fewer words.

Suppose users accept A more often. Inspect whether the assistant really completed the refund or simply made a confident promise. An acceptance click is not a substitute for a state check.

The release decision

Candidate B is eligible for further testing only if its policy and grounding checks pass. The team should then improve its clarity and measure whether the billing workflow reaches the correct next state. Candidate A cannot ship in this scope if its promise is unsupported, regardless of its tone score or pairwise preference.

That is the point of a measurement contract. It does not remove judgment. It puts each judgment in the place where it can do the least damage.

How should you maintain the evaluation as the product changes?

Treat the evaluation as a versioned part of the product, not as a dashboard query that can change silently.

Version the moving parts

Record versions for:

• the system prompt or instruction set;
• the output model and endpoint;
• retrieval sources and indexes;
• tool schemas or external connectors;
• memory and context rules;
• the rubric and its anchors;
• the judge model and judge prompt;
• the dataset and its partitions;
• the release thresholds and veto list.

If a score changes, you need to know which of these changed. Otherwise a comparison between two runs is a comparison between unknown systems.

Keep old cases alive

New cases are useful, but they should not replace the cases that caught the last failure. Keep a fixed regression slice. When the workflow changes, mark cases as obsolete only with a reason and a replacement case.

This is especially important for subjective quality. A rubric can become easier to satisfy as the team adds examples that describe the current model's behavior. Old failures keep the original quality bar visible.

Review the construct, not just the score

Every review cycle should ask:

• Are we still measuring the job users care about?
• Are any criteria duplicated or missing?
• Are users accepting outputs that domain reviewers reject?
• Are reviewers disagreeing about a new type of case?
• Are model judges changing their style preferences?
• Are vetoes catching real risk or creating avoidable false blocks?
• Is the product promise broader than the evaluation contract?

NIST emphasizes that measurement should be tied to the deployment context, updated with relevant experts and users, and documented with validity limits. That is a useful standard for the review questions even when the product is small. NIST's context and validity guidance supports this ongoing review.

Know when to narrow the promise

Sometimes the correct evaluation result is that the feature promises too much. If the system can produce plausible drafts but cannot reliably verify facts, call it a drafting assistant, not an autonomous answerer. If it can suggest options but cannot safely choose one, make the user the decision-maker. If the product cannot observe the outcome, reduce the claim to the part you can test.

This is not a failure of evaluation. It is evaluation doing product work.

What do you still not know after measuring AI output quality?

You still do not know everything.

You may know that the candidate passes the current contract on a representative set. You may know that a judge agrees with human labels on the calibration slice. You may know that a staged group completes the task without a rise in corrections. Those are meaningful facts with scope.

You do not know that the system will behave well on every future prompt. You do not know that a user outcome was caused only by the output. You do not know that a judge's agreement will survive a new domain, language, or model. You do not know that a preference winner is safer or more useful outside the comparison design.

Say those limits in the report. A measurement result becomes more trustworthy when it tells the reader where it stops applying.

The goal is not to discover a single right answer for every open-ended output. The goal is to make the quality decision explicit, testable, and revisable.

A practical starting sequence

If you are starting from an informal “this looks good” review, do this in order:

1. Write the user outcome in one sentence.
2. List the facts, behaviors, and constraints that must hold.
3. Separate hard vetoes from qualities that can trade off.
4. Create three to six rubric criteria with concrete anchors.
5. Add representative passes, failures, valid alternatives, and boundary cases.
6. Choose code, model, human, or mixed graders for each criterion.
7. Blind and label a calibration slice before automating the judge.
8. Run pairwise comparisons only when the decision is comparative, and swap order.
9. Connect the output to a downstream task or user signal.
10. Write the release rule before looking at candidate results.
11. Preserve disagreements and production corrections as new cases.
12. Recheck the contract whenever the model, prompt, context, judge, or user job changes.

This sequence is intentionally ordinary. The hard part is not finding a clever score. It is making the product team agree about what the output is for and what failure the system cannot be allowed to hide.

When I have seen people skip that agreement, they usually start with the evaluation framework or the model API. When I have seen teams ship, they start with a concrete definition of done and then choose the cheapest evidence that can test it.

If you are building an AI feature and need help turning an ambiguous quality goal into a testable contract, learn how Marius Manolachi helps existing teams become capable of building AI products on their own work. The article should leave you able to start without outside help. A consultation or tutoring engagement is useful when the team needs to adapt the method to a live workflow, dataset, or release decision.