# AI judge calibration pack for phone-call evaluation

Use this pack before an AI judge can pass, fail, or route a voice-agent call. Calibrate one criterion at a time. Keep the model, prompt, schema, and evidence inputs locked during each comparison.

Do not send recordings, transcripts, tool payloads, or personal data to a judge unless that processing path is approved. Redact or replace identifiers in calibration fixtures.

## 1. Criterion contract

- Criterion ID: `[stable ID]`
- Name: `[one behavior]`
- Business risk: `[what a wrong verdict could allow or block]`
- Applies when: `[trigger]`
- Not applicable when: `[condition]`
- Pass definition: `[observable condition]`
- Fail definition: `[observable condition]`
- Abstain definition: `[missing, conflicting, or inaccessible evidence]`
- Severity: `[critical/high/medium/low]`
- Approved policy/source version: `[ID and version]`

Allowed evidence:

- [ ] Audio segment with timestamps
- [ ] Transcript or turn events
- [ ] Tool request and response
- [ ] Before and after system state
- [ ] Policy source and applicability facts
- [ ] Handoff or media events

Evidence the judge must not use: `[agent self-report, hidden answer, unrelated metadata, or other leakage]`

## 2. Human labeling guide

Each reviewer labels independently before discussion.

| Field | Reviewer entry |
| --- | --- |
| Verdict | `pass / fail / not_applicable / abstain` |
| Evidence spans | `[timestamps, event IDs, or state fields]` |
| Reason | `[one sentence tied to the contract]` |
| Confidence | `high / medium / low` |
| Ambiguity | `[what the contract does not settle]` |

Adjudication rule: `[who resolves disagreement and whether the contract changes before the case is relabeled]`

## 3. Judge prompt template

Copy this prompt and replace the bracketed fields. Keep output machine-readable.

```text
You are grading one criterion for a voice-agent session.

CRITERION
ID: [criterion ID]
Applies when: [trigger]
Pass: [observable pass definition]
Fail: [observable fail definition]
Not applicable: [condition]
Abstain: evidence is missing, conflicting, or insufficient.

ALLOWED EVIDENCE
[List the exact evidence fields supplied below.]

RULES
- Use only the supplied evidence.
- Do not treat the agent's claim as proof that an external action completed.
- Return not_applicable only when the applicability trigger did not occur.
- Return abstain when the trigger may have occurred but the needed proof is unavailable.
- Cite the smallest evidence spans that support the verdict.

EVIDENCE
[Insert the redacted evidence package.]

Return JSON with exactly these keys:
{"criterion_id":"string","verdict":"pass|fail|not_applicable|abstain","evidence_ids":["string"],"reason":"string"}
```

## 4. Calibration set worksheet

Build the set before looking at the judge's answers.

| Case ID | Stratum | Human gold | Judge verdict | Evidence correct? | Error type | Severity | Action |
| --- | --- | --- | --- | --- | --- | --- | --- |
| CAL-001 | `[workflow/language/route/condition]` | `[verdict]` | `[verdict]` | `[yes/no]` | `[FP/FN/NA error/abstention/evidence error]` | `[severity]` | `[prompt, contract, threshold, route, or data fix]` |

Minimum set design:

- [ ] Clear passes
- [ ] Clear failures
- [ ] Hard borderline cases
- [ ] Cases where the criterion does not apply
- [ ] Cases with intentionally missing evidence
- [ ] Critical subgroups, languages, routes, and workflow branches
- [ ] Confusing agent speech where tool or final-state evidence decides the result

## 5. Confusion matrix and error budget

Count human gold on rows and judge verdicts on columns.

| Human \ Judge | Pass | Fail | Not applicable | Abstain |
| --- | ---: | ---: | ---: | ---: |
| Pass | `[ ]` | `[ ]` | `[ ]` | `[ ]` |
| Fail | `[ ]` | `[ ]` | `[ ]` | `[ ]` |
| Not applicable | `[ ]` | `[ ]` | `[ ]` | `[ ]` |
| Abstain | `[ ]` | `[ ]` | `[ ]` | `[ ]` |

- False-pass rate for critical failures: `[ ]`
- False-fail rate: `[ ]`
- Unsupported evidence citation rate: `[ ]`
- Abstention rate overall and by stratum: `[ ]`
- Human-human agreement before adjudication: `[ ]`
- Human-judge agreement after adjudication: `[ ]`

Do not use one accuracy number as the release rule. A judge that misses rare critical failures can look strong on an easy set.

## 6. Promotion rule

- Auto-pass allowed when: `[measured rule]`
- Auto-fail allowed when: `[measured rule]`
- Human review required when: `[measured rule]`
- Judge must abstain when: `[evidence condition]`
- Critical false-pass ceiling: `[threshold]`
- Required calibration volume by stratum: `[threshold]`
- Model/prompt/schema version: `[locked versions]`
- Promotion owner and date: `[owner/date]`

## 7. Production monitoring

- Random human review rate: `[percentage or count]`
- Always-review strata: `[critical lanes]`
- Drift signal: `[verdict mix, evidence errors, subgroup gap, model update, policy update]`
- Recalibration trigger: `[explicit threshold]`
- Rollback or review route: `[action and owner]`

## Worked fictional example

This is an invented example. A fictional billing agent says, “Your refund is complete,” but the tool event is `timeout` and the account state is unchanged. The criterion is “verified refund completion.” Human gold is `fail`. A judge that returns `pass` from the sentence alone has used disallowed evidence and created a critical false pass. The fix is not simply a lower score threshold; the contract and prompt must require tool and final-state proof.

Voxeval's opinion is simple: a judge becomes a release control only after its errors are measured on the same evidence and risk lanes it will grade.