Evals & ops
Every extractor change runs against a fixed labeled review set before shipping. The gate optimizes for grounding and compliance precision, not headline F1 or cost.
Pipeline runs, 8 weeks
695
321 conversations processed
Total model spend
$3.54
$0.011 per call
Current intent F1
0.87
extract@v2.3 on 240 labeled calls
Unsupported-claim rate
2.1%
claims without a matching evidence span
Regression gate: why the cheaper model was rejected
The v3.0 candidate beat production on intent F1 and cost, and still failed the gate: its unsupported compliance claims nearly tripled. Grounding is the binding constraint because compliance flags trigger coaching and audits.
| Metric | extract@v2.3 (production) | extract@v3.0 (candidate) | Verdict |
|---|---|---|---|
| Intent F1 | 0.87 | 0.89 | candidate better |
| Evidence grounding | 0.93 | 0.86 | production better |
| Unsupported compliance claims | 2.1% | 5.8% | production better, gating |
| Cost per call | $0.014 | $0.010 | candidate better |
| Latency p50 | 1800ms | 1100ms | candidate better |
Decision: reject v3.0. Follow-up candidate v3.1 adds a mandatory critic gate on compliance and escalation fields and is in an expanded 500-call eval now.
Eval history
Small model with mandatory critic gate on compliance and escalation fields. Close to v2.3 quality at 14% lower cost; running expanded 500-call eval before decision.
REJECTED despite better intent F1 and 29% lower cost: unsupported compliance claims nearly tripled (2.1% to 5.8%). Compliance precision is the binding constraint, not cost.
Added mandatory evidence-span citation to the prompt; unsupported claims fell by more than half.
Initial production model over the 240-call labeled review set.
Pipeline cost & latency by agent
| Pipeline agent | Runs | Avg latency | Retried | Total cost |
|---|---|---|---|---|
| ConversationExtractor | 321 | 1764ms | 10 | $3.21 |
| EvidenceValidator | 53 | 1034ms | 0 | $0.33 |
| TranscriptNormalizer | 321 | 34ms | 0 | $0.00 |