A Fair Test Before Fine-Tuning
Inspect the comparison plan: incumbent, credible repair, and a trained candidate only if a meaningful model error remains.
TYPE
Illustrative comparison
SCOPE
One invoice workflow
TIMELINE
Before candidate selection
SOLUTION
Baseline & protected test
Strongest practical baseline
Repair routes to assess
SAMPLE 02 / COMPARISON DESIGN
Three routes. The same reliability bar.
01 / INCUMBENT
Keep the baseline
Current model + application
Record the existing error
Preserve working behavior
02 / CREDIBLE REPAIR
Add missing context
Approved order at inference
Test prompt, rules, retrieval
Choose the strongest repair
03 / TRAINED CANDIDATE
Only if justified
Residual model error remains
Train on eligible families
Evaluate on protected cases
Illustrative test plan. No evaluation scores or customer results are claimed.
The candidate must beat the better of the incumbent and the strongest credible repair on fresh, protected cases.
The OpenTour standard
Illustrative delivery package
01 // OVERVIEW
Test the cause before training
Continue the invoice example. If the model never received the approved order, a context repair may resolve the error without changing the weights. A prompt, rule, or retrieval change may be the right answer.
The sample below records proposed tests and expected behavior. It contains no measured accuracy lift. Development fixtures help diagnose the issue; separate protected real cases determine whether a change earns acceptance.
THE CHALLENGE
A weak repair makes training look better than it is. Compare against the strongest practical alternative under the same conditions.
02 // SAMPLE ARTIFACT
Inspect the comparison plan
Freeze the eligible case families, scorer, minimum useful improvement, and reliability limits before selecting a candidate. Use the same input boundary and evaluation conditions across routes.
Ambiguous service
“Call-out” + onsite order → onsite_labor
Compare the incumbent with an order-context repair.
Missing evidence
No reliable order join → abstain / review
Do not reward a confident unsupported category.
Retained behavior
Confirmed replacement part → materials
Check that fixing services does not break parts.
Release readout
Fresh, protected real case families
Report paired differences and uncertainty; this page has no scores.
03 // HANDOFF
What the scorecard must explain
Report paired fixes and regressions, critical errors, retained behavior, abstentions, latency, total cost, and uncertainty. Show the denominator and exclusions so the conclusion can be checked.
Train only when a material model error remains and the supported route, data rights, time, and compute allow it. A simpler repair can win. An inconclusive comparison stays on hold; the acceptance bar does not change after results arrive.