A Fair Test Before Fine-Tuning

Inspect the comparison plan: incumbent, credible repair, and a trained candidate only if a meaningful model error remains.

OpenTour illustrative sample

TYPE

Illustrative comparison

SCOPE

One invoice workflow

TIMELINE

Before candidate selection

SOLUTION

Baseline & protected test

01

01

Strongest practical baseline

04

04

Repair routes to assess

SAMPLE 02 / COMPARISON DESIGN

Three routes. The same reliability bar.

01 / INCUMBENT

Keep the baseline

Current model + application

Record the existing error

Preserve working behavior

02 / CREDIBLE REPAIR

Add missing context

Approved order at inference

Test prompt, rules, retrieval

Choose the strongest repair

03 / TRAINED CANDIDATE

Only if justified

Residual model error remains

Train on eligible families

Evaluate on protected cases

Illustrative test plan. No evaluation scores or customer results are claimed.

OpenTour mark

The candidate must beat the better of the incumbent and the strongest credible repair on fresh, protected cases.

The OpenTour standard

Illustrative delivery package

01 // OVERVIEW

Test the cause before training

Continue the invoice example. If the model never received the approved order, a context repair may resolve the error without changing the weights. A prompt, rule, or retrieval change may be the right answer.

The sample below records proposed tests and expected behavior. It contains no measured accuracy lift. Development fixtures help diagnose the issue; separate protected real cases determine whether a change earns acceptance.

THE CHALLENGE

A weak repair makes training look better than it is. Compare against the strongest practical alternative under the same conditions.

02 // SAMPLE ARTIFACT

Inspect the comparison plan

Freeze the eligible case families, scorer, minimum useful improvement, and reliability limits before selecting a candidate. Use the same input boundary and evaluation conditions across routes.

TEST SLICE

EXPECTED BEHAVIOR

COMPARISON RULE

TEST SLICE

Ambiguous service

EXPECTED BEHAVIOR

“Call-out” + onsite order → onsite_labor

COMPARISON RULE

Compare the incumbent with an order-context repair.

TEST SLICE

Missing evidence

EXPECTED BEHAVIOR

No reliable order join → abstain / review

COMPARISON RULE

Do not reward a confident unsupported category.

TEST SLICE

Retained behavior

EXPECTED BEHAVIOR

Confirmed replacement part → materials

COMPARISON RULE

Check that fixing services does not break parts.

TEST SLICE

Release readout

EXPECTED BEHAVIOR

Fresh, protected real case families

COMPARISON RULE

Report paired differences and uncertainty; this page has no scores.

03 // HANDOFF

What the scorecard must explain

Report paired fixes and regressions, critical errors, retained behavior, abstentions, latency, total cost, and uncertainty. Show the denominator and exclusions so the conclusion can be checked.

Train only when a material model error remains and the supported route, data rights, time, and compute allow it. A simpler repair can win. An inconclusive comparison stays on hold; the acceptance bar does not change after results arrive.

[READY TO IMPROVE YOUR AI WORKFLOW?]

IMPROVE WHAT YOU’RE

BUILDING

NEXT

Book a focused workflow review to assess recurring errors, your model route, and the records needed to verify a meaningful improvement.

↳ Prefer to send an enquiry?

Tell us about one live workflow, your open-weight model, and where errors create rework. We’ll assess the fit and define a practical starting point.

START WITH [ONE REPEATABLE WORKFLOW]

*Share a workflow overview here. Please do not include customer records or sensitive data.

[READY TO IMPROVE YOUR AI WORKFLOW?]

IMPROVE WHAT YOU’RE

BUILDING

NEXT

Book a focused workflow review to assess recurring errors, your model route, and the records needed to verify a meaningful improvement.

↳ Prefer to send an enquiry?

Tell us about one live workflow, your open-weight model, and where errors create rework. We’ll assess the fit and define a practical starting point.

START WITH [ONE REPEATABLE WORKFLOW]

*Share a workflow overview here. Please do not include customer records or sensitive data.

[READY TO IMPROVE YOUR AI WORKFLOW?]

IMPROVE WHAT YOU’RE

BUILDING

NEXT

Book a focused workflow review to assess recurring errors, your model route, and the records needed to verify a meaningful improvement.

↳ Prefer to send an enquiry?

Tell us about one live workflow, your open-weight model, and where errors create rework. We’ll assess the fit and define a practical starting point.

START WITH [ONE REPEATABLE WORKFLOW]

*Share a workflow overview here. Please do not include customer records or sensitive data.