source: .forge/design/ai/model-choice-evaluation-plan.md · generated — edit the markdown, not this file

Model Choice — Can Every AI Feature Run on Haiku? Evaluation Plan

Status: agreed as the approach (the lead, 2026-11-04); results not yet run. The model per feature stays as ai-provider-strategy.md §2 and AD #19 set it, run on Haiku 4.5 meanwhile (§5 row 10), until the lead signs off a result of this plan. Update 2026-11-04: the lead agreed the order this plan proposes. When the 5.5 models answer, Haiku 5.5 is evaluated first for every feature, and Sonnet 5.5 stays only for a feature whose Haiku 5.5 result fails §4.3’s rule; the aim is the lower cost, with no material loss of accuracy (ai-provider-strategy.md §4d, “Which model to try first”). D-4 is decided: the lead signs off each feature’s result. D-1 (§4.3’s margin) and D-3 (real inquiries from Empire) stay open until the runs are prepared. Author: Dinil (with Claude), 2026-10-22 Update 2026-10-26: AWS will not raise the account’s Bedrock quotas until it has a few billing cycles of usage and spend (P-2 waits with no date). Meanwhile every AI feature runs on Haiku 4.5 (ai-provider-strategy.md §5 row 10), and §4d of that file sets out how the team checks for the models and switches once they answer. P-1, P-3, P-4 and P-5 do not wait on AWS. D-2 is decided: no (the lead, 26 Oct) — no Anthropic API account billed per token, for cost, and the team’s Claude plan seats give the backend no API key — so P-6 is not used and the comparison runs on Bedrock once the models answer there. Where each precondition is tracked: P-1 is BUG-248 (bugs.md); P-3 and P-4 are the owed.md row “Haiku 5.5 readiness in the backend”; the task role’s grant for Haiku 5.5 is the row “Allow the backend’s task role to call Haiku 5.5”. Question: can every AI feature in the CRM and the capture app run on Claude Haiku instead of Claude Sonnet 5.5, to lower the AI bill, with no loss of accuracy or output quality in any feature? Inputs: ai-provider-strategy.md (§2 model per feature, §3 cost, §4c evaluation rules) · forge-et-be/docs/ai/README.md § Evaluation (the live-evaluation runner and the four existing sets) · .forge/client/project-proposal.md §A.3 (the model named per action, and the A.3.1 comparison commitment) · Anthropic’s model and pricing reference as cached 2026-10-06 (Haiku 5.5 price and behaviour) · cloud-gated-validation.md §C-5 (Bedrock quotas).


1. Summary

  1. The realistic candidate is Haiku 5.5, not Haiku 4.5. Anthropic lists Haiku 5.5 at $0.10 / $0.50 per million tokens (input / output, prompts up to 100K tokens) — a tenth of Haiku 4.5 and a twentieth of Sonnet 5.5 ($2 / $10) — and describes it as substantially better than Haiku 4.5 at following instructions. It is already the planned successor to Haiku 4.5 (AD #19). The decision is therefore “Haiku 5.5 everywhere” against “Haiku 5.5 for high-volume work, Sonnet 5.5 for writing and synthesis”. The Bedrock price of Haiku 5.5 must be read from the AWS pricing page before any figure here is quoted to anyone.
  2. What is at stake in money. At the proposal’s expected volumes, the Sonnet 5.5 features cost about $28 a month (drafts about $14, morning briefings about $7.60, account briefings about $4, quote assists about $1.75, lead research about $0.80; §3 of the strategy). The same work on Haiku 5.5 would cost about $1.50 a month. The saving is about $25–30 a month, roughly $300–350 a year, about double that if usage doubles. Prompt caching lowers both sides. This is a real saving on Empire’s bill, but a small one, so the evaluation should cost a few developer-days, not weeks.
  3. “No sacrifice whatsoever” cannot be proved by testing; it can be bounded. Two runs of the same model on the same input already differ, so any comparison shows some differences. The plan’s standard is: on a large enough set of cases, Haiku’s results against Sonnet’s look no worse than Sonnet’s results against a second run of Sonnet, and Haiku fails no safety check that Sonnet passes. §4 turns this into numbers the team agrees before running anything.
  4. The current evaluation sets cannot answer this question as they stand. They are small (16–27 cases), they check minimum safety properties (no invented field values, no prices in a draft, no genuine lead marked as spam) rather than quality relative to Sonnet, each case runs once, and the quality judgement is one person’s 1–5 rating, not blind. §3 lists what has to change.
  5. Decide per feature, not once for all features. Each feature already has its own {model, enabled} setting, so the outcome can be Haiku for some features and Sonnet for others, changed later by configuration. “Haiku for everything” becomes the result if every feature passes, not an assumption. About half the Sonnet spend is email drafts, so if drafts pass, most of the saving is captured even if another feature stays on Sonnet.
  6. A blocker was found while preparing this plan (§2, P-1; filed as BUG-248): the backend’s Sonnet 5.5 path will very likely fail on every call. The backend forces the model to call one named tool (BedrockAiProvider sets a SpecificToolChoice), and Anthropic’s migration guide states that Sonnet 5.5 rejects a forced tool choice with a 400 error. The backend maps that error to MODEL_UNAVAILABLE, so the email draft and the account briefing would fall back to the manual path on every request. Nobody has seen this because the dev account’s Sonnet 5.5 quota is 0. It has to be fixed before Sonnet can serve as the reference in this comparison — and before Sonnet serves anyone in production. Fixed 26 Oct in forge-et-be#316 (P-1); the one live Sonnet 5.5 call is register §B F12 row 10.

2. Preconditions — what must be true before the comparison runs

# Precondition Why Owner
P-1 Fix BUG-248: change the backend’s tool choice from “this named tool” to auto, with the prompt saying the answer must be given through the tool. The existing “no tool call → INVALID_OUTPUT → one retry” rule already covers a model that answers without the tool. Use the same request shape for every model (strategy §4c rule 2). Confirm with one Sonnet 5.5 call once its quota is raised. Done 26 Oct in forge-et-be#316, except the live call (register §B F12 row 10) Sonnet 5.5 rejects a forced tool choice (Anthropic’s Sonnet 5.5 migration guide, breaking change 2), so the reference model cannot run today. Haiku 5.5 accepts a forced tool choice but then skips thinking entirely, so with auto it can also think before answering — a fairer test of what Haiku 5.5 can do Backend PR; whoever owns F12’s seam
P-2 Raise the dev account’s Bedrock quotas for Haiku 5.5 and Sonnet 5.5 (both 0 tokens a minute) and for Haiku 4.5 (10 requests a minute). Enough for about 30 requests a minute is plenty No model comparison can run on Bedrock without it; at 10 requests a minute a 450-call run takes 45 minutes Account holder (Empire IT, Kavindu Anthony); a non-read AWS action, so the lead approves first (CLAUDE.md § ASK FIRST)
P-3 Add Haiku 5.5’s price row (et.ai.prices) at its Bedrock price, including the 10% premium on the us. profile, and decide which profile it is called through (us. or Global, strategy §4b) The runner refuses a model without a price row, and cost per call is one of the results Backend PR
P-4 Handle Haiku 5.5’s differences from Haiku 4.5: a refusal answer from its safety classifiers (no automatic fallback for this model) must map to a failure the feature handles, and a response may begin with thinking blocks, so the tool answer is read by type, not by position (it already is, and forge-et-be#316 adds the test that keeps it so) Otherwise a refusal could be scored as a quality failure, or crash the call Backend PR, with P-1
P-5 Extend the live-evaluation runner (./gradlew liveEvaluation): run each case N times (-Pruns=3), keep every output, and write a paired file — for each case, the output of model A and model B in random order with the model names hidden — for the blind rating in §4 The present runner sends each case once and reports pass/fail only Backend PR (test code only)
P-6 Optional, decided by the lead: if the quota raise is slow, run the comparison on Anthropic’s own API with Altrium-held evaluation keys, which AD #19 allows for exactly this comparison on synthetic or anonymised data; then repeat a short confirming run on Bedrock before any switch Keeps the decision off the AWS critical path Lead

3. What the evaluation sets need

The four existing sets (forge-et-be/src/test/resources/ai/eval/) are the starting point. Each needs the following before it can support this decision.

Size. With 16 cases and no failures, a model that fails 1 time in 6 still has a reasonable chance of passing every case. If a model shows zero failures in n cases, the failure rate it can still have (at 95% confidence) is about 3 in n: about 19% at 16 cases, 3% at 100, 2% at 150. Target 100–150 cases per feature for the zero-tolerance checks. Most new cases can be variations of existing ones (different buyer, country, product, length, missing fields), drafted with a model and checked by a person.

Realism. Every case today is synthetic English. Add, per feature:

  • non-English and mixed-language inquiries (buyers write from Russia, the Middle East, Central Asia and elsewhere), and English written by non-native speakers;
  • long inputs: an account with more than 50 timeline entries, a long email thread, an inquiry with a long signature and quoted history;
  • sparse and messy inputs: forwarded mail, a one-line inquiry, a business card photographed at an angle;
  • attempts to instruct the model through the content (already in the sets; keep and extend).

Real data. Proposal A.3.1 commits the comparison “on real Empire Teas inquiries”. A sample of about 50 real, anonymised inquiries and threads from Empire would show what synthetic cases miss. Real text never goes into a repository (strategy §4c rule 6); it stays in a file outside git, run with the same runner. The source is still to be agreed with Empire (ai-provider-strategy.md §4, “Eval before lock”).

Quality checks, not only safety checks. Each set gains, per case, the facts a good answer must contain (recall) as well as the facts it must not contain, so a correct-but-thin answer scores lower than a complete one.

4. How each run is judged

4.1 Configurations compared

Label Model Notes
S Sonnet 5.5 (Global profile) The reference: what each Sonnet feature uses today
S′ Sonnet 5.5, its second run of the same cases (no extra calls) The control: measures how much Sonnet differs from itself
H5-low, H5-med, H5-high Haiku 5.5 at effort low, medium (its default), high Effort controls how much the model thinks; higher effort costs more output tokens but stays far below Sonnet’s price
H4 Haiku 4.5 The model on dev today; shows how far Haiku 5.5 moves things

Every case runs three times per configuration, to measure how consistent each model is.

4.2 Three kinds of evidence

  1. Automated checks (zero tolerance). The existing scorers, extended per §3: no invented field values, no price in any draft or briefing, no genuine inquiry marked spam, every cited source id real, the dormancy rule, the output schema. Rule: a Haiku configuration fails if it fails any automated check on any case where Sonnet passed, in any of its three runs.
  2. Blind paired rating by people. For each case, a rater sees two outputs labelled A and B in random order, without model names, and records: A better, B better, or about the same; and, separately for each, whether it is usable as is, usable after edits, or not usable. Raters are the four developers, and for email drafts at least one of Empire’s sales people, because only they can judge Empire’s voice. Some pairs are S against S′ (the control), mixed in without the raters knowing.
  3. A model as a second judge, for scale. Claude Opus 5.5 rates every pair against a written rubric per feature. Its ratings count only after they agree with the human ratings on at least 30 pairs per feature (about 80% agreement or better); it never decides a feature on its own.

4.3 The decision rule per feature — agree it before running

A Haiku 5.5 configuration may replace Sonnet 5.5 for a feature only if all of the following hold:

  1. It passes every automated check (§4.2 item 1).
  2. In the blind rating, Sonnet is preferred over Haiku no more often than Sonnet is preferred over its own second run, plus a margin. Suggested margin: 5 percentage points of all pairs. Example: if S beats S′ in 22% of pairs and S′ beats S in 20%, Sonnet’s preference over Haiku may be at most about 27%.
  3. No case where Haiku’s output is “not usable” and Sonnet’s is usable, for the features whose errors reach a buyer or a price (email drafts, the quote assistant). For the others, at most one such case per 100, each one read and explained.
  4. Its 95th-percentile latency fits the 9-second call deadline (et.ai.call.deadline), and its rate of schema failures and refusals is no higher than Sonnet’s.

Of the Haiku configurations that pass, choose the lowest effort that passes. If none passes, the feature stays on Sonnet 5.5, and the result records which cases failed and why, so a later prompt version or model can be retested against them.

The margin in rule 2 is the team’s choice. Setting it to zero means Haiku must be preferred at least as often as Sonnet’s own second run, which a model as good as Sonnet would only meet about half the time by chance. A small margin is what “no measurable loss” means in practice.

5. Per-feature risk and what to test

Feature Model today Built? Risk on Haiku 5.5 What decides it
Inquiry extraction (F12) Haiku 4.5 Done Low — already Haiku Field accuracy and no invented values; non-English and messy inquiries
Lead summary and spam / duplicate check (F11-d) Haiku 4.5 Done Low — already Haiku No genuine lead marked spam; spam caught; the known gap with an instruction-shaped email domain on a lead with no inquiry text (owed.md) retested on Haiku 5.5
Business-card read (capture app, F36) Haiku 4.5 (planned) Not built Low Field accuracy on a photo set: angles, glare, non-Latin scripts
Account briefing (F13) Sonnet 5.5 (dev runs Haiku 4.5) Wave 1 merged 2026-10-22 Medium — summarising given entries is within Haiku’s reach; completeness on long histories is the risk Facts recalled from 50+ entries; no invented facts; usefulness in blind rating
Email follow-up draft (F11-h); feedback-chase draft (F6-c); sequence steps (F25) Sonnet 5.5 (dev runs Haiku 4.5) F11-h done; others not built High — buyer-facing prose in Empire’s voice; about half of the Sonnet spend No price, no invented claim, answers what the buyer asked, tone; Empire’s sales person in the blind rating
Lead research and ranking (F11-c, F11-g); trade-show research and rank (F36) Sonnet 5.5 (planned) Not built High — judging fit from search results; a wrong rank hides a good lead Rank agreement with Sonnet and with a person’s ranking of the same leads; every claim traceable to a search result
Morning briefing (F15-a) Sonnet 5.5 (planned) Not built Medium–High — synthesis across the whole pipeline Nothing missed from a seeded pipeline; correct priority order; no invented numbers
Quote assistant (F14) Sonnet 5.5 (planned) Not built (Phase 2) Highest — reasons across readiness, certifications and pricing; errors reach a buyer Any case where it calls a quote ready when it is not is a failure, for both models

For the features not yet built, the comparison becomes part of the feature: its spec’s evaluation requirement includes the paired Haiku-versus-Sonnet run, and the model is chosen from that run before the feature ships.

6. Order of work and effort

  1. P-1 and P-4 (backend fix and Haiku 5.5 handling) — about a day, one PR, reviewed as usual. This is needed whatever the decision is.
  2. P-2 and P-3 (quotas and price row) — waiting time on AWS, little work.
  3. P-5 (runner: repeated runs, paired blind output) — about a day.
  4. Expand the four existing sets to 100–150 cases each (§3) — about one to two days in all, mostly reviewing generated cases.
  5. Run F12, F11-d, F11-h and F13 across the five model configurations, three times each — about 7,500 model calls plus about 2,000 judge calls, roughly $50–100 of Bedrock and judge spend at list prices (most of it the Sonnet and Opus calls), and a few hours of run time once quotas are raised. Each run needs the lead’s approval (a non-read AWS call).
  6. Blind rating — about 100 pairs per feature for the two Sonnet features, about a minute each: roughly 4–6 hours shared across raters, plus an hour of an Empire sales person’s time for drafts.
  7. Record the decision per feature (§7).

Total: about four to five developer-days, plus waiting on quotas. It fits beside Phase 1 work; it does not need to finish before 20 November unless the team wants Phase 1 to ship on the final models.

7. Recording the decision

  • A per-feature result table in this file (configurations, check results, preference rates, the chosen model and effort, cost per call measured), with each run’s report from build/reports/ai-evaluation/live/.
  • ai-provider-strategy.md §2 and §3 updated with the chosen model per feature and the new monthly cost; sow-cost-annexe.md §5 updated in the same change (strategy §4c rule 8).
  • AD #19 names the split (“Haiku … for high volume, Sonnet 5.5 for synthesis”), so moving any synthesis feature to Haiku refines it: a new AD row with the lead’s sign-off and the same-PR updates .claude/rules/prd.md requires.
  • The issued proposal (§A.3) names Sonnet for six actions. A move to Haiku is a divergence that lowers Empire’s cost; it is stated in a post-issue note (L-001), and Empire is told, since the proposal promised the “side-by-side comparison” and drafting “in Empire Teas’ voice”.
  • The setting changes in application.yml and in each environment’s task definition.

8. After a switch — watching production

Testing covers the cases someone thought of; production shows the rest. For one month after a feature moves to Haiku:

  • Compare, before and after the switch, what the feature already records or can record: for drafts, how much a person edits a draft before sending it and how often they discard it; for extraction, how many fields a person corrects before saving; for triage, how often a spam flag is overturned; for briefings, any “not useful” feedback.
  • Any clear worsening moves the feature back to Sonnet 5.5 by changing one setting, without a deployment of new code.
  • The monthly model-lifecycle check (strategy §4c rule 7) also reviews these numbers.

9. Decisions for the team

# Decision Suggested
D-1 The margin in §4.3 rule 2 5 percentage points
D-2 Whether to run on Anthropic’s API with evaluation keys if the Bedrock quota raise is slow (P-6) Yes, then confirm on Bedrock
D-3 Whether to ask Empire for about 50 anonymised real inquiries and threads, and an hour of a sales person’s time to rate drafts Yes — drafts are the feature most at risk and the largest share of the saving
D-4 Who signs off the per-feature result The lead (AD #19 refinement). Decided 2026-11-04

Generated from .forge/design/ai/model-choice-evaluation-plan.md by .claude/lib/render-docs-html.js. Markdown is canonical.