Why I Still Chose Haiku at Ten Times the Price
Recently, I needed to choose a model for a context-aware definition feature. When a user selects an English word, the system uses its sentence and paragraph as context and returns a Chinese definition of just 1–8 characters.
The task sounds simple, but real testing showed otherwise. In a technical context, race should be translated as “竞态条件,” not “竞争”; in a messaging-system context, Kafka cannot simply be transliterated as “卡夫卡.” The model needs to interpret the term accurately, keep the answer short, follow a strict JSON format, and avoid making the user wait too long.
My initial preference was Gemini 2.5 Flash Lite. Its input and output prices are $0.10 and $0.40 per million tokens, respectively. Assuming 200–400 input tokens and 10–15 output tokens per request, 1,000 calls would cost only about $0.021.
That price was hard to turn down. But after reconsidering the use case, I ranked the selection criteria as accuracy, reliability, speed, and only then cost. Users may not notice a difference of a few dozen milliseconds, but they will be directly affected by an incorrect definition.
The Cheaper Model Was Not Bad
Gemini 2.5 Flash Lite performed well in early testing. None of the 63 calls produced an HTTP error, and every result was parseable. After fixing the routing, 14 of 15 test cases were acceptable, with a median response time of about 1.2 seconds.
The problem was its sensitivity to the prompt and provider.
When asked only for the “basic dictionary meaning,” it translated race in a technical context as “竞争.” It did not consistently return “竞态” until I explicitly required established domain terminology for legal, technical, and academic contexts. Structured-output reliability could also vary noticeably when the same model was called through different providers.
That made me realize that what I actually needed to evaluate was not a model name, but a complete configuration:
Model × provider × prompt × parameters × output validation
A change to any one of these can change the result.
How the Four Models Performed in Practice
I then tested four candidate models side by side on the same set of 15 cases. The cases covered polysemous words, fixed expressions, technical terms, and metaphorical language. Every model received the same prompt and was required to return strict JSON.
| Model | JSON compliance | Median latency | Main issue |
|---|---|---|---|
| Claude Haiku 4.5 | 15/15 | 861ms | Some wording was not precise enough |
| GPT-4.1 Mini | 15/15 | 817ms | Semantic error in a key phrase |
| GPT-5.4 Mini | 15/15 | 966ms | Technical proper noun was only transliterated |
| Gemini 3.7 Flash | 10/15 | 937ms | Repeated structured-output failures |
GPT-4.1 Mini was the fastest, with a median latency 44ms lower than Haiku’s, but it translated bears directly on as “具有.” GPT-5.4 Mini followed the format perfectly, yet rendered Kafka in a technical context simply as “卡夫卡.” Gemini 3.7 Flash answered some cases well, but repeatedly failed to produce valid structured results.
These results also show that valid JSON does not mean the task was completed successfully. A schema can ensure that the program can parse the response; it cannot ensure that the user will not be misled.
Haiku was not perfect either. It translated forked as “分支,” which is less precise than “分叉.” Across the full test set, however, it made no errors that clearly changed the original meaning, all 15 outputs met the formatting requirements, and its response time was fast enough.
Ten Times the Price Does Not Necessarily Mean Ten Times the Cost
Haiku’s per-token price is about ten times that of Gemini 2.5 Flash Lite. The multiple looks substantial on its own, but in absolute terms, Haiku would cost about $0.25–$0.48 per 1,000 calls.
This feature is explicitly triggered by the user, so call volume is limited, and the output is very short. The comparison should convert the difference into added monthly cost and estimate how many errors a more reliable result prevents.
A simple way to frame the decision is:
Monthly incremental cost = monthly call volume × difference in per-call cost between the two models
If the more expensive model reduces expected error costs by more than that increment, it is actually the more economical choice. Error costs include not only retries and manual corrections, but also users losing trust in the feature after misunderstanding the content.
Anthropic’s evaluation guidance likewise recommends evaluating across multiple dimensions, including task fidelity, consistency, latency, per-call cost, and usage frequency. DoiT’s Cost Per Task framework emphasizes that if a cheaper model requires more retries or fails more often, the total cost of completing a task may actually be higher.
These sources do not prove that Haiku is better for every similar task. The final decision still came down to this set of real inputs, the current call frequency, and the acceptable level of error risk.
The Final Choice
In the end, I chose Claude Haiku 4.5, fixed the provider, disabled randomness, limited the output tokens, and validated the results with both JSON Schema and application-level rules.
Haiku did not place first on every individual metric, but all 15 outputs met the format requirements, none clearly changed the original meaning, and median latency was 861 ms. At the current call volume, the additional cost is acceptable.
After launch, I will continue tracking semantic error rate, latency, and monthly volume. If usage or the error distribution changes, I will rerun the same evaluation. This choice applies only to the current task and data.