--- title: "Consistently Third" author: "Shawn Jiang" lang: en date: 2026-09-14 canonical: https://tplog.dev/en/essay/consistent-third-place/ translations: ja: https://tplog.dev/ja/essay/consistent-third-place/ zh: https://tplog.dev/zh/essay/consistent-third-place/ --- # Consistently Third When we evaluate an AI product, do we pay too much attention to its highest score? A coding agent—an AI agent that can read and write code and use tools—completes a complex refactor on its own. A model solves a problem that had resisted every previous attempt. An agent works for several hours and finishes a task that would have taken an engineer a full day. These moments are exciting. They are like first-place finishes: they show us the ceiling of what is possible. But when AI becomes part of actual work, there is another, less glamorous question that may matter more: can it consistently finish third? ## What does “consistently third” mean? By “third,” I do not mean that third place is all we should aim for, or that we should give up on winning. I am using the placing to describe a level of performance: the result is already quite good, usable, and ready to deliver. “Third” is not about the ceiling. It is about a quality floor I hope to rely on. The more important word is “consistently.” Give someone a problem today, and they finish third. Give them a different problem tomorrow, and they finish third again. Next week, they face something unfamiliar but still within their capabilities, and they get the job done. This is not about answering the same question a hundred times and getting the same answer. It is about handling new, changing tasks without easily falling outside an acceptable range. Suppose I am a coach with two competitors. The first might win, then finish fifth or tenth, then suddenly win again. The second rarely wins, but their record over time looks like second, third, third, fourth, third. In a highlight reel, the first competitor is clearly more dazzling. But over a long season, the second offers something valuable: I know roughly what I will get when I put them in. There is a Japanese expression I like: 「計算が立つ」. It means being able to make a reasonable estimate, to know what to count on. Applied to a competitor, it means the coach knows roughly how much they can contribute. That predictability is itself an ability. ## A workflow’s constraints are part of the product This is one source of value in combining AI with workflows: sequences of steps defined in advance. Take a human-designed workflow in Dify. Compared with an agent that decides its own next move, it can seem a little “dumb.” You have to specify the first step and the second; when to take branch A and when to take branch B; where to call tools, where to check results, and what data may pass to the next step. When necessary, you also insert human approval at critical points. Its freedom is deliberately limited. Those limits may look like a compromise on capability, but they are part of the product. Fixed steps, constrained inputs and outputs, restricted tools, and conditional checks are usually not there to produce the occasional stroke of genius nobody anticipated. They are there to reduce variation in results. Suppose a freely exploring agent’s performance ranges from 50 to 100. A well-designed workflow might narrow that range to 70–85. It may give up some chances of scoring 100 while reducing the chance of dropping to 50: trading some ceiling for a higher floor. Still, a fixed process does not automatically produce reliable results. If a check is wrong, or the workflow does not cover a class of inputs, the system can be consistently wrong. Whether the floor has actually risen must be tested on real tasks. In production, predictability may matter more than an occasional high score. What businesses really fear is often not that AI is insufficiently intelligent, but this: I do not know what it will do next time. ## Coding agents have another variable: the coach What fascinates me about coding agents such as Codex and Claude Code is precisely their chance of winning. For a complex problem, you can give the agent the codebase, requirements, and context, then keep discussing the work, adjusting direction, and checking results. Sometimes it produces something far beyond expectations. That is a very different experience from executing a predefined process. But there is an easily overlooked variable: the coach. The same coding agent in different hands can produce completely different results. A good engineer knows when to let it explore and when to interrupt; what context to supply and what looks wrong in the output; and when to change the problem itself rather than keep editing the prompt. The final performance therefore does not depend on the agent alone. Some of the capability lives in the human-agent loop: the repeated process of doing the work, checking it, and adjusting together. The competitor has strong underlying ability, but the coach also needs to read the situation. A good coach can help them keep challenging for first place. If the coach does not know what they are doing, even a strong competitor can go off course. And the coach does more than help them reach a high score. Catching mistakes early, narrowing the task, and requiring verification also protect the floor. ## They emphasize different goals From this perspective, “Which is more advanced, Dify or coding agents?” is not a particularly useful question. What we are really comparing is the emphasis of two ways of working. A predefined workflow puts more emphasis on maintaining a dependable quality floor across recurring tasks whose specific content keeps changing. A coding agent exploring with a human gives the model more room to make decisions along the way, allowing the user to reach results that were not specified in advance. One is more concerned with “Can it do this again next time?” The other is more willing to explore “How good can this attempt be?” Both are valuable. This is not an impermeable boundary between products. A workflow can call an agent at one stage, and a coding agent can use fixed steps, tests, and human approval to improve consistency. ## When should we aim to win? “Consistently third” does not mean that every AI system should become a workflow. If the payoff from winning is high and failure is cheap, I am happy to accept variation. Coding agents can have an important advantage here: if the development environment can be rebuilt quickly, they have considerable room to experiment. If an agent writes the wrong code, I can inspect the diff—the changes to the code—run tests, or revert the changes. In a development environment isolated from production and quick to rebuild, even a broken environment can be rebuilt for another attempt. Starting over is not free. But if recovery is fast enough, I am more willing to let the agent explore boldly. A successful complex refactor might save me several hours. Within those bounds, I will even accept an occasional tenth-place finish, because the gains from winning may cover the cost of trial and error. But a company is not a development environment. You cannot break it, quickly rebuild another one, and act as if nothing happened. Money sent to the wrong place has to be recovered. Incorrect messages sent to customers need an explanation. Damaged trust does not return just because the system has been rebuilt. Suppose an AI workflow handles 10,000 business transactions a day, and a single exception can cause one of these outcomes. An occasional dazzling performance does not make up for those losses. This also marks a boundary for coding agents: rebuilding an environment does not undo the consequences of the code. Once an agent can directly manipulate production data, or changes reach the business without sufficient verification, failure may no longer be cheap. What matters then is knowing that it will probably finish third again tomorrow. Detecting exceptions, blocking risky actions, and handing work to a person when necessary all help protect that dependable quality floor. ## Do not judge only the best attempt When we talk about AI, it is easy to be drawn to its best moments: high leaderboard scores, the smoothest demo, the most impressive example on social media. These matter. They show us where the technological frontier is moving. But focusing only on impressive scores and successful examples makes it easy to overlook differences between tasks and where failures occur. When AI moves from demos into real work, how bad can its worst performance be? And if I hand it the next task, do I know roughly what I will get? A mature AI product does not have to win every time. Sometimes the harder achievement is facing the next unfamiliar problem and still finishing consistently third.