AI won’t deliver the productivity gains everyone expects, because models still aren’t good enough at complex tasks. They need constant nudging, small pushes only a human can give. No prompt, no markdown file, no agent loop can reliably deliver value for a business on its own. It works on side projects. Prototyping gets faster. Every now and then, a couple of iterations even ship a feature without much help. But once real complexity shows up, a human has to step in.
The problem is that if we let models drive delivery, humans stop learning and understanding what’s happening. To jump back into the loop, we have to spend the time we saved on generation actually reading the code. Before, this happened automatically: we wrote code and understood it as we went. Now we move fast at first, without feeling how a retry policy, a timeout, or a cache TTL actually behaves under load. When something breaks, we have to debug and read what the model produced in the first place.
Companies can’t replace their staff with AI if they want to innovate or ship real changes. AI can drive a predictable pipeline of tasks on legacy systems, where everything is already known. Renaming a deprecated field across a hundred call sites, or creating a simple ETL pipeline, is exactly this kind of work. A human in the loop can start the agents there and let them run for long stretches without major interruption. New features and new requirements are different: they need real thinking and understanding, at least the moment something breaks.
Human in the loop is a good mental model for this. Even though the model produces the code, it does it hand in hand with a human, who acts as the memory and the reasoning: the one who tells the automation “that’s not how it should work.” Take that human out, and the loop breaks.
This is a kind of technical debt. When a model ships something nobody reviewed closely, nobody on the team really understands why it works the way it does. When it breaks in production, paying off that debt means reverse-engineering code no one remembers writing or reviewing, which costs far more time than reading it upfront would have. The longer we let models run on autopilot, the further we drift from the code, and the more that debt compounds. Like any other debt, code or financial, the longer we wait, the more it costs.