The Model Was Rarely the Hardest Part

The Hardest Part of Building AI Products Wasn’t the AI

About eighteen months ago, I began helping lead the development of a conversational AI product for a large SaaS platform. We had experienced product leaders, talented engineers, strong domain experts, and an executive team willing to make a serious investment. What we did not have was a proven playbook for building a production-grade AI product, because at the time, almost nobody did.

There were plenty of demonstrations showing what large language models could do. There were conference presentations, breathless predictions, architectural diagrams, and enough new terminology to make even experienced technology leaders wonder whether everyone else had attended a meeting they somehow missed. What remained scarce were practical answers to much harder questions. How do you define good conversational behavior? How do you test something that may answer the same question differently each time? How do you know whether a disappointing response came from the model, the retrieved information, the instructions, or the surrounding product experience? How much independence should the AI have, and where should it stop and ask a person for permission?

We were not building a prototype that would disappear after an executive demonstration. The product had to work with real customer data, respect permissions, survive security reviews, support thousands of organizations, and become reliable enough that people would return to it after the novelty wore off. We moved deliberately through an internal alpha, a customer pilot, and a beta rollout before eventually making it available across tens of thousands of companies and to hundreds of thousands of potential users.

In other words, we were going to learn in production, but we needed to do it without treating our customers as unpaid members of the quality-assurance department.

We Were Building the Product and the Playbook Together

Traditional product development gave us a strong foundation, but some of its assumptions began breaking almost immediately. A requirement such as “provide a useful recommendation” sounded perfectly reasonable until Product, Engineering, Quality Assurance, and a domain expert each produced a different definition of useful. The software could return a polished response that was technically valid, factually questionable, emotionally convincing, and entirely wrong, which is an impressive collection of qualities for a single feature.

We began replacing abstract requirements with concrete examples. If a customer asked this question with this information available, what should a strong answer contain? What should it never say? How should it respond when critical information was missing? When should it acknowledge uncertainty? Which sources should support its recommendation?

Those examples became acceptance criteria, evaluation cases, regression tests, and eventually golden datasets containing hundreds of representative questions. Every meaningful change to a prompt, retrieval strategy, model, or workflow could be tested against the same exam. When something failed in production, the lesson did not remain trapped in a retrospective document that everyone promised to reread someday. It became another permanent test.

We also learned that quality could not be reduced to a single score. The model might reason beautifully over the wrong information. Retrieval might return the correct records but bury them beneath irrelevant context. A response might be accurate but so slow that the user abandoned it. An answer might sound helpful while failing to follow the organization’s methodology. Each layer required its own definition of good, its own measurements, and its own improvement loop.

This changed more than the product. It changed the Product Development Life Cycle around it.

Requirements became examples. Quality assurance expanded into evaluation. Production monitoring became an input to product discovery. Domain experts became continuous participants instead of occasional reviewers. Product managers needed to understand prompts, retrieval, context assembly, latency, model behavior, and cost well enough to make tradeoffs without pretending to be machine-learning engineers. Engineers needed customer context because a technically elegant change could subtly alter the way people experienced and trusted the product.

We were making things up as we went, but not in the sense that we were throwing ideas at production and hoping the legal team had taken the afternoon off. We were practicing disciplined invention under uncertainty. We formed hypotheses, constrained the risk, observed what happened, and then changed both the product and the process used to build it.

Then We Discovered That We Couldn’t See Enough

One of our most important early lessons was also one of the least glamorous. We had not invested deeply enough in observability soon enough.

We had logs, error reports, and conventional application telemetry. Engineering could usually determine whether a request failed and where the failure occurred. What we could not explain was why some customers became enthusiastic, sophisticated users while others tried the product twice and quietly disappeared.

Was the model producing poor answers? Was retrieval missing important information? Did customers misunderstand what the product could do? Was onboarding inadequate? Did an early failure damage trust permanently? Were successful users discovering a workflow that unsuccessful users never found?

The existing tools could tell us pieces of the story, but none could show us the complete journey from an AI interaction to customer behavior. AI telemetry lived in one place, platform telemetry lived somewhere else, and product analytics lived in another system entirely. Each tool was doing its job correctly; unfortunately, the customer did not live inside any one of them.

Because apparently leading the product was not enough to occupy my time, I built an internal application that connected the AI interaction data with our behavioral product analytics. For the first time, we could follow the experience from the questions people asked to the responses they received and the actions they took afterward.

We could see what happened before someone became a power user. We could identify the first successful interaction that appeared to change a customer’s understanding of the product. We could examine what occurred immediately before another customer stopped returning. We could distinguish failures in the model from failures in retrieval, onboarding, discoverability, or user experience.

That changed our roadmap because it changed the questions we were asking. We stopped focusing exclusively on whether the AI had produced a good response and started examining whether the interaction had helped a person become successful.

The distinction mattered. Technical logs are excellent at answering, “What happened to the system?” Product leaders also need to answer, “What happened to the human?”

A Smaller Product Taught Me the Other Half of the Lesson

More recently, I have been building a focused AI application for a professional-services business. The scale is dramatically smaller, but the work is no less real. The application processes confidential source material, constructs detailed profiles, maintains evidence and provenance, and helps an experienced professional make recommendations that would otherwise require hours of manual analysis.

This was not a weekend prototype either. It included tenant isolation, audit history, versioned AI outputs, document processing, retrieval, security controls, automated testing, and a background worker responsible for expensive AI operations. In many ways, it benefited from everything I had learned while building the earlier conversational product.

It also found a completely new way to teach me humility.

The system was designed to process newly available information automatically. From an engineering perspective, this was sensible. New material arrived, the background worker detected it, and the application began updating its knowledge. The automation was doing exactly what automation is generally praised for doing: removing manual steps and completing work without requiring the user’s attention.

The problem was that the user could not always tell when the system was working, what it was processing, or how much AI activity it might generate. At one point, automated processing continued in the background when he was not actively using the application. His most reliable way to stop it was to shut the application down.

That was not evidence that the system was badly engineered or that the user had done something wrong. It was evidence that we had placed the boundary of automation in the wrong place.

The product had permission to do something because the software could do it, not because the user had clearly decided that it should happen at that moment. We had optimized away a click and, in the process, weakened the user’s sense of control.

The redesigned workflow is intentionally less magical. The user can upload several documents without initiating any AI processing. He can review what has accumulated, decide when the context is complete, and then explicitly select a button to process the entire batch. Once he delegates that work, the system can perform the extraction, analysis, consolidation, and profile updates automatically within the boundary he authorized.

That additional button is not unnecessary friction. It is a contract.

It also improves the result. A single document may present an incomplete picture, while several documents considered together provide better context. Waiting for explicit authorization reduces unnecessary processing costs, avoids repeatedly updating a profile with partial information, and gives the user confidence that the system is acting when he expects it to act.

The lesson was not that automation is dangerous. The lesson was that successful automation is not simply work that can happen without a person. It is work that happens after a person clearly delegates it.

Trust Requires Visibility in Both Directions

These two products were developed at very different scales, for different users, and in different business environments. One became available across tens of thousands of organizations. The other serves a highly specialized workflow for a small professional-services business. Yet they led me toward the same conclusion from opposite directions.

In the first product, the team needed better visibility into the AI and the customer journey. We could not improve what we could not understand.

In the second, the user needed better visibility into and control over the AI. He could not trust what he could not clearly authorize.

Trust requires both sides to see clearly. The product team must understand what the AI received, retrieved, decided, returned, and cost, along with how those interactions affected customer behavior. The user must understand what the AI is doing, what information it is using, when it is acting, and where its authority ends.

This is why I have become increasingly uncomfortable with the phrase “human in the loop.” It sounds reassuring, but it avoids the difficult design decisions. Which human? In which loop? What exactly are they reviewing? Are they providing context, approving cost, validating quality, confirming an irreversible action, or simply clicking a ceremonial button after the important decision has already been made?

The real work is deciding where a person must understand, where a person must authorize, and where the AI has earned permission to proceed independently.

The Model Was Rarely the Hardest Part

After living through multiple production AI products, I have become less interested in conversations about which model is smartest. Models matter enormously, but selecting one is only the beginning.

The difficult work happens around the model. It is defining good when no universal benchmark exists. It is building evaluation systems that detect quiet regressions. It is retrieving the correct information and excluding distracting context. It is connecting AI telemetry to human behavior. It is designing authorization boundaries that preserve control without destroying usefulness. It is learning from production without making customers pay for every lesson.

Most importantly, it is recognizing that AI changes how products must be built. The learning loop cannot be an occasional activity at the end of a roadmap cycle. It must be part of the operating system of the product.

When we began that first conversational product eighteen months ago, we did not have a mature playbook to follow. We built one through customer conversations, uncomfortable surprises, evaluation failures, production data, and more than a few moments when the product taught us something we would have preferred to know earlier.

I do not view that as a weakness in the story, because it is the story. Building production AI required us to develop two things simultaneously: the product itself and the process capable of improving it. Along the way, I learned that leadership in this space is less about arriving with all the answers than creating an environment in which a team can discover the right answers quickly, safely, and honestly.

The companies that succeed with AI will not simply be the ones with access to the best models. They will be the ones that learn how to learn faster, turn every failure into a lasting improvement, and build enough visibility and control that customers feel safe relying on the result.

As I consider what comes next in my own career, I am especially interested in organizations confronting those challenges for real. Not experimenting with AI solely for a demonstration, but building products that must earn customer trust, perform reliably in production, and create measurable value at scale. I have lived that journey more than once, from early uncertainty through controlled rollout and broad adoption, and I would welcome the opportunity to help another team navigate it with fewer avoidable scars and a better learning loop from the beginning.

I am still learning, which may be the most honest credential anyone can offer in AI right now. The difference is that these lessons did not come only from reading predictions or building prototypes. They came from putting production AI products in front of real people, watching what happened, and accepting that sometimes the most valuable feature you can add is the button that tells the AI when to begin.

Wishing you all the best

Mike