Before You Build

What AI taught me about judgement under uncertainty

*Author's Note*This essay grew out of a series of conversations with different AI models while developing a small software project. What began as an exploration of AI-assisted software development became an exploration of judgement itself. The examples are real. The conclusions, however, extend well beyond artificial intelligence.

Every time I thought I understood the project, it changed.

The idea seemed almost embarrassingly simple.

I had acquired tens of thousands of high-resolution images of Chinese paintings from museums across China. Existing slideshow software displayed many of them poorly. Long scroll paintings were cropped. Wide landscapes were reduced to narrow strips. I wanted something simple that would let the paintings scroll naturally across a television screen, the way they were meant to be seen.

It sounded like the sort of vibe coding software project that should take a couple of weekends.

Instead, the project refused to stand still.

To help think it through, I used several AI models to guide development. Their responses were intelligent, thoughtful and, frustratingly, quite different. One enthusiastically generated more and more alternative products. Another wanted to define a first release, set a deadline and begin building.

None of these ideas was unreasonable.

The problem was that every new possibility quietly changed the question I was trying to answer.

Looking back, I think they were revealing something much more interesting than a disagreement about software. They were exposing a way of thinking that I had spent much of my professional life studying, but had never seen quite so clearly.

1.      Two Advisers

What surprised me most was not the software idea. It was the advice.

The AI models were like different advisors; encouraging me to think differently about the same uncertainty.

One treated every conversation as an opportunity to expand the search space. A viewer for Chinese paintings became a digital gallery for the home, then a platform for sharing curated exhibitions, and perhaps a commercial product. Each discussion was a brainstorming session to explore more ideas, but sometimes good ideas were lost as new ideas were generated.

Another behaved more like a senior software architect. It wanted to identify the essential features, define a first release, set a deadline and begin delivery. Its instinct was to reduce uncertainty by making choices.

Both approaches were helpful, but they were optimising for different risks. One feared overlooking an important opportunity. The other feared building nothing at all.

My discomfort was not really about deadlines or planning. We were disagreeing about what kind of project this was.

If the objective was to deliver a known product, narrowing the scope and moving towards implementation made sense. If the objective was to discover what the product ought to be, those decisions were premature. That distinction changes how a project should be managed.

2.      Different biases

Looking back, the disagreement was not really between AI systems. It reflected different ways of managing uncertainty.

One sought value by generating alternatives. The other sought value by converting ambiguity into decisions and testing them through action. Neither was unreasonable; they were just pursuing different objectives.

I found myself pursuing a third objective. I was not trying to generate more ideas or begin building immediately. I was trying to discover the right course of action.

Exploration generates alternatives. Execution turns a chosen alternative into something tangible. Between them lies a less visible activity: comparing, testing and refining possibilities until some strengthen and others can be responsibly discarded.

I have begun thinking of this as convergence. It is the disciplined process that allows exploration to end and responsible execution to begin.

3.      Standing on Other’s Shoulders

Is this new?

James March's classic work on exploration and exploitation argued that organisations must balance the search for new knowledge against the efficient use of what they already know. Too much exploration produces endless experimentation. Too much exploitation produces competence without adaptation.

Uncertainty is also well established in project management. Shenhar and Dvir argued that projects require different approaches according to their uncertainty and complexity. Dvir and Lechler later showed that planning matters, but much more the capacity to change plans intelligently as understanding develops.

Benefits management approaches the problem from another direction. Ward and Daniel argue that projects exist to realise benefits, not merely deliver outputs. Completing a project successfully is not the same as creating value.

Herbert Simon made a related observation many years earlier. Managers rarely possess enough information to identify an objectively optimal course of action. Instead, they eventually commit to one that appears good enough given the uncertainty that remains. Daniel Kahneman later showed that even these judgements are subject to systematic biases. AI does not remove either problem. If anything, by generating many more plausible alternatives, it makes the exercise of judgement more visible.

Together, these ideas form a consistent picture. Organisations must explore, eventually execute and adapt as they learn, while remaining focused on the benefits they seek rather than the artefacts they build.

So what, if anything, is new?

The novelty does not lie in any one of these ideas. It lies in watching them play out, almost in real time, through conversations with AI.

AI systems can be prompted to take different roles: explore, critique or execute. Yet, despite repeated attempts to steer them, my AI advisers tended to return to familiar defaults. Whether that reflects training, product design or something else, I cannot say. For the user, however, the practical difference is real.

One consistently encouraged exploration; another encouraged execution. Neither initially recognised that the project itself was still evolving, or that the right advice depended on the uncertainty that remained.

That led me to an unexpected insight: perhaps the first responsibility of an AI project adviser is not to produce a better plan, but to recognise what kind of thinking a project currently requires.

4.      The Missing Discipline

Exploration and execution are familiar ideas. The harder question is how a project moves responsibly from one to the other.

That transition is often presented as simple: generate ideas, select the best and build it. In practice, ideas are rarely comparable. They may solve different problems, create different benefits or serve different users. Before choosing among them, we must understand what they represent.

The art viewer illustrated this. A personal screensaver, a family gift, a household gallery and a commercial platform all initially appeared to be versions of the same idea. They were not. Each rested on a different benefit.

For me, the benefit was clear: I enjoyed living with the art, and existing software displayed some of it badly. For my family, the benefit was uncertain. A beautiful object would place greater weight on the screen; cultural connection would require context and narrative; maintaining contact would make curation and renewal more important than the software itself.

The project could not converge by comparing features because the alternatives did not yet share the same purpose.

Convergence is not merely scope reduction or compromise between imagination and impatience. It is the disciplined search for the right course of action. In projects that means finding the right project: one that balances the value of the benefits sought against the cost, risk and practical difficulty of achieving them.

Each iteration should move the search closer to the right course of action. It should clarify the benefits, expose assumptions, test achievability or eliminate alternatives that no longer deserve attention. The sceptical questions matter as much as the creative ones: what evidence would strengthen this possibility, weaken it or cause us to stop?

Without that discipline, exploration becomes accumulation and every option survives. Premature closure creates the opposite failure: a team selects a solution and begins delivery before establishing that the output is connected to the desired benefit. The project may be well managed and still be misguided.

Convergence preserves alternatives while they continue to increase understanding and closes them when there is sufficient confidence that one course of action offers a better balance of value, achievability and risk. Convergence does not eliminate uncertainty; it reduces it far enough for commitment to become responsible.

It seems this responsibility cannot be handed to a single AI adviser. An explorer may keep widening the field, a builder may narrow it too quickly, and a sceptic may kill an idea before it has developed. The human role is to judge which mode of thinking is most appropriate, orchestrate the different perspectives and decide when the remaining uncertainty is acceptable.

That is the missing discipline: not exploration alone or execution alone, but convergence.

5.      Implications

The immediate lesson is not about artificial intelligence. It is about judgement.

AI can already help us explore possibilities, challenge assumptions, organise information and even execute plans.

What it cannot yet do is exercise judgement.

Judgement is the decision to commit to one course of action despite remaining uncertainty.

That responsibility remains human.

The AI generated possibilities. Judgement determined which possibility was worth pursuing.

The value of AI is therefore not that it replaces judgement. It is that it helps us exercise better judgement by making different ways of thinking visible.

Perhaps that is its greatest contribution.

References

·       March, J. G. (1991). Exploration and Exploitation in Organizational Learning. Organization Science, 2(1), 71–87.

·       Shenhar, A. J., & Dvir, D. (2007). Reinventing Project Management. Harvard Business School Press.

·       Dvir, D., & Lechler, T. (2004). Plans are Nothing, Changing Plans is Everything: The Impact of Changes on Project Success. Research Policy.

·       Ward, J., & Daniel, E. (2012). Benefits Management: How to Increase the Business Value of Your IT Projects (2nd ed.). Wiley.

Kahneman, D. (2011). Thinking, Fast and Slow. Farrar, Straus and Giroux.

Simon, H. A. (1997). Administrative Behavior (4th ed.). Free Press. (Original work published 1947).