STRATEGY & PREPARATION
NUMEZISADVISORY &
ENGINEERING
Contents
DECISION AID · DECISION POINTS
Each step has to earn the next one
The nine phases are not a checklist. Each one removes a different uncertainty and protects the next investment.
No proof, no next step. Going back to an earlier phase is learning, not failure.
Most AI project post-mortems blame the wrong culprit. The final report points at the model: not accurate enough, not reliable enough, "not ready for our line of business". In our engagements, we almost always find something else. The model worked. What was missing was an entire phase of the project: nobody had measured the process before changing it, nobody had defined what would justify stopping, nobody had built a test environment before exposing the system to real users.
When an AI project fails, look for the skipped phase before looking for the faulty model. You will almost always find the phase first.
The problem is not a general lack of method: organisations know how to run projects. It is that AI shifts risk to unusual places. Quality is probabilistic, proof requires a starting measurement, access rights become an architecture question and value does not hold without follow-up. Hence this method, drawn from our engagements: the full anatomy of an enterprise AI project, in nine phases.
The method at a glance
An end-to-end AI project goes through nine phases. Three are about deciding: intent, diagnosis, portfolio. Three are about building: architecture, foundations, pilot. Three are about value: proof, production, long-term value. Each phase produces identifiable deliverables, carries a recurring trap and ends with a verifiable exit criterion. Until that criterion is met, moving on does not remove the work: it pushes it to where it costs more.
The order does not rule out going back: a pilot can send you back to the architecture, and proof can reopen the portfolio. But the decisions are ordered: you do not choose a technical solution before choosing a problem, and you do not expand a system before counting its errors.
The method applies to a document assistant as much as to an agent acting in production; only the weight of each phase changes. A low-risk case goes through all nine phases in a few weeks, with light deliverables; a case that influences decisions affecting customers spends more time on architecture, foundations and the pilot. Skipping a phase never saves time: it shifts a debt into production.
DECIDE
- Intent
- Starting point
- Portfolio
BUILD
- Architecture
- Foundations
- Pilot
PROVE AND EXPAND
- Proof
- Production
- Value
Every phase has an exit criterion.
Phase 1: the decision of intent
Every project starts with an intention; few start with a decision. Phase 1 turns "we need AI" into an explicit choice: which problem, backed by which sponsor, with which criterion for abandoning it. That last point is the most important deliverable, and the one least often produced. Deciding what would justify stopping before you invest costs an hour of clear thinking; deciding it afterwards costs a political negotiation.
Deliverables. A one-page statement of intent: the problem expressed as a process result to be changed, a named sponsor who genuinely sets time aside, the budget for the proof phase and the written criterion for abandoning the project.
The classic trap. The copycat project: launched because a competitor made an announcement, without a problem of your own. The sponsor is a name on a slide, not a person who will make the call when the time comes.
Exit criterion. The sponsor can state, without notes, the result to be changed and the conditions for stopping, and both appear in a document they have signed.
Phase 2: diagnosis and the starting measurement
You cannot improve a process you have not measured; you can only change it. Phase 2 describes the real work, not the official process: who does what, where time is lost, where rework concentrates, which data flows and under which legal constraints. It is also the moment to fix the starting measurement that will make any proof possible.
Deliverables. A description of the process as it is actually carried out; figures (turnaround time, rework rate, exception rate, cost per case, quality gaps); an inventory of the data and software involved, with their owners.
The classic trap. Measuring after the fact. Without a starting measurement, the team will have to prove the pilot's value against people's memories, and memories always lose against the finance director's scepticism. A common variant: describing the process in the quality manual, which nobody follows.
Exit criterion. You know what the process costs today, with figures to back it up, and the person who runs it recognises those figures as their own.
Phase 3: the portfolio and the first case
Even when the brief covers a single use case, it has to be chosen against others. Phase 3 brings together several candidates and scores them on three criteria: gain (economic size, effect for users, strategic importance), effort (access to data, complexity of connections, controls required, ease of testing) and risk (data involved, consequence of a mistake). The first case is not the one that promises the most, but the one that combines a significant gain with the shortest path to reliable proof.
Deliverables. A portfolio scored on gain, effort and risk; a first case chosen and documented; the list of cases set aside, with the reason they were postponed, which saves you from replaying the same debate every quarter.
The classic trap. Choosing the most spectacular case: often customer-facing, often high-risk, almost always the slowest to prove. A visible first failure costs more credibility than three quiet successes earn.
Exit criterion. The sponsor, the process owner and the technical team all defend the same first case, for the same reasons.
Phase 4: architecture and boundaries
Before building, you need to draw the boundaries: which data comes in, which rights apply, which technical solution carries the case and what remains reversible. On the choice of solution, our rule matches the work Anthropic has published on building agents: start with the simplest solution that will do (a single call to a tool, a chain of prompts, a routing step) and only move to an autonomous agent if the case demonstrably requires it. On integration, open standards such as the Model Context Protocol reduce dependence on suppliers and make it easier to switch later.
Deliverables. A diagram of data flows with their owners; a rights model in which the system inherits the user's rights, never more; a choice of solution justified in writing; an analysis of reversibility: what a change of model or supplier would cost.
The classic trap. Overbuilding: creating an autonomous agent where a chain of prompts would have done, then spending months correcting autonomy nobody needed. The opposite trap exists too: ignoring access rights and discovering during the pilot that the assistant shows documents the user should never have seen.
Exit criterion. Every piece of data that enters or leaves the system has a documented path, an owner and a verifiable access boundary.
Phase 5: the foundations
Phase 5 builds what never shows up in a demonstration: properly assigned identities and access, reliable data connections, operational controls and, above all, the test environment. This is what separates a managed project from a blind one: a set of real, representative and deliberately difficult cases that can be rerun after every change. The NIST AI Risk Management Framework says the same thing in its own way: you can only govern what you can describe and measure.
Deliverables. Technical accounts with minimum rights; working, logged access to data; a test set built from real cases, including edge cases; the basic controls (logging, alert thresholds, incident procedure).
The classic trap. Skipping the test environment to "get to the pilot faster". The team then adjusts prompts by gut feeling, unable to tell an improvement from a regression, and the pilot becomes a series of contradictory impressions.
Exit criterion. You can rerun your test set and get a reproducible result before a single real user has touched the system.
Phase 6: the supervised pilot
The pilot does not test the technology: it tests the changed process, its controls and the people who carry it out. Phase 6 opens the system to a limited, named group, with systematic human approval at first, then sampling as confidence builds on facts. Every error is classified: severity, cause, how easy it is to detect. It is this count of errors, not the enthusiasm of the first users, that will feed the decision in phase 7. Supervision is not mistrust of the system: it is the instrument that will allow you to lighten it, with proof in hand.
Deliverables. A written scope (named users, cases covered, cases explicitly excluded); a human approval process with its sampling rules; a log of classified errors; quality thresholds set before launch, not after.
The classic trap. The showcase pilot: three enthusiastic users, no error log, and a decision to expand based on anecdotes. Errors that are not counted during the pilot get counted later as incidents.
Exit criterion. You have an error rate measured by type of case and compared with the starting measurement from phase 2, not a general impression.
Phase 7: proof and the decision to expand
Phase 7 is a decision meeting, not a celebration. Three outcomes are legitimate: expand, because the thresholds have been met; adjust, because the value exists but elsewhere or in a different form; stop, because the proof is lacking, while keeping what has been learnt. A decision to stop based on proof is a management success; expanding through inertia is a failure postponed. The gain is measured as the difference between the pilot's results and the starting measurement from phase 2: that is what turns an estimate into a fact.
Deliverables. A results pack: the difference from the starting measurement, full costs (build, running, supervision), incidents and how they were resolved, actual use; a written decision (expand, adjust or stop) signed by the sponsor.
The classic trap. Expanding through inertia: "the pilot went well", so it is expanded, without comparing results with the thresholds, often because no thresholds were set in phase 6. Stopping was never a real option, just a line on a slide.
Exit criterion. The decision is made on figures and signed, and stopping was genuinely on the table.
Phase 8: going into production
Going into production is not the end of the project: it is the start of running it. Phase 8 turns a pilot that works into a service that lasts: monitoring (traces, cost per request, drops in quality), organising operations (who responds when it breaks, with what commitments), cost control and training the teams in the new process, not just in the tool. This is where long-term economics are decided: a cost per request accepted during the pilot can become indefensible at volume if nobody watches it.
Deliverables. Production monitoring dashboards; an operating procedure with a named owner; budgets and cost alerts; a training plan rooted in the process; the security review before going into production.
The classic trap. The phantom handover: the project team moves on to the next topic, nobody is on call, costs drift silently and quality degrades with nobody watching. The system dies slowly in production, without anyone noticing.
Exit criterion. The system has a named operations owner, a tracked budget and quality measured continuously, not just on days when something goes wrong.
Phase 9: long-term value
The last phase has no end: it is the steady state. Measurements are repeated at regular intervals, because the process, the volumes and the models change. A short monthly review looks at value, costs, risks and trade-offs. And proven gains fund the next cases in the portfolio: it is reinvestment, not enthusiasm, that turns a first project into a programme.
Deliverables. Repeated, dated measurements; a monthly review with a fixed agenda (value, costs, incidents, risks, decisions); a reinvestment rule that links proven gains to the next cases.
The classic trap. Declaring victory at go-live and no longer measuring. Six months later, nobody knows whether the system is still creating value, or whether it ever did. The next budget is then negotiated on faith, in front of a finance director who has none.
Exit criterion. You do not leave this phase; you keep it going. You are in the right rhythm when every monthly review produces at least one decision: expand, adjust, stop or reinvest.
The nine exit criteria in one table
| Phase | You can move on when… |
|---|---|
| 1. Decision of intent | The sponsor can state, without notes, the result to be changed and the criterion for abandoning. |
| 2. Diagnosis and starting measurement | The current cost of the process is quantified and confirmed by its owner. |
| 3. Portfolio and first case | Sponsor, business and technical teams defend the same case, for the same reasons. |
| 4. Architecture and boundaries | Every piece of data has a documented path, an owner and verifiable access. |
| 5. Foundations | The test set produces a reproducible result before any real user. |
| 6. Supervised pilot | The error rate is measured by type of case and compared with the starting measurement. |
| 7. Proof and expansion | The decision to expand, adjust or stop is made on figures and signed. |
| 8. Going into production | Named operations owner, tracked budget, quality measured continuously. |
| 9. Long-term value | Every monthly review produces at least one decision. |
From method to implementation plan
Nine phases fit into an article; carrying them out takes a plan. To show in practice what each phase produces, we have anonymised a complete, 57-page implementation plan drawn from a real engagement: it works through the nine phases deliverable by deliverable, from the statement of intent to the monthly reviews, with the document templates and thresholds we used.
Two rules sum up how to use the method. Do not skip any phase: each one exists because leaving it out has a cost. And do not turn it into bureaucracy: on a simple case, a phase can fit into one meeting and two pages. What matters is not how long a phase lasts, but that its exit criterion is met.
Where to start on Monday
- Name the sponsor and have them write down the criterion for abandoning the project, before any conversation about tools.
- Choose a process and measure it for one to two weeks: turnaround time, rework, exceptions, cost per case.
- Score three to five candidate cases on gain, effort and risk, and keep only one.
- Draw the data flows and access rights for the chosen case on a single page.
- Build a test set from real cases, including the difficult ones, before writing a single prompt.
- Set the pilot's quality thresholds in writing, together with the date of the decision to expand, adjust or stop.
- Put the monthly review in people's diaries now: follow-up that is not scheduled does not exist.
The difference between an AI project that proves and one that merely promises rarely comes down to the choice of model. It comes down to the discipline of the nine phases, and it shows from the very first week.
How we can help
Phases 1 to 3 are prepared with your management team as part of AI Adoption Support. Phases 4 to 9, from architecture through to production and follow-up, come under AI Engineering: a scoping phase of one to three weeks, then a firm quote, or a subscription if several projects follow one another. For an SME, the short version of this method is the 30-day pilot.
Discuss this decision
hello@numezis.com ↗