How to measure the real value of an AI programme

AI programmes often report the number of users, active licences, content produced or an estimate of hours saved. These figures describe activity; they do not prove value to the business.

Value exists when AI changes a result that matters: volume handled, turnaround time, quality, risk, revenue, costs, capacity or the quality of decisions. Measuring it requires a starting point and a credible link between what was deployed and what changed. This article is for management teams overseeing several projects; for a first use in an SME, the ROI guide is enough.

Describe the mechanism before you measure

Start from the business constraint, not the technology. A useful description has four parts:

1. the process in which the solution is used;
2. the mechanism: what changes in the way people work;
3. the result: which measure should improve;
4. the economic link: why that result matters financially or strategically.

Example: better access to approved technical documentation reduces repeated diagnoses, shortens resolution times and frees experts for the difficult cases.

The chain between activity and value is longer than people think: access is opened; the person changes the way they work; the task is done differently; the process produces a different result; that result has a financial effect; part of that effect is actually captured by the company. An assistant that saves time finding a contract clause only creates value if that time is put to use, if the quality of the clause holds up and if the work does not simply shift to review.

Measure before you roll out

Measure the current process using data from your software, supplemented by a sample where that data is incomplete: volume, time per step, waiting time, rework, escalations, errors, quality, user effort. Record the uncertainty: a defensible range is better than a precise figure built on shaky assumptions.

Three layers of indicators

Do not mix the layers, or each team will end up presenting whichever one suits it.

• Usage indicators show whether the mechanism is taking hold: the share of intended users who actually use it, the share of tasks covered, repeat use, the correction rate. A high correction rate is normal during a supervised pilot; it is worrying if it does not fall afterwards.
• Process indicators show whether the way of working has changed: turnaround time, waiting time, hand-offs, rework, exceptions, human effort per case. This is often where real change shows up first.
• Outcome indicators show whether the business result has moved: cost per case, volume handled, capacity freed up, customer satisfaction, losses avoided, revenue. They are slower and noisier, but they are what an investment committee is waiting for.

For each indicator, record an owner, a definition, the source, the frequency, the reference period, the target and the factors that could distort the reading.

Gross, net and captured value

The most common mistake is to treat time theoretically saved as money. Distinguish three levels:

Five hundred hours freed up in a team with no backlog, no hire avoided and no plan to redeploy them are gross value, and little more. Conversely, an improvement in quality that prevents the loss of a customer can be worth a great deal even if little time is saved.

Proof in proportion to the decision

Not every initiative needs a controlled experiment. A low-cost internal trial can rely on a before-and-after comparison, with its limits written down. A significant organisational change deserves a control group, a phased roll-out or a controlled test on tasks.

At a minimum, track volume, headcount, seasonality, other changes to the process and the version of the tool. Break results down by type of case: an average improvement can hide a gain on simple cases and a deterioration on complex ones, and the decision must follow the detail, not the average. Add direct observation: examine a sample of successes, failures and corrections, and ask users which work has disappeared and which review work has appeared.

Thresholds for expanding, adjusting or stopping

Set in advance, these thresholds prevent two mistakes: expanding a popular tool that delivers nothing, and abandoning a good mechanism because its first implementation was weak.

Decide every month

Measurement becomes useful when it changes what the portfolio does. A monthly review can fit on one page per initiative:

1. annual value at the current pace, compared with the forecast;
2. usage, process and outcome indicators;
3. quality, controls and incidents;
4. full cost and cost per completed case;
5. level of confidence and the limits of attribution;
6. recommendation: expand, adjust, stop or wait;
7. the next test and the date of the next decision.

Three uncomfortable questions to ask at every review: what would we have seen without the tool? Which result moved in the targeted process, and not elsewhere? What proof would make us stop? If the team cannot answer, the set-up is still measuring activity.

An example, without shortcuts

Take a department that handles 1,200 cases a month, taking 42 minutes on average, with 12% of cases reworked. A search and drafting aid cuts drafting time, but the team still has to review. The correct analysis does not multiply the advertised minutes by the annual volume.

It first checks that turnaround time is falling, that reworked cases are decreasing, that the review workload is not rising and that service quality is holding. Only then does it put a value on the lasting difference: backlog absorbed, fewer temporary staff, more cases handled with the same headcount. It subtracts the costs of the platform, integration, review and running. Finally, it gives a confidence range and the condition under which the benefit disappears.

This rigour does not make the business case smaller: it makes it fundable. Management funds a mechanism it understands, the teams can improve it, and the company can stop it when the proof no longer follows.

How we can help

We set up this measurement system (starting measurements, indicators, monthly review) as part of AI Adoption Support, at a fixed price on quotation or by subscription, three months minimum. When measurement requires instrumenting an agent or an integration, that comes under AI Engineering. How to bring these decisions into a portfolio is described in how to build an enterprise AI roadmap.