An AI pilot needs a clear stop decision as well as a success target
An AI pilot should be evaluated against a defined task, realistic test cases and acceptance criteria agreed before the demonstration. Review accuracy, usefulness, operating effort and whether the system stays within its boundaries. Decide in advance what would justify expansion, revision or stopping the pilot.
Put the advice into practice
Build a workflow your team can actually rely on
Connect systems and follow-up with agreed rules, human review and exception handling.
Discuss your workflowA convincing demonstration answers whether the system can do something useful under selected conditions. A pilot needs to answer a broader question: whether the organisation can rely on the proposed use within its actual workflow and responsibilities.
Define the task and the comparison
Choose a bounded task and record how it is currently completed. Identify the required output, the people who review it and the errors that matter. Keep the pilot small enough that the team can inspect the work and understand what changes.
A comparison needs more than the speed of the model response. Include preparation, checking, correction and handover. If a faster first draft creates more review work, the result may still be useful, but the evaluation should show where the effort moved.
Agree acceptance criteria before reviewing results
The National AI Centre’s AI adoption implementation guidance calls for clear acceptance criteria, pre-deployment testing, documented outcomes and monitoring. Apply those principles to the purpose and risk of the particular task.
| Criterion | Question the reviewer should answer |
|---|---|
| Accuracy | Are the required facts correct and supported? |
| Completeness | Does the output include the information the task needs? |
| Usefulness | Can the receiving person use it without reconstructing the source? |
| Boundaries | Did the system avoid unauthorised information or actions? |
| Operating effort | How much preparation, review and correction was required? |
| Failure handling | Did uncertain or incomplete cases reach the agreed owner? |
Do not force every criterion into one blended score. A serious boundary failure should remain visible even if the writing quality is strong. Agree which conditions are mandatory for this use.
An illustrative internal-summary pilot
Imagine a hypothetical professional-services team wants AI to prepare a draft summary from approved, non-sensitive project notes. The intended output identifies decisions, open questions and assigned actions for a human reviewer. It is not authorised to send commitments to a client.
The test set includes a clear record, a missing owner, conflicting dates and notes that do not support a conclusion. Reviewers assess whether the draft preserves the facts and flags the unresolved points. A polished sentence that invents a decision is a failure for this task.
This is a teaching scenario, not a result from a private client deployment. The actual information and permissions would need their own review before a real pilot.
Include cases that challenge the proposed use
Test ordinary inputs as well as incomplete, ambiguous and out-of-scope ones. Use approved sample material where possible and avoid introducing real personal or confidential data simply to make the trial feel realistic.
Keep the test cases and criteria stable enough to compare revisions. If the team changes the task, model, source material and review standard simultaneously, it becomes difficult to understand what improved. Record meaningful changes and rerun the cases affected by them.
Also test the handover. The receiving person needs to know that the output is a draft, where its source sits and what they are expected to check. The interface and workflow can create risk even when the text itself appears accurate.
Make the expansion decision explicit
At the end of the pilot, choose a supported next step. Expand a proven bounded use, revise an identified weakness, narrow the task or stop because the current approach does not meet the criteria. Stopping a pilot can be a useful finding when it prevents an unsuitable deployment.
If the use continues, assign ownership for monitoring, source maintenance, product changes and incident review. A one-time test cannot establish that the system will remain appropriate as its inputs and operating environment change.
Use the existing automation savings guide for a fuller calculation of effort after human review. Keep that calculation alongside quality and boundary checks so the expansion decision reflects the whole operating task.
