§ InsightsNo. 117

Piloting an accounts review tool on past signed files

A fair pilot compares a tool's output with review points your own managers recorded on signed files, without turning the counts into accuracy claims.

A reviewer piloting an accounts review tool on a past signed file at a dual-monitor desk

Piloting an accounts review tool on past signed files means running it over accounts your practice has already reviewed, signed and filed, so that you can compare what it flags with what your own managers found. This article is for partners and managers who want a fair test before they commit to any tool. It covers how to choose the files, how to set up the comparison, what to record, and what a pilot cannot tell you. It is vendor neutral and makes no claim about how any tool performs.

What a pilot can and cannot show

A pilot on past files answers a narrow question: on these files, which of the points your reviewers recorded did the tool also raise, and what else did it raise that they had not? It does not show how the tool would behave on other files, and the practice should not quote its counts to clients, insurers or colleagues as a measure of the tool, because a handful of files from one practice are not a sample of anything else.

The ICO makes a related distinction in its AI guidance. It describes statistical accuracy as how often an AI system guesses the correct answer against correctly labelled test data, and it warns that there are trade-offs between precision and recall. A pilot is a small version of that exercise, with your own review notes as the labels. Used that way, it tells you whether the tool is worth a longer trial, and where its output needs a human most. Our guide to choosing an accounts review tool and testing it sets the pilot in the wider selection process.

Before you upload anything

Signed files hold client personal data, so the pilot is processing in its own right. Three steps come first.

  • Put the written terms in place. If the supplier will process personal data on the practice’s behalf, UK GDPR Article 28(3) requires a binding contract covering, among other things, documented instructions, confidentiality and security, and deletion or return of the data when the services end. Our note on client data and AI tools under the UK GDPR covers the questions to ask the supplier.
  • Decide whether a data protection impact assessment is needed. The ICO says that if you are in any doubt, it recommends doing one, and where Article 35 applies it requires the assessment before the processing starts.
  • Minimise the data. The ICO says you should process only the personal data you need for your purpose. If the tool reads accounts and notes, there is no need to send payroll by employee, bank statements or correspondence.

Checked against the ICO pages cited below on 21 November 2026. The ICO says on its DPIA and AI pages that the guidance is under review because of the Data (Use and Access) Act.

Ask the supplier, in writing, to delete the pilot files at the end and to confirm that deletion, and keep a copy of the confirmation. A pilot that leaves client data behind has created the problem the pilot was meant to help you avoid.

Choosing the files

Select files for what they test: a set that mixes ordinary and awkward files will show more than one made up of clean ones.

  • Include more than one size and regime, for example a micro-entity set under FRS 105 and a small company set under FRS 102 Section 1A, if your practice prepares both.
  • Include files with prior period adjustments, a changed accounting policy, a long or short period, and a first-year set, where you have them.
  • Include at least one file that your reviewers passed with few or no points. It shows whether the tool raises noise on a clean set.
  • Use signed files from more than one preparer or reviewer, so the test does not reflect one person’s habits.
  • Use files from a year when the standard you apply today also applied, or state which edition each file was prepared under, because a tool tested against a superseded edition tells you little. Our guide to what the 2026 amendments mean for reviewers explains why the edition matters.

Building the answer key

The answer key is a list, per file, of the review points your own team recorded before sign-off: the review notes, the points raised and cleared, and any errors found later. Build it before you run the tool, and do not change it afterwards to suit the results.

  1. Pull the manager’s review notes and the cleared points from the file.
  2. Give each point a reference, a short description, and a category such as casting, comparatives, policy, disclosure or Companies Act statement.
  3. Record which points were material in your reviewers’ view and which were housekeeping.
  4. Ask someone who did not review the file to check the key for ambiguity, so that each point has one clear meaning.
  5. Record the points that a reviewer would now add if they read the file again. Label them as later additions and keep them apart from the original key.

The signed accounts are the corrected version, so they no longer contain the errors your reviewers found. If you can recover the draft that went to review, test the tool on that. If you can only use the signed set, the key should say that the original errors have been corrected, and you can instead seed known points back into a copy, recording each change. Either way, keep a note of what the tool actually received.

Running the test fairly

  • Fix the inputs before the run. Use the same file format and the same supporting documents for every file, and write down the settings.
  • Keep the people who build the answer key separate from the people who score the output, or at least score blind, with the key hidden until the output is in.
  • Run each file once under normal conditions, then repeat a few to see whether the output varies. The ICO says the test data is usually separated from the data used to train a system, which is a reminder not to tune the tool on the very files you will score.
  • Do not coach the tool with the answer key. If the supplier is helping with the setup, it should not see your key until the scoring is finished.
  • Give the supplier the same files you give any other tool you are comparing, scored by the same key, so the comparison is one of like with like.

What to measure

Count and record each measure, and avoid calculating a single score, because a single score invites a headline.

MeasureWhat you recordWhat it tells you
Known points foundKey points the tool also raised, by categoryWhere it overlaps with your reviewers
Known points missedKey points the tool did not raiseWhat a human still has to catch
Extra points raisedPoints not in the key, each judged by a reviewerNoise, or something your reviewers missed
Reference qualityWhether each point cites the paragraph or section behind itWhether the file can evidence it
Effort to actWhether a reviewer could clear the point from the output aloneHow useful the wording is

The ICO’s AI guidance names the two kinds of error that matter. A false positive is a case the system labels positive that is not, and a false negative is a case it labels negative that is positive. For a review tool, a false positive is a point raised that is not a point, and a false negative is a real point left unflagged. The ICO also defines precision as the percentage of cases identified as positive that are in fact positive, and it says there are trade-offs between precision and recall. The ICO’s example is that aiming to find as many positive cases as possible can come at the cost of more false positives, so look at both and decide which error costs your practice more.

An illustration of the arithmetic

This is hypothetical and is not a benchmark of any tool. Suppose one file has 8 points in its answer key. The tool raises 6 of them, misses 2, and raises 5 more. A reviewer judges 2 of those 5 to be real points the original review missed and 3 to be unnecessary. You would record 6 found, 2 missed, 2 additional real points and 3 unnecessary points, with a note on each. Dividing these into one percentage hides the 2 real additions and the nature of the misses. Keep the table, and read the misses by category.

Reading the results

  • Read every miss. A miss on casting tells you something different from a miss on a related party note, and a miss on a material point is more important than one on housekeeping.
  • Read every extra point. Some will be noise. Some will be genuine points that your review missed, which you should treat as a finding about your own process, and fix on your own files under your quality management system.
  • Record the reviewer’s time and the clarity of the output separately from the hit counts, in the practice’s own words and without turning them into claims.
  • Record where the tool says it does not know. A note of where it was uncertain tells the reviewer where to look first.
  • Repeat on a second set before deciding. The ICO says, of AI systems in use, that monitoring should be proportionate to the impact an incorrect output may have on individuals, and a practice can monitor a tool it adopts in the same spirit.

Write a short pilot note for the file: the files used, the key, the settings, the counts, the extra points and what you decided. Our guide to quality management for accounts preparation covers evidence of review, and a pilot note is the evidence for the decision to adopt a tool. The tool then becomes an input to review. The partner remains responsible for the accounts, as our guide to signing off AI-reviewed accounts explains.

Questions practices ask

How many files do we need for a pilot?

There is no sourced number, and a count would imply a statistical standard that a small practice test cannot meet. Choose enough files to cover your regimes and your awkward cases, and treat the result as a guide to a longer trial and not as proof.

Can we publish the results or quote them to clients?

We would not, because the results describe one tool on a few of your files at one time, and a quoted figure can read as a claim about how the tool performs. Keep the results as an internal record of why you made the decision.

Should the pilot use live files from the current year?

Past signed files are safer because the answers are known and nothing depends on the output. If you later run a live file, treat the tool output as one input and keep the review in your normal process. The data protection steps apply in both cases.

What if the tool finds a real error in a signed set?

Treat it as you would any error found after sign-off: tell the partner, assess whether the accounts need correction, and record what you did, so that the note on file shows what the pilot found.

Does a good pilot mean the tool can replace a manager’s review?

No. A pilot shows overlap with your reviewers on a set of files. The judgement calls and the signature stay with the partner, whatever the pilot shows.

Sources