Ai Automation

How to Compare Automation ROI Tools and Vendor ROI Claims

Published Updated 9 min read

To compare automation ROI tools, or the ROI claims vendors make, put every claim through the same four tests: where each input comes from, whether freed hours are counted as cash, whether payback is counted from kickoff with running costs netted off, and whether the result survives a downside case. A tool or claim that fails any of the four is a marketing figure, not a business case.

This guide gives you the criteria, a worked stress test of a typical vendor claim, the red flags, and a comparison of the main types of ROI tool. The method is the same whatever the category: AP automation, incentive compensation, spend analytics or IT service desk. For the benchmarks and the full calculation method, see our intelligent automation ROI guide with benchmarks and calculator.

The six criteria for any automation ROI tool

Score each tool, or each vendor’s ROI model, against these six. They are the same tests a CFO applies.

CriterionWhat good looks likeWhat fails
Sourced inputsEvery default (cost per invoice, cost per ticket, hourly rate) is visible and names its source”Industry data”, or math hidden in a script
Capacity versus cashFreed hours are discounted unless a hire, overtime or revenue actually movesEvery recovered hour multiplied by a fully loaded rate
Cost linesBuild, integration, change management, data preparation, retraining and running costs as separate linesLicence only, or one “contingency” slider
A formula that fits the workAP is cost per invoice plus discounts; AR is DSO; a service desk is ticket volume times deflection times cost per ticketOne “percent of spend saved” field for everything
Payback from kickoffNo benefit during the build, a ramp after go-live, running costs netted off each monthPayback measured from go-live at full benefit
SensitivityA band on volume and on the discount rate, plus a downside caseOne ROI percentage with no time horizon

Two smaller checks catch a lot. First, is the ROI figure net (value minus cost, divided by cost) or gross? A tool that reports value divided by cost will show a much larger number for the same case. Second, what is the time horizon? A “300% ROI” over five years and over three years are different claims.

How to stress-test a vendor’s ROI claim

A vendor claim is usually built from three inputs: hours saved, an hourly rate, and an implementation price. Each one hides an assumption. Work through them in order.

1. The baseline. Ask where “hours saved” came from. If it is the vendor’s estimate of how long your task takes, replace it with your own measurement: two weeks of logged time per item at normal volume, plus error and rework counts from your systems. The baseline method in our intelligent automation ROI guide sets out the steps.

2. The conversion to cash. Hours become money only when a hire is avoided, overtime stops or the time goes to work that earns revenue. If none of those is planned, discount the hours. Kwestra’s calculator converts half of freed time to value and takes a further 15 percent risk haircut; analyst studies do something similar, for example the Forrester study commissioned by SS&C Blue Prism adjusted one benefit downward by 10 percent for risk. Pick a rule and apply it to every vendor equally.

3. The payback math. Simple payback is implementation cost divided by monthly value, which assumes full value on day one. Recount it from kickoff: zero benefit during the build, partial benefit while exception rules are tuned, and running costs (licence, usage, maintenance) subtracted every month.

4. The hidden costs. Add the lines the claim leaves out: integration with your ERP or ticketing system, data preparation, change management and training, and ongoing tuning and monitoring. Get each one quoted or estimated separately, so you can see which one moves the result.

5. The downside. Halve the headline benefit, double the vendor’s published cost, and re-run. If the case still clears your hurdle rate, it is a decision you can defend.

Worked example: a “three-month payback” claim

The figures below are a hypothetical illustration, not a benchmark. Replace every input with your own.

A vendor claims its tool saves 12,000 hours a year at $50 an hour, or $600,000 a year, against a $150,000 implementation: payback in three months.

StepAdjustmentResult
BaselineYour time study shows the task takes 70% of the vendor’s assumed time8,400 hours a year
Cash conversionHalf the hours go to a hire avoided; the rest has no named destination4,200 hours x $50 = $210,000 a year
Running costLicence, usage and maintenance at $5,000 a month$60,000 a year
Hidden costsIntegration and change management quoted at $45,000$195,000 total implementation
TimingThree-month build, three months at 50% benefit, full benefit from month 7Net $12,500 a month at steady state

Months 4 to 6 recover about $11,250. The remaining $183,750, at $12,500 a month, takes about 15 more months. Payback lands around month 21 from kickoff, not month 3. That may still be a good investment. The point is that it is a different decision from the one the vendor’s number implies, and it is the one finance will check.

For context on what is realistic: Deloitte’s global intelligent automation survey found organizations automating at scale reported an average payback of nine months, while those still piloting expected fifteen. A first workflow should be compared against the pilot figure, not the at-scale one. Our AI automation payback period guide breaks this down by use case.

Red flags in vendor ROI claims

  • No formula. You cannot see how the number was produced, or the defaults have no named source.
  • Hours treated as cash. The headline is hours saved times a loaded rate, with no question about what happens to the hours.
  • Payback from go-live. The clock starts after the build, or the build time is not mentioned.
  • Licence-only cost. Integration, data preparation, change management and tuning are missing, or bundled into one contingency.
  • A single number. One ROI percentage, no horizon, no range, no downside case.
  • A case study presented as a forecast. The figure comes from one customer’s best deployment, without the baseline, the time since go-live or who measured it.
  • A study that does not look like you. A commissioned study’s composite organization is much larger, in another industry or on different systems. The SS&C Blue Prism study’s composite, for example, is a global financial services firm with $4 billion in annual revenue.
  • Guaranteed payback. “Payback in three months, guaranteed” with no conditions on volume, data quality or adoption.

None of these means the vendor is dishonest. Case studies are usually real; they are just not representative of your first year. The fix is the same each time: re-run the claim with your inputs.

Types of automation ROI tool compared

ROI tools are not one category. Each type is useful for something and biased in a predictable direction.

Tool typeWho builds itGood forBuilt-in biasAsk before you rely on it
Vendor web calculatorThe vendor selling the productA quick sense of scaleFull capacity-to-cash conversion, hidden defaults, licence-only costShow me the formula and the source of each default
Vendor-commissioned analyst study (for example a Forrester TEI)An analyst firm, paid by the vendorA documented method with customer interviews and risk adjustmentInterviewed customers and the composite may be larger or more mature than youIs it marked commissioned? How does the composite compare to us?
Vendor case studyThe vendor’s marketing teamProof the product worked somewhereThe best deployment, not the typical oneWhat was the baseline, how long after go-live, and who measured it?
Independent benchmark (APQC, Ardent Partners, MetricNet and similar)Research firms and associations, sometimes vendor-sponsoredChecking whether your baseline is in the normal bandDefinitions, sample and year differ between sourcesHow is the metric defined, which year, and who sponsored it?
In-house spreadsheetYour finance or ops teamYour own numbers, full controlInherits whoever set the assumptions; formula errorsHas someone outside the project reviewed it? Is there a downside case?
Transparent calculator (constants shown, payback from kickoff)Consultancies and some vendors, including KwestraA first-pass case you can defend and editStill built by someone who sells implementationCan you see and change every constant?

Forrester’s Total Economic Impact method models four components (cost, benefits, flexibility and risk), and Forrester states that all TEI studies are clearly marked as commissioned. That makes them more useful than a vendor calculator, as long as you read them as what a mature customer achieved rather than what you will achieve in year one.

Comparing several vendors side by side

When the question is which of three or four vendors delivers the best ROI, do not compare their ROI figures with each other. Each was built on different assumptions. Instead:

  1. Use one model and one baseline. Feed each vendor’s price and claimed effect into the same model, with your measured baseline.
  2. Compare three-year total cost. Licence, usage, implementation, integration and internal time, per year, for three years. Per-seat and per-transaction pricing diverge as volume grows.
  3. Test ease of use on your data. A short pilot on a sample of your own documents, tickets or transactions tells you the real exception rate. Ease of use, for automation, mostly means how much work the exceptions take.
  4. Compare payback from kickoff and the downside case, not the headline ROI.
  5. Check the exit. Who owns the configuration, the data and the workflow logic if you leave?

Run the numbers yourself

For a first estimate from three inputs, use the quick automation ROI calculator. For the full model, with sourced constants, payback from kickoff, three-year NPV and a sensitivity band, use the calculator in our intelligent automation ROI guide. Run each vendor’s claim through it with your own baseline and the comparison becomes straightforward.

Frequently asked questions

01 How do I compare automation vendors on price, ease of use and ROI?

Put every vendor on the same basis. Run each vendor's claim through one model with your own measured baseline, the same three-year horizon and the same discount rate. Compare three-year total cost (not the year-one licence), payback counted from kickoff, and a downside case. Judge ease of use in a short pilot on your own data, by how many items still need a person and how long each exception takes.

02 How should a large enterprise evaluate a vendor's ROI claim?

Ask for the formula and the source of every default, check whether the headline counts freed hours as cash, check whether payback starts at go-live or at signature, and list the running costs the claim leaves out. If the claim comes from a commissioned study, check how closely its composite customer resembles you. Then re-run the claim with your own baseline.

03 Are vendor-commissioned ROI studies worthless?

No. A study such as a Forrester Total Economic Impact report interviews real customers and risk-adjusts its figures, and Forrester marks every one as commissioned. It shows what a mature deployment at the composite organization achieved. Treat it as a ceiling for your case, not a forecast.

04 Does this apply to incentive compensation or spend analytics tools as well?

Yes. The checks are the same for any category that sells on ROI: AP automation, incentive compensation management, spend analytics, IT service desk. What changes is the value formula, so make sure the tool models the lever that category actually moves.

Want a second opinion on a vendor’s ROI case?

Send us the vendor’s numbers and your baseline. We re-run the claim with the method above and tell you what we would expect in year one, including where a custom integration would or would not beat the vendor. Book a working session.

Read next:

Questions? We answer them in person.