AI is everywhere right now. Every company has an AI strategy. Every product roadmap seems to have at least one AI initiative. And as a Product Manager, AI has become part of almost every conversation I have. Whether it’s automation, agents, or productivity tools, the discussion usually ends up in the same place:
“How much more can we get done?”
It’s a reasonable question. After all, one of AI’s biggest promises is productivity. More code shipped. More tickets resolved. More content generated. More work completed in less time.
But lately, I’ve been wondering if we are focusing on the wrong thing. Getting more done is great. Moving faster is great. But neither automatically means we are creating more value. A team can double its output and still have little impact on the outcomes that actually matter to the business and that got me thinking:
Are we measuring AI success the wrong way?
The productivity trap
When organizations evaluate AI initiatives, they often look at:
- Hours saved
- Tasks completed
- Tickets processed
- Throughput improvements
These metrics are useful. They tell us whether work is being done more efficiently, but they don’t tell us whether that work is making a meaningful difference.
A customer support team might handle twice as many tickets. An engineering team might ship features faster. A marketing team might generate content at scale.
But if customer retention doesn’t improve, revenue doesn’t grow, customer satisfaction stays flat, or business outcomes remain unchanged, did the AI investment actually create value?
That’s the distinction I think we are missing. Efficiency and impact are not the same thing and AI makes that distinction even more important. Instead of focusing primarily on productivity metrics, I think organizations should start evaluating AI investments through a business lens. A simple way to do that is through what I call:
Return on AI Automation (RoAI)
RoAI isn’t meant to replace ROI or introduce a new financial concept. Instead, it provides a simple way to apply ROI thinking to AI initiatives. While productivity metrics tell us how efficiently work is being done, RoAI helps answer: is the value created by AI greater than the cost of building and operating it?
RoAI = (Business Value − AI Cost) ÷ AI Cost
Where:
- Business Value is the measurable value generated by the AI capability.
- AI Cost includes development, infrastructure, licensing, maintenance, monitoring, and operational support.
The exact definition of business value will vary depending on the use case. For example, value might come from:
- Higher customer retention
- Increased conversions or revenue
- Improved customer satisfaction
- Reduced operational risk
- Fewer production incidents
- Faster time-to-market
The important thing is that we are measuring outcomes that matter to the business and not simply the amount of work completed by AI.
Let’s look at a simple example:
Imagine an e-commerce company launches an AI chatbot to automate customer support. Not all support tickets create the same business value. Some inquiries are expensive to handle, while others are relatively low cost.
The chatbot successfully resolves 6,000 support tickets per month.
Scenario 1:The chatbot automates 6,000 support tickets, and each ticket would have cost the company $5 to handle.
Business value = 6,000 × $5 = $30,000
The AI costs $10,000 per month to develop, operate, and maintain.
RoAI = ($30,000 − $10,000) ÷ $10,000
RoAI = 2 (or 200%)
This means that for every $1 invested in AI, the company generates $2 of net value.
Scenario 2: The chatbot still automates 6,000 support tickets, but these are lower value inquiries that cost only $1 each to handle.
Business Value = 6,000 × $1 = $6,000
The AI still costs $10,000 per month.
RoAI = ($6,000 − $10,000) ÷ $10,000
RoAI = -0.4 (or -40%)
This means that for every $1 invested in AI, the company loses $0.40.
In both cases, the AI resolves the exact same number of tickets. If we only measured productivity, both implementations would appear equally successful but RoAI tells a different story. In one scenario, the AI creates significant business value. In the other, the value created doesn’t justify the cost of the investment.
That’s why measuring AI success through activity alone can be misleading. The real question isn’t how much work the AI performs – it’s how much value that work creates.
Why RoAI matters
The problem with many AI discussions is that they stop at productivity. We are celebrating the number of tasks automated, tickets resolved, or hours saved. But those metrics don’t answer the real question.
Instead of asking:
“How much work did the AI help us do?”
We ask:
“How much value did the AI help us create?”
That’s a much harder question but it’s also the one that matters because organizations don’t invest in AI simply to increase activity. They invest in AI to improve outcomes.
The hard part: Measuring value
Of course, measuring value isn’t always easy. Calculating costs is relatively straightforward. Calculating business value is where things become more challenging. Organizations need to identify the outcome they’re trying to influence and estimate its impact. For example:
- What’s the value of retaining a customer?
- What’s the value of increasing conversion rates?
- What’s the value of improving customer satisfaction?
- What’s the value of preventing a production outage?
These aren’t always precise calculations. They often involve assumptions, modeling, and estimation. I believe that imperfect measurements tied to business outcomes are still more useful than productivity metrics that may have little connection to actual impact.
Measuring what matters
As organizations continue investing in AI, the challenge isn’t simply automating work – it’s ensuring those automations create meaningful outcome and as Product Managers, we are ultimately accountable for outcomes, not output. The same principle should apply to AI.