How to tell whether an AI agent pilot worked
The four numbers that decide a 30-day AI agent pilot, the baseline to take first, and the bar to set before day one.
Quick answer
Measure four things over 30 days: how many requests users made, how many of those ended in a completed task, how much time each completed task saved compared with doing it by hand, and what users asked for that the agent could not do. Write down the bar for success before the pilot starts, and take a baseline of how long the same tasks take today, or the numbers at the end will prove nothing either way.
Most pilots end with a feeling: people seemed to like it. That is not enough to decide whether to keep paying, or to set a price. A pilot is worth running only if the decision at the end is already written down at the start. Here is how to set one up so the numbers decide.
Before day one: the baseline and the bar
Pick the tasks. Three to five things users do that take several screens: a setup, a report, a change across many records. These are where an agent earns its keep.
Time them today. How long each takes a typical user by hand, including finding where it lives. Ask a few users, or time it yourself as someone who does not know the product well. This is the baseline everything is measured against.
Write down the bar. For example: at least 300 completed tasks in 30 days, saving at least two minutes each. Whatever the number, agree it before anyone sees results.
The four numbers to track
- Requests. How many times users asked for something. It shows people found it and tried it, nothing more.
- Tasks completed. Requests that ended in the thing being done, confirmed by the user. This is the number that matters.
- Time saved. Completed tasks times the minutes each saved against the baseline. Value it at what an hour of your users' time is worth; the ROI calculator does the arithmetic.
- What it could not do. Requests no action could handle, in the users' own words. Not a failure number: it is the most direct list there is of what users expected your product to do.
During the pilot
Read the requests every few days in the first two weeks. You will see phrasing you did not expect, tasks you did not list, and requests the agent should handle but cannot yet. Add the actions people actually ask for; that is usually where the second half of the pilot gets its best numbers.
At day 30: decide
Compare against the bar you wrote down. Above it: keep it, and price it against the time it saves. Below it, look at why before deciding: too few users found it, the wrong tasks were picked, or the tasks were too short to save anything. Each has a different fix.
The verdict
Four numbers, a baseline taken first and a bar agreed in advance. That is exactly how a Verb pilot runs: I set it up with you, it runs free for 30 days on your product, the dashboard shows the requests, the things done and the requests it could not handle, and the price afterwards is agreed from what it measurably saved. See what an in-app AI agent really costs for the other side of the sum.
Frequently asked questions
What metrics matter in an AI agent pilot?
Requests made, tasks completed, time saved per completed task, and the requests it could not handle. Usage alone says people tried it; completed tasks and time saved say it was worth it.
How long should an AI agent pilot run?
Thirty days is enough for most products: long enough to cover a monthly cycle and for the novelty to wear off, short enough to decide while it is still fresh.
How do I price an AI agent after a pilot?
From what it measured: the time it saved users, valued at what that time is worth, against what it costs to run. That is how a Verb pilot is priced: free for 30 days, then agreed from those numbers.
Vishnu, founder of AskVerb @itsvishnups
Building the AI agent that lives inside a SaaS product and does the work its users ask for. Writing down what I learn along the way, including the parts I got wrong first.