How to score an AI opportunity before you build it
Every AI roadmap eventually becomes a spreadsheet. The question is whether the columns mean anything.
Most prioritization exercises collapse into a single vibe-based score — “high impact” — that nobody can defend two weeks later when the budget conversation gets real. A useful frame separates the things that actually trade off against each other, and forces a number where a number is possible.
The four columns that matter
| Column | Question it answers | Where the number comes from |
|---|---|---|
| Impact | If this worked perfectly, how much time or money comes back? | Observed hours in the workflow, not estimates |
| Effort | What does it cost to build and roll out? | Engineering days + integration surface |
| Risk | What happens when it’s wrong? | Blast radius × reversibility |
| Payback | How long until it’s net positive? | Impact ÷ effort, adjusted for adoption lag |
The trap is filling in Impact from intuition. “Support spends a lot of time on refunds” is not a number. “Refund handling consumed 41 hours across the team last week, 60% of it in two steps” is a number, and it changes the ranking.
Impact: measure the work, not the wish
Ask three people how long a task takes and you get three answers, all of them wrong in the same direction — everyone underestimates the boring parts. The interruptions, the tab-switching, the second look because the first answer didn’t feel right.
That’s why the honest version of this column starts from observation. Not a survey. Not a workshop. The actual shape of the work.
Effort: integration surface is the hidden cost
A model call is cheap. What isn’t cheap:
- Auth and permissions — the agent needs to act as someone, and someone has to decide who.
- The unhappy path — what the thing does when it doesn’t know.
- Rollout — the twenty people who have to change what they do on Monday.
- Maintenance — the workflow shifts, and nobody tells the automation.
Rank effort by integration surface, not by model complexity. The model is the easy part now.
Risk: blast radius times reversibility
Two questions, and only two:
- If this is wrong, how far does the wrongness travel before a human sees it?
- When we catch it, how hard is it to undo?
An agent that drafts a reply for review has a tiny blast radius and perfect reversibility. An agent that sends the reply has neither. Same model, same prompt, entirely different risk column — and the difference is a design decision, not a technical one.
The safest agent isn’t the smartest one. It’s the one whose mistakes are cheap to catch and cheap to reverse.
Payback: the column that kills good ideas
Payback is where elegant projects go to die, and that’s the point. A workflow that saves four hours a week and takes six months to build is a worse investment than one that saves ninety minutes and ships on Thursday — for at least the first year, which is the only horizon anyone is actually budgeting against.
Rank by payback, ship the fast ones first, and let the compounding fund the ambitious ones.
What this looks like filled in
The output isn’t a score. It’s an order. A defensible sequence you can walk into a room with, where every row traces back to something you observed rather than something someone asserted.
That’s the whole trick: rank the work you can see, not the tools you’ve heard of.
Nikita started Silow after watching company after company buy AI it never used — the tools were fine, but nobody could say which work was worth automating. He leads product and the company, and writes most of what you read here.