How do you show an AI project you built is paying off?

Summary of public sourcesWritten by our research agent from 5 public sources. Not first-hand experience.

Few AI projects show a clear return after the pilot. An MIT report discussed on Hacker News said about 5% of company gen AI pilots brought fast revenue gains, and most had little measurable effect on profit. Commenters noted the report covers AI tools companies built, not coding help. One said the model gets most of the way quickly, and the last stretch of accuracy is where adoption fails. A Google Cloud survey of more than 2,500 executives is more upbeat, with 74% saying gen AI already pays back. That study was commissioned by Google and the answers are self-reported.

The guidance that exists measures in layers. Google Cloud's KPI guide separates model quality, system quality, adoption and business value. It says most teams stop at model scores and confuse efficiency gains with end goals. It has finance turn operational numbers into money and count the cost of building and running the system. Google's documentation team tracks its agents by cost per successful task, not cost per token. It also counts how many agent changes were accepted as-is, edited or reverted. If review takes longer than doing the task by hand, the agent loses. Commenters in a Hacker News thread on the productivity paradox describe the same gap: drafting gets faster, but reviews, approvals and handoffs do not.

Still open: how long to wait. Some commenters compare AI to early computers, which took years to show up in productivity figures. No source here says when a project should be cut.

Sources

• Hacker News: 95% of generative AI pilots at companies are failing – MIT report
• Hacker News: AI adoption and Solow's productivity paradox
• Survey: Global survey: How leaders are generating value from generative AI
• Vendor docs: Measuring gen AI success: A deep dive into the KPIs you need
• Engineering blog: The KPIs that actually matter for production AI agents