How do you measure whether AI tools are worth what you pay?

Summary of public sourcesWritten by our research agent from 6 public sources. Not first-hand experience.

Most claims of value are self-reported. In the 2025 Stack Overflow survey, 52% of developers said AI tools had a positive effect on their productivity. The top frustration, at 66%, was answers that are almost right but not quite. Practitioners split the same way. Some see large gains on boilerplate and new projects. Others say checking the output eats the time saved.

Companies that measured used several signals. Bitly surveyed engineers before and after its program and treated its objective numbers as directional. Docusign tracked adoption and productivity and found gains stalled when people only had access, then grew with peer support. GitHub's metrics cover adoption, acceptance rates and pull request flow, and its docs say return-on-investment figures are estimates. Practitioners suggest one concrete number per task, or cost per shipped feature. Another test they mention is asking what would break if the tool vanished tomorrow.

Still open: how to count quality and review costs. Headcount cuts are the most visible metric, and people doubt they prove anything.

Sources

• Hacker News: Ask HN: Using AI daily but not seeing productivity gains – is it just me?
• Hacker News: Ask HN: Are employers getting the returns from AI?
• Survey: AI | 2025 Stack Overflow Developer Survey
• Vendor docs: GitHub Copilot usage metrics
• Engineering blog: How We Rolled Out AI Coding Tools at Bitly: Six Months and Counting
• Engineering blog: How We Drove AI Adoption Across Docusign Engineering Without Mandating It