How do you set token budgets for internal AI apps?

Summary of public sourcesWritten by our research agent from 5 public sources. Not first-hand experience.

Cost does not grow in a straight line. Practitioners say retries, tool fanout and growing context stack up, and long-tail requests can dominate the bill. One found cache misses cost far more than expected once they tracked daily usage. OWASP lists runaway usage as a security risk too, under "denial of wallet," where attackers exploit pay-per-use pricing.

The approaches overlap across sources. Measure tokens, retries and context per run during a pilot, then set limits. Budget per session, not per request, and decide in advance what happens at the limit: a smaller model, a shorter answer or a hard stop. OWASP recommends rate limits, user quotas, input size limits and monitoring. Gateway docs such as LiteLLM's show budgets by team, user, key and model, with reset periods and token or request rate limits. Anthropic's Claude Code docs give suggested token rate limits by team size and workspace spend limits.

Still open: a fixed, predictable cost without running your own hardware.

Sources

• Hacker News: Ask HN: How do you budget for token based AI APIs?
• Hacker News: Ask HN: What's your biggest LLM cost multiplier?
• Vendor docs: Budgets, Rate Limits | liteLLM
• Vendor docs: Manage costs effectively | Claude Code Docs
• Standard: LLM10:2025 Unbounded Consumption