Your AI Bill Counts Retries You Never Saw. Log Usage at a Proxy, Not in the App.

Most teams track AI spend in one of two places: the provider dashboard or a counter inside their own code. Both lie a little, and the gap between them is where money disappears.

This week a small tool called Inferock-bench showed up. It's a local proxy that sits between your app and OpenAI, Anthropic, Gemini or any OpenRouter-compatible endpoint, and records token usage and failures for every call, retries included. The point is to line up what you were billed against what your product actually did.

The tool is new and we haven't run it in production, so treat this as a take on the idea rather than a review. But the idea is right.

Where the invisible tokens come from

Your application code logs one request. The SDK underneath it may have made three.

• SDK retries. Most official clients retry on 429s and 5xx errors by default. Your code sees one successful call. The provider may have counted a failed attempt that still burned tokens.
• Timeouts that didn't cancel. You gave up after 30 seconds. The model kept generating. You still paid for the output.
• Agent loops. A tool call fails, the agent tries again with a slightly different prompt, and each pass resends the whole context.
• Streaming that got dropped. The client disconnected halfway. The server finished anyway.

None of that shows up if your counter lives in the handler that wraps the happy path.

Why a proxy beats an in-app counter

An in-app counter only knows what your code chose to record. A proxy sees every byte that leaves the machine, whichever library or agent made the call. That matters more now that a typical project mixes your own code, a coding agent, an SDK someone pulled in, and a framework that makes its own calls.

It also gives you one place to answer boring but expensive questions. Which feature has the highest retry rate? Which model returns the most errors at 9am? How much of last month's bill was failed work?

Run it locally first. Point your dev environment at the proxy for a week and compare the totals against the provider invoice. If the numbers match, great, you know your accounting is sound. If they're off by 10 or 20 percent, you've found a budget line nobody was watching.

What we'd do with the numbers

1. Cap retries explicitly. Set max retries yourself instead of inheriting a default you've never read.
2. Pass cancellation all the way down so a timeout actually stops the generation.
3. Track cost per completed task, not cost per request. A failed request that gets retried is one task with two costs.
4. Alert on retry rate, not just error rate. A rising retry rate is the early warning before a bill spike.

None of this is glamorous. It's the same discipline we apply to database queries and third party APIs, just applied to a line item that's grown fast enough to deserve it.

If you can't reconcile your AI usage with your invoice today, that's the first thing to fix, before you optimise prompts or switch models.

We're here to help founders and teams design and build digital products that are built to scale with you, not slow you down. If you're looking to build something, get in contact with us today!