For a year I have been paying for AI work I could not see.
Not the invoice. The invoice is easy. I mean the attribution. Which model did which piece of my work, and what that piece actually cost. That number was invisible, and you cannot manage what you cannot see.
On July 16 I built the meter. It is called modelmix, mix on the command line, and it shows me, live in my terminal, which model is doing my work and what it costs while it happens.
I designed it, built it, tested it, and switched it on in one day, orchestrated start to finish by Claude inside Claude Code. Sixty-four tests. Real calls verified against all five provider paths. By the end of the day it was measuring its own construction.
Here is the line that made it worth writing about.
[Fable 70% | 578,879 tok, $185.16] [opus 30% | 245,755 tok, $50.20]
That sits in my Claude Code statusline and repaints as work moves through it. That is the actual line from the session that built the tool, captured while I wrote this. I switched between two Claude models mid-session, and the meter shows both, because each segment is priced at its own model's rate. The dollar figures are not what I paid. They are what those exact tokens would cost at published API rates, cache reads and cache writes included. All of it was billed to me at zero, because it ran through my Max plan. As work gets delegated to other providers, more segments appear, each with its share, its token count, and its cost conversion. My total real spend across every routed call on build day: about one cent.
The name comes from the output. A routed job might read Fable 5% | gem 40% | qwen 55%, a single request split across three providers by capability and cost. It mixes. So, mix.
Every provider called the cheapest way it can be reached
The whole design rests on one rule: never pay full API price for a call you can make for free.
Claude runs through claude -p on the Max plan, which costs zero incremental dollars. GPT runs through the codex CLI on the ChatGPT plan, also zero. Gemini and Grok go over REST, because that is the only way to reach them. Ollama runs on localhost against a daemon that stays warm between calls, free and local. Five providers, four billing surfaces, one invocation. User state lives in ~/.modelmix/, deliberately separate from the code, so I can wipe the running state without ever touching the source.
Three columns, and an honest word for "saved"
Every routed call lands in a per-day ledger, one JSON line each, with three cost columns. What I actually paid. What the same tokens would cost at API rates. What they would cost at an all-Claude baseline. "Saved" is the gap between those, and the tool reports it as an estimate, because that is what it is. I did not want a number that flatters me. I wanted a number I could defend in front of a CFO.
The statusline reads only the new bytes of the transcript on each tick, so orchestrator usage gets attributed without being double counted.
Then I ran it on the session that built it. Over five hundred messages by the end of the day, split across two Claude models. About 235 dollars of work at API-equivalent rates, billed to me at zero. The tool's first real output was a precise measurement of the invisible labor that had just produced the tool.
The meter also audited itself. When I asked whether the cost numbers were actually correct, the check against authoritative pricing found three real errors in the first draft: stale per-token rates inherited from an old table, prompt-cache writes that were not being billed at all, and a session's entire total being priced at whichever model spoke last instead of each model at its own rate. Cache writes alone were about 40 dollars of that session. A meter you have not audited is just a number generator with confidence.
The router picks capability first, then punishes cost
Routing is not "use the cheapest model." It is "use the most capable model that fits, then make paid API dollars hurt."
For each task the router takes the highest-capability model that clears the context window, the latency class, and a quality floor. Then it applies a penalty: two quality points subtracted per cent of real API cost, plus a charge for burning Max plan quota. The effect is that free and local models absorb the bulk work and the background jobs, Gemini wins anything that needs its million-token window, and Claude Sonnet on the Max plan sits at the bottom as the terminal fallback, so the router degrades but never hard-fails. There is always a next model to fall to.
The failures are the part worth telling
Anyone can demo the happy path. The reason I trust this thing is the four times it broke and told me the truth.
The local Ollama model ran out of memory on a RAM-starved machine. The router did not crash. It fell through to a cloud model cleanly, exactly as designed.
xAI had silently dropped a model ID that the pricing tables still listed. The tool was quoting a cost for a model that no longer existed. Verifying against the live API, not the docs, is the only reason it got caught.
A 114 megabyte compiled binary lost a startup race to an 11 kilobyte node bundle. The small thing won. So I shipped the small thing.
And a benchmark had to be rebuilt with paired sampling, because on Windows, Defender owns the p95 tail of every process spawn. The slow outliers were not the code. They were the antivirus, and the only way to measure past it was to sample both paths against the same interference.
None of those are in a slide deck. All of them are why the meter reads true.
The switch, and the loop that caught its own bug
There is an on and off switch. mix on, mix off. When it is off, nothing routes and nothing gets measured, which matters, because a measurement tool you cannot disable is a liability, not an asset.
And the delegation loop got dogfooded on itself. I had the router summarize its own design documents through a delegated model. Even the cleanest summary of how this system works was drafted by a delegated model routed through the system it describes. The summary was fine. The act of running it surfaced a real bug in the skill file that drives the delegation. The tool found a defect in the tool by being used as the tool.
This is what agentic engineering actually looks like
I want to be precise about what happened here, because the word "agentic" gets thrown at everything now.
Claude did not just write code from a spec. Over one day it explored the machine to learn what was actually installed and reachable, found reusable patterns already sitting in tools I had built and pulled them forward, called five live provider APIs and checked its own assumptions against what came back rather than against documentation, corrected its own overclaims when the numbers did not hold, and filed the learnings so the next session starts ahead of this one.
That is the shape of the work. Not a machine that types faster. A machine that investigates, verifies, is wrong out loud, and fixes it.
For anyone running an AI-heavy operation, that is the whole point, and the meter is the management surface. You cannot reason about a cost you cannot see, route around a dependency you have not measured, or trust a system that never tells you when it fell back. I built modelmix so I could see all three. The model you pick matters less than the fact that you can watch it work.