Stop Wasting Tokens: How to Measure AI Usage Across Your Engineering Team

Why Leaders Lack AI Visibility

Most engineering leaders can't answer the basic question: is AI working here? Engineers run Claude Code all day, but leadership has no view into how fast they are, what it costs, or whether the right model is being used for the right task. Ash's framing: an AI-first organization has two bottlenecks — the product level (tickets, specs, plans) and the quality level (code review, tests, CI/CD keeping AI slop out of production) — and measuring impact means maximizing both.

Three Lenses: Business ROI, Productivity, Cost

Every metric worth tracking ladders up to one of three lenses. Business ROI is real movement on the business: backlog pushed forward, revenue cycles, support tickets resolved faster. Productivity is people doing their day-to-day work faster with AI — imagine cloning your best principal engineer, with the same evals and harness, across the whole company. Cost is using models well: not defaulting to the most expensive option for every request.

Three Ways to Deploy AI Engineers

The companies getting results place AI engineers in one of three structures:

1. Center of excellence – a closed-off "Delta team" reporting to the CTO that experiments, ships internal AI solutions, and proliferates the ones that work. It also sidesteps the quiet AI naysayers who accept the tools and never use them.
2. Embedded engineers – engineers placed inside sales, marketing, or support who own that team's AI output, build its agents, and teach its people.
3. Enablement leaders – teachers who run training sessions and spread working methods, because handing out tools without a method doesn't change output.

Measure each with the VIC framework: Volume (how much input — deployments, sessions, embedded engineers), Impact (did that volume move productivity or revenue), and Compounding (does it keep growing without intervention — do teams still perform after the embedded engineer leaves?).

The Four ROI Metrics

For the business-ROI lens, four numbers matter: revenue growth from deployed AI projects, cost avoidance (the same headcount doing 3–4x the work), cycle time from idea to completion trending down, and quality — evals and observability on everything you ship, so "we're doing AI" never substitutes for knowing it works.

Trace Everything

AI is a black box unless you trace it. Behind every polished answer are prompts, tool calls, retries, infinite loops, and token burn. Most coding agents expose hooks; the demo wires Claude Code to Langfuse with a single plugin install plus API keys in settings.json, after which every session and turn — role, context, output, tokens, dollars — lands in a dashboard, per user and per repository. The same platform works team-wide: spot who needs prompting help, who is burning tokens, and whether great output is secretly taking seven retries. LangSmith, Braintrust, Datadog, or your own Postgres database all work too.

From Observability to Evals

Once work is traced, score it the way you'd score a person: rubrics, deterministic checks, and non-deterministic checks on merge rate, iterations, prompt quality, token efficiency, and rework. Port merge events back into the observability platform and the full lifecycle — ticket → specs → agent runs → review → merge — becomes measurable. Companies use this to find their natural enablement leaders, target coaching, and build leaderboards. The sequence matters: observability leads to evals, and evals lead to improvement.

Token Economics and Model Routing

Every provider is incentivized to serve the expensive default, but for 80–85% of tasks a cheaper model does the same work. Gauntlet's in-house answer is a proxy layer (AI Control): a local control plane that fronts the coding agent, classifies each request, and routes it across nano / cheap / standard / strong / frontier tiers via OpenRouter — including self-hosted open-weight models (Qwen and Kimi at Gauntlet) for everyday tasks. The metric to watch is cost per successful outcome — cost per merged task, cost per resolved ticket — trending down as routing improves. For IP or privacy concerns, self-host the observability stack; the hooks work the same.

The Scorecard

Put together, the scorecard is: dollar impact (deployed projects moving revenue indicators), productivity (merged work per token, verified by evals), and cost (per successful outcome, driven down by routing). Start with observability this week — even just tracking how your own coding-agent sessions turn into merged work — then add evals, then routing.