feat(metering): capture prompt-cache reads + accurate cross-provider cost - #18
Merged
Conversation
…cost The router billed every input token at full price and discarded the provider's cache-read metric, so it could neither show cache_hot's savings nor meter spend accurately — the dollar figure was overstated on every cache hit. Verified the gap and the fix end to end (openai/gpt-4o-mini via the router, same session, same 1.3k-token prefix): call 1 cost $0.000203 / cached 0; call 2 cost $0.000107 / cached 1280 — same token count, the prefix billed at the cache-read rate. - llm_router_host `_cached_tokens(usage)`: reads cache-READ tokens across shapes — OpenAI-compat (prompt_tokens_details.cached_tokens), Codex Responses (input_tokens_details.cached_tokens), Anthropic (cache_read_input_tokens). - All three backends (wire / codex / streaming) now return `tokens_cached` and `cost_reported` (the provider's own cost, e.g. OpenRouter `usage.cost`). - `_executed_cost_usd` is now accurate across providers: (1) $0 for subscription (codex); (2) the provider's reported cost when present — authoritative, already net of cache discounts, works for ANY provider that reports it; (3) else computed from the ranked price, billing cache-read tokens at a fraction (_CACHE_READ_FACTOR) so a cache hit is not charged at full input price. - x_router now carries `tokens_cached` (both stream and non-stream), so cache efficiency is observable per call (and aggregable in stats via cost_usd). Tests: tests/test_metering.py (5) — reported-cost preference, subscription $0, cached discount in the computed fallback, negative-price clamp, cached-token shapes. Full affected suite green (test_shim/test_compact/test_host 50/0). No engine change; core untouched.
📝 WalkthroughWalkthroughAdds cross-provider cached-token extraction via a new ChangesCached-token metering and per-session accumulation
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Add route_session_meter: a thread-safe per-session accumulator (calls,
tokens_in/out, tokens_cached, cost_usd) mirroring route_cache. shim folds
each executed call into it and exposes the running total on the response as
x_router.session_acc, plus operator views GET /x/session/{sid} and
/x/sessions. Gives per-call AND accumulated spend/cache data per session_id
across all providers.
![coderabbitai[bot]](https://avatars.github.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@route_session_meter.py`:
- Around line 18-40: The in-process session accumulator in observe() is
unbounded, so every new caller-supplied session key can live forever in _acc and
bloat /x/sessions responses. Update route_session_meter.py to add an eviction
policy around _acc in observe() and any session listing code, such as TTL, LRU,
or a maximum size cap, or replace the dict with a bounded shared store. Keep the
existing session aggregation behavior, but ensure stale or excess sessions are
removed so the table cannot grow without limit.
In `@shim.py`:
- Around line 1275-1284: The session meter update in
`_router_response_to_openai()` is only applied for non-streaming responses, so
streaming SSE completions never call `route_session_meter.observe()` or set
`x_router.session_acc`. Thread `session` through the final-chunk assembly path
used by `_final_chunk_parts()` and ensure the observation runs once when the
stream completes, reusing the same `route_session_meter.observe()` logic and
`session` handling already present in `_router_response_to_openai()`.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 7ebec0f6-1f93-4d3c-9245-ffce1c701b0c
📒 Files selected for processing (6)
codex_backend.pyllm_router_host.pyroute_session_meter.pyshim.pystreaming.pytests/test_metering.py
Comment on lines
+18
to
+40
| _lock = threading.Lock() | ||
| _acc: dict[str, dict] = {} # session -> running totals | ||
|
|
||
|
|
||
| def observe(session: "str | None", *, tokens_in=0, tokens_out=0, | ||
| tokens_cached=0, cost_usd=0.0) -> "dict | None": | ||
| """Fold one call's usage into the session's running total; return the new | ||
| accumulated totals (so the caller can put per-call AND acc on the response). | ||
| No-op (returns None) when the caller named no session.""" | ||
| if not session: | ||
| return None | ||
| with _lock: | ||
| a = _acc.get(session) | ||
| if a is None: | ||
| a = {"calls": 0, "tokens_in": 0, "tokens_out": 0, | ||
| "tokens_cached": 0, "cost_usd": 0.0} | ||
| _acc[session] = a | ||
| a["calls"] += 1 | ||
| a["tokens_in"] += int(tokens_in or 0) | ||
| a["tokens_out"] += int(tokens_out or 0) | ||
| a["tokens_cached"] += int(tokens_cached or 0) | ||
| a["cost_usd"] = round(a["cost_usd"] + float(cost_usd or 0.0), 6) | ||
| return dict(a) |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift
Bound the in-process session table.
Every new caller-supplied session creates a permanent _acc entry, and nothing expires or caps it. A client that rotates session IDs will grow this dict for the life of the process, and /x/sessions will serialize the whole thing. Add TTL/LRU/size bounds (or move this meter to a bounded shared store) before relying on it in production.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@route_session_meter.py` around lines 18 - 40, The in-process session
accumulator in observe() is unbounded, so every new caller-supplied session key
can live forever in _acc and bloat /x/sessions responses. Update
route_session_meter.py to add an eviction policy around _acc in observe() and
any session listing code, such as TTL, LRU, or a maximum size cap, or replace
the dict with a bounded shared store. Keep the existing session aggregation
behavior, but ensure stale or excess sessions are removed so the table cannot
grow without limit.
Comment on lines
+1275
to
+1284
| # Per-session meter: fold this call into the session's running total and put | ||
| # BOTH on the response — per-call (above) and accumulated (session_acc). | ||
| if session: | ||
| acc = route_session_meter.observe( | ||
| session, | ||
| tokens_in=response.get("tokens_in") or 0, | ||
| tokens_out=response.get("tokens_out") or 0, | ||
| tokens_cached=response.get("tokens_cached") or 0, | ||
| cost_usd=out["x_router"]["cost_usd"] or 0.0) | ||
| out["x_router"]["session_acc"] = acc |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Streaming calls never reach the session meter.
route_session_meter.observe() only runs here, but the SSE paths build their final payloads through _final_chunk_parts() instead of _router_response_to_openai(). Any stream=true request therefore skips session_acc entirely and never lands in /x/session/{sid}, so the accumulated totals are systematically low for streaming clients. Thread session through the final-chunk path and record the observation once when the stream completes.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@shim.py` around lines 1275 - 1284, The session meter update in
`_router_response_to_openai()` is only applied for non-streaming responses, so
streaming SSE completions never call `route_session_meter.observe()` or set
`x_router.session_acc`. Thread `session` through the final-chunk assembly path
used by `_final_chunk_parts()` and ensure the observation runs once when the
stream completes, reusing the same `route_session_meter.observe()` logic and
`session` handling already present in `_router_response_to_openai()`.
jmlago
added a commit
that referenced
this pull request
Jun 24, 2026
… consumer /v1/session (#22) * feat(metering): session meter wiring, flow cost+cache aggregation, warm routes, consumer /v1/session Post-#18 follow-on (stranded on the merged cache-metrics branch). Lands: - route_session_meter: /x/session endpoints wiring + per-(session,family) warm map (observe_route/warm) for display. - shim: read session from X-Unhardcoded-Session header; meter the session on the STREAMING paths too; emit standard usage.prompt_tokens_details.cached_tokens so OpenAI-compatible clients (opencode) see cache; /x/session returns warm. - llm_router_host: aggregate cost + cached across flow nodes (the synthetic 'flow' chosen has no price); record warm route per node on success. - auth_proxy: consumer-facing GET /v1/session/{sid} (consumer-key authed) so a harness reads cost/tokens/cache/warm without operator /x/* access. Verified live: flow cost_usd 0.034 + cached 2304; GET /v1/session/{sid} returns warm [gpt-5.5/openrouter, z-ai/glm-5.2/openrouter_market]. * fix(metering): scope per-session view to the owning consumer (sid->owner) GET /v1/session/{sid} authed only that the caller was *a* valid consumer, never that the sid belonged to *that* consumer. route_session_meter is keyed by sid alone, so any authed consumer could read any other consumer's session economics (cost/tokens/cache) and `warm[]` — which discloses the real family/provider/served_by peers serving someone else's conversation. Bind sid -> owning consumer key and enforce it: - route_session_meter: add _owner (sid->consumer key), guarded by _lock. observe() gains owner=; records first-writer-wins (setdefault) so a consumer reusing another's opaque sid cannot steal/overwrite ownership. add owner(); reset() clears it. - shim: thread the authed consumer (the ingress proxy's x-llm-router-caller header, captured onto ChatRequest.caller) into BOTH observe sites (streaming _final_chunk_parts and non-streaming _router_response_to_openai) as owner=. - shim /x/session/{sid}: when a caller header is present (the consumer-facing path), only the owner may read; anyone else gets 404 — NOT 403, so the endpoint never confirms another consumer's sid exists. Operator /x/* (no header) stays unscoped. - auth_proxy session_view: forward the authed caller as x-llm-router-caller so the upstream meter scopes the read. Tests: meter-level first-writer-wins binding (test_metering) and an end-to-end endpoint test (test_shim): owner reads 200, other consumer/unknown sid 404, operator unscoped.
jmlago
added a commit
that referenced
this pull request
Jun 25, 2026
…#23) Bumps the embedded Σ_pol engine submodule from 3d49132 (engine #18) to ee31e81, which includes unhardcoded-engine#20: price_in/price_out field observation now reads the candidate's own offer price before the provider-family EMA fallback. Before this the engine read ctx.state.ema[provider|family].price first, so multiple AntSeed seller offers sharing one provider/model-family collapsed to a single family price (seeded here via update_metrics in sources/__init__.py) and a price-ranking policy could not pick the cheapest seller. The host already forwards per-offer prices through the discover hook, so this pointer-only bump unlocks per-offer price selection with no host logic change. Engine suite green at ee31e81 (564/0). Host suite not run locally (no python3 here); CI validates.
jmlago
added a commit
that referenced
this pull request
Jun 26, 2026
…26) Reconstructed from Edgars Nemše's #24, carrying only the family half. The provider-adapter half is in its sibling PR; this one makes discovered OpenRouter marketplace families provider-neutral. What it does: - sources/openrouter.py: a discovered marketplace model's policy-facing family is the provider-neutral name (`openai/gpt-5-mini` -> `gpt-5-mini`), while `wire_model_id` keeps the exact OpenRouter slug for the wire. `service_aliases` (config) handle the canonicalization exceptions where stripping the vendor isn't the right family (dated/suffixed slugs). Curated families served by the static `openrouter` provider are deduped out, so a marketplace row never shadows a curated family. - This pairs with the engine's `provider_eq` (#18): family is the model, provider is a separate axis the algebra filters. It lets a single provider-agnostic policy span curated + marketplace routes for one model, which is the point of the router ("stop hardcoding models"). Core bump: `core` -> 97d0333 (unhardcoded-engine #22), which makes `served_model_id` the offer's wire id (`offer.wire_model_id or model_family`). With a neutral family + a distinct wire slug, the engine now wires the slug on both the curated and discovered paths without every adapter special-casing `wire_model_id`, and the replayable trace records the real wire id. The §3 determinism boundary holds: `model_meta.lua` stays curated-keyed (refresh_model_meta untouched), so an on-chain/genvm host lacking a discovered offer still fails closed. Pinned by test_openrouter_model_meta_still_keyed_by _curated_family and the dedup assertion in test_openrouter_discovery_derives_policy_families_from_raw_model_ids. Verification: nix-shell --run 'python -m pytest tests -q' -> 333 passed, 2 skipped, 0 failed, against the bumped core. No new dependency.
This was referenced Jun 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The router billed every input token at full price and discarded the provider's
cache-read metric, so it could neither show cache_hot's savings nor meter spend
accurately — the dollar figure was overstated on every cache hit. Verified the
gap and the fix end to end (openai/gpt-4o-mini via the router, same session,
same 1.3k-token prefix): call 1 cost $0.000203 / cached 0; call 2 cost $0.000107
/ cached 1280 — same token count, the prefix billed at the cache-read rate.
_cached_tokens(usage): reads cache-READ tokens across shapes— OpenAI-compat (prompt_tokens_details.cached_tokens), Codex Responses
(input_tokens_details.cached_tokens), Anthropic (cache_read_input_tokens).
tokens_cachedandcost_reported(the provider's own cost, e.g. OpenRouterusage.cost)._executed_cost_usdis now accurate across providers: (1) $0 for subscription(codex); (2) the provider's reported cost when present — authoritative, already
net of cache discounts, works for ANY provider that reports it; (3) else
computed from the ranked price, billing cache-read tokens at a fraction
(_CACHE_READ_FACTOR) so a cache hit is not charged at full input price.
tokens_cached(both stream and non-stream), so cacheefficiency is observable per call (and aggregable in stats via cost_usd).
Tests: tests/test_metering.py (5) — reported-cost preference, subscription $0,
cached discount in the computed fallback, negative-price clamp, cached-token
shapes. Full affected suite green (test_shim/test_compact/test_host 50/0).
No engine change; core untouched.
Summary by CodeRabbit
New Features
Bug Fixes