feat: add cost efficiency analytics - #93
Conversation
📝 WalkthroughWalkthroughUsage analytics now tracks cached tokens across storage and aggregation paths. The dashboard exposes cache-hit rate, blended cost, cost per request, route breakdowns, and activity efficiency details. Tests cover cached-token totals and dashboard markers. ChangesCached-token analytics
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant CallsLedger
participant HostStore
participant AuthProxy
participant Dashboard
CallsLedger->>HostStore: aggregate and roll up tokens_cached
HostStore->>AuthProxy: provide cached-token and cost totals
AuthProxy->>Dashboard: render analytics cards and activity details
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
93319bf to
f4ed04e
Compare
![coderabbitai[bot]](https://avatars.github.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (2)
tests/test_host_store.py (1)
59-78: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd coverage for
cost_per_mtok.The fixture validates
tokens_cached,cache_hit_rate, andcost_per_request, but it does not validate the blended cost-per-million metric. For callera, the expected value is66666.6667($3 / 45 * 1_000_000).Proposed test addition
assert agg["totals"]["tokens_cached"] == 15 assert agg["totals"]["cache_hit_rate"] == 0.5 assert agg["totals"]["cost_per_request"] == 1.5 + assert agg["totals"]["cost_per_mtok"] == 66666.6667🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/test_host_store.py` around lines 59 - 78, Extend the existing caller “a” assertions in the analytics aggregation test to validate agg["totals"]["cost_per_mtok"] equals approximately 66666.6667, using the test’s appropriate floating-point comparison for the computed blended metric.tests/test_auth_proxy_dashboard_full.py (1)
1164-1168: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winTest chart behavior, not only HTML markers.
These assertions verify IDs and one CSS token. They do not fail when
renderSeriesreceives incorrectly ordered data or renders bars without the expected cost and cache details. Add a focused assertion for the generated series, or a targeted source assertion if browser execution is unavailable.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/test_auth_proxy_dashboard_full.py` around lines 1164 - 1168, Strengthen the dashboard test around the chart rendering verified by the existing HTML-marker assertions: inspect the generated series passed to renderSeries and assert that its data preserves the expected ordering and includes the required cost and cache details. If browser execution is unavailable, add a targeted assertion against the chart source that validates this series construction rather than only checking element IDs and CSS.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@host_store.py`:
- Line 1082: Update the analytics rollup state handling around
_ANALYTICS_STATE_NAME, rollup_analytics(), analytics_rollup_state(), and
analytics_aggregate() to add a bounded, one-shot migration from existing
"hourly" rows into "hourly-v2-cache". Ensure the migration preserves/backfills
coverage, then mark or remove the legacy state with a safe guard so subsequent
reads cannot repeatedly reconstruct it or loop indefinitely.
---
Nitpick comments:
In `@tests/test_auth_proxy_dashboard_full.py`:
- Around line 1164-1168: Strengthen the dashboard test around the chart
rendering verified by the existing HTML-marker assertions: inspect the generated
series passed to renderSeries and assert that its data preserves the expected
ordering and includes the required cost and cache details. If browser execution
is unavailable, add a targeted assertion against the chart source that validates
this series construction rather than only checking element IDs and CSS.
In `@tests/test_host_store.py`:
- Around line 59-78: Extend the existing caller “a” assertions in the analytics
aggregation test to validate agg["totals"]["cost_per_mtok"] equals approximately
66666.6667, using the test’s appropriate floating-point comparison for the
computed blended metric.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 8bccf7db-695e-4e73-a9d5-5d4140d63534
📒 Files selected for processing (5)
auth_proxy.pyhost_store.pytests/test_auth_proxy_dashboard_full.pytests/test_dashboard_stats_sql.pytests/test_host_store.py
|
|
||
|
|
||
| _ANALYTICS_LOCK_KEY = 0x616E616C_79746963 | ||
| _ANALYTICS_STATE_NAME = "hourly-v2-cache" |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
rg -n -C 5 --glob '*.py' \
"hourly-v2-cache|[\"']hourly[\"']|_ANALYTICS_STATE_NAME|analytics_rollup_state|rollup_analytics|covered_(from|until)"Repository: genlayerlabs/unhardcoded
Length of output: 162
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf 'Tracked files named host_store.py:\n'
git ls-files | rg '(^|/)host_store\.py$' || true
printf '\nSearch relevant identifiers and state names:\n'
rg -n -C 5 "hourly-v2-cache|['\"]hourly['\"]|_ANALYTICS_STATE_NAME|analytics_rollup_state|rollup_analytics|covered_(from|until)" . --glob '*.py' || true
printf '\nFile list around host_store if present:\n'
fd -a 'host_store\.py$' . || trueRepository: genlayerlabs/unhardcoded
Length of output: 9406
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf 'host_store.py relevant rollup/aggregate sections:\n'
sed -n '1080,1205p' host_store.py | cat -n
printf '\nscripts/rollup_analytics.py full:\n'
sed -n '1,70p' scripts/rollup_analytics.py | cat -n
printf '\nTest setup around analytics rollup:\n'
sed -n '1,90p' tests/test_host_store.py | cat -n
printf '\nDiff/stat if available:\n'
git diff --stat HEAD~1..HEAD || true
git diff --unified=80 HEAD~1..HEAD -- host_store.py scripts/rollup_analytics.py tests/test_host_store.py || trueRepository: genlayerlabs/unhardcoded
Length of output: 50380
Add the migration and backfill guard for the rollup state name.
rollup_analytics() now writes "hourly-v2-cache", and analytics_rollup_state() / analytics_aggregate() now read that name. Existing databases will have "hourly" rows, so coverage from those rolls out of the view and becomes unavailable. Add a bounded one-shot migration that copies/populates coverage from "hourly" to "hourly-v2-cache" and delete or ignore the old state with a safe trigger to avoid infinite reconstruction.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@host_store.py` at line 1082, Update the analytics rollup state handling
around _ANALYTICS_STATE_NAME, rollup_analytics(), analytics_rollup_state(), and
analytics_aggregate() to add a bounded, one-shot migration from existing
"hourly" rows into "hourly-v2-cache". Ensure the migration preserves/backfills
coverage, then mark or remove the legacy state with a safe guard so subsequent
reads cannot repeatedly reconstruct it or loop indefinitely.
1 participant
Summary
Rollout
The rollup schema evolves in place with
tokens_cached. A new rollup-state generation triggers the existing bounded backfill automatically (24 hours per five-minute CronJob run), so no manual database migration or full-table query is required.Follow-up telemetry
Historical fallback rate, counterfactual savings, custom-policy fingerprint attribution, and rule-branch activation are not derivable from the current hourly facts. They require persisting bounded fields derived from
decision_traceat ingestion time. This PR does not fabricate those values from the latest-100 memory buffer.Tests
python -m pytest -q— 548 passed, 2 skipped