Skip to content

fix: attribute error/aux rows to the requested model, not 'unknown' - #82

Merged
acastellana merged 1 commit into
mainfrom
fix/error-model-attribution
Jul 16, 2026
Merged

fix: attribute error/aux rows to the requested model, not 'unknown'#82
acastellana merged 1 commit into
mainfrom
fix/error-model-attribution

Conversation

@acastellana

@acastellana acastellana commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

From the 2026-07-15 dashboard review: the "By model family" panel showed 857 requests / 20% err in a single opaque unknown bucket. Cause: failed and non-chat requests carry no served model_family (it comes from the upstream x_router, absent on error), so they all collapsed to unknown with no idea which model they targeted. requested_model was captured (auth_proxy.py:3337) but never used as a fallback.

Fix

Fall back to the requested model when the served family is absent, in all three aggregation paths — kept in lockstep (the SQL-vs-Python parity test enforces it):

  • host_store.usage_aggregate family_k — persistent all-time view
  • auth_proxy._stats live family buckets — recent feed
  • auth_proxy._aggregate_usage_rows — historical replay fold

Served family stays the primary key for 2xx rows; the stored model_family column is untouched (still null on error) — only the display bucketing changes. policy_ir/flow_ir routed requests now show under their route label instead of unknown too.

Tests

New test_error_rows_attribute_to_requested_model_not_unknown pins it; full suite 501 passed, 2 skipped.

Deploy

Ships with the pending router deploy (main 6f80677, devops).

Summary by CodeRabbit

  • Bug Fixes
    • Improved dashboard usage reporting for error and auxiliary events without model-family data.
    • These events are now attributed to the requested model or route instead of being grouped under “unknown.”
    • Runtime and historical usage views now provide more consistent model-family totals.

Failed and non-chat requests carry no served model_family (it comes
from the upstream x_router, absent on error), so every one fell into a
single opaque 'unknown' family bucket — 857 requests / 20% err on the
prod dashboard with no idea which model they targeted.

Fall back to the REQUESTED model in all three aggregation paths, kept
in lockstep (the SQL-vs-Python parity test enforces it):
- host_store.usage_aggregate family_k (persistent all-time view)
- auth_proxy._stats live _stats family buckets (recent feed)
- auth_proxy._aggregate_usage_rows (historical replay fold)

Served family stays the primary key for 2xx rows; the stored
model_family column is untouched (still null on error). New metering
test pins the fallback; full suite 501 passed.
@coderabbitai

coderabbitai Bot commented Jul 16, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 91dbb8ba-4488-4c2a-9e9b-c6cbadde9044

📥 Commits

Reviewing files that changed from the base of the PR and between 6f80677 and f0d1935.

📒 Files selected for processing (3)
  • auth_proxy.py
  • host_store.py
  • tests/test_metering.py

📝 Walkthrough

Walkthrough

Usage aggregation now attributes error and auxiliary rows without model_family to their requested model or route across runtime counters, history-shaped aggregation, and SQL results. A regression test verifies the requested-model bucket and excludes an unknown bucket.

Changes

Usage attribution

Layer / File(s) Summary
Runtime family bucketing
auth_proxy.py
_record_request derives a fallback family key and applies it to global, caller-scoped, and key-scoped model-family counters.
Persistent aggregation and regression coverage
auth_proxy.py, host_store.py, tests/test_metering.py
History-shaped and SQL aggregation fall back to requested_model or route; the regression test verifies error-row attribution to gpt-5.5 without an unknown bucket.

Estimated code review effort: 2 (Simple) | ~10 minutes

Suggested reviewers: jmlago, muncleuscles

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly matches the main change: attributing error/aux rows to the requested model instead of "unknown".
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/error-model-attribution

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f0d1935977

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread host_store.py
# x_router, absent on failure) — fall back to the model the caller ASKED
# for so failures attribute to a real model instead of piling into a single
# opaque "unknown" bucket. Served family stays the primary key for 2xx rows.
" COALESCE(NULLIF(model_family,''),NULLIF(requested_model,''),'unknown') AS family_k,"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Use the fallback key when filtering model buckets

When a window has error/aux rows with model_family NULL and requested_model='gpt-5.5', this new family_k puts gpt-5.5 into filter_options.models; selecting that model sends model=gpt-5.5 back to /dashboard/api/stats, but host_store._usage_where(..., model_family=model) still filters on raw model_family = %s. Those same fallback-attributed rows are then excluded from totals/recent, so drilling into the new bucket shows too few requests or a lower error rate. Please make the model filter use the same fallback expression, or avoid exposing fallback-only keys as filter options.

Useful? React with 👍 / 👎.

@acastellana
acastellana merged commit 7f6bb75 into main Jul 16, 2026
1 check passed
@acastellana
acastellana deleted the fix/error-model-attribution branch July 16, 2026 07:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant