feat(reviewer-eval): mine-telemetry — fail→fix golds + waived decoys from the collector db - #303
Merged
Conversation
…from the collector db Capture point 1 of the corpus-growth charter, plus the historical override-valve decoy slice. Read-side design: the miner SELECTs ~/.claude-usage/usage.db (USAGE_DB override) and reads the #295 diff archive — the collector itself is untouched (deviation from the addendum's 'collector correlates' phrasing, recorded in the module docstring; read-side correlation adds zero write paths). - fail→fix: per-(repo,branch) chronological chains (ISO-aware ordering, ship-id tiebreak); PER-LENS correlation when lens data exists, reviewer-level fallback only on lens-data absence; same-diff overrides route to the decoy path; fixed-by-absence recognized on clean ships. Candidate lenses are an ALLOWLIST on disposition ('blocking' or pre-disposition null). Merge keys carry the candidate kind. pickFailReason falls through to the reviewer reason when issues_json filters to nothing (whitespace-only arrays shadowed it via ??). - Evidence honesty: failReason-only rows flagged + counted; repo allowlist defaults to devkit (frink diffs are private — explicit flag + adapt stage). - Shared miner plumbing extracted to mine-common.mts (jscpd: zero clones). - Atomic merge-by-key output to raw/candidates-telemetry.jsonl (gitignored); fail-open on missing db/sqlite3/archive; bypasses propose.mts by design. Live dry-run (devkit scope): 44 candidates incl. 9 with archived bytes; idempotent re-runs. 48 unit tests; 490/490 suite green; tsc/biome clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
|
Warning Review limit reached
Next review available in: 13 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (4)
📝 WalkthroughWalkthroughAdded shared miner utilities, a telemetry correlation library, and a bench-only telemetry mining CLI. The CLI correlates review outcomes with later fixes, builds candidate rows, merges output atomically, and reports mining statistics. Vitest coverage validates the helper behavior. ChangesTelemetry mining
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant MineTelemetry
participant SQLite
participant DiffArchive
participant CandidatesFile
MineTelemetry->>SQLite: Read telemetry rows
MineTelemetry->>DiffArchive: Load archived diffs
MineTelemetry->>MineTelemetry: Build fail-fix and waived candidates
MineTelemetry->>CandidatesFile: Merge and atomically write candidates
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.github.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@gate-engine/review/eval/reviewers/mine-common.mts`:
- Around line 32-35: Update collectRepoArgs to reject --repo when its next
argument is missing or starts with "--", raising the established usage error
instead of adding it to repoArgs. Preserve accepting non-flag repository values
and ensure mine-bots does not interpret an incomplete --repo pair as an explicit
scope.
- Around line 51-58: Update sqliteJson to enforce read-only database access for
every caller-supplied SQL statement, preferably by opening sqlite3 in enforced
read-only mode or rejecting non-SELECT statements before execution. Preserve
existing JSON result behavior and add a test verifying that an INSERT, UPDATE,
or DDL operation is refused.
In `@gate-engine/review/eval/reviewers/mine-telemetry.mts`:
- Around line 262-272: Update the candidate-row fallback around failLenses and
lensesByShipReviewer so [null] is used only when no lens breakdown exists for
the ship/reviewer; when breakdown entries exist but none are blocking failed
lenses, return no candidate rows instead of creating a reviewer-level candidate.
Preserve the existing blocking-fail filtering and per-lens rows.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 13303e51-d2cf-4f1d-84f1-35c467ab060d
📒 Files selected for processing (5)
gate-engine/review/__tests__/mine-telemetry.test.mtsgate-engine/review/eval/reviewers/mine-bots.mtsgate-engine/review/eval/reviewers/mine-common.mtsgate-engine/review/eval/reviewers/mine-telemetry-lib.mtsgate-engine/review/eval/reviewers/mine-telemetry.mts
norvalbv
added a commit
that referenced
this pull request
Aug 2, 2026
…d module-suffixed extensions (#305) The post-filter kept only .ts/.tsx/.js/.jsx, but devkit's own scanRoots (cli, gate-engine) are pure .mts — jscpd tokenizes mts/cts as typescript and mjs/cjs as javascript and reported the clones, then CODE_EXT discarded every one. `guard-clone scan` returned 0 repo-wide while raw jscpd found real clones (verified during #303's gates): a silently dead gate. With the fix the same scan reports 62 cross-file clones (min-tokens 50), dominated by the review/eval vs review/eval/conventions bench+matcher family. Regression tests: .mts and .mjs clone fixtures must block (exit 1). Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Three findings, all valid. - selectFailLensRows (new, in the lib so it is testable): an empty blocking filter used to fall through to `[null]`, the "no lens breakdown recorded" branch. A fail whose only failing lenses were waived or dropped_out_of_charter therefore minted a reviewer-level gold from exactly the lenses the allowlist had just excluded — contradicting the rule the comment above it states. Now three distinct cases, and the skip is COUNTED in the drop histogram (fail-fix:all-failing-lenses-non-blocking) rather than vanishing silently, which is the same evidence-honesty bar as the rest of the funnel. - collectRepoArgs: `--repo --dev` stored "--dev" as the repository. Both callers treat any non-empty result as an explicit scope replacing their defaults, so that silently narrowed the sweep to a repo that cannot exist and reported a clean run. A missing or flag-shaped value is now a usage error. - sqliteJson: the docstring claimed read-only but nothing enforced it. Now opens the collector db with sqlite3 `-readonly`, so the boundary is held by the engine — a write raises "attempt to write a readonly database" instead of mutating the user's telemetry. The miners are strictly read-side. Tests: 47 -> 61 in mine-telemetry.test.mts, covering all three cases of the lens rule, both malformed --repo shapes, and a refused INSERT/DDL that leaves the db intact. Full gate-engine/review suite 503/503. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
norvalbv
added a commit
that referenced
this pull request
Aug 2, 2026
…clone-gate ruling (#310) benchmarks-grow-from-telemetry gains its convergence record before the release that ships it: capture loop closed end-to-end (#295/#302/#303/#309, first 8 pure-telemetry rows, corpus 128), label-trust precondition met (#304: κ 0.735 post-triage, 4.2% noise floor; cleanlab floor still pending bench pred_probs), and the Target's c-CRAB/CR-Bench known-answer path recorded as falsified (#307) with the replacement candidates awaiting ratification. New axis clone-gate-non-import-code ([VALIDATED]): clones are measured over non-import code, excluded at the jscpd tokenizer — with the six-hole failure of post-hoc fragment classification recorded as the rejected road so a future simplifier can't silently re-vacuous the gate (#305/#308). Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Capture point 1 of the corpus-growth charter, plus the historical override-valve decoy slice — completing the capture trio (#295 archive, #302 waive, this). The loop now mints labeled candidates entirely from your own telemetry, no CodeRabbit involved.
Read-side design: SELECTs
~/.claude-usage/usage.db+ reads the #295 diff archive; the collector is untouched (deviation from the addendum's 'collector correlates' phrasing, recorded in the docstring — read-side correlation adds zero write paths).dropped_out_of_charterlenses can never become golds.--repo+ adapt-stage decision).mine-common.mts(jscpd: zero clones between miners now).Live dry-run (devkit scope): 44 candidates — ~40 correctness fail→fix (9 with full archived diff bytes), commit-guard pair, 1 waived decoy; idempotent re-runs.
Gate provenance: five distinct findings absorbed across four review rounds (per-lens stamping blocker, phantom
'overridden'disposition literal, 3 verbatim clones, whitespace-array reason shadow) — all fixed, none waived. One follow-up the opus escalation surfaced: the clone gate is currently vacuous (clone-detector.mjs scanreports 0 repo-wide while raw jscpd finds real clones) — filed for separate investigation.48 unit tests; 490/490 suite green; tsc/biome clean.
🤖 Generated with Claude Code
Summary by CodeRabbit