feat(evidence): symbol-bounded code_evidence chunks — measured negative for the #93 eval - #140
feat(evidence): symbol-bounded code_evidence chunks — measured negative for the #93 eval#1402233admin wants to merge 3 commits into
Conversation
…ve for the #93 eval Every file used to emit exactly one chunk: startLine 1, endLine <file length>, kind "file". A consumer that matched a chunk got line 1 as its pointer no matter which symbol the chunk's containsSymbols had named. This adds one chunk per extracted symbol, bounded by the declaration line through the line before the next declaration (last runs to EOF), so a chunk hit resolves near the thing it names. The whole-file chunk is retained — it is the only pointer available for a file with no extracted symbols (e.g. every .yml), and eval q10 depends on exactly that. Also emitted honestly: each symbol now maps to both the file chunk and its own symbol chunk, and scorecard fallbackChunkRate stops being hardcoded 1.0. Measured result on the issue #93 eval: this does NOT move q03, and moves nothing else either. Coverage is unchanged (A 5/10, B 8/10, A=2 B=8) and Arm A reads 24.7% more bytes per covered question. Full numbers, prediction, and mechanism are in the PR body. Rung (b) is a negative result. Refs #93
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
|
🔒 Repowise is not analyzing this repository The PR bot is free on public repositories. This one is private, which needs a Pro plan. |
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches 💡 1🛠️ Fix failing CI checks 💡
📝 Generate docstrings
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Code Intel change risk
Top signals
revspec: |
`cargo fmt -p code-intel -- --check` is a CI gate and the new test in tests/native_code_evidence.rs was not formatted. Formatting-only change plus the `code-intel repin --write` cascade it triggers (the test file's sha256 is pinned by orchestration/internalization/native-code-evidence.json, which cascades into the c03-r05-r12 measurement digest). Re-verified after: cargo test --workspace --locked green, `run execute` self-scan exit 0, god files 33 -> 33, repin clean, and the eval harness reproduces byte-identical per-question numbers (Arm A 5/10, Arm B 8/10, A=2 B=8) across two runs. Refs #93
…cy facade CI's retirement packet suite failed 5 of 10 checks: editing legacy/run-code-intel.ps1 moves the frozen snapshot identity of every packet whose frozen source set contains it, and E07 additionally requires rollback-rehearsal/run-code-intel.ps1 to be byte-identical to the live facade. `cargo test --workspace` does not cover this — the suite is a separate PowerShell CI job. Regenerated with each packet's ORIGINAL EvaluatedAt (E02/E03/E04/E08 1785441780, E07 1785748660), following the precedent in #134, so re-freezing does not restamp the governance approvals. Verified: suite passes 8 packets + 2 audits, 0 known-blocked. This is the honest cost of rung (b): it cannot be done in Rust alone, because tests/native_code_evidence.rs compares the Rust and legacy producers' chunks.json byte-for-byte, and touching the legacy producer invalidates five in-flight retirement packets. Recorded in the PR body as an argument against the rung, not as routine churn. Refs #93
|
关闭：这是一次成功的负结果，不合并。 实现者动手前写下预测「q03 不会动」，理由是 代价是实打实的： 三个独立验伪 agent 全部 分支保留作为证据，不删。真正的结论记在后续 issue：Arm A 的开销 99.8% 是产物全量扫描，把产物做得更精确只会让它更贵。 |
1 participant
Refs #93
Rung (b): make
code_evidencechunks carry real per-symbol line ranges instead of one whole-file chunk anchored at line 1.Measured outcome: negative. q03 does not move, and nothing else moves either. Arm A pays 24.7% more bytes per covered question for zero coverage. I recommend this is not merged as-is — see "Recommendation" at the bottom.
Everything below was measured on this machine at base
6a6dd20, each configuration run twice.1. Prediction (written before the after-run, verbatim)
2. Result
Every predicted covered-state held. Nothing changed state in either direction.
no_pointer_found=3,pointer_found_window_excludes_golden_span=2golden_file_outside_top_cap=2Per question — Arm A.
stopisstopped_at_kind;coverednever changes:code_evidence.symbolscode_evidence.symbolscode_evidence.symbolscode_evidence.symbolscode_evidence.chunkscode_evidence.symbolscode_evidence.symbolsq03's pointers are byte-for-byte what they were:
sentrux_gate.rs:1203andsentrux_gate.rs:1434, window read 1,632 bytes, reasonpointer_found_window_excludes_golden_span.Why rung (b) cannot move q03
arm_a_answerbreaks out of the kind loop at the first artifact kind with any match. For q03 that kind iscode_evidence.symbols. The chunks artifact is never opened for q03 — confirmed directly by the baseline's ownstopped_at_kindfield, not inferred. Improving chunks is unobservable there.Even removing that stop rule would not help. Arm A resolves a pointer from
entry["startLine"]and readsstartLine ± 10;endLineis never consulted, so a symbol chunk boundingcoupling_limityields pointer 1203 — exactly what the symbol already yielded. And q03's golden span is prose in the module//!header at lines 16–25, which contains no occurrence ofcoupling_limit. Arm A matches identifier text (symbolid, importfile::target, chunkid); there is no identifier-to-prose edge in this evidence model for it to traverse. Reaching q03 requires evidence that links a symbol to the module documentation that explains it, or a retrieval mode that searches text rather than identifiers — a different rung, not this one.3. Surprises
Three, all investigated rather than booked. None of them moved a covered-state.
3a. The
code_evidence.symbolsartifact grew 970 bytes — but symbol emission did not changeThe prediction said symbols would be unchanged (1,013,379). It measured 1,014,349. I did not book it; I ran a decomposition.
The benchmark is self-hosted —
code-intel . --mode litescans the working tree the eval lives in — so my own patch added source lines to the corpus being measured. To separate "format cost" from "I made the repo bigger", I ran the new binary against the base tree (changes stashed, binary unchanged):7520a936…, size 1,013,379 — byte-identical to the baseline. So symbol emission is unchanged; all 970 bytes are the threefndeclarations this patch adds to the scanned repo (symbol_region_chunks,fallback_chunk_rate, and the new test).files.json,imports.json,coverage.json,ranking.json,inventory.files,repository.snapshot— all byte-identical too. Exactly three artifacts changed:chunks,symbol_chunks,scorecard. That is precisely the set this patch touches, with no collateral.3b. Editing the legacy producer invalidated five in-flight retirement packets
CI caught what
cargo test --workspace --lockedcould not: the PowerShell jobwindows-build-test-packagefailed its Compatibility retirement packets step, 5 of 10 checks — E02, E03, E04, E07, E08, allpacket is stale relative to its frozen source set(E08:snapshot drift).Mechanism: those packets freeze a source set that includes
legacy/run-code-intel.ps1, and E07 additionally asserts that itsrollback-rehearsal/run-code-intel.ps1is byte-identical to the live facade. Rung (b) cannot be implemented in Rust alone —tests/native_code_evidence.rscompares the Rust and legacy producers'chunks.jsonbyte-for-byte — so the legacy facade must change, and changing it invalidates those five approvals.Resolved the sanctioned way, following the precedent set in #134: regenerated each packet with its original
EvaluatedAt(E02/E03/E04/E081785441780, E071785748660), so re-freezing does not restamp the governance timestamps. The suite now passes 8 packets + 2 audits, 0 known-blocked.This is a real cost of the rung, not routine churn. A chunk-format experiment that buys zero coverage should not be re-freezing five in-flight retirement approvals. It is an independent argument against merging.
3c. Arm B's q08 byte count moved +15,161 — through the self-hosting loop, not through retrieval
After the packet regeneration, q08's Arm B cost went 2,206,091 → 2,221,252. Arm A on q08 is untouched, and q08's Arm B verdict is unchanged (
golden_file_outside_top_capbefore and after).Mechanism: q08's keywords are
scan,repin, and the five files Arm B ranks highest are the retirementcompatibility-retirement-deletion-diff.jsonfiles — which embed the PowerShell hunks slated for deletion fromrun-code-intel.ps1. My added lines sit inside that region, so four of those five JSON files grew. Same five files read, ~3.8 KB more each.Worth flagging because q08 is one of only two questions Arm A wins, and it wins precisely because Arm B overspends on those packet files. My patch made Arm B's q08 marginally more expensive. That is a self-hosting artifact, not a retrieval result, and I am not counting it as anything.
No other outcome moved in either direction.
4. Byte cost — this buys nothing and costs real bytes
Artifact sizes on the identical corpus (base tree; old binary vs new), so this is pure format cost:
code_evidence.chunkscode_evidence.symbolscode_evidence.importscode_evidence.symbol_chunksroughly doubles (708,397 on the base corpus → 1,559,657 on the final corpus — two mappings per symbol instead of one). Arm A never reads it, so it costs setup bytes only; the two figures are from slightly different corpora and are not a clean before/after pair.Arm A bytes-per-covered-question: 1,155,861 → 1,441,450 (+24.7%) across an unchanged 5 covered questions. The rise comes entirely from q10, the one covered question that reaches the chunks stage; the other four pay only the +970 self-hosting cost.
Against the acceptance criterion (Arm A ≤ ~2× Arm B on the questions A covers), on q04/q06/q08/q10/q11:
The baseline was already over the ~2× bar at 2.26×; this change pushes it to 2.81×. Coverage did not go up, so no score was bought — but bytes were spent, and the criterion is failed harder than before. (Arm B's figure includes the +15,161 self-hosting movement on q08 described in §3c; without it the ratio is 2.82×. Either way, worse.)
5. Reproducibility
Every configuration run twice. Comparison ignores only the top-level
metablock (timestamps, wall-clock, nonce-bearing run directory) and the two published objects that carry the run nonce by design.setup+questions+aggregate)6a6dd20, base binaryrepository.iterationrepository.iterationdifferAll three artifacts Arm A consumes (
symbols,imports,chunks) were byte-identical across repeated runs in both configurations. No nondeterminism had to be fixed.6. Gates
cargo test --workspace --locked— green, 0 failures.cargo fmt -p code-intel -- --check— clean.legacy/scripts/tests/test-retirement-packets.ps1) — 8 packets + 2 audits pass, 0 known-blocked after the re-freeze in §3b.code-intel run execute— exit 0 before and after.code-intel sentrux --operation check— god files 33 → 33, "No degradation detected".One local-only false alarm worth recording: the packet generators leave scratch copies of the 4400-line facade in
legacy/work/, which is gitignored but not in the gate'sSKIP_DIRECTORIES, so the self-scan counted them and reported god files 33 → 36. Reproduced on a pristineHEADtree, so it is pre-existing environmental noise, not this change; deletinglegacy/work/restores 33 → 33. CI checks out fresh and never sees it.code-intel repin --repo .— clean, no stale pins. The pinsrepinis blind to (issue bug(repin): operationTrace 与 evidenceIds 内嵌 digest 不在扫描面——clean 报告失真 #133:operationTrace[].source.sha256) were verified manually: 96 operationTrace digest pins recomputed against the live tree, 0 stale. One of them,orchestration/internalization/rg.json, pinslegacy/run-code-intel.ps1and was caught only by a failing test, not byrepin— consistent with bug(repin): operationTrace 与 evidenceIds 内嵌 digest 不在扫描面——clean 报告失真 #133.Fixtures that legitimately changed
crates/code-intel-cli/tests/native_code_evidence.rs,native_atom_preserves_…:scorecard.metrics.chunks2 → 3. The fixture is 2 files;index.jshas one extracted symbol (greet) andindex.test.jshas none, so 2 file chunks + 1 symbol chunk. Added an assertion thatfallbackChunkRateis now 2/3 rather than the previously hardcoded 1.0.orchestration/schemas/code-evidence-native-artifacts.v1.schema.json: chunkkind{"const": "file"}→{"enum": ["file", "symbol"]}. Widening, consistent with the schema's declaredadditive-onlycompatibility policy.legacy/run-code-intel.ps1mirrors the new chunk emission.a01_a09_artifacts_match_the_real_legacy_producer_on_the_same_fixturecompares both producers'chunks.jsonbyte-for-byte after canonical sorting over 10 legacy runs; it passes, so the two producers agree exactly, including the regiontextHash.orchestration/arerepin --writeoutput plus the two manual fixes above.No question in
eval/questions.jsonwas edited, no coverage or grading criterion was relaxed, and there is no question id, golden path, or keyword list anywhere in the Rust, the PowerShell, or the harness.eval/BASELINE.mdandeval/baseline-*.jsonare deliberately left at their committed state: the numbers above are the deliverable, and the harness names its output afterHEAD, which cannot exist before the commit.7. Recommendation
Do not merge as-is. On the benchmark this is strictly negative: identical coverage, +24.7% bytes per covered question, and the chunks artifact ×3.56. It also costs a re-freeze of five in-flight retirement packets (§3b), which is a disproportionate price for a format change that buys nothing.
The chunk shape itself is genuinely more honest than a whole-file chunk claiming to point at a symbol, so this is worth keeping on a branch as the reference implementation. If it is ever pursued it should land only alongside a consumer that actually uses
endLine(Arm A ignores it entirely today), and the extractors innative_code_evidence.rsshould be split into their own module first: this patch pushes that file from 748 to 838 non-blank lines, over the 800 threshold. It was already one of the 33 god files, sogod_file_countis unchanged and the gate is green, but growing it further is the wrong direction.For issue #93 the actionable finding is elsewhere: Arm A's ceiling on
whyquestions (0/4) is not chunk granularity. All threeno_pointer_foundcases (q05, q07, q09) fail because their keyword is a match-arm string literal, a Rustconst, and a JSON field name respectively — none of which the line heuristic extracts as a symbol at all — and q03/q12 fail because the answer is prose that no identifier-keyed index points to. Those are extractor coverage and a text-bearing evidence kind, not chunk line ranges.