Bound workspace analysis, AST aggregation, and large-file materialization

[acoliver]

Parent: #3202

Problem

Workspace tools have finite inputs, but several materialize very large files or aggregate unbounded record counts before applying output limits:

• structural-analysis dependencies/references/exports modes scan all files, accumulate all records, and duplicate the aggregate during JSON formatting; their truncated status is not accurate;
• AST grep accumulates all matches before applying maxResults;
• AST read-file reads and parses an explicitly selected file without the regular read_file pre-read size gate;
• edit/patch/insert/delete paths can read complete large target files and create additional copies for diff/backup.

This is a workspace-finite aggregation/file-boundary problem, not a streaming subprocess problem.

Architecture

Reuse only the shared validated budget and truncation metadata contracts established by #3200. Do not force record aggregation or whole-file semantics through the head/tail stream collector.

Extend the existing packages/tools file-size gate used by read_file/read_line_range instead of creating a second gate. Keep record-count and traversal-stop policies in structural/AST tool modules.

Acceptance criteria

• structural-analysis modes enforce file/record/result budgets while traversing and report accurate partial-result metadata;
• AST grep stops accumulating when maxResults is satisfied rather than collecting everything and slicing afterward;
• aggregate serialization avoids unnecessary full-result duplication;
• AST read-file and write/edit/patch helpers reuse a shared pre-read stat/size policy before parsing or copying oversized files;
• regular read_file, read_line_range, and read_many_files behavior remains compatible and their existing acquisition bounds are preserved;
• partial workspace analyses are clearly identified as partial.

Behavioral tests

Use Bun and bun:test with real fixture trees/files:

• structural dependencies/references/exports beyond record and file budgets;
• AST grep with far more matches than requested;
• exact-limit and one-byte-over files for AST read and modification paths;
• abort during a large traversal;
• aggregate output that proves bounded collection without mock-only call-count assertions.

Non-goals

• No AST parser/language redesign.
• No generic streaming result protocol.
• Do not modify the shared head/tail collector for record aggregation.
• No separate file-size utility when the existing gate can be extended.

[coderabbitai]

🔗 Related PRs

#3077 - Keep a failed tool call failed on the way out (Fixes #3063) [merged]
#3103 - Run every test by discovery, not by an allowlist (Fixes #2979) [merged]
#3127 - Release per-turn stream state and prove the memory plateau (Fixes #3114) [merged]
#3183 - fix(typecheck): prevent test errors from bypassing CI (Fixes #3161) [merged]
#3201 - Bound recorded sessions with a safe global janitor (Fixes #3164) [open]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[github-actions]

Plan for #3205: Bound workspace analysis, AST aggregation, and large-file materialization

Status: Draft
Issue: #3205
Parent Epic: #3202
Size: Large
Sizing basis: 7 acceptance criteria spanning multiple tool subsystems within packages/tools; introduces shared bounded-traversal/truncation metadata, extends an existing shared file-size gate across AST/edit paths, adds new behavioral fixtures and tests, and is expected to exceed 500 net LoC across helpers, tool invocations, and tests.

Related PRs/issues

• Shell foreground output acquisition is unbounded across platforms #3200: Shell foreground output acquisition is unbounded across platforms
• Bound tool acquisition before data exhausts process memory #3202: Bound tool acquisition before data exhausts process memory

Semantic relevance: #3200/#3202 establish the repo-wide memory-safety boundary and shared budget/truncation metadata contracts this plan must reuse. This issue applies those contracts to workspace-analysis and AST tool paths inside packages/tools.

Phases

Phase 0.5 — Preflight verification

Prerequisites:

• packages/tools workspace installs and typechecks.
• Existing behavioral tests for structural_analysis, ast_grep, ast_read_file, AST edit/write paths, read_file, read_line_range, and read_many_files are discoverable.
• The shared file-size gate in packages/tools/src/utils/fileUtils.ts is the single reuse point for pre-read size checks.

Verification:

• Run npm run typecheck in packages/tools.
• Run npm run test -- packages/tools/src/tools/structural-analysis/structural-analysis-modes.bun.test.ts packages/tools/src/__tests__/edit-ast-tools.test.ts packages/tools/src/__tests__/filesystem-tools.test.ts packages/tools/src/tools/line-range-tools-issue3036.bun.test.ts.
• Confirm no existing tests rely on unbounded aggregate materialization for large fixtures.

Phase 1 — Shared bounded-analysis contracts in packages/tools

Goal: Introduce one shared budget/truncation metadata contract for workspace-analysis and AST aggregation paths, reused by #3200-style tooling.

Steps:

1. Extend packages/tools/src/tools/structural-analysis/types.ts with bounded traversal metadata:
   • fileBudget?: number
   • recordBudget?: number
   • filesVisited?: number
   • recordsCollected?: number
   • budgetTruncated?: boolean
     Keep existing AnalysisResult.truncated semantics; add partial-result metadata without breaking current serialization.
2. Add a small shared helper in packages/tools/src/tools/structural-analysis/helpers.ts for:
   • checking abort signals before/after file operations,
   • incrementing filesVisited,
   • capping record collection and setting budgetTruncated when a budget is hit.
3. Do not modify the shared head/tail stream collector per non-goals.

Test-first:

• Extend packages/tools/src/tools/structural-analysis/structural-analysis-modes.bun.test.ts with bounded-traversal cases:
  • dependencies/references/exports beyond record and file budgets,
  • abort during large traversal,
  • partial workspace analyses are clearly identified as partial in truncated and metadata.

Verification:

• npm run test -- packages/tools/src/tools/structural-analysis/structural-analysis-modes.bun.test.ts
• npm run typecheck

Phase 2 — Bound structural-analysis aggregation modes

Goal: Enforce file/record/result budgets while traversing and report accurate partial-result metadata.

Steps:

1. Update executeDependencies, executeReferences, and executeExports to:
   • accept optional budgets,
   • stop traversal when filesVisited >= fileBudget or recordsCollected >= recordBudget,
   • set truncated and metadata accurately.
2. Keep callers/callees budgets behavior unchanged except for metadata consistency.
3. Ensure aggregate serialization avoids unnecessary full-result duplication:
   • stream/serialize partial results during traversal rather than building one giant aggregate then JSON-stringifying it.

Test-first:

• Extend structural-analysis-modes.bun.test.ts with:
  • exact budget boundary cases,
  • metadata assertions (filesVisited, recordsCollected, budgetTruncated),
  • aggregate output proving bounded collection without mock-only call-count assertions.

Verification:

• npm run test -- packages/tools/src/tools/structural-analysis/structural-analysis-modes.bun.test.ts
• npm run lint:ci, npm run lint:eslint-guard, npm run typecheck

Phase 3 — Bound AST grep accumulation

Goal: Stop accumulating when maxResults is satisfied.

Steps:

1. Refactor packages/tools/src/tools/ast-grep.ts collectMatches to:
   • accumulate matches up to maxResults,
   • stop searching additional files/nodes once satisfied,
   • preserve abort-signal handling.
2. Update formatSearchResults to use early-termination metadata (totalMatches can remain optional; truncated must be accurate).
3. Add a dedicated behavioral test file:
   • packages/tools/src/tools/ast-grep.behavioral.bun.test.ts
   • Justify creation because no existing AST grep behavioral test file covers bounded accumulation with real fixtures.

Test-first:

• Create ast-grep.behavioral.bun.test.ts with:
  • far more matches than maxResults,
  • exact-limit and one-over-limit assertions,
  • abort during directory traversal,
  • bounded output proving no full-result duplication.

Verification:

• npm run test -- packages/tools/src/tools/ast-grep.behavioral.bun.test.ts
• npm run typecheck

Phase 4 — Reuse shared file-size gate for AST read-file and AST edit/write helpers

Goal: AST read-file and edit/patch/insert/delete paths should reuse the existing read_file/read_line_range pre-read size policy.

Steps:

1. Extract/reuse the pre-read stat/size check from packages/tools/src/utils/fileUtils.ts:
   • validateFileSize already enforces MAX_FILE_SIZE_BYTES (20MB).
   • Expose/reuse this check in AST edit helpers instead of reading full files blindly.
2. Update packages/tools/src/tools/ast-edit/edit-calculator.ts readFileState to:
   • stat the file first,
   • return a bounded-size error consistent with read_file when oversized,
   • preserve freshness/last-modified behavior for compliant files.
3. Update packages/tools/src/tools/ast-edit/ast-read-file-invocation.ts to:
   • stat before full read,
   • refuse oversized files with the same boundary behavior as read_file,
   • keep line-range selection behavior unchanged for files within bounds.
4. Preserve existing read_file, read_line_range, and read_many_files behavior and acquisition bounds.

Test-first:

• Extend packages/tools/src/__tests__/edit-ast-tools.test.ts with:
  • exact-limit and one-byte-over files for AST read and modification paths,
  • assertions that oversized files are refused before parsing/copying.
• Extend packages/tools/src/tools/line-range-tools-issue3036.bun.test.ts or filesystem-tools.test.ts only if needed to prove shared gate behavior; prefer extending existing read-path tests.

Verification:

• npm run test -- packages/tools/src/__tests__/edit-ast-tools.test.ts packages/tools/src/tools/line-range-tools-issue3036.bun.test.ts
• npm run typecheck

Phase 5 — Integration checks and deferred-implementation detection

Goal: Ensure no bounded path silently falls back to unbounded behavior and no TODO/HACK/STUB remains.

Steps:

1. Search packages/tools/src/tools/structural-analysis, ast-grep.ts, ast-edit, and read-many-files.ts for deferred markers; replace with bounded behavior or explicit error returns.
2. Verify callers:
   • packages/core/src/config/toolRegistryFactory.ts
   • packages/agents/src/api/discovery.ts
   • packages/cli/src/zed-integration/zed-tool-handler.ts
     still resolve the tools and that returned ToolResult.metadata exposes new partial-result fields without breaking existing consumers.
3. Verify read_many_files existing acquisition bounds remain intact after shared gate reuse.

Verification:

• grep audit for TODO\|HACK\|STUB\|return; in touched modules.
• npm run test, npm run lint:ci, npm run lint:eslint-guard, npm run typecheck.

Test plan

Existing files to extend

• packages/tools/src/tools/structural-analysis/structural-analysis-modes.bun.test.ts
  • bounded dependencies/references/exports budgets,
  • abort-during-traversal,
  • accurate truncated and partial metadata.
• packages/tools/src/__tests__/edit-ast-tools.test.ts
  • AST read-file/edit paths with exact-limit and one-byte-over files.
• packages/tools/src/__tests__/filesystem-tools.test.ts
  • confirm shared gate behavior remains compatible for read_file/read_line_range if needed.
• packages/tools/src/tools/line-range-tools-issue3036.bun.test.ts
  • confirm line-range behavior unchanged by shared gate reuse if needed.

New files to create

• packages/tools/src/tools/ast-grep.behavioral.bun.test.ts
  • bounded accumulation behavioral tests for ast_grep.

Implementation notes

• Reuse only the shared validated budget and truncation metadata contracts established by Shell foreground output acquisition is unbounded across platforms #3200; do not create a second file-size utility.
• Keep record-count and traversal-stop policies in structural/AST tool modules.
• Do not force record aggregation or whole-file semantics through the head/tail stream collector.
• Preserve existing read_file, read_line_range, and read_many_files behavior and acquisition bounds.

Policy invariance

• No new eslint-disable*, @ts-ignore, @ts-expect-error, @ts-nocheck.
• No ESLint severity downgrades.
• No complexity/size threshold increases.
• No new ignores: blocks.
• Fix underlying causes instead of suppressing warnings.

Prompt for AI agents

Implement Phase 1 first, then Phase 2, then Phase 3, then Phase 4, then Phase 5. For each phase:

1. Add/extend behavioral tests using Bun and bun:test with real fixture trees/files.
2. Implement the bounded behavior in packages/tools only.
3. Run npm run lint:ci, npm run lint:eslint-guard, npm run typecheck, and npm run test for touched paths.
4. Do not modify packages/core, packages/agents, or packages/cli unless integration verification requires non-behavioral adjustments; in that case, keep changes minimal and type/contract-preserving.