Skip to content

Make OCR transport tests failure-safe (Fixes #2744) - #3166

Merged
acoliver merged 3 commits into
mainfrom
issue2744
Aug 8, 2026
Merged

Make OCR transport tests failure-safe (Fixes #2744)#3166
acoliver merged 3 commits into
mainfrom
issue2744

Conversation

@acoliver

@acoliver acoliver commented Aug 8, 2026

Copy link
Copy Markdown
Collaborator

TLDR

Makes the OCR real-loopback transport integration tests failure-safe and event-driven. Each scenario now owns its client requests and active responses, embedded monitor process and temporary directory, and upstream server/connections so failures cannot strand resources. The abort regression destroys the client only after the complete request body and first streamed response chunk are observed, then waits for bounded request/response close events instead of sleeping.

Dive Deeper

  • Adds a test-only OCR scenario scope that registers resources before traffic or callbacks can fail and cleans them in client, monitor, upstream order.
  • Tracks responses before user response callbacks by prepending the ownership listener, including Bun's request-close-before-response-close behavior.
  • Handles throwing pre-traffic request callbacks without converting the original failure into a cleanup timeout.
  • Retains monitor ownership when termination fails so the global fallback can retry reaping the live child and removing its directory.
  • Preserves the exact scenario/startup Error when cleanup succeeds. When cleanup also fails, an AggregateError retains the original failure first and the cleanup failure second.
  • Moves assertions after scenario cleanup where they are not required to drive real traffic.
  • Adds real-loopback regressions for startup failures, assertion/traffic failures, active sibling cleanup, direct callback failures, cleanup error ordering, and monitor fallback cleanup.
  • Preserves the existing streaming, chunked request body, retry, HTTP 429, retry-header, concurrent request, partial-response, upstream-error, and telemetry behavior.
  • Does not change workflow or production behavior, dependencies, or lint policy.

Reviewer Test Plan

Run the focused lifecycle and existing transport suites independently:

bun test --preload ./scripts/tests/test-setup.ts scripts/tests/ocr-transport-lifecycle-2744.test.ts
bun test --preload ./scripts/tests/test-setup.ts scripts/tests/ocr-concurrency-canary-2673.test.ts

Then validate scripts typing and targeted lint:

npx tsc --project tsconfig.scripts.json --noEmit
npx eslint scripts/tests/ocr-concurrency-canary-2673-helpers.ts scripts/tests/ocr-concurrency-canary-2673.test.ts scripts/tests/ocr-transport-lifecycle-2744-helpers.ts scripts/tests/ocr-transport-lifecycle-2744.test.ts

Local evidence:

  • Lifecycle suite passed three sequential runs: 9 tests and 48 assertions per run.
  • Existing OCR canary suite passed: 15 tests and 75 assertions.
  • Full core workspace suite passed: 363 of 363 files.
  • Full build, typecheck, and format passed.
  • ESLint guard, no-new-JS, copyright-year, doc-placement, test-shard, and test-file-coverage guards passed.
  • StepFun smoke test passed with a valid three-line haiku.
  • Root npm run test and npm run lint were attempted locally but did not complete before the shell's hard execution ceiling under concurrent system load. No failure was observed before termination, but these are intentionally not claimed as local passes; PR CI should provide the authoritative full-repository result.

Testing Matrix

🍏 🪟 🐧
npm run ⚠️ focused/core/build/typecheck/format passed; root test/lint incomplete
npx ✅ scripts TypeScript and targeted ESLint
Docker
Podman - -
Seatbelt - -

Linked issues / bugs

Fixes #2744

Summary by CodeRabbit

  • Bug Fixes

    • Improved OCR transport reliability for retries, streaming responses, connection failures, malformed headers, and aborted requests.
    • Improved cleanup after interrupted or failed OCR operations, including monitor processes, temporary resources, and network connections.
    • Preserved relevant errors and combined operation and cleanup failures for clearer diagnostics.
    • Ensured sensitive request data is excluded from telemetry.
  • Tests

    • Expanded coverage for lifecycle handling, concurrent requests, telemetry, timeouts, startup failures, and resource cleanup.

@github-actions github-actions Bot added the maintainer:e2e:ok Trusted contributor; maintainer-approved E2E run label Aug 8, 2026
@coderabbitai

coderabbitai Bot commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your recent review volume is higher than typical usage, so adaptive limits are currently applied.

Next review available in: 9 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 08e611c6-c071-4790-aad3-45f04f82046d

📥 Commits

Reviewing files that changed from the base of the PR and between 0c87ffc and 8e3e152.

📒 Files selected for processing (3)
  • scripts/tests/ocr-concurrency-canary-2673-helpers.ts
  • scripts/tests/ocr-transport-lifecycle-2744-helpers.ts
  • scripts/tests/ocr-transport-lifecycle-2744.test.ts
📝 Walkthrough

Walkthrough

The change adds OCR transport scenario lifecycle helpers, request and response tracking, failure-safe monitor cleanup, and comprehensive lifecycle tests. Existing concurrency canary tests now use scoped resource management.

Changes

OCR transport lifecycle

Layer / File(s) Summary
Proxy callbacks and monitor cleanup
scripts/tests/ocr-concurrency-canary-2673-helpers.ts
ProxyRequestOptions and MonitorResource are exported. Proxy lifecycle callbacks are supported. Monitor cleanup preserves telemetry and failed resources.
Scenario resource orchestration
scripts/tests/ocr-transport-lifecycle-2744-helpers.ts
withOcrScenario tracks transport resources, performs bounded cleanup, and combines scenario and cleanup errors.
Concurrency canary lifecycle migration
scripts/tests/ocr-concurrency-canary-2673.test.ts
Canary tests use scoped lifecycle management while retaining retry, streaming, telemetry, and partial-response assertions.
Lifecycle failure validation and project wiring
scripts/tests/ocr-transport-lifecycle-2744.test.ts, tsconfig.scripts.json
New tests cover startup, callback, abort, cleanup, socket, child-process, and temporary-directory behavior. The scripts project includes the new files.

Estimated code review effort: 4 (Complex) | ~60 minutes

Possibly related PRs

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly describes the main change: making OCR transport tests failure-safe, and references the directly linked issue.
Description check ✅ Passed The description includes all template sections with scope, implementation details, test commands, results, matrix status, and linked issue information.
Linked Issues check ✅ Passed The changes address cleanup, event-driven aborts, partial failures, error preservation, real loopback coverage, and relevant test and lint requirements for issue #2744.
Out of Scope Changes check ✅ Passed The changes are limited to OCR test helpers, OCR tests, and TypeScript inclusion, with no unrelated production, workflow, dependency, or lint-policy changes.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch issue2744

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

Walkthrough

Before this PR, the OCR transport test surface could leave child processes, temp directories, and monitor state behind when an assertion failed or a test exited early. The existing concurrency canary coverage exercised successful transport paths, but a failed startup, timeout, mid-stream error, or client abort could still leak resources or mask cleanup problems. After this PR, the transport lifecycle is covered by a new failure-safe test suite that forces invalid targets, readiness timeouts, upstream crashes, and client aborts, while the concurrency tests reuse scoped cleanup helpers that aggregate cleanup errors and reap children deterministically. The scripts TypeScript configuration is also updated so these new lifecycle tests are compiled and discoverable by the test runner.

Release Notes

Tests

  • Added a failure-safe OCR transport lifecycle test suite covering invalid targets, startup timeouts, upstream mid-stream errors, client aborts, and leaked-resource cleanup paths.
  • Refactored OCR concurrency canary tests to use scoped cleanup helpers that aggregate cleanup errors and reap monitor child processes.
  • Updated the scripts TypeScript configuration to include the new OCR transport lifecycle tests so they are compiled and discovered by the test runner.

Refactor

  • Extracted reusable OCR transport test harness helpers for lifecycle management, cleanup error aggregation, child-process reaping, and preserving failed monitor resources for inspection.

Changes

Layer File(s) Summary
tests scripts/tests/ocr-transport-lifecycle-2744.test.ts, scripts/tests/ocr-concurrency-canary-2673.test.ts Adds a new failure-safe OCR transport lifecycle test suite and refactors existing concurrency tests to use scoped cleanup helpers, preventing resource leaks on assertion failures or early exits.
test-helpers scripts/tests/ocr-transport-lifecycle-2744-helpers.ts, scripts/tests/ocr-concurrency-canary-2673-helpers.ts Introduces failure-safe test harness and helper utilities for OCR transport scenarios, including scoped lifecycle management, aggregated cleanup errors, child-process reaping, and preserved failed resources for retry.
config tsconfig.scripts.json Includes the new OCR transport lifecycle test files in the TypeScript scripts configuration so they are compiled and discoverable by the test runner.
docs project-plans/issue2744.md Documents the behavioral contract and test plan for failure-safe OCR transport lifecycle handling, covering scope ownership, cleanup semantics, and out-of-scope boundaries.

Magnitude

🎯 1 (S)
1135 additions, 408 deletions, 6 changed files across 0 packages, 1 acceptance criterion

Related

No related items found.

Pre-merge Checks

Check Status Note
Title PR title is clear, specific, and includes the linked issue reference.
Description Body includes all expected template sections: TLDR, Dive Deeper, Reviewer Test Plan, Testing Matrix, and Linked issues / bugs.
Linked Issues The modified canary helpers and refactored canary test align with the issue’s failure-safe cleanup goals, but the two core new files named in the PR—scripts/tests/ocr-transport-lifecycle-2744.test.ts and scripts/tests/ocr-transport-lifecycle-2744-helpers.ts—are not present in the actual changes, so full acceptance-criteria coverage cannot be verified from the supplied code.
Out of Scope Missing new lifecycle test suite and helper files; missing project plan file; therefore event-driven abort behavior, startup-failure cleanup, and exact error-ordering claims from the issue cannot be confirmed from actual code changes.

Walkthrough generated by LLxprt PR Review. Planner issue: #2256

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
scripts/tests/ocr-concurrency-canary-2673-helpers.ts (1)

264-276: 🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Handle callback exceptions inside proxyRequest.

If onResponse throws, the exception escapes the asynchronous response listener, leaving the Promise pending and potentially terminating the process. If onRequest throws, the Promise rejects while the request remains undestroyed. Register request handlers before onRequest, catch both callback exceptions, destroy the request or response, and reject the Promise.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@scripts/tests/ocr-concurrency-canary-2673-helpers.ts` around lines 264 - 276,
Update proxyRequest to register request and response error handlers before
invoking onRequest, then wrap both onRequest and onResponse callbacks in
try/catch blocks. On callback failure, destroy the associated request or
response and reject the Promise, ensuring no asynchronous exception escapes and
no request remains open.
🧹 Nitpick comments (1)
scripts/tests/ocr-concurrency-canary-2673.test.ts (1)

495-521: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win

Settle the streaming promise on abnormal response termination.

The promise resolves only on the response end event. It rejects only on a request-level error. If the proxy or upstream destroys the response after headers, neither event fires. The scenario then hangs until the test timeout instead of failing fast. The sibling test at lines 602-618 already handles this case with close and error listeners.

♻️ Proposed fix to bound the streaming wait
             (response) => {
               const chunks: Buffer[] = [];
               response.on('data', (chunk) => {
                 if (chunks.length === 0)
                   firstChunkBeforeCompletion = !upstreamCompleted;
                 chunks.push(chunk);
               });
               response.on('end', () =>
                 resolve(Buffer.concat(chunks).toString()),
               );
+              response.on('error', reject);
+              response.once('close', () => {
+                if (!response.complete) {
+                  reject(new Error('streamed response closed before end'));
+                }
+              });
             },
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@scripts/tests/ocr-concurrency-canary-2673.test.ts` around lines 495 - 521,
Update the response handling in the streaming promise around the request
callback to reject on abnormal response termination by adding response error and
close handling, while resolving only on end. Ensure the close listener does not
reject after normal completion, and preserve the existing chunk collection and
request-level error handling.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@scripts/tests/ocr-transport-lifecycle-2744-helpers.ts`:
- Around line 87-90: In the onRequest error handler, remove the explicit
clientRequests.delete call after createdRequest.destroy(). Keep the failed
request tracked until the trackRequest close listener handles its removal, while
preserving the destroy and rethrow behavior.
- Around line 70-76: Update registerUpstream and registerMonitor to preserve
every registered resource instead of overwriting the existing upstream or
monitor. Store each registration in a collection and ensure the corresponding
cleanup path iterates over and cleans all resources, or explicitly reject
duplicate registrations while retaining the existing cleanup behavior.

In `@scripts/tests/ocr-transport-lifecycle-2744.test.ts`:
- Around line 105-112: Add timeout: 30_000 and killSignal: 'SIGKILL' to the
execFileSync options in the probe helper, ensuring the child process is forcibly
terminated if it hangs while preserving the existing Bun ESM execution setup.

---

Outside diff comments:
In `@scripts/tests/ocr-concurrency-canary-2673-helpers.ts`:
- Around line 264-276: Update proxyRequest to register request and response
error handlers before invoking onRequest, then wrap both onRequest and
onResponse callbacks in try/catch blocks. On callback failure, destroy the
associated request or response and reject the Promise, ensuring no asynchronous
exception escapes and no request remains open.

---

Nitpick comments:
In `@scripts/tests/ocr-concurrency-canary-2673.test.ts`:
- Around line 495-521: Update the response handling in the streaming promise
around the request callback to reject on abnormal response termination by adding
response error and close handling, while resolving only on end. Ensure the close
listener does not reject after normal completion, and preserve the existing
chunk collection and request-level error handling.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 810cba01-c963-40ec-b895-3e4c4e5eb684

📥 Commits

Reviewing files that changed from the base of the PR and between 73579ae and 0c87ffc.

⛔ Files ignored due to path filters (1)
  • project-plans/issue2744.md is excluded by !project-plans/**
📒 Files selected for processing (5)
  • scripts/tests/ocr-concurrency-canary-2673-helpers.ts
  • scripts/tests/ocr-concurrency-canary-2673.test.ts
  • scripts/tests/ocr-transport-lifecycle-2744-helpers.ts
  • scripts/tests/ocr-transport-lifecycle-2744.test.ts
  • tsconfig.scripts.json

Comment thread scripts/tests/ocr-transport-lifecycle-2744-helpers.ts
Comment thread scripts/tests/ocr-transport-lifecycle-2744-helpers.ts
Comment thread scripts/tests/ocr-transport-lifecycle-2744.test.ts
Comment thread scripts/tests/ocr-transport-lifecycle-2744-helpers.ts
Comment thread scripts/tests/ocr-transport-lifecycle-2744-helpers.ts
Comment thread scripts/tests/ocr-transport-lifecycle-2744.test.ts
Comment thread scripts/tests/ocr-transport-lifecycle-2744-helpers.ts
@github-actions

github-actions Bot commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

OpenCodeReview — automatic reviews suspended

Automatic OCR reviews are suspended for this PR after 2 of 2 automatic reviews.

To get more reviews you can:

  • Check the box below to re-enable automatic reviews (resets the counter), or

  • Comment /review, /ocr, or /open-code-review to request a single review on demand.

  • Re-enable automatic reviews


OpenCodeReview — PR #3166

  • Reviewed head SHA: 5c3b588228744770afe4808e5ec09fbee9eb6f2d
  • Merge base: 2bc73580f8e4af3530efe215a847a1ef009a50bb
  • Range: full from 2bc73580f8e4af3530efe215a847a1ef009a50bb
  • Range fallback: checkpoint-missing
  • Scope: selected 6 file(s), +1139/-408; cumulative 6 file(s), +1139/-408
  • Tokens: 628618 total (453472 input, 175146 output, 292608 cache)
  • OCR version: open-code-review v1.8.4 (e78474478) linux/amd64 built at: 2026-08-01T03:27:37Z https://github.com/alibaba/open-code-review
  • Phase: review
  • Exit code: 0
  • Run: https://github.com/vybestack/llxprt-code/actions/runs/31280568441
  • Partial review: 0 of 5 files completed (1 failed); 4 finding(s) (2 posted inline).
  • Artifacts: ocr-review-output contains raw JSON, stdout, stderr, preview, phase, and exit-code diagnostics.

Findings without a resolvable position

  • scripts/tests/ocr-concurrency-canary-2673.test.ts: [bug/low] > If response.destroy() throws during cleanup, the response remains in clientResponses and is never removed. The subsequent bounded waitFor times out and records the failure, but the response object leaks until process exit. The same issue exists for requests if request.destroy() throws. This is a resource leak path in the cleanup logic — a thrown destroy should still remove the item from the tracking set before re-throwing or recording the cleanup failure.
  • scripts/tests/ocr-concurrency-canary-2673.test.ts: [bug/low] > Same resource-leak pattern exists in the request cleanup loop: if request.destroy() throws, the request remains in clientRequests, causing the subsequent bounded waitFor to time out and leaving the request object tracked until process exit. To ensure deterministic cleanup, the request should be removed from clientRequests regardless of whether destroy() succeeds.
  • WARNING: Changed-file coverage 3/5 preview files covered is below the 90% threshold. WARNING: 1 file read/review failure(s) detected.

@acoliver

acoliver commented Aug 8, 2026

Copy link
Copy Markdown
Collaborator Author

PR OCR overflow disposition — Reject: the base proxy helper invokes its request callback immediately after request construction, but the only scenario callback supplied here first registers ownership, and its user-callback catch destroys and synchronously deregisters the real Bun request before rethrowing. The Promise executor preserves that exact callback failure. The dedicated pre-traffic callback regression proves the request is destroyed with no cleanup timeout or leak. Moving the callback before request construction is impossible because the callback receives that request; adding speculative handling for an asynchronous event within the same synchronous stack is not warranted. No source change was made for this finding. The changed-file coverage notice is a review-tool coverage warning rather than a source defect; CI independently ran and passed the scripts test shard.

Comment thread scripts/tests/ocr-concurrency-canary-2673-helpers.ts
Comment thread scripts/tests/ocr-transport-lifecycle-2744-helpers.ts
@acoliver

acoliver commented Aug 8, 2026

Copy link
Copy Markdown
Collaborator Author

Second PR OCR overflow dispositions — Reject for both destroy-throw variants. A request or response is removed from ownership only after it is already destroyed/complete or after its real close event. If destroy itself throws, deleting it from the set would falsely claim release of a resource that is not proven closed and would suppress the bounded ownership failure. The current path retains the original destroy failure plus the subsequent observable cleanup timeout, then continues with monitor and upstream cleanup. The tracking sets are local to the scenario and become collectible after rejection; they do not create a process-lifetime registry leak. Triggering these findings would also require replacing the controlled Bun destroy methods. No source change was made. The 3/5 review coverage warning and one tool file-read failure are review diagnostics, not source findings; final CI scripts tests and lint passed.

@acoliver
acoliver merged commit b751e6b into main Aug 8, 2026
36 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

maintainer:e2e:ok Trusted contributor; maintainer-approved E2E run

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make OCR transport integration tests failure-safe and event-driven

1 participant