Skip to content

Fix node renderer graceful shutdown restarts - #4400

Merged
justin808 merged 16 commits into
mainfrom
jg-codex/fix/node-renderer-shutdown-restart
Jul 3, 2026
Merged

Fix node renderer graceful shutdown restarts#4400
justin808 merged 16 commits into
mainfrom
jg-codex/fix/node-renderer-shutdown-restart

Conversation

@justin808

@justin808 justin808 commented Jul 2, 2026

Copy link
Copy Markdown
Member

Summary

  • count each active node-renderer request once across response, abort, and timeout hooks so graceful worker shutdown cannot wait forever after overlapping hooks
  • make scheduled worker restarts robust to stale workers, send failures, worker error events, and rejected restart cycles while preserving serial restarts
  • treat graceful restart timeouts as documented seconds and clarify the docs
  • ACK worker shutdown receipt so the master skips only the early SIGKILL for draining workers while preserving the hard shutdown deadline

Fixes #4365.
Fixes #4366.
Fixes #4367.
Fixes #4368.

QA Evidence

  • pnpm --dir packages/react-on-rails-pro-node-renderer exec jest --runTestsByPath tests/restartWorkers.test.ts tests/handleGracefulShutdown.test.ts tests/masterStartupFailure.test.ts --runInBand passed: 3 suites, 33 tests
  • pnpm --dir packages/react-on-rails-pro-node-renderer run type-check passed
  • pnpm --dir packages/react-on-rails-pro-node-renderer run build passed
  • pnpm run lint passed
  • pnpm start format.listDifferent passed
  • git diff --check passed
  • pre-commit changed-file hooks passed
  • pre-push branch hooks passed

Known local limitations:

  • bundle exec rubocop currently fails on pre-existing unrelated files under react_on_rails/spike/3313_prism_gemfile_rewriter/; this PR does not touch Ruby files, and pre-push branch lint reported no Ruby files to lint.
  • pnpm --filter react-on-rails-pro-node-renderer run test -- --runInBand currently fails 4 streaming/RSC suites because react_on_rails_pro/spec/dummy/ssr-generated/server-bundle.js and rsc-bundle.js are absent in this checkout. The focused shutdown/restart suites above pass on current head.

Workflow config: known in current checkout. Base branch is main; strict merge ledger command is script/pr-merge-ledger <PR> --strict; hosted CI trigger is +ci-* comments or ready-for-hosted-ci / force-full-hosted-ci labels.

Batch QA Lane

  • qa-batch-i PASS evidence on head ac42546c5284f0bfc532255758822e4ee04fd76d: serial shared-file sequencing verified, focused node-renderer shutdown/restart Jest suite passed, and type-check passed.
  • Review follow-up on head 9d5ea8bf10d7829fe1b1ec0ca23a5c92c8c0a9e1: Claude threads triaged; three low-risk fixes were applied and one [Node Renderer] Master SIGKILLs all workers 2s into graceful shutdown — draining renders truncated, 10s shutdown-hook budget unreachable #4365 ACK/early-kill thread was resolved as an intentional drain-vs-hard-deadline decision.
  • Optimized hosted CI requested with ready-for-hosted-ci on head 9d5ea8bf10d7829fe1b1ec0ca23a5c92c8c0a9e1.
  • Current readiness remains waiting on current-head hosted checks/review gates; repository auto-merge is disabled, so the PR cannot be added to the merge queue until required checks pass.

Summary by CodeRabbit

  • Bug Fixes
    • Improved Node renderer graceful shutdown by tracking worker acknowledgements to avoid unnecessary force-kills.
    • Hardened scheduled worker restarts to better handle missing/dead workers and continue reliably when shutdown messaging fails.
    • Clarified graceful restart timeout behavior so restart timeouts are interpreted in seconds.
  • Documentation
    • Updated the changelog with a new Pro fixed entry covering graceful shutdown and scheduled restarts behavior.
  • Tests
    • Added Jest coverage for the revised timeout semantics and multiple stale/edge restart scenarios.

@coderabbitai

coderabbitai Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

Walkthrough

The PR updates Node renderer graceful shutdown and scheduled restart handling, including shutdown acknowledgement tracking, safer worker restart flow, regression tests, and a changelog entry.

Changes

Node renderer shutdown and restart hardening

Layer / File(s) Summary
Changelog entry
CHANGELOG.md
Adds a Pro fixed changelog entry for Node renderer graceful shutdown and scheduled restarts, with linked issue and PR references.
Graceful shutdown acknowledgements
packages/react-on-rails-pro-node-renderer/src/master.ts
Tracks shutdown acknowledgements per worker, skips acknowledged workers during the early force-kill phase, and clears acknowledgement state on exit.
Restart scheduling wrapper
packages/react-on-rails-pro-node-renderer/src/master.ts
Logs restartWorkers failures and always schedules the next restart cycle afterward.
Restart worker flow
packages/react-on-rails-pro-node-renderer/src/master/restartWorkers.ts
Handles missing or dead workers, converts the graceful restart timeout from seconds to milliseconds, and cleans up restart waits across send, error, and timeout paths.
Restart regression tests
packages/react-on-rails-pro-node-renderer/tests/restartWorkers.test.ts
Adds fake-worker coverage for timeout units and stale-worker restart edge cases.

Estimated code review effort: 4 (Complex) | ~45 minutes

Suggested labels: full-ci

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title is concise and accurately summarizes the main node renderer shutdown and restart fix.
Linked Issues check ✅ Passed The changes address all four linked issues: ACK-aware shutdown killing, seconds-based restart timeouts, once-per-request shutdown counting, and hardened restart-loop error handling.
Out of Scope Changes check ✅ Passed All shown changes stay within the node renderer shutdown/restart fixes and matching test/docs updates.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch jg-codex/fix/node-renderer-shutdown-restart

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@greptile-apps

greptile-apps Bot commented Jul 2, 2026

Copy link
Copy Markdown

Greptile Summary

This PR updates node-renderer worker shutdown and restart behavior. The main changes are:

  • Request completion is counted once across response, abort, and timeout hooks.
  • Scheduled worker restarts handle stale workers and send failures more gracefully.
  • Graceful restart timeouts are treated and documented as seconds.
  • Workers ACK shutdown receipt so the master can treat draining workers differently.

Confidence Score: 4/5

The shutdown ACK path can leave a stuck worker alive past the early kill window.

  • Restart timeout units and stale-worker handling look consistent with the updated tests.
  • Request accounting now avoids double-decrements across overlapping hooks.
  • The ACK is sent before draining completes, but the master uses it to skip the 2s stuck-worker kill.

packages/react-on-rails-pro-node-renderer/src/master.ts

Important Files Changed

Filename Overview
packages/react-on-rails-pro-node-renderer/src/master.ts Adds shutdown ACK tracking and restart rescheduling error handling; the ACK skip can bypass the early stuck-worker kill window.
packages/react-on-rails-pro-node-renderer/src/master/restartWorkers.ts Hardens scheduled restarts against stale workers and send failures, and converts graceful restart timeout seconds to milliseconds.
packages/react-on-rails-pro-node-renderer/src/worker/handleGracefulShutdown.ts Adds per-request completion tracking with a WeakSet and sends a shutdown ACK when the worker receives the shutdown message.
packages/react-on-rails-pro-node-renderer/tests/handleGracefulShutdown.test.ts Adds coverage for ACK sending and overlapping timeout/response request completion hooks.
packages/react-on-rails-pro-node-renderer/tests/masterStartupFailure.test.ts Extends the master harness for ACK skipping and scheduled restart failure rescheduling.
packages/react-on-rails-pro-node-renderer/tests/restartWorkers.test.ts Adds focused tests for restart timeout units and stale-worker send failure paths.
docs/oss/building-features/node-renderer/js-configuration.md Clarifies that gracefulWorkerRestartTimeout values are seconds.
docs/pro/js-memory-leaks.md Clarifies the seconds unit in the memory leak restart example.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
  participant M as Master
  participant W as Worker
  participant S as Supervisor
  M->>W: SHUTDOWN_WORKER_MESSAGE
  W-->>M: ACK before drain completes
  Note over W: Active request may still be stuck
  M->>M: 2s force-kill timer
  M--xW: Skipped because worker ACKed
  S--xM: Supervisor may kill master before 11s
  Note over W: Worker can remain alive without the master
Loading
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
  participant M as Master
  participant W as Worker
  participant S as Supervisor
  M->>W: SHUTDOWN_WORKER_MESSAGE
  W-->>M: ACK before drain completes
  Note over W: Active request may still be stuck
  M->>M: 2s force-kill timer
  M--xW: Skipped because worker ACKed
  S--xM: Supervisor may kill master before 11s
  Note over W: Worker can remain alive without the master
Loading

Reviews (1): Last reviewed commit: "Document node renderer shutdown fixes" | Re-trigger Greptile

Comment thread packages/react-on-rails-pro-node-renderer/src/master.ts
@justin808

Copy link
Copy Markdown
Member Author

+ci-status

@justin808 justin808 added the ready-for-hosted-ci Run optimized hosted GitHub CI for this PR label Jul 2, 2026
Comment thread packages/react-on-rails-pro-node-renderer/src/worker/handleGracefulShutdown.ts Outdated
Comment thread packages/react-on-rails-pro-node-renderer/src/master.ts
Comment thread packages/react-on-rails-pro-node-renderer/src/master.ts
@claude

claude Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Review summary

Reviewed the shutdown/restart logic changes across master.ts, master/restartWorkers.ts, and worker/handleGracefulShutdown.ts. The overlapping-hook request counting fix and the retry/error hardening in restartWorkers are solid improvements. Left 4 inline comments, most-severe first:

  1. master.ts:219skipAcknowledgedWorkers trusts the ACK (sent unconditionally, before any drain check) to exempt a worker from the early 2s force-kill. A worker can ACK while idle, then block its event loop shortly after on an already-in-flight request's synchronous render — deferring its kill to the ~11s hard deadline, which exceeds the 5s supervisor SIGTERM→SIGKILL window this early-kill timer exists to stay inside. This looks like it can reintroduce the orphaned-worker failure mode the timer was built to prevent.
  2. master/restartWorkers.ts:77-80 — the error handler (unlike onSendError/the send() catch) never resets isScheduledRestart = false. If the worker survives the error event and later crashes for real, master.ts's exit handler will treat it as an expected scheduled restart and silently re-fork, skipping the "died UNEXPECTEDLY" error report.
  3. worker/handleGracefulShutdown.ts:22SHUTDOWN_WORKER_ACK_MESSAGE is duplicated as an independent literal in this file and in master.ts, instead of living once in shared/utils.ts like SHUTDOWN_WORKER_MESSAGE. A future drift between the two copies wouldn't be caught by the type checker and would silently disable the ACK handshake.
  4. master.ts:127 (minor) — gracefulShutdownAcknowledgedWorkerIds is scoped to the whole master lifetime and never pruned, so it grows by one entry per scheduled worker restart forever. Low practical impact, but inconsistent with the per-shutdown-call scoping used elsewhere in the file.

@github-actions

github-actions Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

size-limit report 📦

Path Size
react-on-rails/client bundled (gzip) 63.52 KB (0%)
react-on-rails/client bundled (gzip) (time) 63.52 KB (0%)
react-on-rails/client bundled (brotli) 54.49 KB (0%)
react-on-rails/client bundled (brotli) (time) 54.49 KB (0%)
react-on-rails-pro/client bundled (gzip) 64.54 KB (0%)
react-on-rails-pro/client bundled (gzip) (time) 64.54 KB (0%)
react-on-rails-pro/client bundled (brotli) 55.53 KB (0%)
react-on-rails-pro/client bundled (brotli) (time) 55.53 KB (0%)
registerServerComponent/client bundled (gzip) 75.98 KB (0%)
registerServerComponent/client bundled (gzip) (time) 75.98 KB (0%)
registerServerComponent/client bundled (brotli) 65.51 KB (0%)
registerServerComponent/client bundled (brotli) (time) 65.51 KB (0%)
wrapServerComponentRenderer/client bundled (gzip) 68.42 KB (0%)
wrapServerComponentRenderer/client bundled (gzip) (time) 68.42 KB (0%)
wrapServerComponentRenderer/client bundled (brotli) 58.72 KB (0%)
wrapServerComponentRenderer/client bundled (brotli) (time) 58.72 KB (0%)

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9d5ea8bf10

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@github-actions

github-actions Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

CI Status

Head SHA: 9d5ea8bf10d7
Changed files: 10
Docs-only heuristic (matches ci-changes-detector metadata paths): no
ready-for-hosted-ci label: present
force-full-hosted-ci label: absent
Current hosted-CI waiver: not present for this SHA

Optimized hosted CI is enabled for this PR.

justin808 added 2 commits July 2, 2026 09:38
…derer-shutdown-restart

* origin/main:
  Add mechanical parity guards for Ruby↔TS protocol constants (#4412) (#4427)
  Move Node tsconfigs from @tsconfig/node14 to @tsconfig/node18 (#4410) (#4429)
  [Pro] Remove unused addressable and rainbow runtime deps from gemspec (#4416) (#4422)
  Delete finished #3313 Prism Gemfile-rewriter spike (#4421)
  Extract generator scan/tracking helpers (#4405) (#4430)
  Extract install_dependency_group helper in JsDependencyManager (#4403) (#4424)
  Remove obsolete Ruby<2.6 YAML-aliases capability shim (#4417) (#4428)
  Remove inert config.server_render_method option (#4415) (#4423)
  Prune stale knip ignores and enforce binaries in CI (#4408) (#4425)
  Extract shared redux_store kwargs validator (#4402) (#4420)
  Avoid caching async props prerender streams (#4376)
  Release incremental render context on setup failure (#4383)
  Optimize response type emitter snapshots (#4397)
  Skip generated stylesheet metadata for OSS renders (#4395)
  Avoid mutating render option inputs (#4396)
  Changelog: document PR 4282 registry cleanup (#4399)

# Conflicts:
#	CHANGELOG.md
…derer-shutdown-restart

* origin/main:
  Drop deprecation-tombstone config options for 17.0.0 (#4419) (#4432)

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4c406866c3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread packages/react-on-rails-pro-node-renderer/src/master/restartWorkers.ts Outdated
Comment thread packages/react-on-rails-pro-node-renderer/src/master.ts
@claude

claude Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Review summary

Reviewed the shutdown/restart changes in master.ts, master/restartWorkers.ts, and worker/handleGracefulShutdown.ts, plus the associated tests and docs.

What looks solid:

  • The WeakSet-based dedup in handleGracefulShutdown.ts correctly fixes the double-decrement bug when onResponse/onRequestAbort/onTimeout overlap for the same request during shutdown — good regression test coverage (decrements each active request once when timeout and response hooks overlap during shutdown).
  • restartWorkers.ts is meaningfully hardened: stale-worker detection via currentWorker(), send() error handling (both thrown and callback-reported), and an error event handler, all correctly reset isScheduledRestart = false and resolve the per-worker promise exactly once via the finish()/isResolved guard. Traced the sync-vs-async callback ordering (including a mocked send invoking its callback synchronously) and found no double-resolve or listener-leak path.
  • gracefulWorkerRestartTimeout is now correctly treated as seconds (* MILLISECONDS_IN_SECOND), matching the documented units — this was a real pre-existing bug (docs said seconds, code used the raw value as ms).
  • gracefulShutdownAcknowledgedWorkerIds is properly pruned on every cluster.on('exit'), so it doesn't leak across worker restarts.
  • Scheduled-restart promise rejections are now caught and logged instead of silently killing the reschedule loop.
  • SHUTDOWN_WORKER_ACK_MESSAGE is defined once in shared/utils.ts and imported everywhere (no literal duplication).

One confirmed, still-open risk (left as an inline comment on master.ts): the shutdown ACK is sent as soon as a worker receives SHUTDOWN_WORKER_MESSAGE, not once it has actually finished draining. skipAcknowledgedWorkers: true uses that ACK to exempt the worker from the 2s early force-kill, deferring it to the ~11s hard deadline — which can exceed short SIGTERM→SIGKILL grace windows (e.g. Foreman's 5s default, called out in this file's own comments), letting the supervisor hard-kill the master mid-drain instead of the master exiting cleanly. This was already raised in an earlier review pass and the PR description says it was "resolved as an intentional drain-vs-hard-deadline decision" — but that reasoning isn't captured in code/comments, so flagging again for an explicit final sign-off rather than as a new finding.

No security concerns (internal process/IPC lifecycle code, no user-controlled input reaches it) and no performance concerns beyond the shutdown-path timing discussed above.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9f7185ea30

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread packages/react-on-rails-pro-node-renderer/src/master/restartWorkers.ts Outdated
justin808 added 2 commits July 2, 2026 10:23
…derer-shutdown-restart

* origin/main:
  Remove dead methods; prune always-false Rails<5.0 spec branches (#4418) (#4431)

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
packages/react-on-rails-pro-node-renderer/tests/restartWorkers.test.ts (1)

48-162: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Missing test coverage for the empty-worker-set guard.

The four tests here cover the seconds-unit timeout, send-throw, send-callback stale/IPC error, and error-before-exit scenarios, but none exercise the workerIds.length === 0 early-return path in restartWorkers.ts (if (workerIds.length === 0) { log.warn('No workers to restart'); return; }). This is one of the explicit hardening scenarios called out for issue #4368 ("empty worker sets... do not crash the master process or break the restart loop").

✅ Suggested additional test
+  it('resolves without error when there are no workers to restart', async () => {
+    const restartWorkers = loadRestartWorkers({});
+    await expect(restartWorkers(0, 30)).resolves.toBeUndefined();
+  });
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/react-on-rails-pro-node-renderer/tests/restartWorkers.test.ts`
around lines 48 - 162, Add a test for the empty-worker-set guard in
restartWorkers by exercising the early return when no worker IDs are available.
In restartWorkers and loadRestartWorkers, verify that calling the restart flow
with an empty worker map does not throw, logs the “No workers to restart”
warning, and leaves the restart loop untouched. Use the existing restartWorkers
test suite and the workerIds.length === 0 branch as the target behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@packages/react-on-rails-pro-node-renderer/tests/restartWorkers.test.ts`:
- Around line 48-162: Add a test for the empty-worker-set guard in
restartWorkers by exercising the early return when no worker IDs are available.
In restartWorkers and loadRestartWorkers, verify that calling the restart flow
with an empty worker map does not throw, logs the “No workers to restart”
warning, and leaves the restart loop untouched. Use the existing restartWorkers
test suite and the workerIds.length === 0 branch as the target behavior.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 9c0aaa11-cc9a-4bd0-8bcb-9d3c82bf7e26

📥 Commits

Reviewing files that changed from the base of the PR and between c2e91d1 and 48b6705.

📒 Files selected for processing (4)
  • CHANGELOG.md
  • packages/react-on-rails-pro-node-renderer/src/master.ts
  • packages/react-on-rails-pro-node-renderer/src/master/restartWorkers.ts
  • packages/react-on-rails-pro-node-renderer/tests/restartWorkers.test.ts
💤 Files with no reviewable changes (1)
  • packages/react-on-rails-pro-node-renderer/src/master/restartWorkers.ts
✅ Files skipped from review due to trivial changes (1)
  • CHANGELOG.md

…derer-shutdown-restart

* origin/main:
  Document RSC public-page validation and sidecar patterns (#4387)
@claude

claude Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Review summary

This PR hardens the node-renderer master/worker shutdown and scheduled-restart machinery: it makes active-request counting exact-once across onResponse/onRequestAbort/onTimeout (via a WeakSet of in-flight request objects), makes the scheduled restartWorkers() loop tolerant of stale/dead workers and send() failures while keeping restarts strictly sequential, fixes gracefulWorkerRestartTimeout being treated as raw milliseconds instead of the documented seconds, and adds a shutdown ACK so the master's early force-kill timer doesn't kill workers that are still gracefully draining. The new/updated tests in restartWorkers.test.ts, handleGracefulShutdown.test.ts, and masterStartupFailure.test.ts cover the main new behaviors well.

I traced through the activeRequests/activeRequestsCount double-counting fix, the gracefulShutdownAcknowledgedWorkerIds Set lifecycle, and the restartWorkers.ts per-worker promise (finish/onExit/onError/onSendError) for exactly-once resolution, timer leaks, and listener leaks. One concrete bug survived verification (left as an inline comment):

  • packages/react-on-rails-pro-node-renderer/src/master/restartWorkers.ts:77-79onError only logs and never calls finish(). Since a child process's 'error' event isn't guaranteed to be followed by 'exit' (per Node's own docs), and gracefulWorkerRestartTimeout has no default and is optional even when scheduled restarts are enabled, a worker 'error' with no timeout configured can permanently stall the sequential restart loop — and because the outer promise never settles, master.ts's scheduleWorkersRestart never reschedules its next cycle either, silently and permanently disabling the periodic-restart safety net. The existing new test for this path only exercises the case where gracefulWorkerRestartTimeout is set (truthy), so the hang path isn't covered.

No other correctness, security, or performance issues stood out — the ACK-based skip-early-kill design and the WeakSet-guarded double-decrement fix both look sound for the scenarios I could construct.

justin808 added 2 commits July 2, 2026 15:34
…derer-shutdown-restart

* origin/main:
  Preserve streaming LoadError during dependency failures (#4388)
  Wire eslint-rules RuleTester suite into a runner, CI, and knip (#4409) (#4433)
  Handle fire-and-forget RSCRoute retry failures (#4378)
  Fix visible hydration cleanup for detached roots (#4374)
  Avoid full locale default obsolete scans (#4398)
@claude

claude Bot commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Review: Node renderer graceful shutdown restarts (PR 4400)

Went through master.ts, master/restartWorkers.ts, and worker/handleGracefulShutdown.ts in detail, tracing the ACK/timeout/exit event interactions and the request-counting dedup logic. Overall this is a solid, well-scoped fix — the iterative commits ('Address shutdown review edge cases' through 'Avoid wedging scheduled restart after worker errors') show the tricky races were already hunted down and closed one by one. I did not find any correctness bugs blocking merge. Notes below are FYI/optional.

Verified correct

  • Double-decrement fix (handleGracefulShutdown.ts): switching to a WeakSet<object> keyed by the actual request object, with activeRequests.delete(request) guarding every decrement, correctly prevents onTimeout + onResponse (or onRequestAbort + onResponse) firing for the same request from decrementing activeRequestsCount twice. The new 'overlapping hooks' test exercises exactly this. Good fix for issues 4365/4368.
  • restartWorkers.ts seconds vs. ms: confirmed gracefulWorkerRestartTimeout * MILLISECONDS_IN_SECOND matches the (now-clarified) docs, and the old code really did treat the value as raw milliseconds — genuine bug fix, not just a doc tweak.
  • Stale/dead worker handling: currentWorker() correctly skips workers that vanished or are already dead before the scheduled turn, without throwing (previously restartWorkers threw Error('No workers to restart') on an empty cluster.workers, which — since master.ts's scheduleWorkersRestart only had .finally(), no .catch() — would have been an unhandled rejection). The new .catch() in master.ts plus the log.warn + early return in restartWorkers.ts closes that hole.
  • onError timeout interaction: traced the final state — with a gracefulWorkerRestartTimeout configured, an 'error' event on the worker now just logs and lets the pre-armed timeout still fire destroy(); without a timeout, 'error' unblocks the serial loop immediately instead of hanging forever (which is what the pre-PR code effectively did, since it had no 'error' listener at all). Confirmed listener cleanup (finish() removes both 'exit' and 'error' listeners) has no leak paths across the branches (send throws synchronously, onSendError callback, onError, onExit, timeout).
  • ACK-based skip-kill: gracefulShutdownAcknowledgedWorkerIds is correctly scoped per worker id and cleaned up on every cluster.on('exit'), so it can't accumulate stale entries across worker lifecycles. The comment at master.ts:217-221 explaining the drain-vs-hard-deadline tradeoff is a nice touch given that's clearly the thread that got the most discussion.

Minor / non-blocking

  • handleGracefulShutdown.ts:101-103: the activeRequestsCount < 0 warn branch looks unreachable now — since a decrement only happens after a successful activeRequests.delete(request), the count can never go negative (it's bounded by the number of prior adds). Harmless as defensive belt-and-suspenders, but effectively dead code.
  • restartWorkers.ts:84-90 and 104-111: the 'worker turned out to be stale' handling is duplicated almost verbatim between the synchronous catch around .send() and the async onSendError callback (both set isScheduledRestart = false, log the same message, call finish()). Could be factored into one helper, but small enough that I would not block on it.

Testing

QA evidence in the PR description (targeted Jest suites, type-check, build, lint, format) matches what I'd expect to run for this change; I wasn't able to execute the suite myself in this sandbox, but the new tests (restartWorkers.test.ts's stale-worker/send-failure/error-event cases, handleGracefulShutdown.test.ts's overlapping-hooks case, masterStartupFailure.test.ts's ACK/skip-kill and reschedule-after-failure cases) look like they exercise the exact races this PR fixes rather than just re-asserting happy-path behavior.

No blocking findings from this pass.

…derer-shutdown-restart

* origin/main:
  Remove pre-monorepo node-renderer devDep baggage; consolidate test multipart builders (#4435)
@claude

claude Bot commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Review summary

This PR has already been through several rounds of review (5 prior claude review passes plus CodeRabbit/Greptile), and the core fixes hold up well on independent re-verification:

  • handleGracefulShutdown.ts: the WeakSet<object>-keyed activeRequests correctly makes onResponse/onRequestAbort/onTimeout decrement exactly once per request even when hooks overlap (e.g. timeout then response) — confirmed by tracing decrementAndMaybeShutdown's delete()-guarded decrement, and the new overlapping-hooks test exercises exactly this.
  • restartWorkers.ts: stale/dead-worker skipping via currentWorker(), the seconds-not-ms fix (gracefulWorkerRestartTimeout * MILLISECONDS_IN_SECOND), and the finish()/isResolved exactly-once-settle guard across onExit/onError/onSendError/sync-throw all check out — no double-resolve or listener-leak path found.
  • master.ts: gracefulShutdownAcknowledgedWorkerIds is properly pruned on every cluster.on('exit'), and scheduled-restart promise rejections are now caught (.catch(...).finally(...)) instead of risking an unhandled rejection that would have silently killed the reschedule loop.

One still-open finding (left inline on restartWorkers.ts): the onError handler never resets isScheduledRestart = false, unlike the two sibling failure paths (onSendError, the synchronous send() catch). When gracefulWorkerRestartTimeout is unset and a worker emits an unrelated 'error' event, the restart loop moves on without ever confirming that worker exited — it's left running with a stale isScheduledRestart: true. A later, genuinely unrelated crash of that same worker would then be silently misclassified as an expected scheduled restart in master.ts's exit handler, suppressing the "died UNEXPECTEDLY" error report. This was flagged in an earlier review pass on this same PR but doesn't appear to have been addressed in the current head — worth a final look before merge, though it's narrow in scope (only reachable when gracefulWorkerRestartTimeout is left unconfigured).

No security or performance concerns — this is internal process/IPC lifecycle code with no user-controlled input, and the changes only affect shutdown/restart timing paths.

justin808 added 2 commits July 2, 2026 18:09
…derer-shutdown-restart

* origin/main:
  Delete never-wired RenderRequest/JsCodeBuilder/RenderingStrategy layer (#4414) (#4437)
  Delegate deprecated base/ shims to capabilities/ instead of cloning (#4413) (#4436)
@claude

claude Bot commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Review Summary

This PR hardens the node-renderer's graceful shutdown and scheduled-restart paths: (1) a SHUTDOWN_WORKER_ACK_MESSAGE handshake so the master's early force-kill timer skips workers that have acknowledged shutdown (while the hard deadline still applies to everyone), (2) a WeakSet-keyed active-request dedup in handleGracefulShutdown.ts so overlapping onResponse/onRequestAbort/onTimeout hooks for the same request can't double-decrement the counter, (3) a rewrite of restartWorkers.ts that re-fetches each worker fresh from cluster.workers and tolerates stale/dead workers, send() failures, and worker error events without breaking the serial restart loop, and (4) a fix treating gracefulWorkerRestartTimeout as seconds (was previously used directly as milliseconds).

I read through master.ts, master/restartWorkers.ts, and worker/handleGracefulShutdown.ts line-by-line, traced the ACK handshake against both force-kill timers, checked the WeakSet dedup for double/under-counting, and checked gracefulShutdownAcknowledgedWorkerIds for stale-id reuse across restart cycles. I also ran an independent second pass to adversarially check the same candidates. Nothing survived verification as a defect introduced by this diff:

  • ACK handshake vs. force-kill timers: the 2s early-kill timer only skips workers in gracefulShutdownAcknowledgedWorkerIds, and the hard-deadline timer intentionally omits that skip, so ACKed-but-stuck workers are still reaped at the deadline. Working as designed.
  • restartWorkers.ts finish(): always strips both exit/error listeners on every resolution path, no listener leak.
  • WeakSet dedup: activeRequests.delete(request) returning false correctly short-circuits repeat decrements from overlapping hooks; no double-decrement path found.
  • Worker id reuse: cluster assigns strictly incrementing, non-reused worker ids, so gracefulShutdownAcknowledgedWorkerIds can't cross-contaminate a later worker with the same id.

One pre-existing (not introduced by this PR) edge case worth noting for awareness rather than as a blocking issue: in restartWorkers.ts, if gracefulWorkerRestartTimeout is left unset (undefined/0, i.e. "no timeout") and a worker's event loop is permanently blocked (never emits exit or error), the await new Promise(...) in the restart loop never resolves, stalling scheduled restarts for all subsequent workers. This mirrors the exact "blocked event loop" scenario master.ts explicitly guards against via SHUTDOWN_WORKER_FORCE_KILL_TIMEOUT_MS, but the scheduled-restart path only gets that protection when an operator opts into gracefulWorkerRestartTimeout. Might be worth a follow-up to always apply some fallback timeout in restartWorkers.ts, but this behavior predates this PR and isn't a regression.

Overall the changes are well-scoped, the new ACK/dedup mechanisms compose correctly with the existing shutdown timers, and the new Jest coverage (restartWorkers.test.ts, updated handleGracefulShutdown.test.ts/masterStartupFailure.test.ts) exercises the stale-worker, send-failure, and overlapping-hook scenarios directly. No blocking findings.

…derer-shutdown-restart

* origin/main:
  Improve release-finish dry-run fetch handling (#4441)
  Flush RSC payloads before incomplete HTML tails (#4379)
  Handle sync RSC route failures as fetch errors (#4393)
@justin808
justin808 added this pull request to the merge queue Jul 3, 2026
Merged via the queue into main with commit c1602a8 Jul 3, 2026
50 checks passed
@justin808
justin808 deleted the jg-codex/fix/node-renderer-shutdown-restart branch July 3, 2026 07:21
@justin808

Copy link
Copy Markdown
Member Author

Batch I closeout

Immediate maintainer attention: none.

FYI / decisions made:

justin808 added a commit that referenced this pull request Jul 3, 2026
…370' into codex/batch-e-loadable-stats-retry-4371

* origin/codex/batch-e-rsc-parser-flush-4370:
  Add cached static RSC helper and diagnostics (#4386)
  Fix Pro tag revalidation retry after delete failures (#4375)
  Fix node renderer graceful shutdown restarts (#4400)
  Improve release-finish dry-run fetch handling (#4441)
  Flush RSC payloads before incomplete HTML tails (#4379)
  Handle sync RSC route failures as fetch errors (#4393)
  Delete never-wired RenderRequest/JsCodeBuilder/RenderingStrategy layer (#4414) (#4437)
  Delegate deprecated base/ shims to capabilities/ instead of cloning (#4413) (#4436)
  Remove pre-monorepo node-renderer devDep baggage; consolidate test multipart builders (#4435)
  Preserve streaming LoadError during dependency failures (#4388)
  Wire eslint-rules RuleTester suite into a runner, CI, and knip (#4409) (#4433)
  Handle fire-and-forget RSCRoute retry failures (#4378)
justin808 added a commit that referenced this pull request Jul 3, 2026
…nsport

* origin/main:
  Unify Pro component cache fetch behavior (#4384)
  Replace chalk with picocolors in create-react-on-rails-app (#4411) (#4444)
  Fix Pro RSC stylesheet stats retry after read failures (#4401)
  [Pro] Reduce tag-index cache work during streaming (#4443)
  Warn on truncated Pro RSC parser streams (#4392)
  Report response-start send rejections (#4389)
  Add cached static RSC helper and diagnostics (#4386)
  Fix Pro tag revalidation retry after delete failures (#4375)
  Fix node renderer graceful shutdown restarts (#4400)
  Improve release-finish dry-run fetch handling (#4441)
  Flush RSC payloads before incomplete HTML tails (#4379)
justin808 added a commit that referenced this pull request Jul 4, 2026
…w-boundary

* origin/main: (26 commits)
  [Pro] Extract async props settled chunk writer (#4448)
  Fix incorrect defer_generated_component_packs = false migration guidance (#4451)
  Fix Pro renderer transport memory and reuse (#4394)
  Fix Pro RSC loadable stats retry visibility (#4447)
  Unify Pro component cache fetch behavior (#4384)
  Replace chalk with picocolors in create-react-on-rails-app (#4411) (#4444)
  Fix Pro RSC stylesheet stats retry after read failures (#4401)
  [Pro] Reduce tag-index cache work during streaming (#4443)
  Warn on truncated Pro RSC parser streams (#4392)
  Report response-start send rejections (#4389)
  Add cached static RSC helper and diagnostics (#4386)
  Fix Pro tag revalidation retry after delete failures (#4375)
  Fix node renderer graceful shutdown restarts (#4400)
  Improve release-finish dry-run fetch handling (#4441)
  Flush RSC payloads before incomplete HTML tails (#4379)
  Handle sync RSC route failures as fetch errors (#4393)
  Delete never-wired RenderRequest/JsCodeBuilder/RenderingStrategy layer (#4414) (#4437)
  Delegate deprecated base/ shims to capabilities/ instead of cloning (#4413) (#4436)
  Remove pre-monorepo node-renderer devDep baggage; consolidate test multipart builders (#4435)
  Preserve streaming LoadError during dependency failures (#4388)
  ...

# Conflicts:
#	CHANGELOG.md
#	packages/react-on-rails-pro/src/RSCProvider.tsx
#	packages/react-on-rails-pro/src/RSCRoute.tsx
#	packages/react-on-rails-pro/tests/boundedCacheProvider.client.test.tsx
#	packages/react-on-rails-pro/tests/getReactServerComponent.client.test.ts
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment