fix(relay): raise grace limit, add replay backpressure, and NOTICE on oversized frames#1226
Merged
Merged
Conversation
wpfleger96
force-pushed
the
wpfleger/fix-relay-disconnects
branch
from
0a553bf to
c9f49c8
Compare
wpfleger96
changed the title
Staging relay disconnects spiked this week as user count grew. Three independent causes compound into a reconnect storm: 1. MAX_FRAME_BYTES was 64 KiB but NIP-44 encrypted observer/engram events produce ~88 KiB frames after base64 expansion — silent disconnect, no error to client. 2. SLOW_CLIENT_GRACE_LIMIT was 3 consecutive buffer-full events — too aggressive with Istio multi-hop buffering and fan-out bursts from more subscribers per channel. 3. REQ historical replay sends up to 10,000 events in a tight try_send loop with no yielding, overflowing the 1,000-message send buffer and triggering the grace limit mid-replay. Changes: - Raise MAX_FRAME_BYTES default to 512 KiB, configurable via BUZZ_MAX_FRAME_BYTES. Send NOTICE before disconnecting on oversized frames. - Raise SLOW_CLIENT_GRACE_LIMIT default to 15, configurable via BUZZ_SLOW_CLIENT_GRACE_LIMIT. - Lower MAX_HISTORICAL_LIMIT from 10,000 to 2,000. - Yield to the runtime every 100 events during replay so the send_loop can drain between batches.
wpfleger96
force-pushed
the
wpfleger/fix-relay-disconnects
branch
from
c9f49c8 to
ba2b3b4
Compare
wesbillman
approved these changes
Jun 24, 2026
wpfleger96
added a commit
that referenced
this pull request
Jun 24, 2026
…est helper The conn_manager.register() signature has required a grace_limit: u8 argument since #1226. #1227's register_presence_sub helper was added with only 5 args, breaking the buzz-relay test build (E0061). Matches the sibling register_conn helper's value of 3. Co-authored-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follows #1225 (configurable frame limit, now merged). Addresses the remaining relay disconnect causes that #1225 did not cover.
Changes
SLOW_CLIENT_GRACE_LIMITfrom 3 to 15, configurable viaBUZZ_SLOW_CLIENT_GRACE_LIMIT. The previous value of 3 consecutive buffer-fulltry_sendcalls was too aggressive with Istio multi-hop buffering and increased fan-out from more subscribers per channel.send_loopcan drain between batches, preventing buffer overflow mid-replay.MAX_HISTORICAL_LIMITfrom 10,000 to 2,000. Reconnecting clients rarely need 10k events of backfill. This reduces the worst-case replay burst that can overflow the send buffer.["NOTICE","error: frame too large (N bytes, limit M)"]before the connection closes, giving a diagnostic instead of a silent disconnect.Infra follow-up
Separately,
BUZZ_SEND_BUFFERshould be raised from 1,000 to 8,000 inblock-coder-tf-stacksstagingvalues.yamlto further absorb replay and fan-out bursts.