Proposal: proactive context compression and session rollover for RealtimeAgent

[iluv7]

This discussion proposes an implementation direction for #2566: context compression for RealtimeAgent, including whether a compressed checkpoint should trigger a proactive session reconnect.

Proposed approach

Realtime context differs from the standard Agent context because the active conversation is maintained by the remote session. Compressing only state.context does not shorten the context already held by the provider; the summary takes effect only after a reconnect.

The proposed flow is:

1. Start background compression before the provider limit is reached.
2. Keep the current realtime conversation running while compression is in progress.
3. Commit a local checkpoint without losing messages produced during compression.
4. At a safe turn boundary, proactively create a new session using the summary and retained recent messages.
5. Buffer new input audio during the reconnect and replay it in order after the new session is ready.

Providers for which proactive reconnect is not yet reliable can fall back to preparing the checkpoint and using it only after a natural disconnect.

Should we proactively reconnect?

Without a proactive reconnect, local compression does not change the active remote context. The provider may continue growing or automatically truncate the earliest history before the checkpoint is ever used. A proactive reconnect makes the compressed context effective immediately, at the cost of connection latency and additional lifecycle complexity.

To check whether the reconnect overhead is acceptable, we ran 20 sequential trials against DashScope qwen-audio-3.0-realtime-flash. Each trial contained a baseline turn and a reconnect turn. During reconnect, a fixed 2.35-second PCM input was buffered locally and sent after the new session was established.

Results:

Metric	Result
Successful reconnect trials	20/20
Audio frames delivered	39/39 in every trial
connect() p50	1081.53 ms
connect() p95	1097.62 ms
Additional next-turn latency p50	70.38 ms
Additional next-turn latency p95	93.09 ms
Maximum additional next-turn latency	about 102 ms

Although establishing a session took about 1.1 seconds, most of that time overlapped with the user audio, so the observed user-visible overhead remained below 500 ms. No audio loss was observed in these trials.

This is only a smoke test. Twenty successful trials do not establish a 99% success rate. Roughly 300 zero-failure trials would be needed to support a sub-1% failure rate at about 95% confidence. We also need short-input tests, because 300 ms or 500 ms of speech may not hide the reconnect as effectively as the 2.35-second input used here.

Based on the current result, our preference is to support proactive reconnect at a safe boundary, with a per-provider fallback to natural reconnect.

Open question 1: when should compression start?

Realtime providers expose different types of limits, so the trigger should not depend on a single universal token counter.

• If response usage is available, use the latest completed response's input_tokens + output_tokens. Input tokens should not be summed across turns because the latest input count normally already includes conversation history.
• If a provider limits audio turns, use the consumed turn count.
• If it limits accumulated audio duration, use the consumed audio seconds.
• If usage is unavailable, estimate tokens from the local transcript as a trigger signal.
• Session duration should help schedule a reconnect, but does not by itself imply that the content needs summarization.

Available dimensions can be normalized into a pressure value:

context_pressure = max(
    used_tokens / max_context_tokens,
    used_turns / max_audio_turns,
    used_audio_seconds / max_audio_duration_s,
)

A possible default is to start background compression around 70% and schedule rollover no later than around 85%. The thresholds should remain configurable, while provider adapters normalize usage and expose their applicable limits.

Open question 2: what happens to messages created during summarization?

The summary must not replace the live context directly. Otherwise, messages created while the summarizer is running may be lost.

Instead, compression should use an immutable snapshot and a revision/watermark:

1. Snapshot completed turns only.
2. Summarize an older prefix while retaining a recent tail verbatim.
3. Continue appending new messages to the live context.
4. Reconnect with the summary, the retained snapshot tail, and all messages added after the snapshot.

For example:

Snapshot:          [A B C] [D]
New messages:              [E F]
Recovered context: Summary(A B C) + D + E + F

If the snapshotted prefix changes due to interruption handling or message aggregation, the candidate summary should be discarded and retried.

Reconnect boundary and audio handling

The reconnect should happen only when the current user turn and assistant response are complete, no tool call or confirmation is pending, and no assistant audio is still playing.

During reconnect:

1. Route new input audio into a bounded local backlog.
2. Close the old session and establish the new session with the checkpoint.
3. Wait for an explicit provider-ready acknowledgement.
4. Replay the backlog in order and resume normal streaming.

Each connection should also have a session epoch so late terminal or error events from the old session cannot affect the new one. In addition, adapter connect() methods need a clear ready guarantee; returning immediately after sending session.update is not a sufficiently strong contract for reliable backlog replay.

Validation plan

Unit and adapter tests should cover:

• usage normalization without double-counting historical input;
• token, turn, and audio-duration triggers;
• preservation of the retained tail and post-snapshot messages;
• invalidation when the snapshot revision changes;
• reconnect only at safe boundaries;
• isolation of events from old session epochs;
• ordered backlog replay and explicit overflow handling;
• fallback behavior when proactive reconnect fails or is unsupported.

Feedback is especially welcome on the provider capability boundary and whether proactive reconnect should initially be opt-in or enabled by default for validated providers.

[iluv7]

@qbc2016
If you have a moment, could you take a look at this TD?

[qbc2016]

@iluv7 The overall direction looks good to me.

[iluv7]

Thanks for taking a look! I’ll move forward with the implementation based on this direction and submit a PR with the corresponding tests.