[Bug]: Assistant streaming causes full-thread projection scans for every text delta

[cheruvian]

Steps to reproduce

1. Run T3 Code 0.0.32 with a mature thread containing substantial message and activity history.
2. Enable Settings → Assistant output → Stream assistant messages.
3. Start a Codex turn that streams a moderately long response.
4. Observe CPU and RSS for the T3 server process (not only the provider subprocess).
5. Compare with assistant streaming disabled.

The degradation grows with both the number of streamed response chunks and the amount of existing thread history.

Expected behavior

Streaming an assistant response should have approximately constant per-delta projection cost. Assistant text deltas should not repeatedly load and decode complete message, proposed-plan, activity, and approval collections.

Actual behavior

Each provider assistant-text delta is dispatched as a durable thread.message.assistant.delta command and becomes a thread.message-sent event.

Every event is then applied serially to all nine projectors. The threads projector handles every thread.message-sent event by invoking refreshThreadShellSummary, which reloads all messages, proposed plans, activities, and pending approvals for that thread.

Sanitized measurements from a real 0.0.32 database:

• 3,610 logical messages produced 128,972 thread.message-sent events
• Average: 35.7 persisted chunk events per logical message
• Maximum observed: 2,436 events for one 11,646-character message
• Busiest thread: 25,878 orchestration events
• With approximately one new orchestration event per second, the T3 server sustained 110–165% CPU
• Server RSS oscillated between approximately 1.4 GB and 2.6 GB

The observed cost therefore scales with streamed chunks × accumulated thread history. Repeatedly loading and decoding the full activity collection also creates significant allocation and garbage-collection pressure.

Relevant source paths

• apps/server/src/orchestration/Layers/ProviderRuntimeIngestion.ts: when enableAssistantStreaming is true, each assistant delta is individually dispatched (currently around lines 1638–1683).
• apps/server/src/orchestration/Layers/ProjectionPipeline.ts: refreshThreadShellSummary loads all four thread collections (currently around lines 547–589).
• The same projector calls that refresh for every thread.message-sent event (currently around lines 845–860).
• Every event runs nine projectors serially, with a transactional projector cursor update for each one (currently around lines 1601–1692).

This appears to undermine the intent of #1647, which removed an earlier thread-wide scan from message projection.

Issue #2761 is adjacent but appears distinct: that report concerns large WebSocket snapshots and reconnect behavior, while this report concerns synchronous persistence/projection work during active streaming.

Suggested direction

The narrowest fix may be to avoid refreshThreadShellSummary for assistant thread.message-sent events. Assistant text cannot change latestUserMessageAt, pending approvals, pending user input, or actionable proposed-plan state.

More generally, shell-summary fields could be maintained incrementally or through targeted aggregate queries for the event types that can actually affect each field. A useful regression contract would be: projecting an assistant delta performs a constant number of queries independent of thread history and does not query activities, proposed plans, or approvals.

Impact

Major degradation or frequent failure.

Version or commit

T3 Code Alpha 0.0.32.

Environment

macOS on Apple Silicon, desktop app, Codex app-server provider, assistant streaming enabled.

Logs or stack traces

No raw logs or database artifacts are attached because they can contain private agent content. The measurements above contain aggregate counts only.

Workaround

Disable Stream assistant messages. The server checks this setting while processing deltas, so subsequent text is handled by the existing buffered path without terminating the provider session. Starting a fresh thread reduces the history-dependent portion of the cost but does not address the per-delta transaction amplification.

[cheruvian]

One scope clarification from follow-up profiling: disabling assistant streaming removes the largest source of thread.message-sent deltas, but it does not eliminate the same projection amplification for activity events.

thread.activity-appended follows the same shell-summary refresh path, which reloads the thread-wide message/plan/activity/approval inputs. In a sanitized 10-second sample after streaming was disabled, 7 new activity events corresponded with 55 projector runs, 7 shell-summary refreshes, and 62 SQL transaction traces while server CPU remained elevated.

So the fix should cover both high-frequency assistant deltas and activity-appended events, ideally by applying incremental projection updates or coalescing shell refreshes. Closing the renderer eliminated concurrent VCS work but did not eliminate this projection load, helping separate the two issues.

[dain]

I observed a cross-thread consequence of this same projection amplification: a high-activity thread can prevent unrelated newly submitted turns from showing lifecycle or assistant-output events for several minutes.

Sanitized evidence from a current main checkout:

• A new turn was requested at 20:02:29.
• Its provider lifecycle event was timestamped 20:02:38, indicating that the provider started promptly.
• That event was not appended until after a client event from 20:10:18.
• There were 647 orchestration events ahead of it, including 562 thread.activity-appended events.
• One unrelated thread contributed 526 of the queued events.
• The incoming burst totaled only approximately 3.74 MB.

The expensive thread had accumulated:

• 27,303 activity rows / 131.2 MB of activity payloads
• 23,778 message rows / 9 MB of message text

Its 526 queued events consisted of 465 routine tool/context activity events and 61 assistant-message events. None needed a full shell-summary rebuild, but each triggered refreshThreadShellSummary.

A live trace from the same running instance measured thread.activity.append processing for that thread at:

• average: 734 ms
• minimum: 656 ms
• maximum: 890 ms

At the measured average, 526 events account for approximately 386 seconds of serial processing.

The impact crosses threads because ProviderRuntimeIngestion feeds runtime events from every thread through one global sequential DrainableWorker. An unrelated turn’s turn.started and assistant-output events therefore wait behind the busy thread’s full-history projection scans. In the UI, the affected threads remained at “Connecting” despite their providers already working, and no assistant output appeared until the queue caught up.

This also means disabling assistant streaming is only a partial workaround: tool-heavy activity can reproduce the same starvation. The practical workaround was to stop the high-activity session, wait for the queue to drain, and prefer fresh threads for tool-intensive work.

The narrow fix appears to be event-specific shell-summary maintenance:

• Do not call refreshThreadShellSummary for ordinary tool/context activity.
• Do not call it for assistant messages.
• Update latest_user_message_at directly for user messages.
• Recompute pending-input state only for user-input lifecycle events.
• Recompute approval and proposed-plan fields only from their relevant projections.
• Reserve complete summary rebuilding for rare repair operations such as revert or reconciliation.

Per-thread ingestion or lifecycle-event prioritization could provide defense in depth, but eliminating the hot-path full-history scans should address the primary backlog.

[dain]

PR #5855 appears to fix the measured cause of this incident, though the globally sequential ingestion queue means other slow projection paths could still cause cross-thread head-of-line blocking.