fix: strip Gemma reasoning markers from inference streams

[rosetta-livekit-bot]

Description

Ports livekit/agents#6398 to agents-js, stripping Gemma reasoning markers from streamed LiveKit Inference LLM output while preserving visible text.

Changes Made

• Added a stateful streamed thinking-token filter with partial marker buffering for default <think> tags and model-specific Gemma channel markers.
• Wired the filter into inference.LLMStream while preserving the existing safety contract that tool-call chunks do not expose content to TTS.
• Treated omitted intermediate finish_reason values as non-final so split markers remain buffered.
• Ported the source utility and inference regression tests, plus a patch changeset for @livekit/agents.

Source diff coverage

Coverage classification

• livekit-agents/livekit/agents/inference/llm.py: adapted to agents/src/inference/llm.ts. Added the Gemma model-specific reasoning marker map, instantiated one streaming thinking-token filter per LLM stream run, and stripped content before tool-call handling. The JS adaptation retains its existing tool-call-to-TTS safety contract by suppressing content on chunks that contain tool calls.
• livekit-agents/livekit/agents/llm/utils.py: adapted to agents/src/llm/utils.ts. The target had the same LLM utility counterpart but no existing thinking-token stripper, so this PR adds the TypeScript equivalent infrastructure: ThinkingTokenFilter, partial marker buffering, final flush/drop semantics, and stripThinkingTokens.
• tests/test_llm_utils.py: adapted to agents/src/llm/utils.test.ts and agents/src/inference/llm.test.ts. Utility-focused tests were ported into the existing LLM utils test file; focused inference regressions additionally cover omitted finish_reason and tool-call content suppression.

Testing

• Automated tests added/updated.
• pnpm exec vitest run agents/src/inference/llm.test.ts agents/src/llm/utils.test.ts passed: 2 files, 64 tests passed, 1 skipped.
• pnpm --filter @livekit/agents build passed.
• Isolated package lint passed with no errors; the normal linked-worktree command encountered duplicate ESLint plugin discovery.
• Focused Prettier check passed.
• Independent spec-compliance review approved.
• Independent code-quality review approved.

Cue voice E2E

• Session: sid_a151bb355799.
• Exact head: ecad09dd3b576cbdd27fcf365449e94227988041.
• Mode: Cue voice over LiveKit with real deepgram/nova-3 STT, cartesia/sonic-3 TTS, tool execution, room transport, and recorded audio.
• Controlled boundary: a deterministic OpenAI-compatible SSE endpoint drove the production inference adapter as google/gemma-4-31b-it, because a hosted Gemma response cannot reliably reproduce the exact malformed split-marker/tool-content sequence on demand.
• Path-specific result: a split reasoning marker with omitted intermediate finish_reason and a tool-call content leak marker produced only SPOKEN_SAFE_PREFIX. , the expected tool output, and the final safe answer. Persisted events contained none of SECRET_REASONING_MARKER, LEAKED_TOOL_CONTENT_MARKER, <|channel>, or <channel|>.
• TTS accounting: cumulative TTS input reached exactly 90 characters, equal to the two safe assistant messages and excluding all injected leak text.
• Artifacts: ~/.cue-cli/sessions/sid_a151bb355799/events.jsonl, recordings/001_run/result.json, session recording.wav, and command recordings/001_run/recording.wav. Both WAVs were flushed and inspected directly.
• Limitation: the scripted SSE boundary is protocol-compatible but not the production Gemma provider. The WAV was not independently re-transcribed; non-leakage is proven at the production TTS input/transcript path with exact character accounting and non-silent recordings.

Additional Notes

• Source diff size was 3 files and 215 changed lines, below the port-plan POST threshold.
• No new dependencies or unrelated inference changes.

---

Ported from livekit/agents#6398

Original PR description

Fixes #6375

Problem

Gemma channel-thought markers and their hidden reasoning pass through the shared thinking-token filter into customer-visible output, including TTS, transcripts, and chat history.

Root cause

strip_thinking_tokens only recognizes <think> blocks and uses an asyncio.Event, which records whether the stream is inside a reasoning block but cannot retain marker fragments split across chunks. It also does not handle complete or multiple reasoning blocks within one chunk.

Change

• add a stateful streaming filter for existing <think> and Gemma <|channel>thought / <channel|> blocks
• buffer only possible marker prefixes across chunk boundaries
• preserve visible content before and after reasoning blocks
• flush incomplete visible marker-like text when the stream finishes while dropping unclosed reasoning

Tests

• uv run --frozen pytest tests/test_llm_utils.py --unit -q — 11 passed
• uv run --frozen pytest --unit --no-concurrent --ignore=tests/test_room.py -q — 1116 passed, 2 skipped
• make check — formatting, Ruff, and mypy passed

tests/test_room.py could not run locally because this environment has no Docker executable; its nine failures were fixture setup errors before test execution.

Scope

No new dependencies, provider calls, or unrelated inference changes.

[changeset-bot]

🦋 Changeset detected

Latest commit: ecad09d

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 37 packages

Name	Type
@livekit/agents	Patch
@livekit/agents-plugin-anam	Patch
@livekit/agents-plugin-anthropic	Patch
@livekit/agents-plugin-assemblyai	Patch
@livekit/agents-plugin-azure	Patch
@livekit/agents-plugin-baseten	Patch
@livekit/agents-plugin-bey	Patch
@livekit/agents-plugin-cartesia	Patch
@livekit/agents-plugin-cerebras	Patch
@livekit/agents-plugin-deepgram	Patch
@livekit/agents-plugin-did	Patch
@livekit/agents-plugin-elevenlabs	Patch
@livekit/agents-plugin-fishaudio	Patch
@livekit/agents-plugin-google	Patch
@livekit/agents-plugin-hedra	Patch
@livekit/agents-plugin-hume	Patch
@livekit/agents-plugin-inworld	Patch
@livekit/agents-plugin-lemonslice	Patch
@livekit/agents-plugin-liveavatar	Patch
@livekit/agents-plugin-livekit	Patch
@livekit/agents-plugin-minimax	Patch
@livekit/agents-plugin-mistral	Patch
@livekit/agents-plugin-mistralai	Patch
@livekit/agents-plugin-neuphonic	Patch
@livekit/agents-plugin-openai	Patch
@livekit/agents-plugin-perplexity	Patch
@livekit/agents-plugin-phonic	Patch
@livekit/agents-plugin-resemble	Patch
@livekit/agents-plugin-rime	Patch
@livekit/agents-plugin-runway	Patch
@livekit/agents-plugin-sarvam	Patch
@livekit/agents-plugin-silero	Patch
@livekit/agents-plugin-soniox	Patch
@livekit/agents-plugin-tavus	Patch
@livekit/agents-plugins-test	Patch
@livekit/agents-plugin-trugen	Patch
@livekit/agents-plugin-xai	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR