feat: dictation into the composer - #148
Draft
incognitojam wants to merge 151 commits into
Draft
Conversation
incognitojam
force-pushed
the
t3code/add-desktop-dictation-support
branch
3 times, most recently
from
August 15, 2026 20:52
c2be37b to
e3deccf
Compare
yngatech-nightly
Bot
force-pushed
the
main
branch
from
August 16, 2026 01:58
5043b5d to
eedd052
Compare
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* ci: stabilize fork nightly tests * ci: isolate image compression test --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(ci): allow Apple notarization to finish * fix(ci): avoid duplicate Apple notarizations --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* ci: build unsigned Windows x64 in fork nightly * ci: add dry_run input to test fork nightly --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
- Run fork CI and nightly jobs for yngatech/t3code Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
fix(branding): use yngatech fork identity Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* ci: migrate Ubuntu jobs to Blacksmith * ci: keep lightweight jobs on GitHub runners --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* feat(web): show GitHub service outages * fix(web): clarify GitHub status link affordance * feat(web): make GitHub outage alerts opt-in * refactor(web): remove GitHub status preview override * test(desktop): include GitHub alert setting --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* feat(web): show unsent drafts in the sidebar - Add a draft indicator to sidebar thread rows - Surface drafts containing text or attachments, excluding settings-only state * fix(web): brighten the sidebar draft pencil on row hover The pencil sat at a flat secondary-label colour, so on a settled row it read brighter than the row's own title and stayed put while the rest of the row lifted. Match the PR badge's settled treatment: muted at rest, transitioning with the row on hover. The pencil has no state colour of its own, so it lifts to the same foreground the title does, and the active row opts out like the PR badge already does. --------- Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
The terminal guide explains browser-opening behavior but does not help users diagnose a missing injected launcher environment. Add focused troubleshooting for inspecting the terminal environment, refreshing terminals after updates, identifying client/server version mismatches, respecting custom `BROWSER` values, and finding helper-install diagnostics. Validation: documentation-only change; formatting checked. --- Written by an agent (T3 Code, gpt-5.6-sol).
> [!NOTE] > TL;DR: New **Copy transcript** thread action copies the conversation as markdown (`## User` / `## Assistant` under the thread title) — no tool calls, reasoning, or half-streamed replies — for sharing or pasting into a new chat as context. There was no way to get a thread's conversation out of T3 Code. This adds `copy-transcript` to the shared thread action menu (sidebar right-click and chat header), with the serializer in `apps/web/src/lib/threadTranscript.ts`. User prompts are copied as their visible text, without injected terminal/element/issue context blocks; image attachments become `[Attached image: …]` placeholders. Details worth reviewing: - The action fetches a full unwindowed snapshot through a new `createThreadSnapshotCommands` factory in client-runtime. The cached detail atom holds only the last 10 user turns (nothing for never-opened threads), so reading client state would silently truncate. - This is the only asynchronous copy action, so clipboard writes follow "last copy wins": every in-app write bumps a shared epoch (the direct `navigator.clipboard` call sites now route through `writeTextToClipboard`), and a transcript fetch whose epoch moved drops its result. A late-landing transcript is therefore harmless — it only lands when nothing else was copied since the click. Copies made in other applications during the fetch are unobservable from a web page (window bounded by the 6s fetch timeout). - Not wired: LegacySidebar (hand-curated legacy menu) and mobile (the shared factory makes later adoption cheap). Verified with unit tests (serializer, menu builder, clipboard epoch) and an integrated browser pass against seeded fixtures — including reproducing both clipboard race directions on an 8000-message thread and confirming the guard fixes them. ## Before / After | Before | After | | --- | --- | |  |  |  --- Written by an agent (T3 Code, claude-fable-5).
> [!NOTE] > Shows GitHub repository requirements directly in the Checks tab and promotes Auto-merge when a pull request is blocked but otherwise mergeable. ## What changed - Added GitHub repository-policy-aware merge readiness to pull request details. - Marked reported required checks and synthesized waiting rows for required checks that have not reported yet. - Read effective repository and organization rulesets alongside classic branch protection, with graceful GitHub Enterprise degradation. - Replaced the primary Merge action with Auto-merge when GitHub says the pull request is blocked and the viewer may enable auto-merge. - Kept unknown merge states and behind-base branches on their existing merge/update flows. ## Why A green or red job list does not explain whether GitHub will actually allow a merge. Required checks and GitHub’s merge-readiness verdict make the Checks tab answer that question directly, while Auto-merge becomes the obvious action when requirements are still outstanding. ## UI changes Captured against this pull request while its required checks were running. | Before: reported jobs only | After: repository requirements | | --- | --- | |  |  | ## Verification - `vp test run apps/server/src/pullRequest/GitHubPullRequestCli.test.ts apps/server/src/pullRequest/GitHubPullRequestProvider.test.ts apps/server/src/pullRequest/PullRequestService.test.ts apps/server/src/pullRequest/gitHubPullRequestJson.test.ts apps/server/src/pullRequest/pullRequestChecks.test.ts apps/web/src/components/pullRequest/PullRequestChecksTab.test.tsx apps/web/src/components/pullRequest/pullRequestChecks.test.tsx apps/web/src/components/pullRequest/pullRequestDetail.logic.test.ts` - `vp run t3#typecheck` - `vp run @t3tools/web#typecheck` - `vp run @t3tools/contracts#typecheck` - Targeted formatting and lint checks ## Checklist - [x] This PR is focused on GitHub merge requirements and the action they imply - [x] I explained what changed and why - [x] I included before/after screenshots for the UI change - [x] No video is needed because this change adds no motion or timing behavior --- Written by an agent (T3 Code, gpt-5.6-sol).
The web client could start restoring the most recent project from cached thread shells while the server was still resolving its launch-directory bootstrap project. If those automatic navigations overlapped, a late bootstrap route could switch projects after the composer became editable, leaving the user's text behind as a draft in the first project. Coordinate index-draft and server-bootstrap navigation through a one-shot in-memory owner. A bootstrap target that is already known can open normally, but a late bootstrap welcome cannot interrupt another launch navigation. Bootstrap targets are marked handled per environment and thread so later visits to Home continue to open a normal draft. Verification: - `vp test run apps/web/src/launchNavigationStore.test.ts` - `vp run typecheck` in `apps/web` - targeted formatter check Browser evidence was not captured because this is launch-timing behavior and computer-use permission was not requested. --- Written by an agent (T3 Code, gpt-5.6-sol).
> [!NOTE] > TL;DR: Clicking an `http://localhost:5173` link in chat now opens the integrated browser instead of the system browser. Every other host is unchanged. Meta/ctrl-click and the right-click menu still reach the system browser. ## The problem A loopback link an agent writes into chat names a port on the machine the thread runs on. In a remote or SSH environment that is not the machine the system browser runs on, so following the link opened the reader's own empty port instead of the dev server. Locally it worked, which is why it survived this long. ## What changed `apps/web/src/components/chat/loopbackLinkPreview.ts` is a pure predicate: open in the integrated browser when the href passes `isPreviewableUrl`, a thread and preview runtime are available, and no meta/ctrl modifier is held. The anchor renderer in `ChatMarkdown.tsx` consults it after the existing change-request branch; everything else keeps the `target="_blank"` the shell already handles. Two details worth reviewing: - **The loopback test is reused, not rewritten.** `isPreviewableUrl` (`packages/shared/src/preview.ts`) is the same check the terminal-link right-click gate uses, so the two paths cannot drift into disagreeing about what "previewable" means. The open path is the same `openUrlInPreview` the link context menu already used, which brings thread history, right-panel opening, and the remote-environment loopback rewrite in `browserTargetResolver.ts` along unchanged. - **A failed open falls back to the system browser.** The click asked for the link, not for the integrated browser specifically, so leaving it doing nothing would be worse than the old behavior. Interrupted (superseded) commands do not trigger the fallback. Non-loopback hosts deliberately stay external. The integrated browser runs in its own persisted partition, so routing github.com or Vercel there would land the reader on a logged-out page while their real session sits in the system browser. ## Surfaces - **Desktop:** new behavior. - **Web in a plain browser:** unchanged. `isPreviewSupportedInRuntime()` is `Boolean(window.desktopBridge?.preview)`, so `canOpenInPreview` is false and links fall through exactly as before. - **Mobile:** untouched — there is no integrated browser to route to. - **Reverse state:** the right-click menu still offers both destinations, and meta/ctrl-click escapes to the system browser, so every click is overridable in both directions. ## Verification - `vp test run apps/web/src/components/chat/loopbackLinkPreview.test.ts packages/shared/src/preview.test.ts apps/web/src/components/ChatMarkdown.test.tsx` - `vp run @t3tools/web#typecheck` - Targeted lint on the three changed files ## Checklist - [x] This PR is focused on where loopback chat links open - [x] I explained what changed and why - [ ] No before/after image: the link is visually identical before and after, and only its destination changes. Verified manually instead. - [x] No video is needed because this change adds no motion or timing behavior ## Follow-up, not in this PR Terminal links still pop an "Open in preview / Open in browser" menu on every loopback click rather than defaulting to preview. Aligning the two paths changes existing behavior in a different feature and belongs in its own PR. --- Written by an agent (Claude Code, claude-opus-5).
> [!NOTE] > TL;DR: Cmd-clicking an `http://localhost:5173` link in the terminal now opens the integrated browser directly instead of popping a two-item chooser first. Non-loopback links are unchanged. Follow-up to #153, but independent of it. ## The problem Activating a loopback link in the terminal showed a menu at the cursor — "Open in preview" / "Open in browser" — on every single click. Activation is already a deliberate modifier gesture (cmd-click on macOS, ctrl-click elsewhere), so the chooser added a second click to every visit to a dev server you are iterating on, for a decision that is nearly always the same one. Non-loopback URLs never showed the menu; they already went straight to the system browser. So the menu was the only place in the app where following a link asked a question first. ## What changed `openTerminalLinkInPreview` opens loopback links in the integrated browser and hands everything else to the system browser, with no menu in between. That deletes the menu plumbing: `TerminalLinkContextMenuShowError`, and the `localApi` and `position` arguments that only existed to position and show it — 107 lines out, 63 in. The loopback test, the failure fallback, the interrupt handling, and the origin-only error logging (so a link carrying a token in its query is not logged) are all unchanged. ## Reverse state The way back out is the integrated browser's own open-in-system-browser button in `PreviewChromeRow`, which is one click and already there. I deliberately did not invent a modifier for "system browser instead": cmd/ctrl is already spent on activation itself, and a shift variant would be undiscoverable. If it turns out people want the choice back, a setting is the honest shape for it, not a menu on every click. ## Relationship to #153 Same reasoning applied to a different surface, and the two together make the rule uniform: a loopback link opens in the integrated browser because it is the only browser that can reach the environment's ports; every other host opens in the system browser, where the reader is signed in. They touch disjoint files and can land in either order, so this is a separate PR rather than a stacked one. ## Surfaces - **Desktop:** new behavior. - **Web in a plain browser:** unchanged. `isPreviewSupportedInRuntime()` is false, so links fall through to the system browser exactly as before. - **Mobile:** no terminal drawer, nothing to change. ## Verification - `vp test run apps/web/src/components/preview/openTerminalLinkInPreview.test.ts` - `vp run @t3tools/web#typecheck` - `vp check` on the three changed files ## Checklist - [x] This PR is focused on removing one chooser from one gesture - [x] I explained what changed and why - [ ] No before/after image: the terminal link looks identical, and the visible difference is the absence of a menu. Verified manually. - [x] No video is needed because this change adds no motion or timing behavior --- Written by an agent (Claude Code, claude-opus-5).
> [!NOTE] > Adds a project-level **Additional instructions** preference and applies it whenever T3 Code starts an agent session. ## Problem Projects cannot currently provide durable guidance to every coding-agent session, so users must repeat the same instructions in each thread. ## What changed - Add an **Additional instructions** textarea to Project Settings, shared across grouped checkouts. - Persist the preference through orchestration events, projections, snapshots, and a fork migration. - Pass instructions through native channels for Codex, Claude, Grok, and OpenCode. - Prefix the first prompt of fresh Cursor sessions because Cursor ACP does not expose a native system-instruction channel. - Retain instructions when provider sessions are recovered. - Document the behavior and reset flow. ## Verification - Focused orchestration, projection, migration, recovery, and provider adapter tests pass. - Contracts, server, and web typechecks pass. - Targeted lint and formatting checks pass. - Verified in an isolated T3 preview that Project Settings renders the new textarea and explanatory copy. Before/after capture is omitted because the shared preview screenshot endpoint repeatedly timed out. --- Written by an agent (T3 Code, gpt-5.6-sol).
Settled turns currently collapse their work behind a duration-only divider, which makes it hard to tell whether a turn mostly inspected files, ran commands, edited code, used external tools, or compacted context. This adds a compact, end-aligned activity summary to the existing `Worked for …` fold. Counts are derived from the already-collapsed work entries, grouped by activity category, and remain accessible through singular/plural labels. Context compaction uses the semantic theme color and stays visible when ordinary activity metadata clips in narrow panes. Web and desktop receive the change through the shared web client. Mobile remains unchanged in this PR. ## Screenshots ### Before <img width="768" alt="Before: duration-only folded turn divider" src="https://github.com/yngatech/t3code/blob/65ec96f9d26571abbe17a6a4747ab1c19104e066/pr-158-before.png?raw=true"> ### After <img width="768" alt="After: folded turn divider with end-aligned activity counts" src="https://github.com/yngatech/t3code/blob/65ec96f9d26571abbe17a6a4747ab1c19104e066/pr-158-after.png?raw=true"> Tests: - `vp test run apps/web/src/components/chat/MessagesTimeline.logic.test.ts apps/web/src/components/chat/MessagesTimeline.test.tsx` - `vp run --filter @t3tools/web typecheck` - Targeted `vp lint` on the four changed files --- Written by an agent (T3 Code, gpt-5.6-sol).
The Summary tab currently shows check health without the repository's merge verdict, so an optional failing check can make a ready pull request look blocked. The inverse is also possible when checks pass but repository policy still blocks the merge. This puts the host-provided merge verdict first in the existing Summary checks control, keeps non-green check health as supporting context, and preserves the previous checks-only fallback when the host has no verdict. The condensed title row uses the same policy-first presentation with shorter copy. ## Screenshot  ## Verification - `vp test run src/components/pullRequest/PullRequestChecksTab.test.tsx --project unit` - `vp run --filter @t3tools/web typecheck` - Preview: Summary chip reads `Ready to merge · 1 of 9 failing` - Preview: condensed title row reads `Ready · 1 failing` - Preview: the Summary chip opens the policy-aware Checks view --- Written by an agent (T3 Code, gpt-5.6-sol).
> [!NOTE] > Adds a clearly labeled operational CO₂ estimate to Usage on web, desktop, and mobile, with familiar comparisons and the assumptions kept one click away. ## What changed - estimates operational emissions from generated tokens using a documented 0.43 g CO₂ per 1,000 output-token factor - shows phone-charge comparisons below 1 kg and driving-distance comparisons at 1 kg and above - adds a methodology popover on web/desktop and a native information alert on mobile - shares calculation and formatting logic across clients, with focused unit coverage - documents the estimate, exclusions, and source methodology ## Screenshots ### Web / desktop | Before | After | | --- | --- | |  |  | ### Mobile | Before | After | | --- | --- | |  |  | All captures use an isolated environment with synthetic usage data. ## Verification - `vp test run packages/shared/src/usageFormat.test.ts` - `vp run --filter @t3tools/shared typecheck` - `vp run --filter @t3tools/web typecheck` - `vp run --filter @t3tools/mobile typecheck` - integrated web and iOS passes against the same synthetic usage fixture --- Written by an agent (T3 Code, gpt-5.6-sol).
The pull request panel can already know that a thread’s pull request merged while the sidebar still holds its older open snapshot, leaving the thread active until the separate VCS poll catches up. Feed a matching panel merge into the existing thread change-request snapshot, seeding it from the active thread when the sidebar row has not mounted. The shared snapshot reconciliation treats merged as irreversible, so stale VCS open state cannot undo the update. Open/closed transitions remain owned by VCS because cached panel detail cannot safely establish their ordering. No server cache, persistence, polling, or contract changes are required. Verification: - `vp run --filter @t3tools/web typecheck` - Web unit suite: 2,741 tests passed --- Written by an agent (T3 Code, gpt-5.6-sol).
## What Changed Updated the nightly workflow to use `actions/cache@v5` and `softprops/action-gh-release@v3`. ## Why Keeps the nightly release workflow aligned with the latest action versions and current CI dependencies. ## Checklist - [x] This PR is small and focused - [x] I explained what changed and why - [x] I included before/after screenshots for any UI changes - [x] I included a video for animation/interaction changes
## What Changed Replaced the Discord webhook action in the fork nightly workflow with a direct `curl` request that builds the payload using `jq`, retries transient failures, and fails when Discord rejects the notification. ## Why The nightly failure notification was not reliable enough. The new implementation makes the request and payload construction explicit, preserves the failure details and workflow link, and adds retry handling for transient errors. ## Checklist - [x] This PR is small and focused - [x] I explained what changed and why - [x] I included before/after screenshots for any UI changes - [x] I included a video for animation/interaction changes
## What Changed Result toasts for opened pull requests now include the pull request number and open the in-app pull request panel when available. They continue to open the external pull request URL when the panel callback is unavailable. ## Why Previously, clicking “View PR” from a result toast always opened the pull request externally, preventing users from viewing it in the app. ## UI Changes The result toast action now opens the corresponding pull request in the in-app panel. ## Checklist - [x] This PR is small and focused - [x] I explained what changed and why - [ ] I included before/after screenshots for any UI changes - [ ] I included a video for animation/interaction changes
The sidebar's project filter narrowed the thread list but had no say in where new threads went, so picking a project still left the New thread button asking which one you wanted. The filter now decides where the sidebar's button creates, because that button belongs to the filtered list and a draft made outside the scope would have no visible row in it. chat.newLocal keeps the opposite precedence — it starts a thread beside the one on screen, and only falls back to the filter when nothing is open — so a narrowed sidebar never removes the way to work beside your current thread. chat.new still opens the chooser. The button keeps one behavior per state rather than letting a modifier divert it. The filter moved out of component state into a small store, since the shortcuts live in the chat route and the palette mounts at the root. Its self-healing reset now waits while the project list is empty instead of treating "not loaded yet" as "stale filter", which previously let any remount or reconnect drop the filter silently. Also corrects the command palette's new-thread shortcut labels, which have advertised the chooser's shortcut on the direct-create item since chat.new started opening a picker. --- Written by an agent (Claude Code, claude-opus-5).
## What Changed Load the upstream rebase policy script from the immutable `main` snapshot when running a `source_ref` nightly rebase. The workflow now validates that both required policy scripts exist on `main`, writes the rebase script to a temporary file, and sources it from there. Direct runs continue to use the repository's checked-in rebase script. ## Why A stale `source_ref` could otherwise control or roll back the rebase policy script. Loading the script from immutable `main` ensures nightly rebases use the current trusted policy and fail clearly when `main` is missing the required script. ## Checklist - [x] This PR is small and focused - [x] I explained what changed and why - [x] I included before/after screenshots for any UI changes - [x] I included a video for animation/interaction changes
yngatech-nightly
Bot
force-pushed
the
main
branch
from
August 16, 2026 11:27
4e2683b to
0646252
Compare
## What changed Fork Nightly now reapplies upstream-equivalent commits and stops when a fork patch becomes empty, making patch retirement a maintainer-reviewed decision. Successful rebases add a compact patch range-diff to the workflow summary. When a rebased candidate has the same tree as the released nightly but newer ancestry or a reviewed patch retirement, the workflow now promotes that candidate to `main` without publishing another release. It no longer restores the older nightly history. Focused Git fixtures cover clean replay, upstream patch adoption, content conflicts, range-diff reporting, and tree-equivalent promotion through the real backup and lease script. ## Verification - `vp test run scripts/rebase-onto-upstream.test.ts scripts/check-source-stack.test.ts` - `vp run --filter @t3tools/scripts typecheck` - `vp lint scripts/rebase-onto-upstream.test.ts --report-unused-disable-directives` - `shellcheck .github/scripts/rebase-onto-upstream.sh .github/scripts/check-source-stack.sh` - `git diff --check` --- Written by an agent (T3 Code, gpt-5.6-sol).
…168) The compaction marker in the folded-turn work log row was tinted `text-primary/70`. `--primary` maps to the theme's solid-control fill — the send-button background, which pairs with `--primary-foreground` on top of it — so read as text on the chat canvas it lands dimmer than the muted stats it is meant to outrank. That holds in all 10 shipped palettes; three fall below the 3:1 floor outright, and T3 Chat dark is effectively invisible at 1.59:1. The marker now takes `text-foreground` while its siblings stay `text-muted-foreground`, keeping its emphasis by being the brightest thing in the row plus its pinned position beside the chevron. That puts it between 12.4:1 and 16.7:1. The `transition-colors` / `group-hover` pair went with it, since there is nothing left to transition and the row's own `hover:text-foreground` already covers hover. The test asserted the literal `text-primary/70` string; it now asserts the marker carries `text-foreground` and not a fill-role colour. Introduced in #158. Roughly nine other sites read `--primary` as text on the canvas with the same flaw — fixing that class properly needs a contrast-solved accent-text role in `packages/shared/src/themePalettes.ts`, which I left alone. No before/after image: capture kept failing on this view, and the change is one colour token on a 12px glyph. --- Written by an agent (Claude Code, claude-opus-5).
Old timestamps currently remain day counts forever, producing labels such as `1093d ago`. This updates the shared web and mobile relative-time formatters to switch to months after 30 days and years after 12 30-day months. Focused boundary coverage verifies `29d`, `1mo`, `11mo`, and `1y`. ## Screenshots The affected surfaces were rendered locally with public/synthetic fixtures across pull request rows, recent threads, and pull request activity. The collaborative preview screenshot exporter failed during capture, so no image is attached rather than substituting generated evidence. ## Test plan - `vp test run apps/web/src/timestampFormat.test.ts apps/mobile/src/lib/time.test.ts` --- Written by an agent (T3 Code, gpt-5.6-sol).
## Summary Background preview tabs remain parked far outside the Electron compositor. Their DOM automation stays available, but `WebContents.capturePage()` can reject with `UnknownVizError` or never settle because the guest has no captureable surface. This change gives snapshot requests a short-lived, reference-counted compositor lease. Only the target webview moves in-window with `opacity: 0`, the host waits for guest render frames, and the lease is always released after capture. Native snapshot capture is also bounded and single-flight so a stuck promise cannot start overlapping compositor copies. This is the screenshot portion of #170. Recording and preview visibility semantics remain separate follow-ups. ## Testing - `vp test run apps/desktop/src/preview/Manager.test.ts apps/web/src/browser/browserSurfaceStore.test.ts apps/web/src/browser/hostedBrowserWebviewStyle.test.ts` - `vp run --filter @t3tools/web typecheck` - `vp run --filter @t3tools/desktop typecheck` - Targeted lint for all changed files - Electron 41.5.0 compositor probe: five parked-to-leased captures returned non-empty 1600x1200 images in 5-76 ms - Dev desktop MCP verification: visible capture succeeded; two fresh `open: false` tabs stayed `visible: false` before and after valid 1280x800 PNG snapshots; fresh `open: true, show: true` background tabs also captured successfully --- Written by an agent (T3 Code, gpt-5.6-sol). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
> [!NOTE] > Adds the first incremental capability ledger and uses it to flag upstream overlap during Fork Nightly. ## What changed - adds a validated, capability-oriented fork ledger with six initial high-risk or conflict-prone entries - distinguishes fork implementation evidence from upstream contact paths - makes Fork CI validation blocking and Nightly overlap reporting advisory and rename-aware - documents incremental maintenance, review, and retirement of fork capabilities ## Verification - `vp run --filter @t3tools/scripts ledger:check` - `vp test run scripts/fork-feature-ledger.test.ts scripts/rebase-onto-upstream.test.ts` (13 tests) - `vp run --filter @t3tools/scripts typecheck` - targeted lint for the ledger scripts and tests - `git diff --check` - confirmed all 22 initial upstream contact paths exist on `upstream/main` --- Written by an agent (T3 Code, gpt-5.6-sol). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Seventeen findings from a working spike (native/dictation-spike/README.md) drive every design decision: Parakeet GGUF over Apple SpeechAnalyzer (95% vs 64-82% identifier recall, noise-robust), no ASR-level biasing (contextualStrings ignored by SpeechTranscriber; custom LMs ignored entirely), deterministic matcher over LLM post-processing (4ms, cannot hallucinate), ~100-term context-scoped vocabulary (repo-scale damaged a third of utterances), and marked substitutions as a requirement since phrase collisions are deterministic consequences of correct transcription. Corpus audio is gitignored (personal voice data); the ASR transcripts are committed as regression fixtures.
Deterministic recovery of code identifiers from ASR output ("work tree path"
-> worktreePath). Ported from the spike's measured Swift implementation; every
rule cites the finding that paid for it: acronym-aware spoken forms, boundary
stopwords waived for the candidate's own words, length-banded edit budget
(threshold-insensitive, unlike a ratio), punctuation and line boundaries,
0.05 ambiguity margin, and spoken-form collision collapse at vocabulary build.
Substitutions carry output offsets for the revert UI. Tests embed real
Parakeet transcripts of a second speaker as fixtures, including one asserting
the isEmpty/isValid phrase collisions must fire — losing them means recall
broke; the product mitigation is visible, revertable substitutions.
…oint native/dictation: Rust sidecar (transcribe-cpp, Parakeet GGUF), PCM on stdin, JSONL events with cumulative RTF on stdout. Metal on macOS, static CPU on Linux (0.39 RTF single-stream; no GPU required for the single-user case). Staged via the resource-monitor pattern, generalized to stageRustSidecar. DictationEngine keeps one warm sidecar (model load ~10s cold, ~0.6s from page cache), replaces it per utterance, unloads after 5 idle minutes, and ties the process to the request scope so client disconnects cannot orphan it. The endpoint is a WebSocket at /api/dictation/stream: browsers only allow streaming fetch bodies over HTTP/2, so a POST cannot work against the HTTP/1.1 server; binary frames carry PCM without base64 inflation, a text frame finalizes, JSON text frames stream back. Auth via wsTicket or same-origin cookie. 503 with a reason before upgrade when no model is configured (T3CODE_DICTATION_MODEL_PATH or dictationModelPath). macOS packaging gains the audio-input entitlement and NSMicrophoneUsageDescription via extendInfo — without both, packaged builds record digital silence with no error anywhere (spike finding 4).
Capture in the renderer (shared by web and desktop): getUserMedia at the microphone's native rate — forcing 16kHz anywhere in the chain renders silence in Chrome — with an AudioWorklet kept in the render graph through a muted gain, JS resampling to the 16kHz wire format, and a framer that flushes its partial frame on stop. Stop lingers 350ms so the final word survives the click; Escape discards without inserting. Session is an app-wide singleton and the hook guards its async startup window, so overlapping utterances are structurally impossible. Composer: mic button in the footer, live bubble (portalled — the footer cluster is overflow-x-auto and clips absolute children), level meter with a no-signal warning after 2.5 silent seconds naming the diagnosis while it happens, matcher substitutions listed with one-tap revert via a new ChatComposerHandle.replaceRange, stash fallback when the composer refuses input, and composer.dictate on mod+shift+d. Desktop grants media permission on the main window session only, scoped to the app's own origin — never the preview partitions. Vite forwards WebSocket upgrades on /api (ws:true), without which the dictation socket hangs at pending forever. Settings: dictationEnabled (off by default), dictationVocabularyEnabled, dictationMode (hold mode still to come).
Three real defects the PR run surfaced: - DictationEngine tests resolved the sidecar from disk, so they passed locally (where native/dictation is built) and failed in CI with the wrong reason. Both the sidecar and model paths are now set explicitly per test, via a new dictationSidecarPath config field alongside dictationModelPath. - An explicitly configured sidecar path is now authoritative rather than the first entry in a fallback chain: a wrong path should say so, not silently run a different binary found on disk. - process.platform in Effect code, the node: import convention in the spike harness scripts, and repo formatting. Also fmt-checks native/dictation in CI, matching resource-monitor.
incognitojam
force-pushed
the
t3code/add-desktop-dictation-support
branch
from
August 16, 2026 14:08
40f2b5e to
14062c2
Compare
yngatech-nightly
Bot
force-pushed
the
main
branch
from
August 16, 2026 15:11
2698323 to
44e3d33
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Dictation into the composer. Audio never leaves the user's own machines: capture
in the renderer, transcription on their T3 server.
Draft. The feature works end to end and is verified live in Chrome; it ships
off by default under Settings → Experimental, so transcription never starts
unless the user opts in. Desktop artifacts still build and package the sidecar
unconditionally.
Approach
Capture lives in
apps/web, so desktop is covered for free (the desktoprenderer is
apps/web). Transcription lives on the server so every client canshare one implementation, and because the T3 server is the user's own machine
the audio stays on their infrastructure.
Why these choices
Every decision came from a measured spike (
native/dictation-spike/README.md,17 findings, reproducible harness). The load-bearing ones:
64-82% identifier recall on the same recordings, and unaffected by the mic and
room noise that moved Apple 18 points. No macOS/Tahoe/Apple Intelligence
dependency anywhere in the design.
contextualStringsis ignored bySpeechTranscriber;Apple custom language models are ignored entirely (a deliberately corrupted
model file produced no error and no change).
hallucinate. A small on-device model given the same job invented identifiers
and corrupted a correct sentence.
9 of 24 utterances were damaged ("and then" ->
andThen, "is fast" ->isLast).request bodies over HTTP/2; the server speaks HTTP/1.1.
collisions ("the room is empty" ->
isEmpty) are deterministic consequences ofcorrect transcription — six TTS voices across five locales produced identical
results — so their rate rises as transcription improves.
Verified
E2E in Chrome against a real server: streaming transcript, identifier recovery
and revert, warm/idle sidecar cycling, stash fallback, and the unavailable path.
Second speaker (independent contributor) scored 91% with Parakeet vs 73% with
Apple, so the numbers are not fitted to one voice.
Unit tests cover the matcher (against real ASR transcripts committed as
fixtures, including one asserting the phrase collisions must fire), the
resampler and framer, and the engine's availability paths.
Desktop artifact builds compile and package the Rust sidecar unconditionally.
The fork nightly matrix builds those artifacts on macOS arm64, Linux x64, and
Windows x64; the Experimental setting remains the sole runtime gate.
Before this is usable day to day
T3CODE_DICTATION_MODEL_PATHordictationModelPathpointing at a Parakeet GGUF. The opt-in ~600MB downloadflow is unbuilt, so today this is a manual step.
(session permission handler,
t3code-dev://origin, no Vite proxy).Also unfinished
bubble panel; the server should advertise availability so the button can
disable ahead of time instead of connect-then-fail; visual pass with Handy/Amp
prior art; input device picker.
Notes for review
native/dictation-spike/is deliberate: it is throwaway code but the findingsare the justification for every decision above, and the harness reproduces
them. Corpus audio is gitignored (personal voice data); the ASR transcripts are
committed as regression fixtures.
(
skips the primary native probe for cross-architecture Windows payloads)also fails on a pristine
origin/mainand is unrelated.DictationControluses a rawcreatePortal, which is not the house overlaypattern; migrating it is part of the UX round.
Experimentalsettings section, so it sets the precedentfor whatever lands there next.
Written by an agent (T3 Code, gpt-5.6-sol).