perf(search): scan history by recency until N unique - #3553
Merged
Conversation
Invert the interactive dedup strategy. The old query materialized every
unique command and then took the N most recent of them:
GROUP BY command HAVING max(timestamp) ORDER BY timestamp DESC LIMIT N
Because grouping must visit every row before ORDER BY ... LIMIT can apply,
each keystroke (and the initial ctrl-r paint) aggregated the entire history
table regardless of N, dominated by a full scan plus a temp B-tree over all
distinct commands.
Instead, scan rows newest-first and stop once N unique commands have been
collected. This is expressed as a correlated NOT EXISTS that keeps only the
most recent entry per command within the filtered set, wrapping the existing
filter query as a derived table so every filter mode stays correct without
duplicating predicate logic. Ordered by an indexed timestamp scan, SQLite
streams newest-first and terminates as soon as N distinct commands are
emitted.
The (timestamp, id) row-value comparison breaks timestamp ties (one row per
command, matching the previous behaviour) while remaining a sargable range
scan on the (command, timestamp) index, so histories with many duplicates
stay fast too.
Unlike a limit-then-dedup approach, dedup happens before the limit, so the
result set is unchanged: up to N unique commands, identical to the old query.
Measured on a 557k-row history (~369k distinct commands), empty-query
search: ~0.78s warm / ~18s cold (full cache eviction) -> ~0.01s warm /
~0.3s cold.
Contributor
Fossier: Vouched and Approved
|
Contributor
Greptile SummaryThis PR changes SQLite history search to stop scanning once enough unique commands are found. The main changes are:
Confidence Score: 5/5This looks safe to merge.
Important Files Changed
Reviews (1): Last reviewed commit: "perf(search): scan history by recency un..." | Re-trigger Greptile |
ellie
approved these changes
Jun 29, 2026
ellie
left a comment
Member
There was a problem hiding this comment.
looks good, thank you!! nice and neat 🙏
Member
|
/fossier vouch |
This was referenced Jun 29, 2026
taylordotfish
pushed a commit
that referenced
this pull request
Jul 9, 2026
### Bug Fixes - *(ai)* Dispatch skills registered in the slash command registry ([#3593](#3593)) - *(ci)* Fossier install in scan workflow ([#3485](#3485)) - *(i18n)* Fix typos in Russian localization ([#3575](#3575)) - *(nu)* Use `char -u 1b` for ESC in OSC 133 sequences ([#3530](#3530)) - *(nu)* Suppress error when `ATUIN_HISTORY_ID` is missing in `pre_prompt` hook ([#3587](#3587)) - *(pi)* Observe tool events instead of registering a bash tool ([#3557](#3557)) - *(pty-proxy)* Set `$SHELL` to the spawned shell ([#3548](#3548)) - *(search)* Fix terminal clearing with latest Ratatui ([#3578](#3578)) - *(sync)* Skip records that fail to decrypt or decode instead of failing the whole store ([#3569](#3569)) - Atuin hangs when attempting to spawn daemon from Ctrl+R invocation ([#3502](#3502)) - Capture session ID from stream headers rather than final event ([#3531](#3531)) - Doctor resiliency fo runknown platforms + openbsd warning ([#3551](#3551)) - Double input on arrow keys in AI setup prompt on Windows ([#3552](#3552)) - Exclude AI agent commands from zsh-autosuggestions ([#3567](#3567)) - Silence shellcheck SC2016 on literal `$all-user` author filter - Respect `store_failed` when using daemon ([#3571](#3571)) ### Documentation - Highlight `Ctrl-r` keybinding on docs page ([#3489](#3489)) - Document store purge workflow ([#3544](#3544)) - Fix command example typo in documentation ([#3536](#3536)) - Make commented-out lines in `config.toml` match real defaults ([#3583](#3583)) - Add fish shell cleanup step to uninstall instructions ([#3582](#3582)) ### Features - *(doctor)* Add whether daemon is enabled to `doctor` output ([#3572](#3572)) - *(pty-proxy)* Add `--shell` flag to override the spawned shell ([#3327](#3327)) - Setup fossier to stop bot slop prs ([#3482](#3482)) - Capture command output + expose to new `atuin_output` tool ([#3510](#3510)) - Cache user contexts on load until `/reload` ([#3525](#3525)) - Create database integration tests for atuin-server ([#3514](#3514)) - Add `/model` slash command for changing models ([#3576](#3576)) - Add mcp server for history tools and expand search filters ([#3581](#3581)) - Add status bar with model and usage information ([#3591](#3591)) ### Miscellaneous Tasks - *(rustdoc)* Fix Rustdoc warnings ([#3585](#3585)) - *(warnings)* Fix compile warnings with latest dependencies ([#3586](#3586)) - Vouch for all existing contributors ([#3486](#3486)) - Update GitHub app token format - Update to Rust 1.96.1 ([#3568](#3568)) - Adopt `derive_more` to reduce boilerplate across the codebase ([#3573](#3573)) ### Performance - *(search)* Scan history by recency until N unique ([#3553](#3553)) - Add `synchronous(Normal)` + `optimize_on_close` to record store SQLite ([#3577](#3577)) - Remove unnecessary clones in a hot path ([#3580](#3580)) ### Refactor - Implement `From<sqlx::Error>` and clean up `fix_error` ([#3484](#3484)) - Pull `fn into_utc` into `atuin-server-database` crate ([#3487](#3487))
1 task
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Inverts the ctrl-r search from "materialize all unique commands, then pick N most recent" to "scan commands by recency until we have N unique ones". Basically the entire change is the SQL query, the rest is scaffolding so we can put filters in the new subquery where they belong.
Measured on a 557k-row history (≈369k distinct commands), empty-query search goes from ~0.78s warm / ~18s cold to ~0.01s warm / ~0.3s cold.
Related PRs I saw:
Both of the above also split the logic between a SQL query and Rust-side filtering; this one keeps the logic in SQL so the query engine can do its job efficiently.
Checks