Skip to content

perf(search): scan history by recency until N unique - #3553

Merged
ellie merged 1 commit into
atuinsh:mainfrom
jbellis:perf/dedup-early-termination
Jun 29, 2026
Merged

perf(search): scan history by recency until N unique#3553
ellie merged 1 commit into
atuinsh:mainfrom
jbellis:perf/dedup-early-termination

Conversation

@jbellis

@jbellis jbellis commented Jun 28, 2026

Copy link
Copy Markdown
Contributor

Inverts the ctrl-r search from "materialize all unique commands, then pick N most recent" to "scan commands by recency until we have N unique ones". Basically the entire change is the SQL query, the rest is scaffolding so we can put filters in the new subquery where they belong.

Measured on a 557k-row history (≈369k distinct commands), empty-query search goes from ~0.78s warm / ~18s cold to ~0.01s warm / ~0.3s cold.

Related PRs I saw:

  1. perf: deduplicate empty-query results in Rust instead of SQL #3549: directed at the same problem but breaks semantics as noted by the bot reviewer
  2. improve search performance by scanning with the timestamp index iteratively #2698: more code churn, looks like main idea is doing the slow path page-at-a-time, which is better than all-of-history but still worse in principle; corner cases are still bad

Both of the above also split the logic between a SQL query and Rust-side filtering; this one keeps the logic in SQL so the query engine can do its job efficiently.

Checks

  • I am happy for maintainers to push small adjustments to this PR, to speed up the review cycle
  • I have checked that there are no existing pull requests for the same thing

Invert the interactive dedup strategy. The old query materialized every
unique command and then took the N most recent of them:

    GROUP BY command HAVING max(timestamp) ORDER BY timestamp DESC LIMIT N

Because grouping must visit every row before ORDER BY ... LIMIT can apply,
each keystroke (and the initial ctrl-r paint) aggregated the entire history
table regardless of N, dominated by a full scan plus a temp B-tree over all
distinct commands.

Instead, scan rows newest-first and stop once N unique commands have been
collected. This is expressed as a correlated NOT EXISTS that keeps only the
most recent entry per command within the filtered set, wrapping the existing
filter query as a derived table so every filter mode stays correct without
duplicating predicate logic. Ordered by an indexed timestamp scan, SQLite
streams newest-first and terminates as soon as N distinct commands are
emitted.

The (timestamp, id) row-value comparison breaks timestamp ties (one row per
command, matching the previous behaviour) while remaining a sargable range
scan on the (command, timestamp) index, so histories with many duplicates
stay fast too.

Unlike a limit-then-dedup approach, dedup happens before the limit, so the
result set is unchanged: up to N unique commands, identical to the old query.

Measured on a 557k-row history (~369k distinct commands), empty-query
search: ~0.78s warm / ~18s cold (full cache eviction) -> ~0.01s warm /
~0.3s cold.
@github-actions

github-actions Bot commented Jun 28, 2026

Copy link
Copy Markdown
Contributor

Fossier: Vouched and Approved

@ellie vouched for @jbellis. Future PRs from this contributor will be trusted.

@greptile-apps

greptile-apps Bot commented Jun 28, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR changes SQLite history search to stop scanning once enough unique commands are found. The main changes are:

  • Moves filtering into an inner history query.
  • Applies deduplication, ordering, and pagination in an outer query.
  • Replaces command grouping with a NOT EXISTS check for the newest row per command.

Confidence Score: 5/5

This looks safe to merge.

  • No blocking issues found in the changed code.
  • The reviewed search paths keep filters inside the derived query and apply duplicate handling in the outer query as intended.

Important Files Changed

Filename Overview
crates/atuin-client/src/database.rs Refactors search SQL construction around a filtered subquery and outer dedup/order/limit logic.

Reviews (1): Last reviewed commit: "perf(search): scan history by recency un..." | Re-trigger Greptile

@ellie ellie left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good, thank you!! nice and neat 🙏

@ellie
ellie merged commit ef54a8b into atuinsh:main Jun 29, 2026
26 checks passed
@ellie

ellie commented Jun 29, 2026

Copy link
Copy Markdown
Member

/fossier vouch

taylordotfish pushed a commit that referenced this pull request Jul 9, 2026
### Bug Fixes

- *(ai)* Dispatch skills registered in the slash command registry
([#3593](#3593))
- *(ci)* Fossier install in scan workflow
([#3485](#3485))
- *(i18n)* Fix typos in Russian localization
([#3575](#3575))
- *(nu)* Use `char -u 1b` for ESC in OSC 133 sequences
([#3530](#3530))
- *(nu)* Suppress error when `ATUIN_HISTORY_ID` is missing in
`pre_prompt` hook
([#3587](#3587))
- *(pi)* Observe tool events instead of registering a bash tool
([#3557](#3557))
- *(pty-proxy)* Set `$SHELL` to the spawned shell
([#3548](#3548))
- *(search)* Fix terminal clearing with latest Ratatui
([#3578](#3578))
- *(sync)* Skip records that fail to decrypt or decode instead of
failing the whole store
([#3569](#3569))
- Atuin hangs when attempting to spawn daemon from Ctrl+R invocation
([#3502](#3502))
- Capture session ID from stream headers rather than final event
([#3531](#3531))
- Doctor resiliency fo runknown platforms + openbsd warning
([#3551](#3551))
- Double input on arrow keys in AI setup prompt on Windows
([#3552](#3552))
- Exclude AI agent commands from zsh-autosuggestions
([#3567](#3567))
- Silence shellcheck SC2016 on literal `$all-user` author filter
- Respect `store_failed` when using daemon
([#3571](#3571))


### Documentation

- Highlight `Ctrl-r` keybinding on docs page
([#3489](#3489))
- Document store purge workflow
([#3544](#3544))
- Fix command example typo in documentation
([#3536](#3536))
- Make commented-out lines in `config.toml` match real defaults
([#3583](#3583))
- Add fish shell cleanup step to uninstall instructions
([#3582](#3582))


### Features

- *(doctor)* Add whether daemon is enabled to `doctor` output
([#3572](#3572))
- *(pty-proxy)* Add `--shell` flag to override the spawned shell
([#3327](#3327))
- Setup fossier to stop bot slop prs
([#3482](#3482))
- Capture command output + expose to new `atuin_output` tool
([#3510](#3510))
- Cache user contexts on load until `/reload`
([#3525](#3525))
- Create database integration tests for atuin-server
([#3514](#3514))
- Add `/model` slash command for changing models
([#3576](#3576))
- Add mcp server for history tools and expand search filters
([#3581](#3581))
- Add status bar with model and usage information
([#3591](#3591))


### Miscellaneous Tasks

- *(rustdoc)* Fix Rustdoc warnings
([#3585](#3585))
- *(warnings)* Fix compile warnings with latest dependencies
([#3586](#3586))
- Vouch for all existing contributors
([#3486](#3486))
- Update GitHub app token format
- Update to Rust 1.96.1
([#3568](#3568))
- Adopt `derive_more` to reduce boilerplate across the codebase
([#3573](#3573))


### Performance

- *(search)* Scan history by recency until N unique
([#3553](#3553))
- Add `synchronous(Normal)` + `optimize_on_close` to record store SQLite
([#3577](#3577))
- Remove unnecessary clones in a hot path
([#3580](#3580))


### Refactor

- Implement `From<sqlx::Error>` and clean up `fix_error`
([#3484](#3484))
- Pull `fn into_utc` into `atuin-server-database` crate
([#3487](#3487))
@ellie ellie mentioned this pull request Jul 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants