atuin history gc?

[AlJohri]

I seem to have accidentally re-run atuin import zsh after already using atuin for a period of time. I installed atuin on a new computer and kept syncing and was getting 0 results. I wasn't sure what was going on until I realized that the new installation was using the sync v2. I have since upgraded all installations to use sync v2, however as part of trying to figure out what was going on, I had re-run the atuin import zsh command and then run atuin sync. This appears to have doubled the size of my history from 150k records to 300k records.

Is there anyway I can "garbage collect" down to a list of unique entries and clean up my atuin history?

[AlJohri]

It seems like its taking almost 0.5 to 1 second when I press CTRL-R (not sure how to check exactly). Would love to figure out some way to trim some of this history down or speed up the TUI so CTRL-R is instantaneous.

As a proxy for timing, it takes about 5 seconds to run atuin history list:

❯ atuin history list | wc -l
366974

❯ time atuin history list &> /dev/null
atuin history list &> /dev/null  4.87s user 2.81s system 172% cpu 4.448 total

[AlJohri]

Looks like my slowdown is likely related to getting the unique set of commands: See #475

[zuzzurro]

I have the same issue. I am actually running "atuin import zsh" multiple times in order to incrementally save my zsh history before I actually really start using atuin and commands are added multiple times. Some sort of deduplication based on the time and command would be extremely appreciated.

[zuzzurro]

Any feedback on this ticket at all?

[ellie]

@AlJohri in response to your issue, I'd suggest trying to craft a atuin search --delete that specifically targets your newly-imported data. You should be able to delete all records after a certain date.

If that is not possible, then I'm afraid the only real way forwards would be to delete and start from scratch :/

Due to how the importers work at the moment, it's not that straightforward to correctly identify duplicate imports vs commands that have been ran a few times.

Otherwise, if you're able to profile the startup/searching with your data, I'd really appreciate it! You may find that the prefix search mode performs much faster.

I am actually running "atuin import zsh" multiple times in order to incrementally save my zsh history before I actually really start using atuin and commands are added multiple times.

@zuzzurro I'm afraid this use case isn't supported. Importing is for migrating old data to Atuin, and not for keeping Atuin up-to-date. Given that the original data soruce doesn't always have enough information to effectively de-dupe, it's not that straightforward.

Your request isn't quite the same as the OP, so I'd appreciate you opening a new issue if there's anything you wish to have help with.

[zuzzurro]

I asked before trying to solve the issue myself....
But let me ask you one question. In the zsh import scenario spotting duplicate records should not be that hard since history items come with a time? So if the cmd is the same and the time is the same, why importing it twice? I may be also used to the fact that in zsh I have the "histignorealldups" that in atuin doesn't exist if I'm not wrong...

[ellie]

@zuzzurro #2290

[zuzzurro]

Thanks. As I said my personal perspective is probably skewed by the fact that in zsh I don't care about saving multiple copies of the same command... so no hard feelings at all if it nobody else cares about it.

[AlJohri]

If that is not possible, then I'm afraid the only real way forwards would be to delete and start from scratch :/

Due to how the importers work at the moment, it's not that straightforward to correctly identify duplicate imports vs commands that have been ran a few times.

What if we ignored the distinction between commands that have been run a few times and duplicate imports and deduplicated it anyway? That way I still have at least 1 copy of every command I have ever run accessible through prefix or exact match search. And I can build up the more frequent commands again over time into the history.

Is this possible to do? Are there any implications with doing this that I am missing?

[AlJohri]

Otherwise, if you're able to profile the startup/searching with your data, I'd really appreciate it! You may find that the prefix search mode performs much faster.

Is there a guide on profiling atuin I can follow? Or some high level steps.

[emilsenan]

For me I solved the issue (entries duplicated by import) with following quick and dirty hack:

• clone the project
• replace the body of the search function with an SQL statement to select all duplicates
• build the project
• cargo run -- search YOLO --delete

Here's the corresponding patch if someone ever need it :)

[lilydjwg]

I've made a dedup subcommand like this:

$ atuin history dedup --help
Delete duplicate history entries (that have the same command, cwd and hostname)

Usage: atuin history dedup [OPTIONS] --before <BEFORE> --dupkeep <DUPKEEP>

Options:
  -n, --dry-run            List matching history lines without performing the actual deletion
  -b, --before <BEFORE>    Only delete results added before this date
      --dupkeep <DUPKEEP>  How many recent duplicates to keep
  -h, --help               Print help

This can be used to remove too many duplicates as they are not very useful but slow things down. (I'm making another change to improve performance dramatically when there aren't too many duplicates.)

[zuzzurro]

Great news! I'm going to try this ASAP and report here..