Order MongoDB chat memory by message sequence - #6720
Open
MacAlsandair wants to merge 1 commit into
Open
Conversation
Ordering relied on the timestamp field alone. MongoDB stores Instant with millisecond precision and saveAll() writes the whole conversation in one batch, so all of its messages get the same timestamp, and a sort on duplicate keys has no defined order. Store each message's position in a sequence field and sort on that, as JdbcChatMemoryRepository does. The field is nullable and timestamp remains the secondary key, so documents written earlier keep the order they had and need no migration. See spring-projects#4740 Signed-off-by: Alexander Makarov <alexander.makarov@nightsong.li>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
MongoChatMemoryRepositoryorders a conversation by itstimestampfield alone. MongoDB storesjava.time.Instantas a BSON date with millisecond precision, andsaveAll()writes the whole conversation in a single batch, so all of its messages end up with the same timestamp.Sorting on duplicate keys has no defined order. From the manual: "When sorting on a field which contains duplicate values, documents containing those values may be returned in any order [...] If consistent sort order is desired, include at least one field in your sort that contains unique values." So a conversation can come back with its turns out of order, and since every save rewrites the whole conversation, a bad read is written straight back.
Checked against mongo:8.0.6 by replaying the current write path with 20 messages: 20 distinct
Instants go in, differing in nanoseconds, and 1 distinct value comes back out, on every run.Every other repository keeps an explicit ordering key — JDBC
sequence_id, Neo4jidx, Redis a counter, Cassandra a native list. This adds the same for MongoDB: each document stores its position in asequencefield, and reads sort on it. The equivalent problem was fixed for JDBC in #4740.Existing data is unaffected.
sequenceis nullable andtimestampremains the secondary sort key, so documents written by earlier versions all tie onsequenceand keep the exact ordering they had; they get real sequence numbers on their next save. No migration is required, and the upgrade notes cover an optional backfill.The auto-configured main index becomes
conversationId,sequence,timestampso the sort stays index-covered.Verified with
MongoChatMemoryRepositoryIT(13 tests) andMongoChatMemoryAutoConfigurationIT(2 tests), including one that reads raw pre-upgrade documents to pin the compatibility behaviour.