bug fix for index miss on dolt_diff_* tables

[zachmu]

No description provided.

[github-actions]

	Main	PR
covering_index_scan_postgres	2182.93/s	2185.65/s	+0.1%
groupby_scan_postgres	155.88/s	153.96/s	-1.3%
index_join_postgres	714.46/s	708.13/s	-0.9%
index_join_scan_postgres	933.70/s	917.59/s	-1.8%
index_scan_postgres	33.83/s	33.46/s	-1.1%
oltp_delete_insert_postgres	913.72/s	915.48/s	+0.1%
oltp_insert	822.23/s	786.07/s	-4.4%
oltp_point_select	3773.98/s	3739.04/s	-1.0%
oltp_read_only	3673.95/s	3651.30/s	-0.7%
oltp_read_write	2696.03/s	2717.69/s	+0.8%
oltp_update_index	838.64/s	842.62/s	+0.4%
oltp_update_non_index	930.68/s	916.38/s	-1.6%
oltp_write_only	2014.63/s	1960.79/s	-2.7%
select_random_points	2240.58/s	2202.76/s	-1.7%
select_random_ranges	1705.06/s	1701.68/s	-0.2%
table_scan_postgres	33.70/s	33.43/s	-0.9%
types_delete_insert_postgres	890.97/s	894.41/s	+0.3%
types_table_scan_postgres	14.66/s	14.68/s	+0.1%

[itoqa]

Commit: 29e2488: 14 test cases ran, 5 failed ❌, 9 passed ✅.

Summary

The run covers core committed-change retrieval, boundary and endpoint filtering, index use, type conversion safety, rollback behavior, and invalid or unusual membership filters. Happy-path and safety behaviors are healthy, but ordering comparisons lose efficient history access and several valid membership queries fail during planning rather than returning results.

Not safe to merge yet — this PR introduces multiple user-visible query failures across IN and NOT IN filtering, along with a broad performance regression for ordered commit filters. These are concentrated in important diff-query behavior and require correction before merging.

Tests run by Ito

Result	Severity	Type	Description
❌		Binary	The filters returned the expected boundary rows, but their query plans did not use the commit index. Each plan used a filter over the diff table without IndexedTableAccess or a commit range.
❌		Membership	The filtered query shows a planner error and returns no results, even though the same diff table returns the expected modified and added rows without the IN filter.
❌		Membership	The diff table showed the expected baseline modified and added rows, including a NULL previous-row ID for the added row. Adding the valid NOT IN filter caused both EXPLAIN and the query to fail with plan is not resolved because of node '*plan.Project', so excluded commits and NULL handling could not be evaluated.
❌		Membership	Duplicate IN, NULL-member IN, duplicate NOT IN, and empty-membership queries all failed instead of producing SQL-consistent rows or an empty result.
❌		Membership	The diff rows were available without a filter, but adding any tested NOT IN exclusion caused a planner error instead of returning the rows prescribed by SQL three-valued logic.
✅	—	Binary	Filtering the change history to the current commit returned only the expected modified row: id 1 changed to value 3.
✅	—	Binary	Filtering by the earlier commit and filtering by the later commit each returned the right change. Each filter used its matching index, and the results matched the non-indexed checks.
✅	—	Cast	Filtering a committed change by its target commit uses the expected index and returns the modified row.
✅	—	Cast	Nested casts and function-wrapped casts did not become column targets, while a simple literal comparison still returned the correct row through the safe indexed path.
✅	—	Cast	Invalid value conversions returned errors without showing partial diff rows, and the database session stayed usable for later queries.
✅	—	Cast	Literal HEAD, HASHOF('HEAD'), and a commit-derived hash returned the same changed row and used the same to_commit index range.
✅	—	Diff	The diff query returned the expected added row and modified row for the two adjacent commit pairs.
✅	—	Diff	Filtering by either commit endpoint returns only the matching history row and uses the matching index.
✅	—	Diff	Reversing the two commit endpoints returned no rows, while the same commits in time order returned the expected changed row.

Tip

Reply with @itoqa to send us feedback on this test run.

[github-actions]

	Main	PR
Total	42090	42090
Successful	18406	18406
Failures	23684	23684
Partial Successes1	5327	5327

	Main	PR
Successful	43.7301%	43.7301%
Failures	56.2699%	56.2699%

Footnotes

1. These are tests that we're marking as Successful, however they do not match the expected output in some way. This is due to small differences, such as different wording on the error messages, or the column names being incorrect while the data itself is correct. ↩

[Hydrocharged]

LGTM!

[itoqa]

Ito Diff Report — 29e2488 → 3850270: 24 test cases ran, 3 new failures ❌, 5 fixed ✅, 16 passing ✅.

Diff Summary

The run covers database query behavior across filtering, casting, existence checks, transaction handling, error recovery, concurrency, and numeric boundaries, including normal flows and edge cases. Most exercised behavior remains correct, including returned data, connection recovery, transaction outcomes, and error handling, but some correlated existence queries retain unfinished execution plans instead of the intended optimized joins.

Merge with caution — the PR introduces several related medium-severity planning failures: correlated existence queries still return correct results but miss required join transformations, creating a performance risk for affected queries. No data correctness or recovery failures were found, but the concentrated planner regressions should be addressed before treating the change as fully safe; unrelated evidence issues are only flag-for-later caveats.

Tests run by Ito

Result	State	Severity	Type	Description
❌	❌ New Failure		Exists	The query returned the expected matching rows, but EXPLAIN kept an Exists Subquery node. The repository test expects a correlated EXISTS expression to be decorrelated into a SemiJoin before the final boolean cast.
❌	❌ New Failure		Exists	The query returned the expected true and false values, but its plan did not finish transforming the nested, correlated, and negated checks into the join forms required by the database.
❌	❌ New Failure		Exists	The query returned the expected three rows and boolean values, but the plan still showed an Exists Subquery. The expected behavior is for correlated checks to be rewritten into a completed join plan before the final boolean cast.
✅	❌->✅ Fixed	—	Binary	All four ordering filters returned the expected changed row, and the query plan used the indexed column with the correct range.
✅	❌->✅ Fixed	—	Membership	The query used the commit index and returned only the two requested commit groups, with rows 4 and 5 and no unrelated history.
✅	❌->✅ Fixed	—	Membership	The query left out both selected commit groups and kept NULL values for the old row ID on added rows.
✅	❌->✅ Fixed	—	Membership	Duplicate commit values returned each matching row once. NULL and empty membership checks followed normal SQL rules, including zero matches for empty IN and all five rows for empty NOT IN.
✅	❌->✅ Fixed	—	Membership	The query excluded the requested commits and matched the equivalent filter in every case tested. NULL followed normal SQL rules, and the non-NULL case returned rows 1, 2, and 3.
✅	Passing	—	Cast	Filtering a diff by the converted commit value succeeds and returns the requested HEAD row.
✅	Passing	—	Cast	Nested casts and casts around values or functions kept their normal expression behavior. The queries returned the expected rows without treating unsafe expressions as index targets.
✅	Passing	—	Cast	An invalid value was rejected cleanly, and the same connection returned the expected row and count afterward.
✅	Passing	—	Cast	The symbolic commit value and the value returned by a commit operation produced the same query plan and returned the same row.
✅	Passing	—	Exists	A direct EXISTS query returned true when a row was present and false when no row was present. The query plan also showed both results as BOOLEAN.
✅	Passing	—	Exists	Filtering an aliased diff table by its commit value returned the same three WORKING rows as the unaliased query.
✅	Passing	—	Isolation	The database accepted READ UNCOMMITTED, READ COMMITTED, REPEATABLE READ, SERIALIZABLE, and SNAPSHOT transactions. Each transaction completed, committed its insert, and the final table contained all six rows.
✅	Passing	—	Isolation	Read-only transactions reject writes, while read-write transactions can insert and commit rows. The earlier failure came from checking an unsupported command and an unavailable local target, not from the supported transaction flow.
✅	Passing	—	Isolation	A SNAPSHOT transaction started successfully and reported read committed behavior, as expected for this product. Invalid isolation syntax was rejected safely, and the same connection then inserted and read back row 9.
✅	Passing	—	Sqlstate	The database returned all seven expected PostgreSQL error codes, and each failed statement was followed by a successful query on the same connection.
✅	Passing	—	Sqlstate	An invalid number conversion returned the expected error, and later direct, extended, and prepared conversions succeeded on the same database session.
✅	Passing	—	Sqlstate	The database accepts the largest supported decimal LIMIT and OFFSET values, rejects values above the int64 range with a clear error, and continues to handle ordinary LIMIT queries.
✅	Passing	—	Sqlstate	Eight separate database connections rejected invalid number conversions with the expected error, then completed all valid conversions and ordinary queries successfully.
✅	Passing	—	Transaction	Prepared statements sent through the database protocol commit together at Sync, roll back the whole batch after an error, and allow later work to continue.
✅	Passing	—	Transaction	After one prepared request failed, the server ignored the remaining work in that batch, recovered at Sync, and ran the next valid insert exactly once.
✅	Passing	—	Transaction	After a statement error, rolling back to the savepoint restored the transaction. The earlier row stayed available, the transaction committed, and a new connection could read it.
⏸️	Skipped	—	Binary	Filtering the change history to the current commit returned only the expected modified row: id 1 changed to value 3.
⏸️	Skipped	—	Binary	Filtering by the earlier commit and filtering by the later commit each returned the right change. Each filter used its matching index, and the results matched the non-indexed checks.
⏸️	Skipped	—	Diff	The diff query returned the expected added row and modified row for the two adjacent commit pairs.
⏸️	Skipped	—	Diff	Filtering by either commit endpoint returns only the matching history row and uses the matching index.
⏸️	Skipped	—	Diff	Reversing the two commit endpoints returned no rows, while the same commits in time order returned the expected changed row.

Tip

Reply with @itoqa to send us feedback on this test run.