[c] Support vector search materialized read in C FFI bindings#537
Conversation
a2e6b72 to
7097120
Compare
JunRuiLee
changed the title
…e_read Extend VectorSearchBuilder::execute_read() to materialize rows for data-evolution (global-index) vector search, matching the primary-key path, so both storage types return projected user columns plus a unified score column. - Unify the user-visible score column to __paimon_search_score (constant SEARCH_SCORE_COLUMN), matching the engine metadata column name. Reject _ROW_ID, _PKEY_VECTOR_POSITION, __paimon_search_score, and the legacy _PKEY_VECTOR_SCORE alias as read projections. - Add attach_scores_by_row_id: join materialized rows to their (rank, score) by global _ROW_ID, reorder to the search's best-first rank (not re-sorted by score), append the score column, and strip _ROW_ID. Fail loud on missing / null / wrong-type / unknown _ROW_ID or a row count that disagrees with the scored result; never emit NaN or silently drop a row. - Add the data-evolution branch to execute_read: run the scored search, build the row-range read exposing _ROW_ID, materialize, and attach scores. Validate ids fit i64 (via to_row_ranges) before building the score map. A filter stays unsupported here and fails loud inside the scored search; the primary-key filter path is unchanged. Covers rank-order fidelity, row-id alignment, and the fail-loud branches with unit and end-to-end tests.
Add a C FFI vector-search surface over the Rust VectorSearchBuilder, so C/C++/Go consumers can run a vector search and read back the matched rows. - Builder handle over owned state (table + accumulated query params) plus setters for vector column, query vector, limit, options, and an optional scalar filter. The query vector and options are copied at the setter; the filter is consumed, matching the read-builder contract. - A single terminal, paimon_vector_search_builder_execute_read, runs the search and returns the existing streaming record-batch reader (paimon_result_record_batch_reader); consumers drain it with the existing reader-next / arrow-batch-free / reader-free functions. The entry is storage-agnostic and does no table-type routing — the core decides. Errors (including a filter on the append / data-evolution path) surface through the result's error field; an empty result is a normal end-of-stream reader. - Compile-time ABI signature guards pin every exported symbol. Integration tests compare the C stream against an independent Rust reference across primary-key and data-evolution tables, cover a primary-key filter that excludes a neighbor, the data-evolution filter error, setter validation, and the empty end-of-stream case.
7097120 to
f264394
Compare
leaves12138
left a comment
leaves12138
left a comment
There was a problem hiding this comment.
The focused core and C FFI tests pass locally, but I found two correctness/API issues in the new data-evolution materialized-read path: default projection can produce duplicate reserved score columns, and invalid vector-column names can silently return an empty reader. Please address these before merge.
| for name in names { | ||
| if name == PKEY_VECTOR_POSITION_COLUMN | ||
| || name == PKEY_VECTOR_SCORE_COLUMN | ||
| || name == SEARCH_SCORE_COLUMN |
There was a problem hiding this comment.
This reserved-name validation only runs for Some(names). With the default projection, a valid table column named __paimon_search_score is retained, and attach_scores_by_row_id then appends another field with the same name. I reproduced an output schema containing two __paimon_search_score columns. The legacy _PKEY_VECTOR_SCORE alias is likewise rejected only for explicit projections. Please validate the resolved/default field list as well, and add default-projection collision coverage for both PK and DE materialization.
There was a problem hiding this comment.
Good catch — fixed in e3dbea0. resolve_materialize_read_type now runs the reserved-name check on the resolved field list (both the explicit-projection and default cases), so a user column literally named __paimon_search_score / _PKEY_VECTOR_SCORE is rejected before it can collide with the appended score column. The raw-request check is kept for the explicit-projection path so a requested reserved name still gets the clear "reserved column" error rather than a confusing "not found" from field resolution. Added default-projection collision coverage for both the primary-key and data-evolution materialization paths.
| // Data-evolution (global-index) vector search: materialize rows from the | ||
| // scored global row-ids and attach the unified score column. A non-vector | ||
| // column or a set filter fails loud inside execute_scored below. | ||
| self.execute_de_vector_read().await |
There was a problem hiding this comment.
This fallback can turn an invalid vector-column name into a successful empty reader. evaluate_batch_vector_search returns an empty result when field lookup fails, and the new test explicitly expects with_vector_column("other") on the empty table to succeed. Through the C API, a typo therefore looks like normal EOF instead of an input error, contrary to the execute_read fail-loud contract. Please validate that the DE target field exists and has a supported vector type before returning an empty result, with core/C coverage for unknown and scalar columns.
There was a problem hiding this comment.
Fixed in e3dbea0. execute_read on the data-evolution path now validates up front that the target column exists and is a FLOAT vector column (ARRAY<FLOAT> or VECTOR<FLOAT>) before doing any search, so an unknown, scalar, or non-float (e.g. ARRAY<INT>) column fails loud with InvalidInput instead of surfacing as an empty EOF reader through the C API. execute_scored's existing empty-on-unknown-field behavior is left unchanged. Added core coverage (unknown / scalar / non-float-array) and C FFI coverage (unknown and scalar columns mapping to InvalidInput).
…ed vector read Address two review findings on the materialized read path: - The reserved output-column-name check ran only for an explicit projection, so a user table column literally named __paimon_search_score (or the legacy _PKEY_VECTOR_SCORE alias) survived the default projection and collided with the score column appended during materialization, producing two identically named columns. The check now also runs on the resolved field list, covering the default projection, for both primary-key and data-evolution reads. - The data-evolution execute_read fell through to a search that returns an empty result for an unknown field, so a typo'd or scalar vector column looked like a normal empty (EOF) read instead of an input error — violating execute_read's fail-loud contract, especially through the C API. execute_read now validates up front that the target column exists and is a FLOAT vector column (ARRAY<FLOAT> or VECTOR<FLOAT>); unknown, scalar, and non-float-element columns fail loud. Tests: default-projection reserved-name collision (PK and DE); unknown, scalar, and ARRAY<INT> columns fail loud on the core execute_read; unknown and scalar columns fail loud through the C API.
d9530de to
e3dbea0
Compare
leaves12138
left a comment
leaves12138
left a comment
There was a problem hiding this comment.
Re-reviewed the follow-up fixes. Reserved-column validation now covers the resolved default projection, invalid data-evolution vector targets fail loudly with appropriate input errors, and the focused core/C FFI regression tests pass locally. Looks good to me.
3 participants
Purpose
Part of #514. Expose vector search to C/C++/Go consumers (e.g. Doris) via
paimon-c. Primary-key vector search produces materialized rows (physical(file, position)coordinates, no global row ids), so the unified user-facing shape across storage types is materialized rows + a score column — matching Java Spark, where both primary-key and data-evolution reads surface rows plus a__paimon_search_scoremetadata column. This PR wires that through:execute_read()gains a data-evolution branch, and a single C entry returns the existing Arrow-stream reader.Brief change log
feat(table):VectorSearchBuilder::execute_read()now materializes rows for both primary-key and data-evolution vector search. Unified user-visible score column__paimon_search_score(was PK-only_PKEY_VECTOR_SCORE, kept as a rejected projection alias). Data-evolution rows are joined to their scores by global_ROW_ID, reordered to the search's best-first rank (not re-sorted by score), then_ROW_IDis stripped; any missing / null / wrong-type / unknown id, or a row count that disagrees with the scored result, fails loud (never a NaN score or a silently dropped row). A filter on the data-evolution path fails loud (parity is a follow-up); the primary-key filter path is unchanged.feat(c): newbindings/c/src/vector_search.rs— a builder handle over owned state, setters (with_vector_column/with_query_vector/with_limit/with_options/with_filter), and a single terminalpaimon_vector_search_builder_execute_readreturningpaimon_result_record_batch_reader(reuses the existing Arrow C Data Interface reader; consumers drain it with the existingpaimon_record_batch_reader_next/paimon_arrow_batch_free/paimon_record_batch_reader_free). The entry is storage-agnostic — C does no table-type routing; the core decides. Errors surface via the result'serrorfield; an empty result is a normal end-of-stream reader. Compile-time ABI signature guards pin every exported symbol.Tests
_ROW_ID, count mismatch); reserved-projection rejection.execute_read()reference across primary-key and data-evolution tables; a primary-key filter that excludes a neighbor; the data-evolution filterInvalidInputerror; setter validation; and the empty end-of-stream case.cargo test -p paimonandcargo test -p paimon-cgreen;cargo build -p paimon-datafusionclean (cross-crateSendgate);cargo clippy -D warningsandcargo fmt --checkclean.API and Format
No on-disk format change. New C ABI symbols only (add-only, each guarded by a compile-time signature assertion). This replaces the earlier scored (row-ids + scores) C entry, which was never released.
Documentation
Doc comments on the new FFI functions; no user-facing docs change.