feat(connectors): add SurrealDB sink connector#3453
Open
countradooku wants to merge 7 commits into
Open
Conversation
SurrealDB is a document database target for Iggy connector users, so the sink writes batches with deterministic record ids and bulk INSERT IGNORE to keep runtime redelivery idempotent without per-message round trips. Constraint: User explicitly requested the latest SurrealDB Rust SDK and chose to keep it despite BUSL-1.1 license-validation warnings for SurrealDB crates. Constraint: Local Docker daemon was unavailable, so real-container integration execution could not run here. Rejected: Per-message SDK writes | too many round trips and weaker batching throughput. Rejected: Using the testcontainers SurrealDB module | module source hardcodes an older SurrealDB image. Confidence: medium Scope-risk: moderate Directive: Keep record ids deterministic across releases; changing build_record_id breaks replay idempotency. Tested: cargo fmt --all; cargo sort --no-format --workspace; cargo clippy --all-features --all-targets -- -D warnings; cargo check --all --all-features; cargo test -p iggy_connector_surrealdb_sink; cargo test -p integration --no-run connectors::surrealdb; cargo test --locked --doc; cargo doc --no-deps --all-features --quiet; taplo/license/shellcheck/version/diff/binary checks; prek install Not-tested: Docker-backed SurrealDB integration execution, because Docker daemon was not running locally.
countradooku
force-pushed
the
feat/surrealdb-sink-connector
branch
from
7b8305a to
48c3a9d
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #3453 +/- ##
============================================
- Coverage 74.70% 72.48% -2.23%
Complexity 937 937
============================================
Files 1256 1255 -1
Lines 124863 123669 -1194
Branches 100585 99434 -1151
============================================
- Hits 93285 89640 -3645
- Misses 28578 31201 +2623
+ Partials 3000 2828 -172
🚀 New features to boost your workflow:
|
Contributor
|
/author |
github-actions
Bot
added
S-waiting-on-author
Contributor
|
Please check the pre-checks failure |
Contributor
Author
|
Sure |
HawkEye maps Rust files to the double-slash license style, so the block comments in the new SurrealDB connector files were treated as missing headers by CI. Constraint: CI runs the updated HawkEye-based license check with strict header matching. Confidence: high Scope-risk: narrow Tested: PATH=/opt/homebrew/bin:/Users/radudiaconu/.vite-plus/bin:/Users/radudiaconu/.codex/tmp/arg0/codex-arg0uTrL1r:/Users/radudiaconu/Library/pnpm/bin:/Users/radudiaconu/.opencode/bin:/opt/homebrew/opt/ruby/bin:/opt/homebrew/opt/ruby/bin:/opt/homebrew/opt/ruby/bin:/opt/homebrew/lib/ruby/gems/4.0.0/bin:/Users/radudiaconu/.local/bin:/Users/radudiaconu/Library/Application Support/Herd/bin/:/Users/radudiaconu/.bun/bin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/pkg/env/global/bin:/Library/Apple/usr/bin:/usr/local/share/dotnet:~/.dotnet/tools:/opt/homebrew/bin:/opt/zerobrew/bin:/Users/radudiaconu/.zerobrew/bin:/Users/radudiaconu/.cargo/bin:/Users/radudiaconu/Library/Application Support/JetBrains/Toolbox/scripts:/Users/radudiaconu/Library/Android/sdk/platform-tools:/Applications/Codex.app/Contents/Resources ./scripts/ci/license-headers.sh --check; cargo fmt --all --check; git diff --check
The Rust pre-merge machete job reported that the SurrealDB sink crate declared toml without using it. Removing the dev-dependency is simpler than adding an ignore entry. Constraint: CI runs cargo machete --with-metadata and fails on unused dependencies. Rejected: Add cargo-machete metadata ignore | the dependency is genuinely unused. Confidence: high Scope-risk: narrow Tested: cargo sort --no-format --workspace; cargo test -p iggy_connector_surrealdb_sink; PATH=/opt/homebrew/bin:/Users/radudiaconu/.vite-plus/bin:/Users/radudiaconu/.codex/tmp/arg0/codex-arg0uTrL1r:/Users/radudiaconu/Library/pnpm/bin:/Users/radudiaconu/.opencode/bin:/opt/homebrew/opt/ruby/bin:/opt/homebrew/opt/ruby/bin:/opt/homebrew/opt/ruby/bin:/opt/homebrew/lib/ruby/gems/4.0.0/bin:/Users/radudiaconu/.local/bin:/Users/radudiaconu/Library/Application Support/Herd/bin/:/Users/radudiaconu/.bun/bin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/pkg/env/global/bin:/Library/Apple/usr/bin:/usr/local/share/dotnet:~/.dotnet/tools:/opt/homebrew/bin:/opt/zerobrew/bin:/Users/radudiaconu/.zerobrew/bin:/Users/radudiaconu/.cargo/bin:/Users/radudiaconu/Library/Application Support/JetBrains/Toolbox/scripts:/Users/radudiaconu/Library/Android/sdk/platform-tools:/Applications/Codex.app/Contents/Resources ./scripts/ci/license-headers.sh --check; cargo fmt --all --check; git diff --check; cargo metadata confirms toml is absent from iggy_connector_surrealdb_sink
Contributor
Author
|
/ready |
github-actions
Bot
added
S-waiting-on-review
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a SurrealDB sink connector for writing Iggy messages into SurrealDB using the latest SurrealDB Rust SDK version available during implementation,
3.1.4.The connector supports deterministic record IDs, bulk
INSERT IGNOREwrites for idempotent replay, configurable batch sizing, root/namespace/database/no-auth modes, optional table and offset-index definition, payload modes (auto,json,text,base64), metadata/header/checksum/origin timestamp fields, retry/backoff handling, and runtime metrics logging.This also wires the connector into workspace membership, connector docs, example runtime config, binary artifact builds, edge-release output, version bump scripts, and Docker-backed integration test scaffolding.
Tests
cargo fmt --allcargo sort --no-format --workspacecargo clippy --all-features --all-targets -- -D warningscargo check --all --all-featurescargo test -p iggy_connector_surrealdb_sinkcargo test -p integration --no-run connectors::surrealdbcargo test --locked --doccargo doc --no-deps --all-features --quiet./scripts/ci/taplo.sh./scripts/ci/license-headers.sh./scripts/ci/shellcheck.sh./scripts/ci/binary-artifacts.sh --check./scripts/extract-version.sh --checkgit diff --checkprek installKnown Review Items
./scripts/ci/third-party-licenses.sh --validate --manifest core/connectors/sinks/surrealdb_sink/Cargo.tomlreports BUSL-1.1 license failures from SurrealDB SDK crates. This PR intentionally keeps the SDK dependency because the connector targets the latest SurrealDB SDK.