Skip to content

perf(persistence): bound JDBC hasChildren existence check with LIMIT#4973

Merged
dimas-b merged 1 commit into
apache:mainfrom
anxkhn:perf/jdbc-haschildren-limit
Jul 16, 2026
Merged

perf(persistence): bound JDBC hasChildren existence check with LIMIT#4973
dimas-b merged 1 commit into
apache:mainfrom
anxkhn:perf/jdbc-haschildren-limit

Conversation

@anxkhn

@anxkhn anxkhn commented Jul 5, 2026

Copy link
Copy Markdown
Contributor

JdbcBasePersistenceImpl.hasChildren only needs to know whether a parent entity
has at least one child, but it does far more work than that. It builds a SELECT
over every entity column with no row limit, then hands the result to
DatasourceOperations.executeSelect, which collects the entire ResultSet into
a list (stream.forEach(results::add)) before the method simply returns
results != null && !results.isEmpty(). Every matching child row is therefore
fetched, transferred over the wire, and deserialized in full, including the
potentially large properties and internal_properties JSON columns, only to be
discarded.

This path runs when dropping a catalog or a namespace
(AtomicOperationMetaStoreManager.hasChildren). When dropping a namespace the
check matches all child types (tables and views included), so dropping a
namespace that holds many tables loads every one of them into memory just to
decide NAMESPACE_NOT_EMPTY. That adds needless latency and allocation and risks
OutOfMemoryError on large namespaces. The JDBC layer already has a bounded
existence-check pattern next door: QueryGenerator.generateEntityTableExistQuery
uses LIMIT 1 for the table-level check.

This change bounds the existence check:

  • Add a LIMIT-aware generateSelectQuery(projections, tableName, whereClause, int limit)
    overload in QueryGenerator. It mirrors the existing 4-arg public overload and
    threads an optional limit through a new private builder, so every existing caller
    keeps emitting no LIMIT (the previous private builder now delegates with null).
    The LIMIT is appended after any ORDER BY, which is valid clause order for all
    supported backends. The shared builder rejects non-positive limits.
  • Call it from hasChildren with a limit of 1. Behavior is unchanged: with at
    most one row returned, !results.isEmpty() still means "has at least one child",
    so the method still returns true when a child exists and false otherwise.

LIMIT is standard SQL accepted by all supported backends (PostgreSQL,
CockroachDB, H2). The projection is intentionally left unchanged: ModelEntity
reads every column when converting a row, so narrowing the projection would need
a separate converter and a larger change. LIMIT 1 alone removes the full scan,
transfer, and materialization.

I kept the projection full and the interface surface additive rather than changing
any existing method signature, per the project's evolution guidelines.

Checklist

  • 🛡️ Don't disclose security issues! (contact security@apache.org)
  • 🔗 Clearly explained why the changes are needed, or linked related issues
  • 🧪 Added/updated tests with good coverage, or manually tested (and explained how)
  • 💡 Added comments for complex logic
  • 🧾 Updated CHANGELOG.md (if needed)
  • 📚 Updated documentation in site/content/in-dev/unreleased (if needed)

Copilot AI review requested due to automatic review settings July 5, 2026 09:57
@github-project-automation github-project-automation Bot moved this to PRs In Progress in Basic Kanban Board Jul 5, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR improves performance in the relational JDBC persistence layer by turning an unbounded “fetch all children rows” existence check into a bounded query using LIMIT 1, reducing unnecessary row transfer/materialization when determining whether a parent entity has any children.

Changes:

  • Add a new QueryGenerator.generateSelectQuery(..., int limit) overload to append a LIMIT clause for bounded existence checks.
  • Update JdbcBasePersistenceImpl.hasChildren to use the new overload with limit = 1.
  • Add a unit test asserting the generated SQL includes LIMIT 1 and preserves parameter binding order.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File Description
persistence/relational-jdbc/src/main/java/org/apache/polaris/persistence/relational/jdbc/QueryGenerator.java Adds a LIMIT-capable SELECT generator overload and threads limit through the internal builder.
persistence/relational-jdbc/src/main/java/org/apache/polaris/persistence/relational/jdbc/JdbcBasePersistenceImpl.java Uses LIMIT 1 for hasChildren to bound result size for existence checks.
persistence/relational-jdbc/src/test/java/org/apache/polaris/persistence/relational/jdbc/QueryGeneratorTest.java Adds a test verifying LIMIT 1 is appended and parameters are ordered correctly.

@dimas-b dimas-b left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice improvement 👍 Thanks, @anxkhn !

JdbcBasePersistenceImpl.hasChildren generated a SELECT over all entity
columns with no row limit and then materialized the entire ResultSet into
a list, only to return whether it was empty. Checking whether a parent has
children therefore fetched, transferred, and deserialized every child row
in full, including the large properties and internal_properties JSON
columns. This path runs when dropping a catalog or a namespace, so dropping
a namespace holding many tables loaded all of them into memory just to
decide NAMESPACE_NOT_EMPTY, adding needless latency and allocation and
risking OOM on large namespaces.

Add a LIMIT-aware generateSelectQuery overload in QueryGenerator and use it
from hasChildren with a limit of 1, mirroring the existing
generateEntityTableExistQuery pattern. LIMIT is standard SQL accepted by all
supported backends (PostgreSQL, CockroachDB, H2). Behavior is unchanged: the
method still returns true when at least one child exists and false
otherwise.

Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com>
@anxkhn
anxkhn force-pushed the perf/jdbc-haschildren-limit branch from dfb7e53 to d8067a0 Compare July 16, 2026 06:14
@anxkhn

anxkhn commented Jul 16, 2026

Copy link
Copy Markdown
Contributor Author

Pushed a revision addressing the latest review/CI feedback.

1 similar comment
@anxkhn

anxkhn commented Jul 16, 2026

Copy link
Copy Markdown
Contributor Author

Pushed a revision addressing the latest review/CI feedback.

@github-project-automation github-project-automation Bot moved this from PRs In Progress to Ready to merge in Basic Kanban Board Jul 16, 2026
@dimas-b
dimas-b merged commit 1383eef into apache:main Jul 16, 2026
25 checks passed
@github-project-automation github-project-automation Bot moved this from Ready to merge to Done in Basic Kanban Board Jul 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants