feat: make StatisticsRegistry the default operator-statistics path

[asolimando]

Which issue does this PR close?

• Closes Pluggable operator-level statistics propagation (StatisticsRegistry) #21443.
• Part of Epic: Statistics improvements #8227.

Rationale for this change

StatisticsRegistry (#21483) was opt-in behind datafusion.optimizer.use_statistics_registry and ran as a separate tree walk from the main statistics computation, so the two could disagree and the registry only reached code that explicitly consulted it. Since #23051 collapsed statistics traversal into a single StatisticsContext walk, the registry can just be folded into that walk instead. This makes it the default path: providers are consulted at each node before falling back to the operator's statistics_from_inputs, so there's one walk and the flag is no longer needed.

What changes are included in this PR?

• The registry is consulted at each node of the single StatisticsContext walk: the first provider that returns Computed sets the node's stats, otherwise the walk falls back to the operator's statistics_from_inputs. join_selection uses this walk.
• The walk carries ExtendedStatistics (core Statistics plus a type-keyed extension map) internally; the statistics_from_inputs signature is unchanged. Extensions are empty by default (the walk never forwards them, so a stale extension can't propagate silently); providers opt into propagation, e.g. PassthroughStatisticsProvider across cardinality-preserving operators.
• StatisticsProvider gains a defaulted, partition-aware compute_statistics_with_args (overall-only by default, delegates per partition so a partition-blind provider can't leak an overall estimate into one partition).
• use_statistics_registry is deprecated (kept but ignored, warns on use); configs.md, information_schema.slt, and the 55.0.0 upgrade guide are updated.
• DefaultStatisticsProvider is deprecated and dropped from the built-in provider chain; the walk's native fallback to statistics_from_inputs makes it redundant.
• Two examples under datafusion-examples/examples/statistics/, each with a before/after EXPLAIN flip: a row-count override provider (core Statistics) and a custom NDV-sketch provider carried through the extension map.

Are these changes tested?

Unit tests in operator_statistics/statistics (provider chaining, partition-aware providers, per-partition UnionExec with a registry), the statistics_registry.slt file, and both examples. Full extended suite passes locally.

Are there any user-facing changes?

• datafusion.optimizer.use_statistics_registry is deprecated and ignored (accepted with a warning, removed later). Default behavior is unchanged: no providers registered means the registry is a no-op. See the 55.0.0 upgrade guide.
• API additions are non-breaking: StatisticsProvider gains a defaulted compute_statistics_with_args, and DefaultStatisticsProvider is deprecated (not removed). For default behavior register nothing, or StatisticsRegistry::default_with_builtin_providers() for the built-in providers. See the 55.0.0 upgrade guide.

---

Disclaimer: I used AI to assist in the code generation, I have manually reviewed the output and it matches my intention and understanding.

[github-actions]

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details

     Cloning apache/main
    Building datafusion-common v54.0.0 (current)
       Built [  27.460s] (current)
     Parsing datafusion-common v54.0.0 (current)
      Parsed [   0.051s] (current)
    Building datafusion-common v54.0.0 (baseline)
       Built [  26.071s] (baseline)
     Parsing datafusion-common v54.0.0 (baseline)
      Parsed [   0.051s] (baseline)
    Checking datafusion-common v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   0.820s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [  55.595s] datafusion-common
    Building datafusion-physical-optimizer v54.0.0 (current)
       Built [  30.722s] (current)
     Parsing datafusion-physical-optimizer v54.0.0 (current)
      Parsed [   0.018s] (current)
    Building datafusion-physical-optimizer v54.0.0 (baseline)
       Built [  31.208s] (baseline)
     Parsing datafusion-physical-optimizer v54.0.0 (baseline)
      Parsed [   0.019s] (baseline)
    Checking datafusion-physical-optimizer v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   0.127s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [  63.151s] datafusion-physical-optimizer
    Building datafusion-physical-plan v54.0.0 (current)
       Built [  29.361s] (current)
     Parsing datafusion-physical-plan v54.0.0 (current)
      Parsed [   0.117s] (current)
    Building datafusion-physical-plan v54.0.0 (baseline)
       Built [  29.282s] (baseline)
     Parsing datafusion-physical-plan v54.0.0 (baseline)
      Parsed [   0.120s] (baseline)
    Checking datafusion-physical-plan v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   0.733s] 223 checks: 222 pass, 1 fail, 0 warn, 30 skip

--- failure type_marked_deprecated: #[deprecated] added on type ---

Description:
A type is now #[deprecated]. Downstream crates will get a compiler warning when using this type.
        ref: https://doc.rust-lang.org/reference/attributes/diagnostics.html#the-deprecated-attribute
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.48.0/src/lints/type_marked_deprecated.ron

Failed in:
  Struct DefaultStatisticsProvider in /home/runner/work/datafusion/datafusion/datafusion/physical-plan/src/operator_statistics/mod.rs:285

     Summary semver requires new minor version: 0 major and 1 minor checks failed
    Finished [  60.596s] datafusion-physical-plan
    Building datafusion-sqllogictest v54.0.0 (current)
       Built [ 148.296s] (current)
     Parsing datafusion-sqllogictest v54.0.0 (current)
      Parsed [   0.019s] (current)
    Building datafusion-sqllogictest v54.0.0 (baseline)
       Built [ 147.157s] (baseline)
     Parsing datafusion-sqllogictest v54.0.0 (baseline)
      Parsed [   0.019s] (baseline)
    Checking datafusion-sqllogictest v54.0.0 -> v54.0.0 (no change; assume patch)
     Checked [   0.089s] 223 checks: 223 pass, 30 skip
     Summary no semver update required
    Finished [ 297.836s] datafusion-sqllogictest

[codecov-commenter]

Codecov Report

❌ Patch coverage is 90.30303% with 16 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (main@7f6cc60). Learn more about missing BASE report.

Files with missing lines	Patch %	Lines
...usion/physical-plan/src/operator_statistics/mod.rs	85.05%	11 Missing and 2 partials ⚠️
datafusion/physical-plan/src/statistics.rs	95.38%	0 Missing and 3 partials ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main   #23651   +/-   ##
=======================================
  Coverage        ?   80.65%           
=======================================
  Files           ?     1086           
  Lines           ?   366567           
  Branches        ?   366567           
=======================================
  Hits            ?   295655           
  Misses          ?    53288           
  Partials        ?    17624           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.