Skip to content

feat: basic support of schema evolution in copy for parquet - #19094

Merged
youngsofun merged 5 commits into
databendlabs:mainfrom
youngsofun:evolution
Dec 12, 2025
Merged

feat: basic support of schema evolution in copy for parquet#19094
youngsofun merged 5 commits into
databendlabs:mainfrom
youngsofun:evolution

Conversation

@youngsofun

@youngsofun youngsofun commented Dec 11, 2025

Copy link
Copy Markdown
Member

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

feats:

  1. table alter and data append is committed together atomicly.

example:

# prepare data
create or replace table t1(a int);
create or replace stage s1;
copy into @s1 from (select 1 as a, 2 as b,  3 as c)

# ENABLE_SCHEMA_EVOLUTION
ALTER TABLE t1 SET options(ENABLE_SCHEMA_EVOLUTION = true);

# do copy
copy into t1 from @s1/ file_format=(type=parquet);

not supported yet:

  • adding field in tuple type
  • merge types, e.g. int8 and int16

Tests

  • Unit Test
  • Logic Test
  • Benchmark Test
  • No Test - Explain why

Type of change

  • [] Bug Fix (non-breaking change which fixes an issue)
  • New Feature (non-breaking change which adds functionality)
  • Breaking Change (fix or feature that could cause existing functionality not to work as expected)
  • Documentation Update
  • Refactoring
  • Performance Improvement
  • Other (please describe):

This change is Reviewable

@github-actions github-actions Bot added the pr-feature this PR introduces a new feature to the codebase label Dec 11, 2025
@youngsofun
youngsofun marked this pull request as draft December 11, 2025 02:12
@youngsofun
youngsofun marked this pull request as ready for review December 11, 2025 08:24
@youngsofun
youngsofun requested review from SkyFan2002, dantengsky and sundy-li and removed request for SkyFan2002 December 11, 2025 08:25

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/query/sql/src/planner/binder/copy_into_table.rs
Comment thread src/meta/app/src/principal/user_stage.rs
Comment thread src/query/service/src/interpreters/interpreter_copy_into_table.rs Outdated
@youngsofun
youngsofun merged commit d7bf3cf into databendlabs:main Dec 12, 2025
87 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-feature this PR introduces a new feature to the codebase

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants