feat: partition_by option for Hive-partitioned Parquet output

[cigrainger]

Summary

Add partition_by: option to to_parquet/3 for Hive-style partitioned directory output. Works both single-node and distributed.

Usage

# Single-node
Dux.from_parquet("events.parquet")
|> Dux.to_parquet("/output/events/", partition_by: [:year, :month])
# Creates: /output/events/year=2024/month=01/data_0.parquet

# Distributed — each worker writes to the same Hive directories
Dux.from_parquet("s3://input/**/*.parquet")
|> Dux.distribute(workers)
|> Dux.to_parquet("/output/events/", partition_by: :region)
# Creates: /output/events/region=US/w0_0.parquet, region=US/w1_0.parquet, etc.

Details

• Single-node: appends PARTITION_BY ("col") to DuckDB COPY options
• Distributed: uses FILENAME_PATTERN "w{idx}_{i}" per worker to prevent file name collisions when multiple workers write to overlapping partition directories
• Column names are properly quoted (handles reserved words like group)
• Supports single column (atom/string) or list of columns
• Combines with :compression and other COPY options

Test plan

• Single partition column creates Hive directory structure
• Multiple partition columns create nested directories
• partition_by with compression option
• Round-trip: partition_by write then partition-pruned read
• Distributed: each worker writes Hive-partitioned output, correct totals
• 600 non-distributed tests pass, Credo clean
• 6 distributed write peer tests pass

🤖 Generated with Claude Code