The different table format compactions & it's impact in conversion

[Shekharrajak]

Have anybody tested the compaction for different table format and how it will be handled. There could be multiple scenerios :

1. Source table format let's say Iceberg table format is compacted, will we need to run table conversion (say Iceberg to Hudi) for the whole table again ?
2. What if there was table update before the compaction and then conversion run, how the xtable will handle that ?
3. How different the compaction look like in each table format and how we can related them if there is compaction in one table format ?
4. Are we trying to make each table format tightly coupled with each other and operations in one table format must be done in all other table formats or we can skip in other table format ?

Open for more discussions, ideas and how it behaves in these scenerios.

[the-other-tim-brown]

@Shekharrajak currently there is no support for translating the log files or delete vectors from one format to another. There is any open issue though that you can follow for updates: #339

Can you elaborate on what you mean by compaction here? I am more used to the compaction from Hudi.

[Shekharrajak]

@the-other-tim-brown , For example Iceberg have different methods to improve the performance by maintening the table data files, snapshots and other metadata : https://iceberg.apache.org/docs/1.6.1/maintenance/ - this can be done by owner of iceberg table independently that can cause the conversion of iceberg to hudi (let's say) - since we have changed the data files, metadata, expired the snapshot or other details.

[the-other-tim-brown]

When the data files that make up the currently active view are changed, a new snapshot is produced in Iceberg. This is synced to the target tables using the regular sync process.

There is a retention that you can set in the configuration that will trigger the Expire Snapshots code for you if you are using an Iceberg target. We leave the removal of data files up to the source system though to avoid any data loss or inconsistencies.