refactor!: txn-specific write_metadata_schema

[zachschuermann]

What changes are proposed in this pull request?

On the way to supporting stats in writes, we must allow for the write_metadata_schema to be specified per-txn/table instead of globally since the schema for write_metadata_schema will eventually include a stats column which is a function of the table schema.

Concretely, this PR

1. removes the global transaction::get_write_metadata_schema()
2. adds write_metadata_schema() to WriteContext (and plumbs the schema through)

Note that the actual naming will change in #1019

This PR affects the following public APIs

The (static) transaction::get_write_metadata_schema() is now a method: WriteContext::write_metadata_schema() (and the WriteContext is derived from a specific Transaction.

How was this change tested?

minor modifications to existing UT

[zachschuermann]

review with whitespace hidden

[codecov]

Codecov Report

Attention: Patch coverage is 96.29630% with 3 lines in your changes missing coverage. Please review.

Project coverage is 84.71%. Comparing base (bbca626) to head (9d087f7).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
kernel/src/transaction.rs	95.08%	0 Missing and 3 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1021      +/-   ##
==========================================
+ Coverage   84.70%   84.71%   +0.01%     
==========================================
  Files          92       92              
  Lines       23067    23266     +199     
  Branches    23067    23266     +199     
==========================================
+ Hits        19538    19710     +172     
- Misses       2568     2578      +10     
- Partials      961      978      +17     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[scovich]

I'm not understanding how responsibility is divided between kernel and engine here? It seems like the engine provides the write schema? I guess that makes sense (e.g. engine can choose not to write columns that have default values), but we don't validate that it's a subset of the table schema?

[scovich]

Now that we take &'a self, I think we can remove the named lifetimes?
At worst we might need + '_ for the iterators?

[zachschuermann]

Hm, the compiler is yelling that anonymous lifetimes are unstable in impl trait..

[zachschuermann]

I'm not understanding how responsibility is divided between kernel and engine here? It seems like the engine provides the write schema? I guess that makes sense (e.g. engine can choose not to write columns that have default values), but we don't validate that it's a subset of the table schema?

I was attempting to 'push down' the write metadata schema (the schema which we agree is how the engine communicates the metadata about files it wrote during appends) from being global to now per-table. The engine/kernel interaction is intended to be:

1. kernel provides the WriteContext from a Transaction to include the physical schema and transforms such that the engine can apply the schema + transform to its logical data to write valid parquet files, then
2. engine hands back some Box<dyn EngineData> which abides by our write_metadata_schema including metadata about the writes it performed. currently this is basically just a bunch of paths, etc. but will soon include a stats column whose schema is a function of the table's (physical) schema.

To be clear this is an unnecessary code change in isolation (the schema is constant for all transactions currently) but this paves the way for us to add the stats column which depends on the physical schema. I suppose one could argue that we could just allow engines to add their own stats and not need to specify it in the write_metadata_schema, but given we use that as the API for adding files to a table, it seemed to make sense to add it.

[scovich]

Who provides "our write_metadata_schema" tho? And how do ensure it's valid/compatible with the table?

[zachschuermann]

Who provides "our write_metadata_schema" tho? And how do ensure it's valid/compatible with the table?

we (kernel) do(es)! this is the Transaction::write_metadata_schema -> when we do stats in the future this will be generated from the table schema