Skip to content

Implement saving TableMetadata to new location. #1388

@liurenjie1024

Description

@liurenjie1024
Contributor
No description provided.

Activity

CTTY

CTTY commented on Jun 3, 2025

@CTTY
Contributor

I think this will need to be implemented for different catalogs individually, because catalogs have different ways to persist metadata. Or we are mainly talking about adding traits to Catalog here?

liurenjie1024

liurenjie1024 commented on Jun 4, 2025

@liurenjie1024
ContributorAuthor

I think this will need to be implemented for different catalogs individually, because catalogs have different ways to persist metadata. Or we are mainly talking about adding traits to Catalog here?

I'm thinking to put some common code like saving table metadata to a location, loading table metadata from a location to a common place.

DerGut

DerGut commented on Jun 9, 2025

@DerGut
Contributor

Other implementations use a metastore catalog that encapsulates some of this shared logic. For example, there is pyiceberg's MetastoreCatalog._write_metadata or Java's BaseMetastoreTableOperations.writeNewMetadata.

I'd be happy to take a look at this. I've already drafted a version of it while working on MemoryCatalog::update_table, but would like to move it a shared place instead.

Whether or not we would want to add something like a MetastoreCatalog trait with default implementations is a different discussion, but I think such a change would build a good foundation for it.

liurenjie1024

liurenjie1024 commented on Jun 12, 2025

@liurenjie1024
ContributorAuthor

Other implementations use a metastore catalog that encapsulates some of this shared logic. For example, there is pyiceberg's MetastoreCatalog._write_metadata or Java's BaseMetastoreTableOperations.writeNewMetadata.

I'd be happy to take a look at this. I've already drafted a version of it while working on MemoryCatalog::update_table, but would like to move it a shared place instead.

Whether or not we would want to add something like a MetastoreCatalog trait with default implementations is a different discussion, but I think such a change would build a good foundation for it.

I don't think we need another trait. Maybe we just need a TableMetadataWriter which could be created from Table would be enough.

DerGut

DerGut commented on Jun 14, 2025

@DerGut
Contributor

I agree! I was considering to start with a new module iceberg::catalog::metastore that contains functions like

pub async fn write_metadata(file_io: &FileIO, location: &str, metadata: &TableMetadata) -> Result<()>
pub async fn read_metadata(file_io: &FileIO, location: &str) -> Result<TableMetadata>

but I like the idea of a metadata writer even better!

added a commit that references this issue on Jul 23, 2025
9277a17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Participants

      @liurenjie1024@DerGut@CTTY

      Issue actions

        Implement saving `TableMetadata` to new location. · Issue #1388 · apache/iceberg-rust