Skip to content

Example of updating schema in TFX #144

Closed
@albertnanda

Description

@albertnanda

I have created a schema using SchemaGen. I want to update one of the fields from required to optional. How can I do that?
https://www.tensorflow.org/tfx/tutorials/tfx/components_keras in this tutorial, every field is required. How can I make one of the fields optional.

Another question is how do I access the "domain" information from context.show(schema_gen.outputs['schema'])
programmatically.

Activity

Mikehem

Mikehem commented on Nov 3, 2020

@Mikehem
  1. Changing schema feature from optional to required and vice versa

first get the feature from the schema

feature = tfdv.get_feature(schema, feature_name) # this will be the column

for changing from required to optional add a value between 0.0 to 1.0

feature.presence.min_fraction = 0.9

for changing from optional to required

feature.presence.min_fraction = 1.0

display the schema to check if the changes have occured as desired

tfdv.display_schema(schema)

  1. Getting the domain values from SchemaGen

This may be a workaround for now. When you run the context, you will get a URI for the run metadata. You can also get it by running this -> schema_gen.outputs['schema'].to_json_dict()['artifacts'][0]['artifact']['uri'] + '/schema.pbtxt'

schema_uri = ''path to tfx pipeline/tfx/pipeline/SchemaGen/schema/19/schema.pbtxt'
schema_1 = tfdv.load_schema_text(schema_uri)
tfdv.get_domain(schema_1, "company")

self-assigned this
on Nov 4, 2020
rmothukuru

rmothukuru commented on Nov 4, 2020

@rmothukuru

@albertnanda,
Can you please let us know if @Mikehem's comment worked for you. Thanks!

albertnanda

albertnanda commented on Nov 5, 2020

@albertnanda
Author

It works, closing this issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

    Development

    No branches or pull requests

      Participants

      @Mikehem@albertnanda@rmothukuru@amahendrakar

      Issue actions

        Example of updating schema in TFX · Issue #144 · tensorflow/data-validation