vdk-impala: Introduce checks for insert template#2198
Merged
Conversation
Add functionality to the insert processing templates in order to allow quality checks to be made before the data is inserted into the target table. Currently, the checks done on the insert template processing step are not covering if the semantics of the data is correct. Therefore, bad data could go into the target table which could be unwanted behavior. More details explained in #1361 What: -Adding functionality to handle the insert template behavior if the user provides checks Tests: provided positive and negative regression tests Signed-off-by: Stefan Buldeev sbuldeev@vmware.com
for more information, see https://pre-commit.ci
Contributor
|
The title of the PR says snapshot, not insert template :) |
antoniivanov
reviewed
Jun 5, 2023
...gins/vdk-impala/src/vdk/plugin/impala/templates/load/fact/insert/02-handle-quality-checks.py
Show resolved
Hide resolved
antoniivanov
suggested changes
Jun 5, 2023
Contributor
antoniivanov
left a comment
antoniivanov
left a comment
There was a problem hiding this comment.
The functionality looks good. Now I am focusing on some small improvements on maintainability and ease-of-troubleshooting of the change.
...gins/vdk-impala/src/vdk/plugin/impala/templates/load/fact/insert/02-handle-quality-checks.py
Outdated
Show resolved
Hide resolved
...gins/vdk-impala/src/vdk/plugin/impala/templates/load/fact/insert/02-handle-quality-checks.py
Outdated
Show resolved
Hide resolved
sbuldeev
changed the title
https://github.com/vmware/versatile-data-kit into person/sbuldeev/add-quality-checks-to-insert-template
for more information, see https://pre-commit.ci
https://github.com/vmware/versatile-data-kit into person/sbuldeev/add-quality-checks-to-insert-template
for more information, see https://pre-commit.ci
antoniivanov
approved these changes
Jun 9, 2023
https://github.com/vmware/versatile-data-kit into person/sbuldeev/add-quality-checks-to-insert-template
for more information, see https://pre-commit.ci
murphp15
approved these changes
Jun 15, 2023
projects/vdk-plugins/vdk-impala/src/vdk/plugin/impala/templates/load/fact/insert/README.md
Outdated
Show resolved
Hide resolved
https://github.com/vmware/versatile-data-kit into person/sbuldeev/add-quality-checks-to-insert-template
sbuldeev
deleted the
person/sbuldeev/add-quality-checks-to-insert-template
branch
murphp15
pushed a commit
that referenced
this pull request
Jun 21, 2023
Why: Add functionality to the insert processing templates in order to allow quality checks to be made before the data is inserted into the target table. Currently, the checks done on the insert template processing step are not covering if the semantics of the data is correct. Therefore, bad data could go into the target table which could be unwanted behavior. More details explained in #1361 What: -Adding functionality to handle the insert template behavior if the user provides checks Tests: provided positive and negative regression tests Signed-off-by: Stefan Buldeev sbuldeev@vmware.com --------- Signed-off-by: Stefan Buldeev sbuldeev@vmware.com Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add functionality to the insert processing templates in order to allow quality checks to be made before the data is inserted into the target table.
Currently, the checks done on the insert template processing step are not covering if the semantics of the data is correct. Therefore, bad data could go into the target table which could be unwanted behavior.
More details explained in
#1361
What:
-Adding functionality to handle the insert template behavior if the user provides checks
Tests: provided positive and negative regression tests
Signed-off-by: Stefan Buldeev sbuldeev@vmware.com