Improved stats inspection #101
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
aykut-bozkurt
force-pushed
the
aykut/stats-udf
branch
2 times, most recently
from
5f53ea9 to
2721808
Compare
aykut-bozkurt
changed the base branch from
main
to
aykut/dest-receiver-bytes-written
aykut-bozkurt
force-pushed
the
aykut/stats-udf
branch
from
2721808 to
af49018
Compare
aykut-bozkurt
force-pushed
the
aykut/dest-receiver-bytes-written
branch
from
1fc60b1 to
44f4190
Compare
aykut-bozkurt
force-pushed
the
aykut/stats-udf
branch
2 times, most recently
from
90593f3 to
f373345
Compare
aykut-bozkurt
force-pushed
the
aykut/dest-receiver-bytes-written
branch
from
44f4190 to
92a3ef0
Compare
aykut-bozkurt
force-pushed
the
aykut/stats-udf
branch
from
f373345 to
47d17cb
Compare
marcoslot
reviewed
Mar 11, 2025
marcoslot
reviewed
Mar 11, 2025
| > { | ||
| let uri = parse_uri(&uri); | ||
|
|
||
| ensure_access_privilege_to_uri(&uri, true); |
Collaborator
There was a problem hiding this comment.
random thought: wonder whether we should have a separate role for public URLs
(probably not needed for now)
Member
Author
There was a problem hiding this comment.
still a network connection. might think on it though
marcoslot
reviewed
Mar 11, 2025
src/parquet_udfs/stats.rs
Outdated
| .get_basic_info() | ||
| .has_id() | ||
| { | ||
| continue; |
Collaborator
There was a problem hiding this comment.
does not having a filed ID imply not having stats?
marcoslot
reviewed
Mar 11, 2025
aykut-bozkurt
changed the base branch from
aykut/dest-receiver-bytes-written
to
main
aykut-bozkurt
force-pushed
the
aykut/stats-udf
branch
3 times, most recently
from
22d6a6b to
cfb3d2c
Compare
aykut-bozkurt
force-pushed
the
aykut/stats-udf
branch
from
cfb3d2c to
1d1d248
Compare
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #101 +/- ##
==========================================
+ Coverage 91.97% 92.43% +0.46%
==========================================
Files 84 85 +1
Lines 10651 11288 +637
==========================================
+ Hits 9796 10434 +638
+ Misses 855 854 -1 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
marcoslot
reviewed
Mar 13, 2025
src/parquet_udfs/stats.rs
Outdated
| }) if !is_adjusted_to_u_t_c | ||
| ); | ||
|
|
||
| let is_timetz = matches!( |
Collaborator
There was a problem hiding this comment.
doing all these checks upfront seems a bit strange, should they perhaps move into the match below?
Member
Author
There was a problem hiding this comment.
done, still prefer matches to match {}
aykut-bozkurt
force-pushed
the
aykut/stats-udf
branch
2 times, most recently
from
97c2b26 to
28b8448
Compare
Previously, we printed column statistics for each column per row group via `parquet.metadata(uri)`. With the new udf `parquet.column_stats(uri)`, we print stats for each column aggregated by row groups. Stats for some of the types were printed in a text format that cannot be converted to actual Postgres type. This PR also makes sure the output format is convertible to the actual Postgres type.
aykut-bozkurt
force-pushed
the
aykut/stats-udf
branch
from
28b8448 to
648d129
Compare
marcoslot
approved these changes
Apr 2, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Previously, we printed column statistics for each column per row group via
parquet.metadata(uri). With the new udfparquet.column_stats(uri), we print the column stats for each column aggregated by row groups.Stats for some of the types were printed in a text format that cannot be converted to actual Postgres type. This PR also makes sure the output format is convertible to the actual Postgres type.