Add license and notice to spark client jar and push jar to maven #1830
Conversation
gh-yzou
requested review from
adutra,
ashvina,
dennishuo,
dimas-b and
eric-maynard
as code owners
gh-yzou
requested review from
jackye1995,
jbonofre,
vvcephei,
collado-mike,
snazy,
RussellSpitzer,
takidau,
MonkeyCanCode,
flyrain,
ebyhr,
ajantha-bhat,
HonahX,
singhpk234 and
pingtimeout
as code owners
| @@ -71,7 +71,7 @@ bin/spark-shell \ | |||
| --conf spark.sql.catalog.<spark-catalog-name>.scope='PRINCIPAL_ROLE:ALL' \ | |||
| --conf spark.sql.catalog.<spark-catalog-name>.token-refresh-enabled=true | |||
| ``` | |||
| Assume the released Polaris Spark client you want to use is `org.apache.polaris:polaris-iceberg-1.8.1-spark-runtime-3.5_2.12:1.0.0`, | |||
| Assume the released Polaris Spark client you want to use is `org.apache.polaris:polaris-spark-3.5_2.12:1.0.0`, | |||
There was a problem hiding this comment.
Are we sure spark can download the bundle jar with the following config? Can we test it out? I'm not sure how do we test it against local mvn repo though, one way is to merge this into nightly maven repo first, then test it out.
--packages org.apache.polaris:polaris-spark-3.5_2.12:1.0.0
There was a problem hiding this comment.
I haven't tested it out yet. once it is into nightly maven repo, i can test it out and update the doc if needed.
| archiveClassifier = null | ||
| archiveBaseName = | ||
| "polaris-iceberg-${icebergVersion}-spark-runtime-${sparkMajorVersion}_${scalaVersion}" | ||
| archiveClassifier = "bundle" |
There was a problem hiding this comment.
Think a bit more, I think it'd be nice to have the iceberg version in the package name. Otherwise, debugging would be tricky. Users couldn't figure out the Iceberg lib version easily.
There was a problem hiding this comment.
Wondering if we could apply the Iceberg version as we did for Scala version.
There was a problem hiding this comment.
The name of the jar now seems <project_name>-<polaris_version>-.jar, and the scala_version is embedded in the project_name, such as spark-3.5_2.12. I don't think it is a good idea to have iceberg version as project name, since the iceberg version might gets updated more frequently.
i don't think we can touch polaris_version also, then classifier is the only place . however, it seems wired that the iceberg version is after the polaris version. I think what we can do is publish on the webpage about the iceberg client compatibility. Given in long term, we may not need iceberg version in the package name also.
There was a problem hiding this comment.
Users can open up the jar file to figure out the embedded Iceberg version. But my concern is that a lot of users don't know that or couldn't do that easily. I think it's still valuable to provide the iceberg versions as it's embedded. Unless we figure out a way to not embed it.
I don't think it is a good idea to have iceberg version as project name, since the iceberg version might gets updated more frequently.
We need to release a new version for this plugin every time we update the embedded Iceberg version.
There was a problem hiding this comment.
it seems wired that the iceberg version is after the polaris version.
True. The only option is to play with the project name.
There was a problem hiding this comment.
It's not a blocker to me for this PR. We could figure it out later.
+1 on option 1. Let's get this in first. |
github-project-automation
bot
moved this from PRs In Progress
to Ready to merge
in Basic Kanban Board
github-project-automation
bot
moved this from Ready to merge
to Done
in Basic Kanban Board
This PR does following: