RS_MapAlgebra for raster processing

[melikasajadian]

Hi,

When I try to code map algebra operation according to the sample notebook here using RS_MapAlgebra, I get the following error:

InferredExpressionException: Exception occurred while evaluating expression RS_MapAlgebra - inputs: [GridCoverage2D["geotiff_coverage", GeneralEnvelope[(470000.0, 7227000.0), (471000.0, 7228000.0)], DefaultProjectedCRS["WGS 84 / UTM zone 6N"]]
Caused by: RuntimeException: Failed to run map algebra
Caused by: JiffleException: Runtime source error for source: package it.geosolutions.jaiext.jiffle.runtime;
Caused by: CompileException: Line 12, Column 15: it.geosolutions.jaiext.jiffle.runtime.String
Caused by: ClassNotFoundException: it.geosolutions.jaiext.jiffle.runtime.String
Caused by: ClassNotFoundException: it.geosolutions.jaiext.jiffle.runtime.String

Apparently, the jiffle runtime is missing. I am using Apache Sedona 1.7.1 in Databricks (I tried with other versions of Sedona but the error persists). Is there a way to solve this error, for example installing the library on the cluster separately or through the init-script?

It is not a critical error since I can calculate for example NDVI through RS_Divide, RS_Add, RS_Subtract.

Thank you.

Cheers,
Melika

[jiayuasu]

@melikasajadian Did you add the geotools-wrapper jar to Databricks?

[jiayuasu]

https://sedona.apache.org/latest/setup/maven-coordinates/#use-sedona-shaded-fat-jars

[melikasajadian]

@jiayuasu Thank you for the response. Yes, both jar files are in the init_script of the cluster following the instruction here with runtime 15.4 LTS (includes Apache Spark 3.5.0, Scala 2.12). I did verify if the jar files are also available in classpath entries in the Spark UI:

/databricks/jars/sedona-spark-shaded-3.5_2.12-1.7.1.jar
/databricks/jars/geotools-wrapper-1.7.1-28.5.jar

As I mentioned for example this code works perfectly fine:
ndgi_df = spark.sql("select *, RS_Divide(RS_Subtract(RS_BandAsArray(raster, 2), RS_BandAsArray(raster, 1)), RS_Add(RS_BandAsArray(raster, 2), RS_BandAsArray(raster, 1))) AS ndgi from neon")
ndgi_df.createOrReplaceTempView("ndgi")
display(ndgi_df)

Only the below one gives an error that it cannot find the geosolutions jiffle class:
ndgi_df = sedona.sql("""
SELECT
RS_MapAlgebra(raster, 'D', 'out = ((rast[1] - rast[0]) / (rast[1] + rast[0]));')
AS raster
FROM neon
""")
ndgi_df.createOrReplaceTempView("ndgi")
display(ndgi_df)

Is there anything else that I can check?

[Kontinuation]

I have reproduced this on Databricks 15.4 LTS, but not on open source Apache Spark 3.5.4. Turns out that it is a issue related to janino version on Databricks.

Jiffle uses janino to compile jiffle expressions to JVM byte code, and a fix in janino janino-compiler/janino@6e8a97d must be present to make jiffle work with Spark. This fix only presents in janino >= 3.1.1. Unfortunately Databricks 15.4 LTS uses janino 3.0.16 and caused this problem.

We may have to shade janino inside of geotools-wrapper to get rid of this problem on Databricks.

[Kontinuation]

No. According to the release note, 16.4 LTS and the latest 17.0 Beta still uses janino 3.0.16.

[jiayuasu]

@Kontinuation should we shade jiffle altogether in geotools-wrapper given this issue https://github.com/apache/sedona/pull/1919/files/2a9fc558b9fad5eaf47e0b74e2e1f2e5432c5e78#r2054442668

[jiayuasu]

@Kontinuation looks like Spark 3.X still uses Janino 3.0.16 so Jiffle won't work on Spark 3.3 at all: #1966

I wonder how our current jiffle tests passed on Spark 3.3 since the Janino versions are incompatible.

[Kontinuation]

@Kontinuation looks like Spark 3.X still uses Janino 3.0.16 so Jiffle won't work on Spark 3.3 at all: #1966

I wonder how our current jiffle tests passed on Spark 3.3 since the Janino versions are incompatible.

It only happens when using Janino 3.0.16 and Spark ExecutorClassLoader at the same time. This only happens when running Apache Spark 3.3.x in scala repl mode (when running spark-shell for example). Running in pyspark mode or running JVM tests in scalatest won't trigger this problem.

Databricks runtime runs notebooks in scala repl mode, so we observed this problem on Databricks runtime. We can also reproduce this in Apache Spark 3.3.x by running RS_MapAlgebra in spark-shell.

[Kontinuation]

The failure of CI you mentioned is because we specified the version of Janino in dependencyManagement. It is OK to pin the version of Janino when we have dropped Spark 3.3, but it conflicts with the version of Janino Spark 3.3 uses after backporting the commit to this branch. I'll fix the pom.xml to resolve this problem later.

[jiayuasu]

@melikasajadian the latest 1.7.2 release should be able to fix this. please give it a try.