Comments (5)
@remibaar Makes sense to me. Would you please update the doc of Sedona website and create a PR? I am happy to accept it!
from sedona.
After some further investigation I see the Databricks runtime also contains H3 functionality. For this it uses com.uber h3
version 3.7.0
. Could this be conflicting with the version 4.1.1
which is being used by Sedona? It would explain it as polygonToCells
is not available in version 3.x
of H3.
from sedona.
I managed to solve the issue! Indeed it was related to the version of H3 that was being installed in the Databricks runtime.
By adjusting the init script, I remove the older H3 jar from the Databricks jars. This solves the issue.
This is the code for my new init script:
%sh
# Create init script
cat > /dbfs/FileStore/sedona/scripts/sedona-init.sh <<'EOF'
#!/bin/bash
#
# File: sedona-init.sh
#
# On cluster startup, this script will copy the Sedona jars to the cluster's default jar directory.
# In order to activate Sedona functions, remember to add to your spark configuration the Sedona extensions: "spark.sql.extensions org.apache.sedona.viz.sql.SedonaVizExtensions,org.apache.sedona.sql.SedonaSqlExtensions"
# Remove default H3 version of databricks, as it is not compatible with Sedona > 1.5.0
rm -f /databricks/jars/*com.uber*h3*.jar
# Copy jars
cp /dbfs/FileStore/sedona/jars/*.jar /databricks/jars
EOF
Note: This will break the builtin H3 functionality of Databricks. But I believe the H3 functions of Sedona supersedes those of the built-in H3 of Databricks. The builtin H3 functions will now throw a NoClassDefFoundError
I will keep this issue open, because I am going to create a PR for a change in the docs.
https://github.com/apache/sedona/blob/master/docs/setup/databricks.md
from sedona.
The main reason is that we shaded the uber-h3 jar into sedona-spark-shaded
which leads to conflicts. Another alternative to fix this is that: use sedona-spark
jar which does not shade anything, and manually download all dependency jars of Sedona: https://github.com/apache/sedona/blob/master/pom.xml#L139
from sedona.
Another alternative to fix this is that: use
sedona-spark
jar which does not shade anything, and manually download all dependency jars of Sedona
Please correct me if I am wrong. With this method you also will not be able to use both the H3 of Sedona and the H3 of Databricks. Because they use different major versions (Sedona uses 4.1.1, Databricks uses 3.7.0), which are incompatible.
My personal recommendation would be to remove the H3 3.7.0 jar from the Databricks runtime. This disables the H3 functions of Databricks, but allows the use of the H3 functions of Sedona.
In my opinion the H3 functions of Sedona are more feature complete.
For example one of the features I need is the fullCover
of the ST_H3CellIDs
function. Which is not available at the Databricks implementation, but is at Sedona
from sedona.
Related Issues (20)
- RS_AsBase64 returns empty string HOT 3
- Request for addition of function: `ST_LineLocatePoint` HOT 6
- rs_setbandnodatavalue not working as expected HOT 2
- Error reading file greater than 2GB HOT 1
- Installing apache-sedona Python package does not install all dependencies (Sedona 1.5.0) HOT 2
- [Feature] Add ST_IsValidDetail as built-in function.
- 1.5.0 spark shaded protobuf version conflict with application HOT 2
- Support of 3D?
- Feature request: ST_Snap HOT 1
- Microsoft Fabric HOT 32
- Apache Sedona backend for Ibis project?
- Save / Load indexed spatial & partitioned Rdd HOT 2
- Clarify SedonaUtils documentation HOT 1
- Write a geotiff raster , but just found _SUCCESS file in output HDFS dir HOT 3
- Noisy warnings on optional imports
- Map RDD and GeometryType (not an instance of type GeometryType()) HOT 2
- When I resample a raster to very width*height raster , Serde.serialize is limited in 2GB HOT 2
- When I use Bicubic alg to resample a raster with no data pixel , the output raster would be filled unexpected data in no data pixel HOT 3
- fieldNames - AttributeError: Not available before 1.0.0 sedona version HOT 4
- Undeclared IPython dependency HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from sedona.