Comments (5)
The problem cannot be reproduced by simply launching pyspark locally using pyspark --packages org.apache.sedona:sedona-spark-shaded-3.4_2.12:1.6.0,org.datasyslab:geotools-wrapper:1.6.0-28.2
. The java.lang.NoSuchMethodError
exception raised by spark executor is really weird, since this constructor method do exist in org.datasyslab:geotools-wrapper:1.6.0-28.2
. BTW, Sedona 1.6.0 is still using GeoTools 28.2, we reverted the GeoTools version upgrade before release.
After trying lots of possible misconfigurations, I finally reproduced this problem locally by putting both geotools-wrapper-1.6.0-28.2.jar and geotools-wrapper-1.6.0-31.0.jar to the classpath. Please make sure that only geotools-wrapper-1.6.0-28.2 is actually being used, and remove geotools-wrapper-1.6.0-31.0.jar from the cluster to avoid any misbehavior.
from sedona.
@golfalot In the first release candidate of Sedona 1.6.0, we wanted to upgrade Sedona to use geotools 31.0 and that's why geotool-wrapper 1.6.0-31.0 exists. However, given the current user base of Sedona, we decided to revert this breaking change in the final release candidate of Sedona 1.6.0 and fall back to geotools-wrapper 1.6.0-28.2.
The exact geotools-wrapper version to use is always mentioned here: https://sedona.apache.org/1.6.0/setup/maven-coordinates/
Unfortunately, Maven Central does not allow anyone to retract a published package so we cannot delete geotools-wrapper 1.6.0-31.0
from sedona.
I don't currently have the skills to debug this beyond this issue report. I hope the info provided is sufficiently comprehensive, do not hesitate to request further info from me.
An observation that is possibly material: geotools-wrapper
release 1.6.0-28.2
Exclude GeographicLib
here https://github.com/jiayuasu/geotools-wrapper/releases/tag/1.6.0-28.2
from sedona.
I also noted another change that could be material
https://docs.geotools.org/stable/userguide/welcome/upgrade.html
GeoTools 30.x
Replace constructor Envelope2D(crs,x,y,width,height) with ReferencedEnvelope.rect(x,y,width,height,crs)
from sedona.
Yes! thank you @Kontinuation , exactly right. geotools-wrapper-1.6.0-31.0.jar
was present on the Synapse Spark Pool, removing it , and leaving only geotools-wrapper-1.6.0-28.2.jar
fixes the issue.
-
It was probably naive of me to leave it there and assume simply referencing the desired version in my config statement would suffice to disambiguate. I shall remember this important lesson!
-
Why was
1.6.0-31.0
there in the first place ?
This is my first time getting up and running with Sedona, once I figured out what jars I needed, I trawled Maven to find the latest versions of the two jars for my target versions of Spark and Scala. https://mvnrepository.com/artifact/org.datasyslab/geotools-wrapper/1.6.0-31.0 -
How did I get the idea to downgrade to
1.6.0-28.2
?
When I headed over to https://github.com/jiayuasu/geotools-wrapper I saw that latest release was in fact1.6.0-28.2
and NOT1.6.0-31.0
-
I wonder if it would be helpful for other users if version
1.6.0-31.0
were completely deleted from Maven ? As this version did not work for me.
Thank you once again for the great guidance, I appreciate the time you spent on this.
from sedona.
Related Issues (20)
- after latest update of sedona snow on snowflake side all functions are gone HOT 2
- datatype geometry is not supported when 'create table xxx (geom geometry)' HOT 3
- AttributeError: 'sedona' has no attribute 'read' HOT 2
- St_isempty(geometry) finds non null geometries but does not find null geometries. HOT 2
- ST_Snap example code does not work HOT 2
- Flink Sedona,geomTbl.execute().print() happen error: HOT 2
- Sedona fails to write Delta Lake on Databricks 15.3 Beta: ClassCastException HOT 7
- ST_IsPolygonCW, ST_IsPolygonCCW, ST_ForcePolygonCW and ST_ForcePolygonCCW fails on Polygons without interior ring
- There was garbled code when reading Chinese from shp file HOT 3
- sedona's docker image can not run successfully on k8s HOT 2
- Cannot run sedona examples into spark-shell HOT 3
- Using setMaster(“ spark://master:7077) An error occurred during operation, but there is no problem running locally HOT 3
- Sedona website homepage layout bug in mobile view
- Sedona website favicon checker issues and missing files
- An error occurred when submitting the file. The actual directory does exist.
- Initialization of sedona 1.6.0 is quite slower than sedona 1.5.1 HOT 2
- 1.6.0 documentation needs update? HOT 6
- st_union - understanding difference with PostGIS HOT 6
- Issue with st_dump? HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from sedona.