NPE while loading small input location csv about sedona HOT 6 CLOSED

Hi @deil87 , when loading small files, have you tried to use the default RDD partition number? Or have you tried to reduce the number of partitions to 1?

from sedona.

@deil87 It would be better if you can post the printed stack trace.

from sedona.

Yes! At first I've tried default and then tried this
objectRDD.rawSpatialRDD.coalesce(1).persist(StorageLevel.MEMORY_ONLY)
Didn't help.

Sure, stack trace is better :)
java.lang.NullPointerException at org.datasyslab.geospark.knnJudgement.GeometryDistanceComparator.compare(GeometryDistanceComparator.java:44) at scala.math.LowPriorityOrderingImplicits$$anon$7.compare(Ordering.scala:153) at org.apache.spark.util.collection.Utils$$anon$1.compare(Utils.scala:35) at org.spark_project.guava.collect.Ordering.max(Ordering.java:551) at org.spark_project.guava.collect.Ordering.leastOf(Ordering.java:667) at org.apache.spark.util.collection.Utils$.takeOrdered(Utils.scala:37) at org.apache.spark.rdd.RDD$$anonfun$takeOrdered$1$$anonfun$30.apply(RDD.scala:1422) at org.apache.spark.rdd.RDD$$anonfun$takeOrdered$1$$anonfun$30.apply(RDD.scala:1419) at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:796) at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:796) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) at org.apache.spark.rdd.RDD.iterator(RDD.scala:287) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) at org.apache.spark.scheduler.Task.run(Task.scala:99) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

from sedona.

@deil87

Hi, this issue is not made by GeoSpark. When GeoSpark RDD has only one partition and the K in KNN query is larger than 1/2 number of total records, rdd.takeOrdered will crash. I will investigate more on this issue but this will not impact GeoSpark.

Fixing this issue is pretty simple. You just need to load a little bit more data, more than 100 or 1000 records, this issue will get solved. Since GeoSpark focuses on big spatial data, no real or test dataset has less than 100 or 1000 records.

from sedona.

I've already figure it out) But unfortunately I'm the one who likes to write tests and uses small datasets as input. And I've stuck) Maybe comment about this "restriction" would be useful.
Thank you for your quick answers!

from sedona.

@deil87 Thanks for your suggestion. I will add comments in the code and documents.

from sedona.