There was garbled code when reading Chinese from shp file about sedona HOT 3 CLOSED

I found a solution to the issue of garbled text in issue #190. Adding the code:

System.setProperty("sedona.global.charset","utf8"); 

solved the problem,
but the configuration mentioned on the official website using:

spark.driver.extraJavaOptions -Dsedona.global.charset=utf8 
spark.executor.extraJavaOptions -Dsedona.global.charset=utf8 

did not work.

from sedona.

spark.driver.extraJavaOptions works when submitting the Spark application using spark-submit or using PySpark (#1345), it will alter the Java system properties of the newly spawned Spark driver process. If you are running the main function in local mode all by yourself, System.setProperty is the proper way of setting Java system properties.

Although System.setProperty works in this local setup, it does not work when submitting the Spark application to a cluster. The DBF files are parsed by executors, and calling System.setProperty on the driver won't alter the Java system properties of executors.

The columns of the DataFrame converted from SpatialRDD object are all strings, this is the shortcoming of how we parse DBF files and hold user data in SpatialRDD. The attributes in DBF files are all converted to strings and we don't keep track of their original data types in SpatialRDD. A more proper way to support Shapefiles is by implementing a Shapefile reader based on Spark DataSourceV2, which directly loads Shapefiles as DataFrames.

from sedona.

spark.driver.extraJavaOptions works when submitting the Spark application using spark-submit or using PySpark (#1345), it will alter the Java system properties of the newly spawned Spark driver process. If you are running the main function in local mode all by yourself, System.setProperty is the proper way of setting Java system properties.

Although System.setProperty works in this local setup, it does not work when submitting the Spark application to a cluster. The DBF files are parsed by executors, and calling System.setProperty on the driver won't alter the Java system properties of executors.

The columns of the DataFrame converted from SpatialRDD object are all strings, this is the shortcoming of how we parse DBF files and hold user data in SpatialRDD. The attributes in DBF files are all converted to strings and we don't keep track of their original data types in SpatialRDD. A more proper way to support Shapefiles is by implementing a Shapefile reader based on Spark DataSourceV2, which directly loads Shapefiles as DataFrames.

Thanks!

from sedona.