Unable to use sedona.global.charset in ShapefileReader about sedona HOT 4 OPEN

@adamaps If you are running Sedona in a cluster mode, this needs to be set via spark.executorEnv.[EnvironmentVariableName]: https://spark.apache.org/docs/latest/configuration.html

In your case, you might want to try this:

spark.executorEnv.sedona.global.charset utf8

spark.executorEnv is a runtime config that can be set after your SparkSession or SedonaContext has been initiated.

spark.conf.set("spark.executorEnv.sedona.global.charset","utf8")
sedona.conf.set("spark.executorEnv.sedona.global.charset","utf8")

from sedona.

Thank you for the quick response, @jiayuasu !

I tested the following in client mode before creating the Sedona SparkSession/SparkContext (via a local Docker container):

conf = SparkConf()
conf.set("sedona.global.charset", "utf8")  # I have other conf settings not shown here
spark = SparkSession.builder.config(conf=conf).getOrCreate()
sedona = SedonaContext.create(spark)

And I tested both of the following in cluster mode after creating the Sedona SparkSession/SparkContext (via AWS Glue):

spark.conf.set("spark.executorEnv.sedona.global.charset","utf8")
sedona.conf.set("spark.executorEnv.sedona.global.charset","utf8")

Unfortunately I still see the same issue in both cases.
Are you able to replicate (or reject) the issue using the attached shapefile sample?

shapefile_sample.zip

from sedona.