Support Python >= 3.3 in Dataproc (ensure spark-submit shell has access to global env vars) about initialization-actions HOT 7 CLOSED

@nehalecky first, thanks so much for all the work on the conda init-action! I think I have some good news: after lots (and lots) of banging my head on my desk, I've got a solution to the PYTHONHASHSEED issue.

You need to add this property to spark-defaults.conf:
spark.executorEnv.PYTHONHASHSEED=0
This is, as far as I can tell, the only way to share environment with the executors in YARN deploy modes. Since PYTHONHASHSEED is basically required for Py3 deployments -- as we've learned the hard way -- I'm not sure why it wouldn't be set/available in those environments by default. Feels like an oversight.

Two ways to set this:

1. When creating a dataproc cluster, add it as a property (for example gcloud beta dataproc clusters create --properties spark:spark.executorEnv.PYTHONHASHSEED=0 ...)
2. Explicitly add it to the conf file during startup. I put this line in my initialization action:
   echo "spark.executorEnv.PYTHONHASHSEED=0" >> spark-defaults.conf

As far as I can tell, you CAN'T add it as a property when submitting a job to dataproc. gcloud will reject it, saying Warning: Ignoring non-spark config property: spark:spark.executorEnv.PYTHONHASHSEED=0. Maybe the strange format (the env var is part of the key, not part of the value, so it's a dynamic property) is why it's rejected. ccing @dennishuo @evilsoapbox in case I'm just making a mistake.

So far, this has worked in my very limited testing. Please let me know if it works for you!

from initialization-actions.

Reviewing some of the other comments on PYSPARK_PYTHON -- I actually haven't run into this issue (weird -- but I'm using 1.6.0 in the preview image and maybe the issue has been fixed there) but there is a complementary property in spark-defaults.conf called spark.yarn.appMasterEnv. I think it might work to set spark.yarn.appMasterEnv.PYSPARK_PYTHON=/path/to/python. Since it's also a dynamic property, I'm guessing it'll work when creating a cluster but not when submitting a job -- maybe best to write it explicitly.

from initialization-actions.

@jlowin, thank you much for the reply—I can sympathize with struggle (and hope your head doesn't hurt too much).

I finally got around to testing this, what an amazing find with spark.executorEnv.PYTHONHASHSEED! I was able to resolve the issue using the --properties parameter on cluster creation, however, I couldn't get it to work using your PR. I'm going to make a few notes on the PR, so I'll hop over there now.

from initialization-actions.

@jlowin Oh my, this actually worked. The errors are gone.
I need to check if it worked well, but I believe it did :)

from initialization-actions.

Another way around this issue is using: export SPARK_YARN_USER_ENV=PYTHONHASHSEED=0 on the master node. This will propagate to the executors.

from initialization-actions.

This is also now fixed by https://issues.apache.org/jira/browse/SPARK-13330 but won't be released until Spark 2.2.0.

from initialization-actions.

Closing this issue as Dataproc 1.2 (new default) has Spark 2.2

from initialization-actions.