Comments (7)
@nehalecky first, thanks so much for all the work on the conda init-action! I think I have some good news: after lots (and lots) of banging my head on my desk, I've got a solution to the PYTHONHASHSEED
issue.
You need to add this property to spark-defaults.conf
:
spark.executorEnv.PYTHONHASHSEED=0
This is, as far as I can tell, the only way to share environment with the executors in YARN deploy modes. Since PYTHONHASHSEED
is basically required for Py3 deployments -- as we've learned the hard way -- I'm not sure why it wouldn't be set/available in those environments by default. Feels like an oversight.
Two ways to set this:
- When creating a dataproc cluster, add it as a property (for example
gcloud beta dataproc clusters create --properties spark:spark.executorEnv.PYTHONHASHSEED=0 ...
) - Explicitly add it to the
conf
file during startup. I put this line in my initialization action:
echo "spark.executorEnv.PYTHONHASHSEED=0" >> spark-defaults.conf
As far as I can tell, you CAN'T add it as a property when submitting a job to dataproc. gcloud
will reject it, saying Warning: Ignoring non-spark config property: spark:spark.executorEnv.PYTHONHASHSEED=0
. Maybe the strange format (the env var is part of the key, not part of the value, so it's a dynamic property) is why it's rejected. ccing @dennishuo @evilsoapbox in case I'm just making a mistake.
So far, this has worked in my very limited testing. Please let me know if it works for you!
from initialization-actions.
Reviewing some of the other comments on PYSPARK_PYTHON
-- I actually haven't run into this issue (weird -- but I'm using 1.6.0 in the preview image and maybe the issue has been fixed there) but there is a complementary property in spark-defaults.conf
called spark.yarn.appMasterEnv
. I think it might work to set spark.yarn.appMasterEnv.PYSPARK_PYTHON=/path/to/python
. Since it's also a dynamic property, I'm guessing it'll work when creating a cluster but not when submitting a job -- maybe best to write it explicitly.
from initialization-actions.
@jlowin, thank you much for the replyโI can sympathize with struggle (and hope your head doesn't hurt too much).
I finally got around to testing this, what an amazing find with spark.executorEnv.PYTHONHASHSEED
! I was able to resolve the issue using the --properties
parameter on cluster creation, however, I couldn't get it to work using your PR. I'm going to make a few notes on the PR, so I'll hop over there now.
from initialization-actions.
@jlowin Oh my, this actually worked. The errors are gone.
I need to check if it worked well, but I believe it did :)
from initialization-actions.
Another way around this issue is using: export SPARK_YARN_USER_ENV=PYTHONHASHSEED=0
on the master node. This will propagate to the executors.
from initialization-actions.
This is also now fixed by https://issues.apache.org/jira/browse/SPARK-13330 but won't be released until Spark 2.2.0.
from initialization-actions.
Closing this issue as Dataproc 1.2 (new default) has Spark 2.2
from initialization-actions.
Related Issues (20)
- [livy] update livy init action for 2.1 HOT 1
- [rapids] please update to work with latest dask-rapids v22.12 HOT 2
- [gpu] Driver does not install on 2.2 Rocky/Ubuntu images HOT 1
- [zeppelin] not supported on 2.1+ image versions HOT 1
- Error on wget livy binary naming HOT 5
- [spark-rapids] Drop Spark 2.x support in spark-rapids.sh
- [gpu] apt-get update Init script seeing broken repositories HOT 2
- [bigtable] apt-get update Init script seeing broken repositories
- [cloud-sql-proxy] Running the Cloud SQL Proxy as a persistent service
- Update initialization scripts to install latest RAPIDS `23.12` OR `24.02` HOT 2
- [gpu] Add tests for GPU agent HOT 1
- initialization actions which use apt-get update fail due to purged oldoldstable backports repository HOT 10
- rstudio.sh is unable to get the receive keys. Maybe due to invalid repo key. HOT 1
- Dataproc "apt-get update" failed on ubuntu20 HOT 4
- dataproc Initialization actions for trino and hudi
- [gpu] Driver installation breaking in Dataproc 2.1 image during initialization HOT 15
- MLVM and horovod init action fails for 2.1 and 2.2 HOT 1
- [bigtable] bigtable.sh references deprecated Hortonworks Nexus resources HOT 2
- [hue] mysql installer prompts for user input HOT 1
- [gpu] Rocky kernel-devel is sometimes moved to the vault HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ๐๐๐
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google โค๏ธ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from initialization-actions.