GCP Initialization Action about einstein HOT 6 CLOSED

Parallelly, I was hoping to explore gcloud compute ssh to see if that would help us in any way, in case this doesn't fall through.

from einstein.

I tried multiple times yesterday to create a cluster using initialization action gs://uga-dsp/scripts/conda-dataproc-bootstrap.sh. But it started giving some errors.

1. Creating the conda folder
2. Downloading the anaconda
3. All the files are getting deleted (probably an error is occurring during installation and everything is getting deleted)

from einstein.

Today, I was successful to create the cluster with following specs

1. region: us east
2. default worker nodes, memory
3. initialization action: gs://uga-dsp/scripts/conda-dataproc-bootstrap.sh

Please try to create a cluster and check if you are facing any problems

from einstein.

Two things I can point to there;
(a) Can you point to a specific error in the logs which might have been leading to this? We could then reach out to Dr. Quinn accordingly.
(b) Is it more like a chance thing, where the script is working some times, and the other times, it is not? If so, do you think it is worth to spend time on it, taking into consideration the deadlines?

from einstein.

I don't have any specific logs relating to the error. Also I don't think it is a chance thing because the initialization action never failed today as I tested by creating 5 clusters today.

The important thing i noticed is if you try to run the VM instances as soon as you created a cluster with cluster status 'provisioning', the anaconda is not getting installed. It is taking almost 10 min before the cluster is getting ready with everything installed.

from einstein.

That's weird, considering it didn't take as much time during P1.
Okay, we will go ahead and close this issue here, and come back to addressing consistency in initialization actions when we are well ahead with the code base.

from einstein.