DataprocSpawner enables JupyterHub to spawn single-user Jupyter notebooks that run on Dataproc clusters. This provides users with ephemeral clusters for data science without the pain of managing them.
- Product Documentation
- DISCLAIMER: DataprocSpawner only supports zonal DNS names. If your project uses global DNS names, click this for instructions on how to migrate.
In order to use this library, you first need to go through the following steps:
- Select or create a Cloud Platform project.
- Enable billing for your project.
- Enable the Google Cloud Dataproc API.
- Setup Authentication.
Python >= 3.6
git clone https://github.com/GoogleCloudPlatform/dataprocspawner
cd dataprocspawner && pip install .
Generate
jupyterhub_config.py
for JupyterHubjupyterhub --generate-config
Within
jupyterhub_config.py
, set the spawner and GCP project. The project must be set.c.JupyterHub.spawner_class = 'dataprocSpawner.DataprocSpawner' c.DataprocSpawner.project = '{GCP project ID}'
To start JupyterHub, run the command:
jupyterhub
Visit https://localhost:8000
in your browser, and sign in with your unix credentials.
- Running JupyterHub locally is likely to run into issues due to firewall rules, use at your own risk/configuration.
To lessen the headache of setup, use the spawner by running JupyterHub within a Docker container that lives in a GCE VM.
Consider storing images on GCR for ease of deployment through GCE.
The VM must allow full access to all Cloud APIs for the spawner to work.
- Install gcloud, part of the Google Cloud SDK
- Create a VM on GCE
If using a Docker image hosted on GCR:
gcloud beta compute instances create-with-container {VM name} --container-image={image URL} --container-arg="--DataprocSpawner.project={GCP project ID}" --scopes=cloud-platform --zone us-central1-a
The spawned notebook by default listens to port 12345 for a connection from the hub. To set a custom port, include this in the gcloud command:
--container-args="--Spawner.port={port number}"
If manually building the Docker image:
gcloud beta compute instances create {VM name} --image-family=cos-stable --image-project=cos-cloud --scopes=cloud-platform --zone us-central1-a
- The container-optimized OS comes with Docker preinstalled.
- The full list of zones can be found here
Visit the Google Cloud Platform Console for GCE
Create an instance
- SSH into the VM
gcloud compute ssh {VM name}
JupyterHub will be running once the VM has been created. No additional commands are necessary. The following is for configuring JupyterHub.
Find the name of the running Docker container
docker ps
Run bash in the running container
docker exec -it {container name} bash
Make changes as desired to
jupyterhub_config.py
(vim, cat, etc.) and exit the containerInstalling vim while inside the conainer:
apt-get update apt-get install vim
Restart the container for changes in
jupyterhub_config.py
to take effectdocker restart {container name}
Check JupyterHub's logs to ensure changes took effect
docker logs -f {container name}
- Clone the DataprocSpawner repo, includes a
Dockerfile
andjupyterhub_config.py
git clone https://github.com/GoogleCloudPlatform/dataprocspawner
- Clone the DataprocSpawner repo, includes a
Add additional configurations to either file, do not change the existing contents.
- If using a GCE instance running a container-optimized OS, allow connections from JupyterHub's REST API (defaults to port 8081)
sudo iptables -w -A INPUT -p tcp --dport 8081 -j ACCEPT
- Build a Docker image from the
Dockerfile
docker build -t jupyterhub .
- Build a Docker image from the
- Run a Docker container using the image
docker run -it --net=host jupyterhub
- The project can be passed as a container argument to Docker instead of setting it within
jupyterhub_config.py
. docker run -it --net=host jupyterhub --DataprocSpawner.project={GCP project ID}
- The project can be passed as a container argument to Docker instead of setting it within
If the Docker image will be used repeatedly, consider pushing the image to GCR.
Cloning the repo, building a Docker image, and pushing it to GCR can be done on a local machine. Follow the instructions for an existing Docker image from above to then use the pushed image on GCE.
DataprocSpawner defaults to port 12345, the port can be set within
jupyterhub_config.py
. More info in JupyterHub's documentation.c.Spawner.port = {port number}
The region default is
us-central1
for Dataproc clusters. The zone default isus-central1-a
. Usingglobal
is currently unsupported. To change region, pick a region and zone from this list and include the following lines injupyterhub_config.py
:c.DataprocSpawner.region = '{region}' c.DataprocSpawner.zone = '{zone that is within the chosen region}'