DataprocSpawner enables JupyterHub to spawn single-user Jupyter notebooks that run on Dataproc clusters. This provides users with ephemeral clusters for data science without the pain of managing them.

• Product Documentation
• DISCLAIMER: DataprocSpawner only supports zonal DNS names. If your project uses global DNS names, click this for instructions on how to migrate.

In order to use this library, you first need to go through the following steps:

1. Select or create a Cloud Platform project.
2. Enable billing for your project.
3. Enable the Google Cloud Dataproc API.
4. Setup Authentication.

Python >= 3.6

git clone https://github.com/GoogleCloudPlatform/dataprocspawner
cd dataprocspawner && pip install .

1. Generate jupyterhub_config.py for JupyterHub

   jupyterhub --generate-config

2. Within jupyterhub_config.py, set the spawner and GCP project. The project must be set.

   c.JupyterHub.spawner_class = 'dataprocSpawner.DataprocSpawner'
   c.DataprocSpawner.project = '{GCP project ID}'

To start JupyterHub, run the command:

jupyterhub

Visit https://localhost:8000 in your browser, and sign in with your unix credentials.

• Running JupyterHub locally is likely to run into issues due to firewall rules, use at your own risk/configuration.

To lessen the headache of setup, use the spawner by running JupyterHub within a Docker container that lives in a GCE VM.

Consider storing images on GCR for ease of deployment through GCE.

The VM must allow full access to all Cloud APIs for the spawner to work.

1. Install gcloud, part of the Google Cloud SDK
2. Create a VM on GCE

• If using a Docker image hosted on GCR:

  gcloud beta compute instances create-with-container {VM name} --container-image={image URL} --container-arg="--DataprocSpawner.project={GCP project ID}" --scopes=cloud-platform --zone us-central1-a

  • The spawned notebook by default listens to port 12345 for a connection from the hub. To set a custom port, include this in the gcloud command:

    --container-args="--Spawner.port={port number}"

• If manually building the Docker image:

  gcloud beta compute instances create {VM name} --image-family=cos-stable --image-project=cos-cloud --scopes=cloud-platform --zone us-central1-a

  • The container-optimized OS comes with Docker preinstalled.
  • The full list of zones can be found here

1. Visit the Google Cloud Platform Console for GCE

2. Create an instance

   

• Check the box under 'Container'

  

• Provide the URL to the container image and set the project

  
  • A custom port can also be set by adding another command argument

• Set the access scopes

  

• Hit create!

1. SSH into the VM

gcloud compute ssh {VM name}

JupyterHub will be running once the VM has been created. No additional commands are necessary. The following is for configuring JupyterHub.

• Find the name of the running Docker container

  docker ps

• Run bash in the running container

  docker exec -it {container name} bash

• Make changes as desired to jupyterhub_config.py (vim, cat, etc.) and exit the container

  • Installing vim while inside the conainer:

    apt-get update
    apt-get install vim

• Restart the container for changes in jupyterhub_config.py to take effect

  docker restart {container name}

• Check JupyterHub's logs to ensure changes took effect

  docker logs -f {container name}

• Clone the DataprocSpawner repo, includes a Dockerfile and jupyterhub_config.py

  git clone https://github.com/GoogleCloudPlatform/dataprocspawner

• Add additional configurations to either file, do not change the existing contents.

  • Dockerfile
  • jupyterhub_config.py

• If using a GCE instance running a container-optimized OS, allow connections from JupyterHub's REST API (defaults to port 8081)

  sudo iptables -w -A INPUT -p tcp --dport 8081 -j ACCEPT

• Build a Docker image from the Dockerfile

  docker build -t jupyterhub .

• Run a Docker container using the image

  docker run -it --net=host jupyterhub

  • The project can be passed as a container argument to Docker instead of setting it within jupyterhub_config.py.

    docker run -it --net=host jupyterhub --DataprocSpawner.project={GCP project ID}

  • If the Docker image will be used repeatedly, consider pushing the image to GCR.

Cloning the repo, building a Docker image, and pushing it to GCR can be done on a local machine. Follow the instructions for an existing Docker image from above to then use the pushed image on GCE.

• DataprocSpawner defaults to port 12345, the port can be set within jupyterhub_config.py. More info in JupyterHub's documentation.

  c.Spawner.port = {port number}

• The region default is us-central1 for Dataproc clusters. The zone default is us-central1-a. Using global is currently unsupported. To change region, pick a region and zone from this list and include the following lines in jupyterhub_config.py:

  c.DataprocSpawner.region = '{region}'
  c.DataprocSpawner.zone = '{zone that is within the chosen region}'

This is not an official Google product.