Interactive Spark lecture using RISE and PySpark.

• Anaconda
• (Optional) Spark 2+ referenced inside a SPARK_HOME environment variable for Spark Streaming.

The following guide has Windows users in mind.

We provide you with a environment.yml which Anaconda can use to create a Python environment named pyspark-interactive-lecture.

conda create -n pyspark-interactive-lecture python=3.7 pip-tools
conda activate pyspark-interactive-lecture
pip install -r requirements.txt

After installation, you should get access to invoke in a GNU terminal, and be able to see all defined tasks for the project with invoke -l.

Documentation on a task with invoke <task> -h.

If you wish to use decktape for exporting your slides as a Reveal.js slidedeck with decktape, you need to prepare your environment. Node.js + npm is required for the following.

npm install

This section is needed if you plan to use native Spark Streaming.

Download Spark with invoke downloadSpark, this will download the archive in bin/ then uncompress it in bin/spark.

In conf/log4j.properties, set log4j.rootCategory=WARN, console.

In conf/spark-defaults.conf, set

spark.sql.shuffle.partitions   4

$ invoke -l
Available tasks:

  clean                  Clean irrelevant directories
  decktape               Specialized export of RISE notebook to a PDF file under the build/ directory
  downloadSpark          Download Spark archive in bin/
  launchSparkStreaming   Launch Spark Streaming structured_network_wordcount.py example
  nbconvert              Convert your lecture notebook to a HTML file, stored in the static/ directory.
  notebook               Launch jupyter notebook to edit notebook files. Ideal for modifying pyspark.ipynb

Run a Jupyter Notebook session : invoke notebook.

If you need to pass a string of arguments : invoke notebook -a "--port=9000"

The invoke command will automatically send bin/spark as the SPARK_HOME environment variablen so you need to have downloaded Spark inside bin/spark before, which is normally easily done in the previous section. If you wish to change that use the --spark_home flag : invoke notebook -s path/to/spark.

We use a personalized nbconvert template pyspark-interactive-lecture.tpl to generate a correct Reveal.js HTML file from the notebook file.

$ invoke nbconvert -h
Usage: inv[oke] [--core-opts] nbconvert [--options] [other tasks here ...]

Docstring:
  Convert your lecture notebook to a HTML file, stored in the src/ directory. With -s/--serve argument, the HTML file is served by a local server as a Reveal.js slideshow.

Options:
  -a STRING, --transition=STRING
  -f STRING, --font-awesome-url=STRING
  -r STRING, --reveal-url-prefix=STRING
  -s, --serve
  -t STRING, --theme=STRING

invoke nbconvert will convert the notebook to a HTML file inside the static/ directory. You can then visualize them by double-clicking on the file, or with python -m http.server.

invoke nbconvert --serve to launch the HTML file with a local server for serving the slides as a Reveal.js slideshow.

Run the following command to run decktape on a background Jupyter notebook: invoke decktape