A few years ago, I wrote and recorded the edX course Implementing Predictive Analytics with Spark in Azure HDInsight, which teaches you how to use the Spark MLLib library to build machine learning solutions in a Spark Azure HDInsight cluster.

Microsoft now also offers Spark capabilities in the Azure Databricks service. This repo contains versions of the lab files that have been modified to use Azure Databricks.

If you're completely new to Spark and Databricks, you might want to check out the Introduction to Databricks Labs first!

The following videos show some of the key concepts and techniques used in the labs. You can watch these to get an overview of using Spark to do machine learning in Azure Databricks; or if you prefer, you can skip straight to the labs. To watch the videos, click the image (or right-click / CTRL+click to open the video in a separate browser tab).

.

Start by following the steps in the Setup Guide to provision your Azure environment and download the labfiles used in the lab exercises. Then complete the labs in the following order:

1. Lab 1 - Exploring Data with Spark.
2. Lab 2 - Building Supervised Learning Models.
3. Lab 3 - Evaluating Supervised Learning Models.
4. Lab 4 - Recommenders and Clustering.
5. Lab 5 - Using the MML Spark Library.