Working with real-time data streams, and deriving real-time insights using custom machine learning models have become increasingly important for many organizations. There are numerous real-time data platforms currently available (e.g. Kafka, Hadoop, Spark), but the one we will be focusing on in this workshop in particular is Azure Stream Analytics. In addition to diving in to Azure Stream Analytics, we will also explore the open-source cross-plattform library ML.NET, which we will use to build our custom machine learning models and look at an alternative solution using Azure Machine Learning Service.
Expand for instructions to set up prerequisites
- Download the .NET Core SDK
- Go to the following page to download the SDK
- Select the correct tab for your operating system (e.g. Windows, Linux or Mac)
- Click on the Build Apps download option
- Open the installer once the download is complete and follow provided instructions
- Install VS Code
- Go to the following page to download the VS Code
- Select the correct installation for your operating system (e.g. Windows, Linux or Mac)
- Open the installer once the download is complete and follow provided instructions
- Open VS Code once the installation is complete
- Install the C# Extension
- In VS Code, select View -> Extensions
- Search for C#
- Click Install
- Install the Azure Functions Extension
- In VS Code, select View -> Extensions
- Search for Azure Function
- Click Install
- Install the ML.NET CLI
- In VS Code, select Terminal -> New Terminal to open a new terminal window
- In the terminal, enter
dotnet tool install -g mlnetand hit enter
- Clone the repository
- In VS Code, select Terminal -> New Terminal to open a new terminal window
- In the terminal, enter
cd C:\and hit enter - In the terminal, enter
git clone https://github.com/aslotte/mldotnet-real-time-data-streaming-workshop.gitand hit enter to clone the repository to the C: drive.
Note: Feel free to clone the repository elsewhere, just make sure to adjust the path in instructions to follow. Furthermore, the repository is also available on provided USB memory sticks, in case the internet bandwidth is not sufficient.
- Download the data
- There are three (3) ways to get the data we will be working with. Please choose the most convenient for you:
- Create a free Azure subscription
- Go to Azure to create a free trial account
- Enter your contact information and click Next
- Fill in your credit card information and click Next.
Note that this is only used to verify your identify, you'll not be charged.
- Check the checkbox to agree to terms and conditions and click Sign-up
- Create an Outlook e-mail
- Go to Outlook to create a free Outlook account
- Follow the provided instructions
- Download Azure Storage Explorer (required for part 3)
- Download Azure Storage Explorer. Make sure to select the correct OS.
- Open the installer
- Follow the provided instructions
Note to macOS users: If the web site downloads an .exe file even after selecting the macOS option please, download the macOS version from here.
As a financial institution, detecting fraud is imperative to ensure safe and continuous operations for the bank and its customers.
In this workshop we will be looking at detecting fradulent transactions in real-time. We will be training our model based on publicly available data from Kaggle and integrating this custom machine learning model in a real-time data pipeline, supported by Azure Stream Analytics.
- Part 1: Machine Learning in .NET
- Introduction to Machine Learning and ML.NET
- Explore the data with Jupyter Notebooks and Pandas
- Train a machine learning model using ML.NET
- Train a machine learning model using AutoML CLI
- Part 2: Setting up a real-time data streaming pipeline
- Introduction to Stream Processing and Azure Stream Analytics
- Introduction to Azure Resource Management (ARM) Templates
- Part 3: ML.NET + Azure DevOps = MLOps
- Introduction to MLOps
- Set up a CI/CD pipeline for model training
- Part 4: ML.NET + Jupyter
- Introduction to ML.NET in Jupyter Notebooks
- Train a machine learning model using ML.NET and Jupyter Notebooks
- Part 5: Machine Learning in Azure
- Introduction to Azure Machine Learning Service
- Train a machine learning model using Azure ML Visual Interface
- Train a machine learning using Azure AutoML
- Train a machine learning using Jupyter Notebooks and Scikit Learn
- Part 6: Consume ONNX Model from Jupyter Notebook in ML.NET
- Consume an exported ONNX model in ML.NET, which was trained with Scikit Learn (Python)
Reminder: Remember to remove your resource group once finished with this workshop, not to incur additional costs.
This workshop is currently valid for ML.NET v1.4.0
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent