1. Project description
2. Results and learnings
   2.1. Main challenge and initial assumptions
   2.2. Dataset
   2.3. Training and evaluation results
   2.4. Model deployment and usage
3. Run the example
   3.1. Prerequisites
   3.2. Train and evaluate the model
   3.3. Deploy web service
4. Code highlights

[back to the top]

This case study shows how to create a model for text analysis and classification and deploy it as a web service in Azure cloud in order to automatically classify support tickets.
This project is a proof of concept made by Microsoft (Commercial Software Engineering team) in collaboration with Endava.
Our combined team tried 3 different approaches to tackle this challenge using:

• Azure Machine Learning Studio - drag-and-drop machine learning tools
• Microsoft Cognitive Toolkit (CNTK) - deep neural networks framework
• Azure Machine Learning Workbench - with Python and classic machine learning algorithms

In this repository we will focus only on AML Workbench and Python scripts used to solve this challenge.

• How to clean and prepare text data and featurize it to make it valuable for machine learning scenarios
• How to strip the data from any sensitive information and also anonymize/encrypt it
• How to create a classification model using Python modules like: sklearn, nltk, matplotlib, pandas
• How to create a web service with a trained model and deploy it to Azure
• How to leverage Azure Machine Learning Services and AML Workbench to make working on ML projects easier, faster and much more efficient

• Karol Żak (GitHub) - Software Development Engineer, Microsoft
• Filip Glavota - Software Development Engineer, Microsoft
• Ionut Mironica (GitHub) - Senior Developer, Endava
• Bogdan Dinu - Senior Development Consultant, Endava
• Bogdan Marin (GitHub) - Senior Developer, Endava
• Florin Vinca - Senior Developer, Endava
• Ioana Raducanu - BI Analyst Developer, Endava
• Andreea Tipau - Developer, Endava

[back to the top]

Disclaimer: This POC and all the learnings you can find bellow is an outcome of close cooperation between Microsoft and Endava. Our combined team spent total of 3 days in order to solve a challenge of automatic support tickets classification.

[back to the top]

• Main challenge we tried to solve was to create a model for automatic support tickets classification for Endavas helpdesk solution. As Endava stated: currently helpdesk operators waste a lot of time evaluating tickets and trying to assign values to properties like: ticket_type, urgency, impact, category, etc. for each submitted ticket
• The dataset we used is Endavas internal data imported from their helpdesk system. We were able to collect around 50k classified support tickets with original messages from users and already assigned labels
• In our POC we focused only on tickets submited in form of an email, similar to the one bellow:

[back to the top]

• For the sake of this repository, data have been stripped out of any sensitive information and anonymized (encrypted). In the original solution we worked on a full dataset without any encryptions. You can download anonymized dataset from here.

• Example of anonymized and preprocessed data from AML Workbench view:
  

• AML Workbench gives some powerfull and easy to use tools for data preparation. And bellow you can see a sample data transformation flow we used while preparing our dataset:
  

• After evaluating the data in AML Workbench we quickly discovered that distribution of values for most of columns we wanted to classify is strongly unbalanced with some of the unique values represented by even as low as 1-2 samples. There are multiple technics to deal with that kind of issues but due to limited amount of time for this POC we were not able to test them in action.

• Distribution of values for each column:

  ticket_type	business_service

  impact	urgency

  category	sub_category1

  sub_category2

[back to the top]

In order to train our models, we used AML Workbench and Azure Machine Learning Services to run training jobs with different parameters and then compare the results and pick up the one with the best values.:

To train models we tested 2 different algorithms: SVM and Naive Bayes. In both cases results were pretty similar but for some of the models, Naive Bayes performed much better (especially after applying hyperparameters) so at some point we decided to work with NB only.

Below you can find some of the results of models we trained to predict different properties:

• ticket_type

  We started from predicting the least unbalanced (and most important from Endavas business point of view) parameter which is ticket_type and after training the model and finding the best hyperparameters using GridSearchCV (which improved precision and recall by around 4%), we were able to achieve some really good results which you can see below:

      class  precision    recall  f1-score   support

          0       0.97      0.95      0.96      2747
          1       0.98      0.99      0.98      6967

avg / total       0.98      0.98      0.98      9714

• business_service

  business_service property is one of the unbalanced features with very low amount of samples/per class for most values.
  We started from running the training on a subset of our dataset where we removed business_service values which were represented by less then 100 samples.
  Unfortunately that didn't help much and we still had a lot of classes that were not at all recognized. So we continued to increase the minimum required number of samples per class untill we started to see some miningfull results:

      class  precision    recall  f1-score   support

         32       0.66      0.95      0.78      1625
         36       0.63      0.73      0.68       792
          4       0.63      0.50      0.56       501
         40       0.68      0.59      0.63       465
         46       0.88      0.62      0.73       344
         51       0.96      0.15      0.25       301
         56       0.84      0.66      0.74       374
         63       0.62      0.13      0.21       367
         66       1.00      0.02      0.04       257
         67       0.67      0.67      0.67       574
         68       0.43      0.76      0.55       727
         70       0.84      0.49      0.62       352
         73       0.93      0.76      0.84       372

avg / total       0.70      0.65      0.62      7051

• category | impact | urgency

  To predict category, impact and urgency we took the same approach as with business_service property but results looked even worse. It's obvious that such level of unbalance within the data makes it impossible to create a model with any miningful results.
  If you would only look at mean/average value of precision and recall you could wrongly assume that results are quite well but if you would check the values of support for each class it would become clear that because one class which covers 70-90% of our data, the results are completely rigged:

 'category'  precision    recall  f1-score   support

         11       1.00      0.03      0.06       120
          3       0.00      0.00      0.00        30
          4       0.82      0.98      0.89      6820
          5       0.87      0.65      0.74      1905
          6       0.65      0.09      0.15       543
          7       0.00      0.00      0.00       207
          8       0.95      0.49      0.65        43
          9       0.00      0.00      0.00        43

avg / total       0.80      0.82      0.79      9711

   'impact'  precision    recall  f1-score   support

          0       0.00      0.00      0.00       112
          1       0.00      0.00      0.00         8
          2       0.00      0.00      0.00        39
          3       0.98      1.00      0.99      9578

avg / total       0.97      0.98      0.98      9737

  'urgency'  precision    recall  f1-score   support

          0       0.00      0.00      0.00       335
          1       0.55      0.18      0.28      1336
          2       0.85      0.98      0.91      8066

avg / total       0.78      0.84      0.79      9737

2.4. Model deployment and usage

[back to the top]

Final model will be used in form of a web service running on Azure and that's why we prepared a sample RESTful web service written in Python and using Flask module. This web service makes use of our trained model and provides API which accepts email body (text) and returns predicted properties.

You can find a running web service hosted on Azure Web Apps here: https://endavaclassifiertest1.azurewebsites.net/.
The project we based our service on with code and all the deployment scripts can be found here: karolzak/CNTK-Python-Web-Service-on-Azure.

Sample request and response in Postman:

3. Run the example

3.1. Prerequisites

[back to the top]

• Download content of this repo

  You can either clone this repo or just download it and unzip to some folder

• Setup Python environment

  In order to run scripts from this repo you should have a proper Python environment setup. If you don't want to setup it locally you can use one of the Data Science Virtual Machine images (both on Linux and Windows) on Azure. All of them come in with most popular data science and machine learning tools and frameworks already preinstalled and ready for you.

• Install dependencies

  Make sure to install all the dependencies for this project. You can easily do it by using requirements.txt file and running this command:

  pip install -r requirements.txt

  Please do report issue if you'll find any errors or missing modules, thanks!

• Download Endava support tickets dataset (all_tickets.csv)

  You can download the dataset from here or by executing 1_download_dataset.py script. If you decide to download it manually, just make sure to put it under:

  project
  └───datasets
      └───all_tickets.csv
  

  Endavas support tickets dataset is already cleaned and stripped out of any unnecessary words and characters. You can check some of the preprocessing operations that were used in 0_preprocess_data.py script.

3.2. Train and evaluate the model

[back to the top]

To train the model you need to run 2_train_and_eval_model.py script. There are some parameters you could posibly play around with - check out code highlights section for more info.

3.3. Deploy web service

[back to the top]

Inside webservice folder you can find scripts to setup a Python based RESTful web service (made with Flask module).

Deeper in that folder you can also find download_models.py script which can be used to download some already trained models that will be used by the web service app.

In order to deploy it to an environment like Azure App Service you can check this GitHub repo for some inspiration.

4. Code highlights

[back to the top]