In this lab, you'll explore applying several types of kernels on some more visual data. At the end of the lab, you'll then apply your knowledge of SVMs to a real world dataset!
You will be able to:
- Create a non-linear SVM in scikit-learn
- Interpret the results of your SVM in scikit-learn
- Apply SVM to a real-world data set
To start, reexamine the final datasets from the previous lab.
from sklearn.datasets import make_blobs
from sklearn.datasets import make_moons
import matplotlib.pyplot as plt
%matplotlib inline
from sklearn import svm
from sklearn.model_selection import train_test_split
import numpy as np
plt.figure(figsize=(10, 4))
plt.subplot(121)
plt.title("Four Blobs")
X_3, y_3 = make_blobs(n_samples=100, n_features=2, centers=4, cluster_std=1.6, random_state = 123)
plt.scatter(X_3[:, 0], X_3[:, 1], c = y_3, s=25)
plt.subplot(122)
plt.title("Two Moons with Substantial Overlap")
X_4, y_4 = make_moons(n_samples=100, shuffle = False , noise = 0.3, random_state=123)
plt.scatter(X_4[:, 0], X_4[:, 1], c = y_4, s=25)
plt.show()
Recall how a radial basis function kernel has 2 hyperparameters: C
and gamma
. To further investigate tuning, you'll generate 9 subplots with varying parameter values and plot the resulting decision boundaries. Take a look at this example from sci-kit learn as inspiration. Each of the 9 plots should look like this:
Note that the score represents the percentage of correctly classified instances according to the model.
# Create a loop that builds a model for each of the 9 combinations
# Prepare your data for plotting
# Plot the prediction results in 9 subplots
#__SOLUTION__
# Plot the prediction results in 9 subplots
plt.figure(figsize=(11, 11))
for (k, (C, gamma, clf)) in enumerate(details):
# evaluate the predictions in a grid
Z = clf.predict(x1x2)
Z = Z.reshape(X1_C.shape)
# visualize decision function for these parameters
plt.subplot(3, 3, k + 1)
plt.title("gam= %r, C= %r, score = %r" % (gamma, C, round(clf.score(X_4,y_4), 2)))
# visualize parameter's effect on decision function
plt.contourf(X1_C, X2_C, Z, alpha = 1)
plt.scatter(X_4[:, 0], X_4[:, 1], c=y_4, edgecolors='gray')
plt.axis('tight')
Repeat what you did before but now, use decision_function
instead of predict
. What do you see?
# Plot the decision function results in 9 subplots
Recall that the polynomial kernel has 3 hyperparameters:
-
$\gamma$ , which can be specified using keywordgamma
-
$r$ , which can be specified using keywordcoef0
-
$d$ , which can be specified using keyworddegree
Build 8 different plots using all the possible combinations between there two values for each:
-
$r= 0.1$ and$2$ -
$\gamma= 0.1$ and$1$ -
$d= 3$ and$4$
Note that decision_function()
cannot be used on a classifier with more than two classes, so simply use predict()
again.
# Create a loop that builds a model for each of the 8 combinations
# Prepare your data for plotting
# Plot the prediction results in 8 subplots
Build a support vector machine using the Sigmoid kernel.
Recall that the sigmoid kernel has 2 hyperparameters:
-
$\gamma$ , which can be specified using keywordgamma
-
$r$ , which can be specified using keywordcoef0
Look at 9 solutions using the following values for
-
$\gamma= 0.001, 0.01$ and$0.1$ -
$r = 0.01, 1$ and$10$
# Create a loop that builds a model for each of the 9 combinations
# Prepare your data for plotting
# Plot the prediction results in 9 subplots
- The polynomial kernel is very sensitive to the hyperparameter settings. Especially setting a "wrong" gamma can have a dramatic effect on the model performance
- Our experiments with the Polynomial kernel were more successful
Explore the same parameters you did before when exploring polynomial kernels
- Do a train test split of 2/3 train vs 1/3 test.
- Train the model on the training set, plot the result and theh accuracy score.
- Next, plot the model with the test set and the resulting accuracy score. Make some notes for yourself on training vs test performance and selecting an appropriate model based on these results.
# Perform a train test split, then create a loop that builds a model for each of the 8 combinations
# Prepare your data for plotting
# Plot the prediction results in 8 subplots on the training set
# Now plot the prediction results for the test set
Until now, you've only explored data sets with 2 features to make it easy to visualize the decision boundary. Remember that you can use Support Vector Machines on a wide range of classification data sets, with more than 2 features. While you will no longer be able to visually represent decision boundaries (at least, if you have more than 3 feature spaces), you'll still be able to make predictions.
To do this, you'll use the salaries dataset again (in salaries_final.csv
).
This dataset has 6 predictors:
-
Age
: continuous. -
Education
: Categorical. Bachelors, Some-college, 11th, HS-grad, Prof-school, Assoc-acdm, Assoc-voc, 9th, 7th-8th, 12th, Masters, 1st-4th, 10th, Doctorate, 5th-6th, Preschool. -
Occupation
: Tech-support, Craft-repair, Other-service, Sales, Exec-managerial, Prof-specialty, Handlers-cleaners, Machine-op-inspct, Adm-clerical, Farming-fishing, Transport-moving, Priv-house-serv, Protective-serv, Armed-Forces. -
Relationship
: Wife, Own-child, Husband, Not-in-family, Other-relative, Unmarried. -
Race
: White, Asian-Pac-Islander, Amer-Indian-Eskimo, Other, Black. -
Sex
: Female, Male.
Simply run the code below to import and preview the dataset. Be sure to note the data type produced bydmatrices
. dmatrices
is often used for preprocessing data with continuous and categorical predictors.
import statsmodels as sm
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
import pandas as pd
salaries = pd.read_csv("salaries_final.csv", index_col = 0)
salaries.head()
.dataframe tbody tr th {
vertical-align: top;
}
.dataframe thead th {
text-align: right;
}
Age | Education | Occupation | Relationship | Race | Sex | Target | |
---|---|---|---|---|---|---|---|
0 | 39 | Bachelors | Adm-clerical | Not-in-family | White | Male | <=50K |
1 | 50 | Bachelors | Exec-managerial | Husband | White | Male | <=50K |
2 | 38 | HS-grad | Handlers-cleaners | Not-in-family | White | Male | <=50K |
3 | 53 | 11th | Handlers-cleaners | Husband | Black | Male | <=50K |
4 | 28 | Bachelors | Prof-specialty | Wife | Black | Female | <=50K |
target = pd.get_dummies(salaries.Target, drop_first=True)
xcols = salaries.columns[:-1]
data = pd.get_dummies(salaries[xcols], drop_first=True)
Now build a simple linear SVM using this data. Note that using SVC, some slack is automatically allowed, so the data doesn't have to perfectly linearly separable.
- Create a train-test-split of 75-25
- Standardize the data
- Fit an SVM model, make sure that you set "probability = True"
- After you run the model, calculate the classification accuracy score
# Split the data into a train and test set
# Standardize the data
# Fit SVM model.
# ⏰ This cell may take several minutes to run
# Calculate the classification accuracy score
Warning: It takes quite a while to compute this! The score is slightly better than the best result obtained using decision trees, but at the cost of computational resources. Changing kernels can make computation times even longer.
Great, you've got plenty of practice in on Support Vector Machines! In this lab you explored kernels and applying SVMs to real-life data!