In this lab, we'll use our knowledge of clustering to perform market segmentation on a real-world dataset!
You will be able to:
- Identify and explain what Market Segmentation is, and how clustering can be used for segmentation
- Use clustering algorithms to create and interpret a market segmentation on real-world data
In this lab, we're going to work with the [Wholesale Customers Dataset] from the UCI Machine Learning Datasets Repository. This dataset contains data on wholesale purchasing information from real businesses. These businesses range from small cafes and hotels to grocery stores and other retailers.
Here's the data dictionary for this dataset:
Column | Description |
---|---|
FRESH | Annual spending on fresh products, such as fruits and vegetables |
MILK | Annual spending on milk and dairy products |
GROCERY | Annual spending on grocery products |
FROZEN | Annual spending on frozen products |
DETERGENTS_PAPER | Annual spending on detergents, cleaning supplies, and paper products |
DELICATESSEN | Annual spending on meats and delicatessen products |
CHANNEL | Type of customer. 1=Hotel/Restaurant/Cafe, 2=Retailer. (This is what we'll use clustering to predict) |
REGION | Region of Portugal that the customer is located in. (This column will be dropped) |
One benefit of working with this dataset for practice with segmentation is that we actually have the ground-truth labels of what market segment each customer actually belongs to. For this reason, we'll borrow some methodology from Supervised Learning and store these labels separately, so that we can use them afterwards to check how well our clustering segmentation actually performed.
Let's get started by importing everything we'll need.
In the cell below:
- Import pandas, numpy, and matplotlib.pyplot, and set the standard alias for each.
- Use numpy to set a random seed of
0
. - Set all matplotlib visualizations to appear inline.
Now, let's load our data and inspect it. You'll find the data stored in wholesale_customers_data.csv
.
In the cell below, load the data into a DataFrame and then display the head to ensure everything loaded correctly.
raw_df = None
Now, let's go ahead and store the 'Channel'
column in a separate variable, and then drop both the 'Channel'
and 'Region'
columns. Then, display the head of the new DataFrame to ensure everything worked correctly.
channels = None
df = None
Now, let's get right down to it and begin our clustering analysis.
In the cell below:
- Import
KMeans
fromsklearn.cluster
, and then create an instance of it. Set the number of clusters to2
- Fit the cluster object.
- Get the predictions from the clustering algorithm and store them in
cluster_preds
k_means = None
cluster_preds = None
Now, let's use some of the metrics we've learned about to check the performance of our segmentation. We'll use calinski_harabasz_score
and adjusted_rand_score
, which can both be found inside sklearn.metrics.cluster
.
In the cell below, import these scoring functions.
Now, let's start with CH Score, to get the variance ratio.
Although we don't have any other numbers to compare this to, this is a pretty low score, suggesting that our clusters aren't great.
Since we actually have ground-truth labels in this case, we can actually use the adjusted_rand_score
to tell us how well the clustering performed. Adjust Rand Score is meant to compare two clusterings, which the score can interpret our labels as. This will tell us how similar our predicted clusters are to the actual channels.
Adjusted Rand Score is bounded between -1 and 1. A score close to 1 shows that the clusters are almost identical. A score close to 0 means that predictions are essentially random, while a score close to -1 means that the predictions are pathologically bad, since they are worse than random chance.
In the cell below, call adjusted_rand_score
and pass in our channels
and cluster_preds
to see how well our first iteration of clustering did.
According to these results, our clusterings were essentially no better than random chance. Let's see if we can improve this.
Recall that the results of K-Means Clustering is heavily affected by scaling. Since the clustering algorithm is distance-based, this makes sense. Let's use a StandardScaler
object to scale our dataset and then try our clustering again and see if the results are different.
In the cells below:
- Import a StandardScaler object and use it to transform our dataset.
- Create another K-Means object, fit it to our scaled data, and then use it to predict clusters.
- Calculate the Adjusted Rand Score of our new predictions and our labels.
That's a big improvement! Although it's not perfect, we can see that scaling our data had a significant effect on the quality of our clusters.
Since clustering algorithms are distance-based, this means that dimensionality has a definite effect on their performance. The greater the dimensionality of the dataset, the the greater the total area that we have to worry about our clusters existing in. Let's try using some Principal Component Analysis to transform our data and see if this affects the performance of our clustering algorithm.
Since you've already seen PCA in a previous section, we won't hold your hand through section too much.
In the cells below:
- Import PCA from the appropriate module in sklearn
- Create a
PCA
instance and use it to tranform our scaled data. - Investigate the explained variance ratio for each Principal Component. Consider dropping certain components to reduce dimensionality if you feel it is worth the loss of information.
- Create a new
KMeans
object, fit it to our pca-transformed data, and check the Adjusted Rand Score of the predictions it makes.
NOTE: Your overall goal here is to get the highest possible Adjusted Rand Score. Don't be afraid to change parameters and rerun things to see how it changes.
Question: What was the Highest Adjusted Rand Score you achieved? Interpret this score, and determine the overall quality of the clustering. Did PCA affect the performance overall? How many Principal Components resulted in the best overall clustering performance? Why do you think this is?
Write your answer below this line:
The highest ARS should be ~0.23, which suggests that the clusters are better than random chance, but far from perfect. Overall, the quality of the clustering algorithm did a lot better than the first algorithm we ran on unscaled data. The best performance was achieved when reducing the number of Principal Components down to 4. The increase in model performance is likely due to the reduction in dimensionality. Although dropping the last 2 PCs means that we lose about 6% of our explained variance, this proved to be a net-positive trade-off for the reduction in dimensionality it provided.
Now that we've tried doing market segmentation with K-Means Clustering, let's end this lab by trying with HAC!
In the cells below, use Agglomerative Clustering to make cluster predictions on the datasets we've created, and see how HAC's performance compares to K-Mean's performance.
NOTE: Don't just try HAC on the PCA-transformed dataset--also compare algorithm performance on the scaled and unscaled datasets, as well!
In this lab, we used our knowledge of clustering to perform a market segmentation on a real-world dataset. We started with a cluster analysis with poor performance, and then implemented some changes to iteratively improve the performance of the clustering analysis!