Recursive hierarchical clustering to overcome missing variables

The purpose of this script is to do Exploratory analysis of how similar the demand of each country is to one another. Instead of using actual country or product features, I am using the approach of comparing weighted demand of SKU products to represent preference of a country relative to another.

I wish to believe the problem with the missing data is that the products might not be offered to the countries in the first place. If that is the case, it is unfair to compare no sales for a country that could not have had demand should there be no sellers. Yet, the dropna() method reduces my dataset drastically, dropping valuable information.

This inspired me to think of a novel method of iteratively doing clustering by a filtered set, tagging the cluster similarity in a correlation matrix, subsequently observing the results.

The logic runs as follows:

1. Randomly select 7 countries
   
2. Keep common products offered as features
   
3. Weight sold quantity values down
   
4. Perform clustering technique (KMeans in my case)
   
5. Tag if same (+1) or different (-1) if two countries are in the same or different cluster
   
6. Loop Steps 1-5 for 100 times (specified by user)
   
7. Calculate average correlation matrix table
   
8. Plot correlation matrix using a gradient chart
   
   

I am using the public dataset Online Retail Data Set for the purpose of this tutorial.