Hi @natalielhtdata, this issue describes the next step for analysing the survey data.

The input data can be found on our Google Drive here (will send you the link on Slack). You can use the survey_data.csv file

The main output should be a list of categorised issues and interventions. So, basically we want to add a new column to survey_data.csv called category

The code in PR #1 already clusters the issues and interventions into some categories. However, we need to make some improvements:

• Removing generic verbs from the interventions (eg, "decrease", "increase")
• Improving the clustering method to assign all points to a cluster (at the moment the HDBSCAN method puts some points in a "noise" cluster). You can do this by using HDBSCAN by reassigning the "noise" points to their closest clusters, or you can also try k-means clustering. To do all of this, you can actually copy the clustering module from here which will make it very easy to do. You can see an example of using this module here - and you can also check in with Rosie if you have questions.
• At this point we should have all the points clustered into categories - here we should make decision whether the clustering results make sense, or whether we need to pivot to a different method (ie, classifying the issues using OpenAI API).
• If the clustering seems OK, then we should finally give the clusters good names. This could be done manually, but we can also use OpenAI API by providing all the issues and interventions and asking for a short name.

If you have issues with setting up Python on your laptop, you can also initially use Google Colab