Laura Vodden

Machine learning models work by identifying patterns in data and making predictions by attempting to replicate them (Wei, 2020). These patterns are assumed to reflect reality, because they emerge from data concerning actual events and phenomena that have already occurred. In Natural Language Processing (NLP), these patterns can be easily biased (Sun et al., 2019; Wei, 2020). Because NLP deals with the link between computer understandings of human language, it is subject to a complex combination of factors including ambiguity, context and nuance (Hirschberg and Manning, 2015). It is true that languages follow certain rules (grammar, syntax), however these rules are flexible due to cultural differences, local dialects, colloquialism and so forth (Chowdhury, 2003). Literary and lingual devices like irony and sarcasm further complicate understanding (Chowdhury, 2003). Finally, implicit cultural bias can result in patterns that cause bias in machine learning predictions (Zhao et al., 2018). Issues like systemic racial prejudice and gender role bias can and do make their way into machine learning predictions (Sun et al., 2019; Wei, 2020). An example of this machine learning bias is the association of ‘black’ to ‘criminal’ and ‘Caucasian’ to ‘police’, or ‘doctor’ to ‘man’ and ‘nurse’ to ‘woman’ (Ethayarajh, 2020; Wei, 2020).

Despite gender equity making great strides globally in the past several decades, there remains a well-documented gap in society when it comes to female participation in STEM careers (Beede et al., 2011; Robnett, 2016; Wu et al., 2020). Long-held public beliefs regarding gender roles, combined with social expectations associated with these roles has resulted in a lower participation rate of women than men in ‘STEM’ related fields (Robnett, 2016; Sun et al. 2019). In recent years, there has been a renewed push, driven by the accessibility of social media, to raise awareness of this gap and encourage women to enter these careers, and to work to identify hurdles to their progression into more senior roles (Australian Government, 2019). However, as this report show, there is still a significant bias towards men in STEM data, and studies show that a lack of representation can discourage women from pursuing certain careers (Wu et al., 2020).

This report will investigate gender bias in Natural Language Processing and machine learning. The data will be derived from Twitter, due to its role as a repository of global attitudes and sentiments over time (Pozzi et al. 2016). Specifically, it will investigate the association of gender to Tweets relating to ‘science’, ‘technology’, ‘engineering’ and ‘mathematics’ – the so-called ‘STEM’ fields.

This analysis shows that, within STEM-related topics, the language used to discuss men and women in STEM is not significantly different (this result may not be universal; a dataset containing Tweets about engineering and childcare may yield different results). While sentiment towards women in STEM is positive, they are still significantly underrepresented in the data and in real life (Robnett, 2016). The bias towards classifying STEM topics as relating to men has implications for NLP tasks, because this bias is learned and amplified by machine learning algorithms (Sun et al., 2019), and fed back into society, perpetuating potentially damaging (not to mention old fashioned) stereotypes regarding women in STEM (Wei, 2020).

Agarwal, B., & Mittal, N. (2016). Prominent feature extraction for sentiment analysis. Springer.

Albon, C. (2017, December 20). Plot the validation curve. Chris Albon. https://chrisalbon.com/machine_learning/model_evaluation/plot_the_validation_curve/

Anurag, A. (2018, August 17). Random Forest Analysis in ML and when to use it. New Gen Apps. https://www.newgenapps.com/blog/random-forest-analysis-in-ml-and-when-to-use-it/#:~:text=Random%20forest%20algorithm%20can%20be,a%20large%20proportion%20of%20data.

Australian Government. (2020). Advancing Women in STEM strategy. Industry.gov.au. https://www.industry.gov.au/data-and-publications/advancing-women-in-stem-strategy/snapshot-of-disparity-in-stem/women-in-stem-at-a-glance

Beri, A. (2020, May 28). Sentimental analysis using VADER: Interpretation and classification of emotions. Towards Data Science. https://towardsdatascience.com/sentimental-analysis-using-vader-a3415fef7664

Chowdhury, G. (2003). Natural language processing. Annual review of information science and technology, 37(1), 51-89.

Donges, N. (2019, June 16). A complete guide to the random forest algorithm. Built In. https://builtin.com/data-science/random-forest-algorithm

Ethayarajh, K. (2020, November 11). Measuring Bias in NLP with Confidence. Stanford AI. http://ai.stanford.edu/blog/bias-nlp/#:~:text=Countless%20studies%20have%20found%20that,when%20given%20the%20right%20prompt.

Gautam, G., & Yadav, D. (2014). Sentiment analysis of twitter data using machine learning approaches and semantic analysis. In Seventh International Conference on Contemporary Computing (IC3) (pp. 437-442). IEEE.

Goel, V. (2018, November 2) Applying machine learning to classify an unsupervised text document. Towards Data Science. https://towardsdatascience.com/applying-machine-learning-to-classify-an-unsupervised-text-document-e7bb6265f52

Hirschberg, J., & Manning, C. (2015). Advances in natural language processing. Science, 349(6245), 261-266.

Kreiger, J. (2020, January 13). Evaluating a Random Forest model. Medium.com. https://medium.com/analytics-vidhya/evaluating-a-random-forest-model-9d165595ad56

Liaw, A., & Wiener, M. (2002). Classification and regression by Random Forest. R News, 2(3), 18-22.

Malik, U. (2018, January 8). Text Classification with Python and Scikit-Learn. Stack Abuse. https://stackabuse.com/text-classification-with-python-and-scikit-learn/

Meinert, R. (2019, June 6). Optimizing Hyperparameters in Random Forest Classification: What hyperparameters are, how to choose hyperparameter values, and whether or not they’re worth your time. Towards Data Science. https://towardsdatascience.com/optimizing-hyperparameters-in-random-forest-classification-ec7741f9d3f6

Neethu, M. S., & Rajasree, R. (2013). Sentiment analysis in twitter using machine learning techniques. In Fourth International Conference on Computing, Communications and Networking Technologies (pp. 1-5). ICCCNT.

Pozzi, F. A., Fersini, E., Messina, E., & Liu, B. (2016). Sentiment analysis in social networks. Morgan Kaufmann. Robnett, R. D. (2016). Gender bias in STEM fields: Variation in prevalence and links to STEM self-concept. Psychology of Women Quarterly, 40(1), 65-79.

Silman, J. (2019, March 23). Scrape tweets using Selenium. Medium. https://medium.com/@jamessilman/scrape-tweets-using-selenium-3f713873439

Sun, T., Gaut, A., Tang, S., Huang, Y., El Sherief, M., Zhao, J., & Wang, W. Y. (2019). Mitigating gender bias in natural language processing: Literature review. arXiv preprint arXiv:1906.08976.

Wei, J. (2020, September 2). Bias in Natural Language Processing (NLP): A Dangerous But Fixable Problem. Towards Data Science. https://towardsdatascience.com/bias-in-natural-language-processing-nlp-a-dangerous-but-fixable-problem-7d01a12cf0f7

Yiu, T. (2019, June 12). Understanding Random Forest: How the Algorithm Works and Why it Is So Effective. Towards Data Science. https://towardsdatascience.com/understanding-random-forest-58381e0602d2