Introduction In any machine learning task, cleaning or preprocessing the data is as important as model building if not more. And when it comes to unstructured data like text, this process is even more important.

Objective of this kernel is to understand the various text preprocessing steps with code examples.

Some of the common text preprocessing / cleaning steps are:

Lower casing Removal of Punctuations Removal of Stopwords Removal of Frequent words Removal of Rare words Stemming Lemmatization Removal of emojis Removal of emoticons Conversion of emoticons to words Conversion of emojis to words Removal of URLs Removal of HTML tags Chat words conversion Spelling correction So these are the different types of text preprocessing steps which we can do on text data. But we need not do all of these all the times. We need to carefully choose the preprocessing steps based on our use case since that also play an important role.

For example, in sentiment analysis use case, we need not remove the emojis or emoticons as it will convey some important information about the sentiment. Similarly we need to decide based on our use cases