This Python script provides functions to preprocess social media text, specifically Reddit and StackOverflow posts, as well as Twitter text.

This script requires Python 3 and the re module. There are no additional dependencies.

To use this script, first import it in your Python code:

from preprocessing import preprocess

Then, call the preprocess() function with the text you want to preprocess as the argument. By default, the function assumes the text is Twitter text, but you can set the is_twitter argument to False if you are preprocessing Reddit or StackOverflow posts:

text = 'RT @user: This is a test tweet with a link https://t.co/123456 and an emoji 😂'
emojis, preprocessed_text = preprocess(text)

The function returns a tuple containing two items:

1. A string containing all extracted emoji characters separated by spaces.
2. The preprocessed text.

Extracts all emoji characters from the text.

emojis = extract_emoji(text)

Removes markdown formatting and symbols from the text.

preprocessed_text = remove_markdown(text)

Removes Twitter-specific symbols from the text.

preprocessed_text = remove_twitter(text)

Preprocesses social media text by first extracting all emoji characters, then removing markdown formatting and symbols, and finally removing Twitter-specific symbols (if is_twitter is set to True).

emojis, preprocessed_text = preprocess(text, is_twitter=True)

@misc{preprocess,
  title = {Preprocessing Social Media Text},
  author = {Heng Ee Tay},
  year = {2023},
  howpublished = {GitHub repository},
  url = {https://github.com/hengee/preprocessing_for_social_media},
  type = {python source code}
}