This code allows you to generate an Arabic corpus based on keywords related to the subject of your study, cleaned and ready. Your corpus will be stored in a json file in the same path as the ".py" is running. The output contains:

• Corpus Name
• Words count : total number of words and diffrent words
• Articles count: number of wikipedia articles returned into the corpus
• Articles: a dictionary containing all the textual data returned stored as { Article's title : Content}

1- Requirments:

• You may need to enable these two commandes if your punkt isn't installed, once dowloaded and installed nltk will work perfectly fine: #1- import nltk #2- nltk.download('punkt')
• Python 2.7 or later
• Internet connection
• At least 20mb of free storage

2- Libraries used:

• Wikipedia: API for wiki articles, installation is required (pip install wikipedia or pip3 if it doesn't work)
• re.sub: function that alllow as to delete portions of text that are or not in the text
• word_tokenize: we use to tokenize (split) our texts into a list of words
• request exception to handle connection errors and timeouts
• sleep for time out exeptions handling
• langdetect: a tool that detects languages, so that we don't return any article that is not written in arabic

3- How to use:

• Call the "BuildCorp" function with its two parameters:
  • Keywords: a string with arabic owrds describing your research or a subject
  • Nmae: name of the json file that will be created after generation
• The more keywords you insert, the bigger and more precise the corpus will be

For more information, please contact:

• [email protected]