This work is supported by the Ministry of Higher Education and Scientific Research in Algeria (Project C00L07UN100120180002) Conception & Supervision : L. Ouahrani & D. Bennouar / Contributor : Abdennour BenHamida.

This code allows to generate a corpus from the web based on keywords related to the subject of the study, cleaned and ready to use. The corpus will be stored in a json file in the same path as the "WebCorp.py" is running. The output contains:

• Corpus Name
• Words count : total number of words and diffrent words
• Articles count: number of wikipedia articles returned into the corpus
• Articles: a dictionary containing all the textual data returned stored as { Article's title : Content}

1- Requirments:

• You may need to enable these two commandes if your punkt isn't installed, once dowloaded and installed nltk will work perfectly fine: #1- import nltk #2- nltk.download('punkt')
• Python 2.7 or later
• Internet connection
• At least 20mb of free storage

2- Libraries used:

• Bs4: BeatifulSoup4 is a python library used for webscrapping (extracting web content from html or xml pages)
• re.sub: function that alllow as to delete portions of text that are or not in the text
• word_tokenize: we use to tokenize (split) our texts into a list of words
• request exception to handle connection errors and timeouts
• sleep for time out exeptions handling
• langdetect: a tool that detects languages, so that we don't return any article that is not written in arabic
• json: storing results

3- How to use:

• Call the "toJson" function with its two parameters:
  • Keywords: a string with words describing your research or a subject (a bigger keychain leads to a much more detailed corpus and richer content)
  • Nmae: name of the json file that will be created after generation
• The more keywords you insert, the bigger and more precise the corpus will be

For further questions or inquiries about this code, you can contact:

• [email protected]