rajeshias / all-repo-non-code-data-downloader Goto Github PK

Currently, the script builds up all the data in repo_data, issues_data, and pr_data and then writes out all the data afterwards, only once all the data is collected. This scales poorly with the number of repositories, especially when the script is used to iterate over ALL GitHub repos. Instead, the data for a particular repo should be retrieved and then immediately written out to disk, with those variables then being reset for the next repo.

Hardcoding '\' is not a good way, because it won't work in linux based OSes, always prefer os.path.join

The current code has many options that are useful for testing. However, the goal of the project is to iterate over all public repos. Therefore, the first 50 or so lines of code that handle user input to adjust the scope of the scraping is uneeded. Instead, the code should iterate over the repos provided by the iteratable g.get_repos().

When the user does not specify a root path (e.g., ""), the script should save the results in the current working directory.

Add a requirements.txt file with all the dependencies that must be installed via PyPI.