handle rate limiting about innersource-crawler HOT 3 CLOSED

At least when searching for repos on github.com, you can search like this:
org:meltwater is:public topic:drone

That would return all public repos from my org meltwater containing the topic drone.

I see that in your code you use this right now:
all_repos = gh.repositories()

Instead it might be something similar to this, of course replacing things accordingly based on your .env:
all_repos = gh.search_repositories("org:meltwater is:public topic:drone")

Note: I didn't try this out yet.

from innersource-crawler.

I noticed that in the current implementation you are fetching all repos accessible via the GH Access Token, rather than searching for repos with a given topic only.

Searching by topic would likely bring down the amount of API calls you need quite a bit as well.

Did you choose the current implementation because you don't want to limit this to a single GitHub org?

from innersource-crawler.

@spier Good question! I did it this way because I couldn't figure out how to only fetch repos with a given topic. That would be much more efficient! Any ideas on how to do that?

from innersource-crawler.