Scorecard scalability limitation: Reduce GitHub API calls about scorecard HOT 8 CLOSED

• Provide a scorecard network service that collects results for projects periodically.

This is the approach we've taken so far. There's a cron that runs every day on ~100 projects and publishes the results in GCS/BigQuery.

I'd be happy to add the envoy deps into that list, it's here: https://github.com/ossf/scorecard/blob/main/cron/projects.txt

from scorecard.

Adding the envoy deps here: #84

from scorecard.

GitHub API supports conditional requests https://docs.github.com/en/rest/overview/resources-in-the-rest-api#conditional-requests

Most responses return an ETag header. Many responses also return a Last-Modified header. You can use the values of these headers to make subsequent requests to those resources using the If-None-Match and If-Modified-Since headers, respectively. If the resource has not changed, the server will return a 304 Not Modified.
Making a conditional request and receiving a 304 response does not count against your Rate Limit, so we encourage you to use it whenever possible.

Making a conditional request and receiving a 304 response does not count against your Rate Limit, so we encourage you to use it whenever possible.

https://github.com/google/go-github supports Conditional requests https://github.com/google/go-github#conditional-requests

As we are scaling more and more projects this would add a lot of value.

https://github.com/gregjones/httpcache

cc @inferno-chromium

from scorecard.

A visual representation of the proposed solution.

1. k8s cron job runs on a schedule.
2. Initial run fetches information using httpcache as a middleware, which caches the HTTP response initially in a large disk (PVC), probably move to Redis later as a cache instead of disk.
3. Subsequent cron runs will utilize the httpcache for checking content modification and load it from the cache if it isn't modified, which reduces the hitting the Rate Limit of the GitHub API.

from scorecard.

Awesome! What would the cache keys be? URLs?

from scorecard.

Awesome! What would the cache keys be? URLs?

Here it is.
https://github.com/gregjones/httpcache/blob/901d90724c7919163f472a9812253fb26761123d/httpcache.go#L42

from scorecard.

This should reduce the GitHub API usage #227

from scorecard.

Close this, since the solution is being tracked/implemented in #318

from scorecard.