Sometimes we got an exit status 128 from git pull about publiccode-crawler HOT 7 CLOSED

@sebbalex found out that we have orgs in manual-reuse.yml that are already in the onboarding list. That explains the locking of the repos because the crawler is trying to update the same local copy concurrently.

We need to:

• Remove the duplicate orgs
• Check for duplicates before starting the crawl process, for good measure.

from publiccode-crawler.

That caused by git not being able to connect to the repo: Failed to connect to github.com port 443: Connection timed out.

I think we are DoS-ing ourselves launching a go routine for every repo and getting starved for network bandwidth.

from publiccode-crawler.

@sebbalex IMO we can close this one. #189 and #191 solved the root cause and #188 now provides the complete error message.

from publiccode-crawler.

@sebbalex What errors are we getting lately?

from publiccode-crawler.

Issue is still there, today we got 6 repositories in error, detail:

time="2020-10-06T00:11:04Z" level=error msg="[UniversitaDellaCalabria/uniAuth] error while cloning: cannot git pull the repository: exit status 128: fatal: Unable to create '/var/crawler/data/repos/api.github.com/UniversitaDellaCalabria/uniAuth/gitClone/.git/index.lock': File exists.\n\nAnother git process seems to be running in this repository, e.g.\nan editor opened by 'git commit'. Please make sure all processes\nare terminated then try again. If it still fails, a git process\nmay have crashed in this repository earlier:\nremove the file manually to continue.\n"

from publiccode-crawler.

@sebbalex What's the timestamp of the lockfile?

from publiccode-crawler.

As we discussed offline I could not find any lock files in the disk. We need to dig deeper.

from publiccode-crawler.