google / goblet Goto Github PK

When using goblet to serve a repository for a large concurrent CI setup, it might be desirable to make use of https://git-scm.com/docs/git-config#Documentation/git-config.txt-uploadpackpackObjectsHook to wrap around git-pack-objects and cache the output per unique request.

Multiple requests (from CI workers) asking for a same combination of objects would only trigger a single git-pack-objects which will simultaneously write to both process stdout and to an on-disk cache dir. Subsequent requests can be served from the cache dir instead of having to re-trigger git-pack-objects, which will save a lot of CPU time.

Prior art could be found in Gitlab's Gitaly https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/design_pack_objects_cache.md

Hello,

I wanted to say hi and discuss a project that my team has been working on at Renault in a similar timeframe

https://gitlab.com/grouperenault/git_cdn

our design looks very similar, and it makes sense to share experience.

git_cdn is written in python asyncio, and we did not experience much performance bottleneck with the python.
Most of our load comes from the git upload-pack process.

In order to mitigate this load from upload-pack, we had to implement caching of the bigger upload-packs.
This is because our CI users are doing a lot of clone from scratch for reasons that are hard to negociate..

our design also implements cache of LFS files, which can be handy.

We do not support protocol v2 yet, and to my understanding goblet only supports v2.

We would be glad to have a discussions with you either in this issue or in a live hangout session.

Expected Behavior

Use https://github.com/go-git/go-git

Actual Behavior

Project is using gopkg.in/src-d/go-git.v4 v4.13.1 which is fairly old.
Src-d is shutdown and the project has moved.

From V4 to V5, there were several memory issues fixed. Upgrading is a good idea

Steps to Reproduce the Problem

Specifications

• Version:
• Platform:

Expected Behavior

Current code should pass staticcheck linting

Actual Behavior

git_protocol_v2_handler.go:133:55: should use time.Since instead of time.Now().Sub (S1012)
managed_repository.go:118:41: should use time.Since instead of time.Now().Sub (S1012)
reporting.go:96:40: should use time.Since instead of time.Now().Sub (S1012)
reporting.go:148:9: infinite recursive call (SA5007)

Steps to Reproduce the Problem

1. Run staticcheck

Specifications

• Version:
• Platform:

Expected Behavior

bazel test //... should succeed

Actual Behavior

        fetch_test.go:57: cannot execute a git command
            Error: exit status 128
            Args: []string{"/usr/local/bin/git", "-c", "http.extraHeader=Authorization: Bearer valid-server-auth-token", "push", "-f", "http://[::]:42989/", "master:master"}
            Output: fatal: unable to access 'http://[::]:42989/': URL using bad/illegal format or missing URL

Steps to Reproduce the Problem

1. Start the project in Github Codespace
2. Install Bazelisk
3. Run bazel test //...

Specifications

• Version:
• Platform: