g-node / gin-repo Goto Github PK

G-Node Infrastructure - Repository server

License: Other

Go 96.41% Shell 2.14% Python 1.44%

gin-repo's Introduction

About gin-repo

gin-repo was a git/git-annex repository server for the GIN (G-Node INfrastructure) services prototype and is no longer actively developed.

The current GIN project is developed at the gin-gogs github page, the service can be found at gin.g-node.org.

A detailed description of the GIN project can be found at the G-Node projects site.

References

gin-cli: The GIN command line client (actively developed).
gin-auth: The GIN authentication server (prototype, no longer actively developed).
gin-ui: The GIN web interface based on VueJs (prototype, no longer actively developed).

gin-repo's People

Contributors

Stargazers

Watchers

Forkers

gicmo stoewer mpsonntag achilleas-k cgars

gin-repo's Issues

Handle virtual repo paths that start with`/~/` for `git-annex-shell`

getRepoVisibility: Statement cannot be reached

This statement cannot be reached, since s.repos.GetRepoVisibility never returns an os.IsNotExist error.

Add Coveralls intergration

Similar to gin-auth Coveralls integration.

Unify logging

Currently the project uses golang library log, fmt.Fprint and a custom repod.Server method log throughout the project.

Changes handling collaborators

Currently collaborators are added with repository permission level can-push which cannot be changed.

Add PUT route /users/{user}/repos/{repo}/collaborators/{username}/permission to update the repository permission level of a collaborator. The user facilitating this change has to at least have the is-admin repo access level for the repository in question. Only valid store.repo.AccessLevel are supported.
Edit listRepoCollaborators to include the repository permission level of a collaborator to the response JSON.

Migrate data from portal to gin-repo

Details TBD.

Refactor Repo Visibility

Repo "Visibility" shall be refactored to "Private".

Change proto.wire.Repo.Visibility and all its usages to Private.
Change usage in getRepoVisibility and setRepoVisibility from string to bool.

Only try to authenticate user for the "git" user

Currently we try to lookup keys for all users that try to authenticate not only the git user. This will make login impossible for users that uploaded their keys. :(

Get the current uid (os. Getuid()) and pass %u to gin-shell. Compare the two and bail if they don't match.

Refactor repo.BranchExists to support more features e.g. tags

Change repo.BranchExists here, to something like RefExists to make it reusable for e.g. tags.

Find the original comment here.

Complete git object graph model

The model objects of the git objects in our git packages are not directly connected (they refer to each other via their ids (i.e. SHA1s). Also, opening the same object (i.e. the same id) twice will lead to two different instances.
For some operations it is better to represent the objects as a tree structure in memory. Initial support for this was done in git/graph.go (CommitGraph, CommitNode). Support for the rest of the git objects are missing:

Tree
Blob
Tag
Ref (maybe)

Alternatively we could refactor the existing base git objects to be the graph nodes (i.e. merge their functionality with *Node.

repod: repository exists check

repod.getRepoVisibility requires a repository exists check.

Push docker images from travis to dockerhub

Repository collaborators REST endpoints

Create collaborators REST endpoints:

GET collaborators for a repository.
PUT a new collaborator.
DELETE an existing collaborator.

Tie into git hooks

We should create a universal git hook command line tool to be put in place in the server git repositories that forwards all information passed to the hook (and its identity) to the gin-repod server so we can it can take appropriate action.

Possibility to make repo public on create

Make commit object and tree structs for proto

We could use commit object and tree structs for handling requests.
Currently, this info is built directly into a string (see

gin-repo/cmd/gin-repod/repo.go

Lines 405 to 415 in fbae86c

 w.Header().Set("Content-Type", "application/json") 

 out.WriteString("{") 

 out.WriteString(fmt.Sprintf("%q: %q,\n", "type", "commit")) 

 out.WriteString(fmt.Sprintf("%q: %q,\n", "tree", obj.Tree)) 

 for _, parent := range obj.Parent { 

 out.WriteString(fmt.Sprintf("%q: %q,\n", "parent", parent)) 

 } 

 out.WriteString(fmt.Sprintf("%q: %q,\n", "author", obj.Author)) 

 out.WriteString(fmt.Sprintf("%q: %q,\n", "commiter", obj.Committer)) 

 out.WriteString(fmt.Sprintf("%q: %q", "message", obj.Message)) 

 out.WriteString("}")

Should we have a couple of structs in proto (as with all other response structs) for this? Also, isn't it simpler to marshal the struct instead of constructing a string buffer?

Use common JSON response on error

Depends on gin-core G-Node/gin-core#1.

Refactor existing responses and implement responses where they are still missing.

UserStore: add method to check if user exists

gpgsig support for git objects

Example:

tree bfd89dbc4b90e556713340092c20ff46d0b82368
parent 55cb2b438e594002e8465e1373acea081762d76e
author cgars <[email protected]> 1476889145 +0200
committer cgars <[email protected]> 1476890106 +0200
gpgsig -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v1

 iQIcBAABAgAGBQJYB437AAoJEHQliMt6qIEXdbUQAKccdTeQZBpqB0gSdiKedqbl
 D3k+6548IPgZsXKdM75M2jv9dnClHK1i/8BP0TfF5yILzvvv5ED/GwbGfLPSNuv6
 zBYqiAOFL3cCTZkaxyBtzbTQba0i8FaQaUPoybWTFtuDtqK2i5dK+BHEwwHzA5So
 g/JSJXf1jjZI/jBDnPASw+3rGRdaK31KacppB/e7mYppZHu8FxikdgEzINxTyRm1
 sKHE9euX/fs6T/Tfw7W4qMm9nJmIWOm5R8nsYwKeHACE+UwbZe8tKNjfAQAh89FW
 iTMMoA4A73Mdu0Sz6y8YLLhYiQxCF0wkyAid2poYj+bB2HAYnszptGD+Ny0vjXgl
 R+IjzOWDnNGDgemfAF0cZ3+jzkrwldlPySmIlXtR6BckzdZh+kKs1HoJKs9LsaDx
 k1T4159T7HjQBIOn/73UTXfGW4arW7qg4Jo1EGA6mV7B3msz4rOqkyJQT5WRCUpI
 3xspXTPd1S4Q3pphGkKMYskrPqRv9LM9CG7jM/YSG/6MwN+1VBOdkTvhFNJNsZLI
 20ZC8sJKtR2YdHeliby/K9LqdEBa655sUOoVvCsJk7vpWSfl43iIS+wlQTcUsi09
 k9LAGntpvqRgiqlN9fJiv10+97z751p9lBcF2NoFj7fS6YOuFmfem0ApASGBDw4m
 yy26RY9yRJsx7gaR7fHc
 =Iuow
 -----END PGP SIGNATURE-----

Imp: remove routes for individual hooks and add the fire route

Need to add a field to git.Commit, parse and write it properly.

Broken JSON from gin-repod objectToWire

Entry ID here is missing a comma:

out.WriteString(fmt.Sprintf("%q: %q\n", "id", entry.ID))

Mode here has a comma too many:

out.WriteString(fmt.Sprintf("%q: \"%08o\",\n", "mode", entry.Mode))

Maximim blob (file) size restriction

We don't want big files be directly checked in into git (this is what git annex is for, also it hurts git performance[1]). For this we need to:

install an update hook
in the update hook walk all commits, their trees and their blobs and check the blob's sizes
reject if any blob's size > maxfilesize

The steps above depend on issue #19 and #20

[1] http://stackoverflow.com/questions/17888604/git-with-large-files/19494211#19494211

Path based API for repo browsing

The idea is to be able to do (REST) call (GET) on /user/repo/this/is/the/branch/this/is/the/path and get a directory listening (or the object info, data). We need to do two things

Match the beginning of the path to a git reference or commit
resolve to path to the git object in question

Step 1. could possibly be made more easy by separating the tag from the path via a double slash (//).

Update repository settings

Enable PATCH CORS for repod.
Add a new PATCH REST endpoint /users/{user}/repos/{repo}/settings and update any changed repository settings.

Verified commits break tests

In some cases (see eg. here and when verified commits get packed the pack_test.go test, which tests against the repository it is in if i get it correctly, does fail.

Ill confess that Git and verified commits are a complex issue (see here and here and it might well be that I should not sign my commits whatsoever, at least not if not everybody does so, however, i guess signed commits and Pack files might become an issue again at some point in time.

As a quick "fix": Shall i remove (rewrite) the signed commits for the time being?
Ideas @gicmo @mpsonntag ?

Cleanup hacks from deployed on sever

diff --git a/cmd/gin-repod/intern.go b/cmd/gin-repod/intern.go
index 2d7c89b..7168b72 100644
--- a/cmd/gin-repod/intern.go
+++ b/cmd/gin-repod/intern.go
@@ -2,7 +2,6 @@ package main

 import (
        "encoding/json"
-       "fmt"
        "log"
        "net/http"
        "os"
@@ -20,7 +19,7 @@ func (s *Server) repoAccess(w http.ResponseWriter, r *http.Request) {

        if err != nil || query.Path == "" || query.User == "" {
                w.WriteHeader(http.StatusBadRequest)
-               fmt.Fprintf(os.Stderr, "Error precessing request: %v", err)
+               s.log(DEBUG, "Error processing request: %v", err)
                return
        }

@@ -28,6 +27,7 @@ func (s *Server) repoAccess(w http.ResponseWriter, r *http.Request) {

        if err != nil {
                w.WriteHeader(http.StatusBadRequest)
+               s.log(DEBUG, "Error parsing repo id: %v", err)
                return
        }

@@ -37,6 +37,7 @@ func (s *Server) repoAccess(w http.ResponseWriter, r *http.Request) {
                if os.IsNotExist(err) {
                        w.WriteHeader(http.StatusNotFound)
                } else {
+                       s.log(DEBUG, "Error processsing request: %v", err)
                        w.WriteHeader(http.StatusBadRequest)
                }

@@ -55,7 +56,7 @@ func (s *Server) repoAccess(w http.ResponseWriter, r *http.Request) {
 }

 func (s *Server) lookupUser(w http.ResponseWriter, r *http.Request) {
-       log.Printf("lookupUser: %s @ %s", r.Method, r.URL.String())
+       s.log(DEBUG, "lookupUser: %s @ %s", r.Method, r.URL.String())

        query := r.URL.Query()

diff --git a/cmd/gin-repod/main.go b/cmd/gin-repod/main.go
index 76f901e..44866be 100644
--- a/cmd/gin-repod/main.go
+++ b/cmd/gin-repod/main.go
@@ -256,7 +256,7 @@ Options:
        args, _ := docopt.Parse(usage, nil, true, "gin repod 0.1a", false)
        fmt.Println(args)

-       s := NewServer(":8888")
+       s := NewServer(":8082")
        s.SetupRoutes()
        s.SetupServiceSecret()
        s.SetupStores()
diff --git a/cmd/gin-shell/main.go b/cmd/gin-shell/main.go
index 045be78..09fd5a6 100644
--- a/cmd/gin-shell/main.go
+++ b/cmd/gin-shell/main.go
@@ -28,7 +28,7 @@ Options:

        log.SetOutput(os.Stderr)

-       client := client.NewClient("http://localhost:8888")
+       client := client.NewClient("http://localhost:8082")

        if token, err := makeServiceToken(); err == nil {
                client.AuthToken = token
diff --git a/store/ginauth.go b/store/ginauth.go
index 0cc981d..a1df6a5 100644
--- a/store/ginauth.go
+++ b/store/ginauth.go
@@ -5,6 +5,7 @@ import (
        "fmt"
        "io"
        "io/ioutil"
+       "os"
        "net/http"
        "strings"

@@ -25,6 +26,10 @@ func close(b io.ReadCloser) {

 func (store *GinAuthStore) LookupUserBySSH(fingerprint string) (*User, error) {

+       fingerprint = strings.Replace(fingerprint, "/", "_", -1)
+
+       fmt.Fprintf(os.Stderr, "GAS: looking up fingerprint: %v\n", fingerprint)
+
        address := fmt.Sprintf("%s/api/keys/%s", store.URL, fingerprint)
        res, err := http.Get(address)

diff --git a/store/repo.go b/store/repo.go
index 4bdeeea..bf0dcad 100644
--- a/store/repo.go
+++ b/store/repo.go
@@ -21,6 +21,10 @@ func (id RepoId) String() string {
 }

 func RepoIdParse(str string) (RepoId, error) {
+       // FIXME: we are remove leading, trailing slashes
+       //        maybe that is not a good idea
+       str = strings.Trim(str, "/ ")
+
        if count := strings.Count(str, "/"); count != 1 {
                return RepoId{}, fmt.Errorf("malformed id: wrong number of components: %d", count)
        }
diff --git a/store/user.go b/store/user.go
index 0d25d21..98bda82 100644
--- a/store/user.go
+++ b/store/user.go
@@ -18,5 +18,5 @@ type UserStore interface {
 }

 func NewUserStore(base string) (UserStore, error) {
-       return newLocalUserStore(base)
+       return newGinAuthStore("http://localhost:8081")
 }

Short tests fails because gin.secret is missing

go test -v -short ./... will fail due to the introduction of Test_RepoExists, which creates a local user store. That in turned expects the gin.secret file to be there but since the server won't have started, it actually won't be there. We need to call auth.CreateSharedSecret before.

Pass address of gin-repod to gin-shell

Currently the address is hardcoded, we should have a command line parameter for that.

Handle virtual repo paths that end in .git

URL schemes: Revisit old and design new if needed

The current url schemes in use were done ad hoc, and we should verify that they are the way we want them to be and are consistent also with future needs.

Replace "git log" call

As suggested in this comment replace the git log call here to reduce calls to git binaries.

Depends on #20.

Unified repo and user name checking

We should make sure that we check the contents of user- and repo-names consistently and across services (i.e. gin-auth).

The places where we currently do checks are:

GET repoDescription REST endpoint

Returns

{
"Name": "Repository name",
"Owner":"Repository owner",
"Description": "Repository description",
"Head": "[current head]",
"Public": boolean value
}

Refactor test environment

Currently the test repositories are set up only once when the very first test is run. This means tests changing repository settings are not independent, if repository settings are changed, repositories are created or deleted, currently theses changes all have to be reset at the end of test. This leads to an overhead when creating a test and potential cross reactions when a test is not properly reset.

Figure out a more robust way of handling the test setup.
Refactor the tests that already reset the test environment.

	w.Header().Set("Content-Type", "application/json")
	out.WriteString("{")
	out.WriteString(fmt.Sprintf("%q: %q,\n", "type", "commit"))
	out.WriteString(fmt.Sprintf("%q: %q,\n", "tree", obj.Tree))
	for _, parent := range obj.Parent {
	out.WriteString(fmt.Sprintf("%q: %q,\n", "parent", parent))
	}
	out.WriteString(fmt.Sprintf("%q: %q,\n", "author", obj.Author))
	out.WriteString(fmt.Sprintf("%q: %q,\n", "commiter", obj.Committer))
	out.WriteString(fmt.Sprintf("%q: %q", "message", obj.Message))
	out.WriteString("}")