src-d / core-retrieval Goto Github PK

View Code? Open in Web Editor NEW

1.0 1.0 7.0 143 KB

Internal utility library for rovers and borges

License: Apache License 2.0

Makefile 2.65% Go 94.15% Shell 3.20%

core-retrieval's People

Contributors

Stargazers

Watchers

Forkers

ajnavarro smola mcarmonaa erizocosmico jfontan jeroenherczeg isabella232

core-retrieval's Issues

Add a configurable profiler

See this for related discussion in borges: src-d/borges#109 (review)

Do not panic in construct functions

Currently some construct functions panic with it's not able to connect to a service. For example:

// Broker returns a queue.Broker for the default queue system.
func Broker() queue.Broker {
	if container.Broker == nil {
		b, err := queue.NewBroker(config.Broker)
		if err != nil {
			panic(err)
		}

		container.Broker = b
	}

	return container.Broker
}

// Database returns a sql.DB for the default database. If it is not possible to
// connect to the database, this function will panic. Multiple calls will always
// return the same instance.
func Database() *sql.DB {
	if container.Database == nil {
		container.Database = database.Must(database.Default())
	}

	return container.Database
}

If these functions can fail they should return an error so connection can be retried or a proper message shown by the program that uses it.

Redesign Copiers so that high level logic can be reused

As @rporres said on #57, there's some common high level logic that should be reused across all copiers. For example, copying to a .tmp file and then renaming. Instead of doing this one time, we're implementing this in each copier.

I propose redesigning the copiers to be more like building blocks that are assembled by a Copier that can have a RemoteFs (or something like that) that is the one providing client-specific logic to the copier.

Copier would be something like this, containing all the high-level logic:

type Copier struct {
  remote RemoteFs
  local LocalFs
}

func (Copier) CopyToRemote(ctx context.Context, src, dst string, localFs billy.Filesystem) error {}
func (Copier) CopyFromRemote(ctx context.Context, src, dst string, localFs billy.Filesystem) error {}

And the RemoteFs/LocalFs (name TBD) something like this:

type Fs interface {
  Open(path string) (io.Reader, error)
  WriteTo(path string) (io.Writer, error)
  Rename(src, dst string) error
}

type RemoteFs interface {
  Fs
  isRemote()
}

type LocalFs interface {
  Fs
  isLocal()
}

With this logic, all we need is a io.Copy that works with context.Context and the logic becomes (in pseudo-code):

CopyToRemote(ctx, src, dst) {
  r, err = local.Open(src)
  if err != nil { return }

  w, err = remote.WriteTo(withTmp(dst))
  if err != nil { return }

  err = copyCtx(ctx, r, w)
  if err != nil { return }

  remote.Rename(withTmp(dst), dst)
}

It becomes really easy to build high-level logic with very low level building blocks.

In fact, this could be simplified even further into a function, independent of remotes and locals:

type Copier func(ctx context.Context, src, dst string) error

func (c Copier) Copy(ctx context.Context, src, dst string) error { return c(ctx, src, dst) }

func NewCopier(from, to Fs) Copier {
  return func(ctx context.Context, src, dst string) error {
    // code from above
  }
}

What do you think? /cc @src-d/data-retrieval @smola

Add RootRepositoriesTempDir to local copier

HSDFCopier does not write directly into the output siva file. It instead does the copy to a temporary directory and does a rename. This is done so no siva files are corrupted in case of borges error or stop.

Do the same in LocalCopier.

https://github.com/src-d/core-retrieval/blob/master/repository/copier.go#L55

RootedTransactioner: files saved into the siva file has a bad path

If we check the content of a random siva file created by the RootedTransactioner we can see:

IN tmp/sourced/160963116/transactioner/HEAD
IN tmp/sourced/160963116/transactioner/config
IN tmp/sourced/160963116/transactioner/refs/heads/competing-consumers/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/tags/v2.1.1/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/tags/v3.0.5/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/tags/v1.0.1/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/tags/v2.0.2/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/tags/v2.2.0/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/tags/v0.1/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/tags/v0.3.0/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/tags/v2.2.1/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/tags/v2.4.1/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/tags/v0.3.1/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/tags/v0.5.0/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/tags/v2.0.0/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/tags/v3.0.0/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/tags/v3.0.6/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/heads/example-event-type/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/tags/v2.0.1/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/tags/v2.1.0/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/heads/master/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/heads/porotobuf-2.4.1/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/tags/v0.4.0/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/tags/v2.2.2/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/tags/v3.0.4/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/tags/v4.0.0/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/tags/v4.1.0/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/tags/v0.2.0/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/tags/v1.0.0/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/tags/v2.3.0/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/tags/v3.0.1/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/tags/v3.0.2/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/tags/v4.1.1/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/tags/v2.0.3/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/tags/v2.1.2/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/tags/v2.4.0/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/tags/v3.0.3/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/tags/v3.0.7/015ccfd3-3949-1d1b-9336-322afcfeb69c

the tmp/sourced/160963116/transactioner sould not be into the siva file, because when we try to open a repository we will send a path like this: tmp/sourced/933892433/transactioner/refs. This path will be totally different, because the local path is created randomly.

Default hadoop version for tests does not exist

If HADOOP_VERSION is not set the default version that the test setup script downloads is 2.7.2 but is not available to download anymore. The closest version is 2.7.4.

use HDFS docker container instead of a binary to do the integration tests

To avoid Travis build fails because the HDFS binary disappears, we need to use a docker container to do the HDFS integration tests.

add rooted repository transactioner for HDFS

Remove TempDir from repository.Fs interface

This method is not used anymore. Temporary *.siva.copy files are done directly to the destination directory.

https://github.com/src-d/core-retrieval/blob/master/repository/copier.go#L30-L31

Delete it also from NewHDFSFs arguments:

https://github.com/src-d/core-retrieval/blob/master/repository/copier.go#L163-L167

And from the config struct RootRepositoriesTempDir:

https://github.com/src-d/core-retrieval/blob/master/container.go#L27

the environment variable for HDFS is wrong

it is CONFIG_H_D_F_S and should be CONFIG_HDFS

Add Init column to repository

Repositories will now be stored in a single siva file. The name of the siva file will be the Init that will be calculated the first time it is downloaded. The Init column type is SHA1.

More info: src-d/borges#389

hdfs: Implement overwrite flag on client.Create method

We are using this library to send siva files to HDFS: github.com/colinmarc/hdfs. We will need to be able to set a flag to overwrite=true in some cases into the client.Create method. We can do a PR to these library with the improvement.

Not compatible with go-git > 4.7

The change is small. Now filesystem.NewStorage doesn't return error and also needs a cache as argument. In sivaRootedTransactioner.Begin:

	c := cache.NewObjectLRUDefault()
	sto := filesystem.NewStorage(fs, c)

We may also want to pass the cache to sivaRootedTransactioner to make it configurable from borges.

Maintainer update ?

RootedTransactioner: packfile is not added to siva file.

After go-billy.v3 migration, packfiles are not copied into the siva file. This behavior doesn't cause any error, just the packfile is not there.