src-d / core-retrieval Goto Github PK
View Code? Open in Web Editor NEWInternal utility library for rovers and borges
License: Apache License 2.0
Internal utility library for rovers and borges
License: Apache License 2.0
See this for related discussion in borges: src-d/borges#109 (review)
Currently some construct functions panic with it's not able to connect to a service. For example:
// Broker returns a queue.Broker for the default queue system.
func Broker() queue.Broker {
if container.Broker == nil {
b, err := queue.NewBroker(config.Broker)
if err != nil {
panic(err)
}
container.Broker = b
}
return container.Broker
}
// Database returns a sql.DB for the default database. If it is not possible to
// connect to the database, this function will panic. Multiple calls will always
// return the same instance.
func Database() *sql.DB {
if container.Database == nil {
container.Database = database.Must(database.Default())
}
return container.Database
}
If these functions can fail they should return an error so connection can be retried or a proper message shown by the program that uses it.
As @rporres said on #57, there's some common high level logic that should be reused across all copiers. For example, copying to a .tmp file and then renaming. Instead of doing this one time, we're implementing this in each copier.
I propose redesigning the copiers to be more like building blocks that are assembled by a Copier
that can have a RemoteFs
(or something like that) that is the one providing client-specific logic to the copier.
Copier would be something like this, containing all the high-level logic:
type Copier struct {
remote RemoteFs
local LocalFs
}
func (Copier) CopyToRemote(ctx context.Context, src, dst string, localFs billy.Filesystem) error {}
func (Copier) CopyFromRemote(ctx context.Context, src, dst string, localFs billy.Filesystem) error {}
And the RemoteFs
/LocalFs
(name TBD) something like this:
type Fs interface {
Open(path string) (io.Reader, error)
WriteTo(path string) (io.Writer, error)
Rename(src, dst string) error
}
type RemoteFs interface {
Fs
isRemote()
}
type LocalFs interface {
Fs
isLocal()
}
With this logic, all we need is a io.Copy
that works with context.Context
and the logic becomes (in pseudo-code):
CopyToRemote(ctx, src, dst) {
r, err = local.Open(src)
if err != nil { return }
w, err = remote.WriteTo(withTmp(dst))
if err != nil { return }
err = copyCtx(ctx, r, w)
if err != nil { return }
remote.Rename(withTmp(dst), dst)
}
It becomes really easy to build high-level logic with very low level building blocks.
In fact, this could be simplified even further into a function, independent of remotes and locals:
type Copier func(ctx context.Context, src, dst string) error
func (c Copier) Copy(ctx context.Context, src, dst string) error { return c(ctx, src, dst) }
func NewCopier(from, to Fs) Copier {
return func(ctx context.Context, src, dst string) error {
// code from above
}
}
What do you think? /cc @src-d/data-retrieval @smola
HSDFCopier
does not write directly into the output siva file. It instead does the copy to a temporary directory and does a rename. This is done so no siva files are corrupted in case of borges error or stop.
Do the same in LocalCopier
.
https://github.com/src-d/core-retrieval/blob/master/repository/copier.go#L55
If we check the content of a random siva file created by the RootedTransactioner
we can see:
IN tmp/sourced/160963116/transactioner/HEAD
IN tmp/sourced/160963116/transactioner/config
IN tmp/sourced/160963116/transactioner/refs/heads/competing-consumers/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/tags/v2.1.1/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/tags/v3.0.5/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/tags/v1.0.1/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/tags/v2.0.2/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/tags/v2.2.0/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/tags/v0.1/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/tags/v0.3.0/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/tags/v2.2.1/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/tags/v2.4.1/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/tags/v0.3.1/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/tags/v0.5.0/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/tags/v2.0.0/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/tags/v3.0.0/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/tags/v3.0.6/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/heads/example-event-type/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/tags/v2.0.1/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/tags/v2.1.0/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/heads/master/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/heads/porotobuf-2.4.1/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/tags/v0.4.0/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/tags/v2.2.2/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/tags/v3.0.4/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/tags/v4.0.0/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/tags/v4.1.0/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/tags/v0.2.0/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/tags/v1.0.0/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/tags/v2.3.0/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/tags/v3.0.1/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/tags/v3.0.2/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/tags/v4.1.1/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/tags/v2.0.3/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/tags/v2.1.2/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/tags/v2.4.0/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/tags/v3.0.3/015ccfd3-3949-1d1b-9336-322afcfeb69c
IN tmp/sourced/160963116/transactioner/refs/tags/v3.0.7/015ccfd3-3949-1d1b-9336-322afcfeb69c
the tmp/sourced/160963116/transactioner
sould not be into the siva file, because when we try to open a repository we will send a path like this: tmp/sourced/933892433/transactioner/refs
. This path will be totally different, because the local path is created randomly.
If HADOOP_VERSION
is not set the default version that the test setup script downloads is 2.7.2
but is not available to download anymore. The closest version is 2.7.4
.
To avoid Travis build fails because the HDFS binary disappears, we need to use a docker container to do the HDFS integration tests.
This method is not used anymore. Temporary *.siva.copy
files are done directly to the destination directory.
https://github.com/src-d/core-retrieval/blob/master/repository/copier.go#L30-L31
Delete it also from NewHDFSFs
arguments:
https://github.com/src-d/core-retrieval/blob/master/repository/copier.go#L163-L167
And from the config struct RootRepositoriesTempDir
:
https://github.com/src-d/core-retrieval/blob/master/container.go#L27
it is CONFIG_H_D_F_S and should be CONFIG_HDFS
Repositories will now be stored in a single siva file. The name of the siva file will be the Init
that will be calculated the first time it is downloaded. The Init
column type is SHA1
.
More info: src-d/borges#389
We are using this library to send siva files to HDFS: github.com/colinmarc/hdfs
. We will need to be able to set a flag to overwrite=true in some cases into the client.Create
method. We can do a PR to these library with the improvement.
The change is small. Now filesystem.NewStorage
doesn't return error and also needs a cache as argument. In sivaRootedTransactioner.Begin
:
c := cache.NewObjectLRUDefault()
sto := filesystem.NewStorage(fs, c)
We may also want to pass the cache to sivaRootedTransactioner
to make it configurable from borges.
After go-billy.v3 migration, packfiles are not copied into the siva file. This behavior doesn't cause any error, just the packfile is not there.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.