tywkeene / autobd Goto Github PK

Write tests for index/index.go

Really annoying and complicated, add the ability to do this within autobd.

Write a complete readme about what autobd is, isn't, and what it is supposed to become

Write a detailed readme for each API endpoint and how they're expected to behave

Ensure files aren't transferred while they are being written to on the server.

Switch to a docker image that already has git/golang installed. Wouldn't be an issue but I'm on a very slow connection.

Write tests for client/client.go

Add support for delta compression

This will require configuration file support (via toml) so we can specify a certificate.
Probably also a good idea to add command line flags as well.

Write a nice clean and shiny web interface to the autobd-server.
Some start features may be:
A nice formatted index of files and metadata about files on the server
Status of nodes registered with the server
Controlled access to files on the server (i.e user accounts and restrictions to what each user may access.)

Add a -no-inflate which will not inflate or unpack the tar content sent to the node.

No point in existing if there's no point in existing.

Allow for a node to write its *Node structure to a file and use it in a future instance.

Write the client side handling of timers and api calls.

Go through each project and justify it's existence, refactor, or remove it

Colored output, tags, logging levels, etc.

Find or write a package

Add settings like 'update_interval' and 'timeout' options to the config.

These will be used in node.go

Add support for sending file usage/network usage/benchmark data and other data to influxdb for analysis and testing.

gzip'd file transfer over http via the '/sync?grab=file' endpoint

Find an appropriate authorization method

Add a '-gen-config' option that will write a config file with some 'sane' defaults

Break up helpers.go into multiple files.
This will probably be logging.go for logging functions and packing.go for tar related functions

WriteFile() and GetQuery() will go into api.go, since it's the only file that requires them.

Add a log level option that allows for suppressing or extending log messages that are output

Autobd-sever chokes on generating a sha512 checksum for large files (>1GB)

Autobd (API 0.0.6) (Node 0.0.6) (git commit 3dd117eba4-dirty)
fatal error: runtime: out of memory

runtime stack:
runtime.throw(0x6f1d8e, 0x16)
    /usr/lib/go/src/runtime/panic.go:566 +0x95
runtime.sysMap(0xc4fa2e0000, 0x6d0f0000, 0x0, 0x8589d8)
    /usr/lib/go/src/runtime/mem_linux.go:219 +0x1d0
runtime.(*mheap).sysAlloc(0x83f4c0, 0x6d0f0000, 0x40e949)
    /usr/lib/go/src/runtime/malloc.go:407 +0x37a
runtime.(*mheap).grow(0x83f4c0, 0x36871, 0x0)
    /usr/lib/go/src/runtime/mheap.go:726 +0x62
runtime.(*mheap).allocSpanLocked(0x83f4c0, 0x36871, 0x45102a)
    /usr/lib/go/src/runtime/mheap.go:630 +0x4f2
runtime.(*mheap).alloc_m(0x83f4c0, 0x36871, 0x7f0100000000, 0x7ffed2029a68)
    /usr/lib/go/src/runtime/mheap.go:515 +0xe0
runtime.(*mheap).alloc.func1()
    /usr/lib/go/src/runtime/mheap.go:579 +0x4b
runtime.systemstack(0x7ffed2029a70)
    /usr/lib/go/src/runtime/asm_amd64.s:314 +0xab
runtime.(*mheap).alloc(0x83f4c0, 0x36871, 0x100000000, 0x7ffed2029ad8)
    /usr/lib/go/src/runtime/mheap.go:580 +0x73
runtime.largeAlloc(0x6d0e2000, 0x845100, 0xc420000201)
    /usr/lib/go/src/runtime/malloc.go:774 +0x93
runtime.mallocgc.func1()
    /usr/lib/go/src/runtime/malloc.go:669 +0x3e
runtime.systemstack(0x83a700)
    /usr/lib/go/src/runtime/asm_amd64.s:298 +0x79
runtime.mstart()
    /usr/lib/go/src/runtime/proc.go:1079

goroutine 1 [running, locked to thread]:
runtime.systemstack_switch()
    /usr/lib/go/src/runtime/asm_amd64.s:252 fp=0xc4200479e0 sp=0xc4200479d8
runtime.mallocgc(0x6d0e2000, 0x0, 0x818300, 0x2a6)
    /usr/lib/go/src/runtime/malloc.go:670 +0x903 fp=0xc420047a80 sp=0xc4200479e0
runtime.rawbyteslice(0x6d0e0a2b, 0x0, 0x0, 0x0)
    /usr/lib/go/src/runtime/string.go:307 +0xc2 fp=0xc420047ac0 sp=0xc420047a80
runtime.stringtoslicebyte(0x0, 0xc48d1be000, 0x6d0e0a2b, 0xc420047b40, 0xc48d1be000, 0x6d0e0a2b)
    /usr/lib/go/src/runtime/string.go:145 +0xcb fp=0xc420047b08 sp=0xc420047ac0
io.WriteString(0x7ff841e7e800, 0xc4200cc000, 0xc48d1be000, 0x6d0e0a2b, 0xc48d1be000, 0x6d0e0a2b, 0x0)
    /usr/lib/go/src/io/io.go:291 +0xe1 fp=0xc420047b60 sp=0xc420047b08
github.com/tywkeene/autobd/index.GetChecksum(0xc42000d620, 0x27, 0x0, 0x0, 0x0, 0x0)
    /home/savior/src/go/src/github.com/tywkeene/autobd/index/index.go:45 +0x2bf fp=0xc420047c98 sp=0xc420047b60
github.com/tywkeene/autobd/index.NewIndex(0xc42000d620, 0x27, 0x6d0e0a2b, 0xecf8927d6, 0x1c90b30b, 0x83a020, 0x1a4, 0xc420047d50)
    /home/savior/src/go/src/github.com/tywkeene/autobd/index/index.go:55 +0x16d fp=0xc420047cf0 sp=0xc420047c98
github.com/tywkeene/autobd/index.GetIndex(0xc4200b8ba0, 0x1e, 0x3, 0x3e, 0x0)
    /home/savior/src/go/src/github.com/tywkeene/autobd/index/index.go:77 +0x2d0 fp=0xc420047de8 sp=0xc420047cf0
github.com/tywkeene/autobd/locks.InitLocks(0xc4200b8ba0, 0x1e, 0xc4200111a4)
    /home/savior/src/go/src/github.com/tywkeene/autobd/locks/locks.go:96 +0x55 fp=0xc420047ea8 sp=0xc420047de8
main.init.1()
    /home/savior/src/go/src/github.com/tywkeene/autobd/main.go:35 +0x153 fp=0xc420047f40 sp=0xc420047ea8
main.init()
    /home/savior/src/go/src/github.com/tywkeene/autobd/main.go:90 +0x60 fp=0xc420047f48 sp=0xc420047f40
runtime.main()
    /usr/lib/go/src/runtime/proc.go:172 +0x1bf fp=0xc420047fa0 sp=0xc420047f48
runtime.goexit()
    /usr/lib/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc420047fa8 sp=0xc420047fa0

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
    /usr/lib/go/src/runtime/asm_amd64.s:2086 +0x1

The manifest directory endpoint returns a json encoded directory listing of the requested directory.

Ensure the node-side code communicates with the server API in a consistent manner and returns appropriate errors in all cases

When transferring a file from the sync endpoint to the client node, if a file or directory is specified that is inside a directory or directory tree that doesn't exist on the client, the client will return an error when receiving the file or directory

File permissions end up getting messed up.
Probably going to just take the perms from the manifest and write the file with them.

Probably via raft or some other predefined solution. This is too complex to rewrite.

Pack directories in a tar archive and transfer them over http via the '/sync?grab=dir' endpoint

If a node requests a manifest, and then a sync when (large) files are being written to in the server's filesystem, the server will fail to pack the directory, returning nothing or partial files/only some files.

Probably might need to find a way to delay the sync until the write is finished. e.g an inotify descriptor that watches the top and will 'lock' syncs until the filesystem isn't being written to.

This also poses some other problems, like if there's a huge and very long write happening, sync will be postponed for a long time, resulting in a very large sync when it's done. This could also be solved by an inotify descriptor for every directory, but I really don't want to go down that road again.

Not sure why I didn't do this before.

The version endpoint returns the server version, api version and git commit of the server in a json encoded struct

The way autobd returns errors to API clients is a bit too revealing. Either find a way to make them less transparent or remove the functionality altogether

Implement a command line flag -one-shot that will connect to the remote server(s) specified in either the config or command like, ask for a manifest, sync, and quit

Go into depth about the design and implementation of autobd.

Write clear and succinct comments with emphasis on the main moving parts
api/api.go
client/client.go
index/index.go
node/node.go

It's pretty bad right now. Definitely needs to be reworked and moved into more manageable pieces

Write a command line tool that interacts with an autobd server instance

Write tests to ensure all endpoints behave as expected and return appropriate errors in all cases.

The git tree is detected as dirty in the build script if built in a container. Both the travis builder and the container built by the Dockerfile shipped with autobd produce this behavior.

The current CompareManifest() function isn't recursive. Make it, you know, recursive.

Add volume mounting to the Dockerfile

toml config file support

Periodically update each node's status on the server.

Allow for the autobd-server side to write a list of nodes to disk