Giter VIP home page Giter VIP logo

knoxite's Introduction

knoxite

Build Status Coverage Status Go ReportCard GoDoc

knoxite is a secure data storage & backup system.

Join the discussion on IRC in #knoxite (on irc.libera.chat) or our Gitter chat room!

๐Ÿ”’ It's secure

knoxite uses AES-encryption to safely store your data.

๐Ÿ’ช It's flexible

You can always extend your storage size or move your stored data to another machine.

๐Ÿš€ It's efficient

knoxite cleverly de-duplicates your data before storing it and supports multiple compression algorithms.

๐Ÿ”— It's connected

You can use multiple storage backends, even parallely: local disks, Dropbox, Amazon S3 & others.

โค๏ธ It's OpenSource

knoxite is free software. Contribute and spread the word!

Installation

Make sure you have a working Go environment. Follow the Go install instructions.

To install knoxite, simply run:

git clone https://github.com/knoxite/knoxite.git
cd knoxite
go build ./cmd/knoxite/

Or use your favourite package manager:

# Arch Linux (btw)
yay -S knoxite-git

Run knoxite --help to see a full list of options.

Getting started

Initialize a repository

First of all we need to initialize an empty directory (in this case /tmp/knoxite) as a repository:

$ knoxite -r /tmp/knoxite repo init
Enter password:
Created new repository at /tmp/knoxite

knoxite encrypts all the data in the repository with the supplied password. Be warned: if you lose this password, you won't be able to access any of your data.

Initialize a volume

Each repository can contain several volumes, which store our data organized in snapshots. So let's create one:

$ knoxite -r /tmp/knoxite volume init "Backups" -d "My system backups"
Volume 66e03034 (Name: Backups, Description: My system backups) created

List all volumes

Now you can get a list of all volumes stored in this repository:

$ knoxite -r /tmp/knoxite volume list
ID        Name                              Description
----------------------------------------------------------------------------------------------
66e03034  Backups                           My system backups

Storing data in a volume

Run the following command to create a new snapshot and store your home directory in the newly created volume:

$ knoxite -r /tmp/knoxite store [volume ID] $HOME -d "Backup of all my data"
document.txt          5.69 MiB / 5.69 MiB [#########################################] 100.00%
other.txt             4.17 MiB / 4.17 MiB [#########################################] 100.00%
...
Snapshot cebc1213 created: 9 files, 8 dirs, 0 symlinks, 0 errors, 1.23 GiB Original Size, 1.23 GiB Storage Size

When errors occur while storing individual data-chunks knoxite still tries to complete the store operation for the remaining chunks. You can toggle this behaviour to immediately exit on the first erroroneus data-chunk by setting the --pedantic command line flag.

List all snapshots

Now you can get an overview of all snapshots stored in this volume:

$ knoxite -r /tmp/knoxite snapshot list [volume ID]
ID        Date                 Original Size  Storage Size  Description
----------------------------------------------------------------------------------------------
cebc1213  2016-07-29 02:27:15       1.23 GiB      1.23 GiB  Backup of all my data
----------------------------------------------------------------------------------------------
                                    1.23 GiB      1.23 GiB

Show the content of a snapshot

Running the following command lists the entire content of a snapshot:

$ knoxite -r /tmp/knoxite ls [snapshot ID]
Perms       User   Group          Size  ModTime              Name
----------------------------------------------------------------------------------------------
-rw-r--r--  user   group      5.69 MiB  2016-07-29 02:06:04  document.txt
-rw-r--r--  user   group      4.17 MiB  2016-07-29 02:05:22  other.txt
...

Show the content of a snapshotted file

With the following command you can also print out the files content to stdout:

$ knoxite -r /tmp/knoxite cat [snapshot ID] document.txt
This is the sample text stored in document.txt

Restoring a snapshot

To restore the latest snapshot to /tmp/myhome, run:

$ knoxite -r /tmp/knoxite restore [snapshot ID] /tmp/myhome
document.txt          5.69 MiB / 5.69 MiB [#########################################] 100.00%
other.txt             4.17 MiB / 4.17 MiB [#########################################] 100.00%
...
Restore done: 9 files, 8 dirs, 0 symlinks, 0 errors, 1.23 GiB Original Size, 1.23 GiB Storage Size

Cloning a snapshot

Adds target file or directory to an existing snapshot:

$ knoxite -r /tmp/knoxite clone [snapshot ID] $HOME
document.txt          5.89 MiB / 5.89 MiB [#########################################] 100.00%
other.txt             5.10 MiB / 5.10 MiB [#########################################] 100.00%
...
Snapshot aefc4591 created: 9 files, 8 dirs, 0 symlinks, 0 errors, 1.34 GiB Original Size, 1.34 GiB Storage Size

Mounting a snapshot

You can even mount a snapshot (currently read-only, read-write is work-in-progress):

$ knoxite -r /tmp/knoxite mount [snapshot ID] /mnt

Backup. No more excuses.

Configuration System

Knoxite comes bundled with a configuration system. You can declare shorthands for your repositories and provide default values for settings like encryption, compression or excludes. For more information refer to the documentation on our website or take a look into the knoxite config command.

Optional environment variables

Optionally you can set the KNOXITE_REPOSITORY and KNOXITE_PASSWORD environment variables to provide default settings for when no options have been passed to knoxite.

knoxite's People

Contributors

bloodchiefy avatar craftamap avatar darthmochi avatar glaxx avatar ma-hartma avatar muesli avatar penguwin avatar prof-bloodstone avatar rsteube avatar treilik avatar waaaaargh avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

knoxite's Issues

continue backup operations in case of source failure

I have some files in my homedir which are not readable by my user (leftovers from some build process under a different user). When knoxite encounters those files it just stops. It would be great if operations could pursue anyway (logging the error).
I think this should be the default behaviour with a 'stop on error' option.

list snapshot issue

My repository is under /data/nox
I run the command under home dir

root@zmbox:~# knoxite -r /data/nox/ volume list
Enter password:
ID        Name                              Description
--------------------------------------------------------------------------------------------
ebf2dec3  jindie                            database-backup
root@zmbox:~# knoxite -r /data/nox/ snapshot list ebf2dec3
Enter password:
open snapshots/d3554c93: no such file or directory
Unable to load snapshot from any storage backend

under the repository folder it works

root@zmbox:~# cd /data/nox/
root@zmbox:/data/nox# knoxite -r /data/nox/ snapshot list ebf2dec3
Enter password:
ID        Date                 Original Size  Storage Size  Description
------------------------------------------------------------------------------------------------------------
d3554c93  2017-04-23 11:46:02       1.45 GiB    186.20 MiB  init backup
------------------------------------------------------------------------------------------------------------
                                    1.45 GiB    186.20 MiB

storage backend - multiple entries for same storage url

After trying to add an s3 bucket multiple times it would error with:
Storing repository failed

But I see that they show up when doing a 'repo info':

knoxite -r ~/knoxite -p test repo info

Storage URL Available Space

/home/user/knoxite 1.00 GiB
http://s3.amazonaws.com/mybucket 0B
s3://id:[email protected]/us-east-1/mybucket 0B
s3://id:[email protected]/us-east-1/mybucket 0B
s3://id:[email protected]/us-east-1/mybucket 0B
s3://id:[email protected]/us-east-1/mybucket 0B


there should also be a way to remove them I suppose

Streamline commands

Don't make required parameters an option, e.g.:

$ knoxite repo init /some/path

instead of

$ knoxite -r /some/path repo init

Windows binary

Dear @knoxite,
Could you be so kind to generate .exe for the rest of us who are mere Windows users w/o compiler?

Panic with progressbar and gzip

Trying -c gzip it panic when printing the progressbar
I can give more info if needed.

panic: runtime error: makeslice: len out of range

goroutine 1 [running]:
panic(0x7f7c00, 0xc420078f30)
    /usr/local/go/src/runtime/panic.go:500 +0x1a1
strings.Repeat(0xc420051508, 0x1, 0xfffffffffffffff2, 0x1, 0x27)
    /usr/local/go/src/strings/strings.go:420 +0x49
github.com/muesli/goprogressbar.(*ProgressBar).Print(0xc420051880)
    /home/wilk/go/src/github.com/muesli/goprogressbar/progressbar.go:87 +0x5df
main.CmdStore.store(0x0, 0x0, 0x7ffe4bc426f4, 0x4, 0x0, 0x0, 0x0, 0xa65cc0, 0xc420051b50, 0xc4200a0d00, ...)
    /home/wilk/go/src/github.com/knoxite/knoxite/knoxite/store.go:103 +0x7e3
main.CmdStore.Execute(0x0, 0x0, 0x7ffe4bc426f4, 0x4, 0x0, 0x0, 0x0, 0xa65cc0, 0xc4200ad320, 0x2, ...)
    /home/wilk/go/src/github.com/knoxite/knoxite/knoxite/store.go:157 +0x50f
main.(*CmdStore).Execute(0xc420080640, 0xc4200ad320, 0x2, 0x9, 0x1, 0x0)
    <autogenerated>:3 +0x8a
github.com/jessevdk/go-flags.(*Parser).ParseArgs(0xc4200b0420, 0xc42008e010, 0x9, 0x9, 0x41aed8, 0x50, 0x837340, 0x1, 0xc4200aa8c0)
    /home/wilk/go/src/github.com/jessevdk/go-flags/parser.go:315 +0x8e6
github.com/jessevdk/go-flags.(*Parser).Parse(0xc4200b0420, 0xc4200aa8c0, 0x2, 0x2, 0xc42016be90, 0xc400000000)
    /home/wilk/go/src/github.com/jessevdk/go-flags/parser.go:185 +0x74
main.main()
    /home/wilk/go/src/github.com/knoxite/knoxite/knoxite/main.go:51 +0x1c9

lzma support

Can you add some other compression method like LZMA ?
Thank you.

Cannot install, stacktic/dropbox not found

Seems that the package github.com/stacktic/dropbox doesn't exists.

โžœ  knoxite git:(master) go get -u -v
go build && go test -v
github.com/knoxite/knoxite (download)
Fetching https://bazil.org/fuse?go-get=1
Parsing meta tags from https://bazil.org/fuse?go-get=1 (status code 200)
get "bazil.org/fuse": found meta tag get.metaImport{Prefix:"bazil.org/fuse", VCS:"git", RepoRoot:"https://github.com/bazil/fuse"} at https://bazil.org/fuse?go-get=1
bazil.org/fuse (download)
Fetching https://bazil.org/fuse/fs?go-get=1
Parsing meta tags from https://bazil.org/fuse/fs?go-get=1 (status code 200)
get "bazil.org/fuse/fs": found meta tag get.metaImport{Prefix:"bazil.org/fuse", VCS:"git", RepoRoot:"https://github.com/bazil/fuse"} at https://bazil.org/fuse/fs?go-get=1
get "bazil.org/fuse/fs": verifying non-authoritative meta tag
Fetching https://bazil.org/fuse?go-get=1
Parsing meta tags from https://bazil.org/fuse?go-get=1 (status code 200)
Fetching https://bazil.org/fuse/fuseutil?go-get=1
Parsing meta tags from https://bazil.org/fuse/fuseutil?go-get=1 (status code 200)
get "bazil.org/fuse/fuseutil": found meta tag get.metaImport{Prefix:"bazil.org/fuse", VCS:"git", RepoRoot:"https://github.com/bazil/fuse"} at https://bazil.org/fuse/fuseutil?go-get=1
get "bazil.org/fuse/fuseutil": verifying non-authoritative meta tag
Fetching https://golang.org/x/net/context?go-get=1
Parsing meta tags from https://golang.org/x/net/context?go-get=1 (status code 200)
get "golang.org/x/net/context": found meta tag get.metaImport{Prefix:"golang.org/x/net", VCS:"git", RepoRoot:"https://go.googlesource.com/net"} at https://golang.org/x/net/context?go-get=1
get "golang.org/x/net/context": verifying non-authoritative meta tag
Fetching https://golang.org/x/net?go-get=1
Parsing meta tags from https://golang.org/x/net?go-get=1 (status code 200)
golang.org/x/net (download)
github.com/dustin/go-humanize (download)
github.com/klauspost/shutdown2 (download)
github.com/DataDog/zstd (download)
github.com/klauspost/reedsolomon (download)
github.com/klauspost/cpuid (download)
github.com/minio/highwayhash (download)
Fetching https://golang.org/x/sys/cpu?go-get=1
Parsing meta tags from https://golang.org/x/sys/cpu?go-get=1 (status code 200)
get "golang.org/x/sys/cpu": found meta tag get.metaImport{Prefix:"golang.org/x/sys", VCS:"git", RepoRoot:"https://go.googlesource.com/sys"} at https://golang.org/x/sys/cpu?go-get=1
get "golang.org/x/sys/cpu": verifying non-authoritative meta tag
Fetching https://golang.org/x/sys?go-get=1
Parsing meta tags from https://golang.org/x/sys?go-get=1 (status code 200)
golang.org/x/sys (download)
github.com/nu7hatch/gouuid (download)
github.com/restic/chunker (download)
github.com/ulikunitz/xz (download)
github.com/minio/minio-go (download)
github.com/go-ini/ini (download)
github.com/mitchellh/go-homedir (download)
Fetching https://golang.org/x/crypto/argon2?go-get=1
Parsing meta tags from https://golang.org/x/crypto/argon2?go-get=1 (status code 200)
get "golang.org/x/crypto/argon2": found meta tag get.metaImport{Prefix:"golang.org/x/crypto", VCS:"git", RepoRoot:"https://go.googlesource.com/crypto"} at https://golang.org/x/crypto/argon2?go-get=1
get "golang.org/x/crypto/argon2": verifying non-authoritative meta tag
Fetching https://golang.org/x/crypto?go-get=1
Parsing meta tags from https://golang.org/x/crypto?go-get=1 (status code 200)
golang.org/x/crypto (download)
Fetching https://golang.org/x/crypto/blake2b?go-get=1
Parsing meta tags from https://golang.org/x/crypto/blake2b?go-get=1 (status code 200)
get "golang.org/x/crypto/blake2b": found meta tag get.metaImport{Prefix:"golang.org/x/crypto", VCS:"git", RepoRoot:"https://go.googlesource.com/crypto"} at https://golang.org/x/crypto/blake2b?go-get=1
get "golang.org/x/crypto/blake2b": verifying non-authoritative meta tag
Fetching https://golang.org/x/net/http/httpguts?go-get=1
Parsing meta tags from https://golang.org/x/net/http/httpguts?go-get=1 (status code 200)
get "golang.org/x/net/http/httpguts": found meta tag get.metaImport{Prefix:"golang.org/x/net", VCS:"git", RepoRoot:"https://go.googlesource.com/net"} at https://golang.org/x/net/http/httpguts?go-get=1
get "golang.org/x/net/http/httpguts": verifying non-authoritative meta tag
Fetching https://golang.org/x/net/idna?go-get=1
Parsing meta tags from https://golang.org/x/net/idna?go-get=1 (status code 200)
get "golang.org/x/net/idna": found meta tag get.metaImport{Prefix:"golang.org/x/net", VCS:"git", RepoRoot:"https://go.googlesource.com/net"} at https://golang.org/x/net/idna?go-get=1
get "golang.org/x/net/idna": verifying non-authoritative meta tag
Fetching https://golang.org/x/text/secure/bidirule?go-get=1
Parsing meta tags from https://golang.org/x/text/secure/bidirule?go-get=1 (status code 200)
get "golang.org/x/text/secure/bidirule": found meta tag get.metaImport{Prefix:"golang.org/x/text", VCS:"git", RepoRoot:"https://go.googlesource.com/text"} at https://golang.org/x/text/secure/bidirule?go-get=1
get "golang.org/x/text/secure/bidirule": verifying non-authoritative meta tag
Fetching https://golang.org/x/text?go-get=1
Parsing meta tags from https://golang.org/x/text?go-get=1 (status code 200)
golang.org/x/text (download)
Fetching https://golang.org/x/text/transform?go-get=1
Parsing meta tags from https://golang.org/x/text/transform?go-get=1 (status code 200)
get "golang.org/x/text/transform": found meta tag get.metaImport{Prefix:"golang.org/x/text", VCS:"git", RepoRoot:"https://go.googlesource.com/text"} at https://golang.org/x/text/transform?go-get=1
get "golang.org/x/text/transform": verifying non-authoritative meta tag
Fetching https://golang.org/x/text/unicode/bidi?go-get=1
Parsing meta tags from https://golang.org/x/text/unicode/bidi?go-get=1 (status code 200)
get "golang.org/x/text/unicode/bidi": found meta tag get.metaImport{Prefix:"golang.org/x/text", VCS:"git", RepoRoot:"https://go.googlesource.com/text"} at https://golang.org/x/text/unicode/bidi?go-get=1
get "golang.org/x/text/unicode/bidi": verifying non-authoritative meta tag
Fetching https://golang.org/x/text/unicode/norm?go-get=1
Parsing meta tags from https://golang.org/x/text/unicode/norm?go-get=1 (status code 200)
get "golang.org/x/text/unicode/norm": found meta tag get.metaImport{Prefix:"golang.org/x/text", VCS:"git", RepoRoot:"https://go.googlesource.com/text"} at https://golang.org/x/text/unicode/norm?go-get=1
get "golang.org/x/text/unicode/norm": verifying non-authoritative meta tag
Fetching https://gopkg.in/kothar/go-backblaze.v0?go-get=1
Parsing meta tags from https://gopkg.in/kothar/go-backblaze.v0?go-get=1 (status code 200)
get "gopkg.in/kothar/go-backblaze.v0": found meta tag get.metaImport{Prefix:"gopkg.in/kothar/go-backblaze.v0", VCS:"git", RepoRoot:"https://gopkg.in/kothar/go-backblaze.v0"} at https://gopkg.in/kothar/go-backblaze.v0?go-get=1
gopkg.in/kothar/go-backblaze.v0 (download)
github.com/google/readahead (download)
github.com/golang/glog (download)
github.com/pquerna/ffjson (download)
github.com/stacktic/dropbox (download)
# cd .; git clone https://github.com/stacktic/dropbox /Users/prometherion/go/src/github.com/stacktic/dropbox
Cloning into '/Users/prometherion/go/src/github.com/stacktic/dropbox'...
fatal: could not read Username for 'https://github.com': terminal prompts disabled
package github.com/stacktic/dropbox: exit status 128
github.com/jlaffaye/ftp (download)
github.com/muesli/goprogressbar (download)
Fetching https://golang.org/x/crypto/ssh/terminal?go-get=1
Parsing meta tags from https://golang.org/x/crypto/ssh/terminal?go-get=1 (status code 200)
get "golang.org/x/crypto/ssh/terminal": found meta tag get.metaImport{Prefix:"golang.org/x/crypto", VCS:"git", RepoRoot:"https://go.googlesource.com/crypto"} at https://golang.org/x/crypto/ssh/terminal?go-get=1
get "golang.org/x/crypto/ssh/terminal": verifying non-authoritative meta tag
Fetching https://golang.org/x/sys/unix?go-get=1
Parsing meta tags from https://golang.org/x/sys/unix?go-get=1 (status code 200)
get "golang.org/x/sys/unix": found meta tag get.metaImport{Prefix:"golang.org/x/sys", VCS:"git", RepoRoot:"https://go.googlesource.com/sys"} at https://golang.org/x/sys/unix?go-get=1
get "golang.org/x/sys/unix": verifying non-authoritative meta tag
github.com/muesli/gotable (download)
github.com/spf13/cobra (download)
github.com/spf13/pflag (download)
storage/dropbox/storage_dropbox.go:21:2: cannot find package "github.com/stacktic/dropbox" in any of:
	/usr/local/Cellar/go/1.11/libexec/src/github.com/stacktic/dropbox (from $GOROOT)
	/Users/prometherion/go/src/github.com/stacktic/dropbox (from $GOPATH)

Issues getting knoxite

Seeing this, under macOS:

madhatter:bin david$ go get github.com/knoxite/knoxite
# github.com/knoxite/knoxite/lib
../src/github.com/knoxite/knoxite/lib/stat_unix.go:16:44: cannot use s.Dev (type int32) as type uint64 in return argument
../src/github.com/knoxite/knoxite/lib/stat_unix.go:18:44: cannot use s.Nlink (type uint16) as type uint64 in return argument
../src/github.com/knoxite/knoxite/lib/stat_unix.go:21:44: cannot use s.Rdev (type int32) as type uint64 in return argument

Identify unchanged files when updating snapshots

Don't try to store unchanged files again when updating a snapshot.

Currently the de-duplication will prevent collecting & storing redundant data, but we could identify unchanged data locally and avoid the unnecessary encryption and chunking steps.

implement 'exclude' option in the store and clone operations

Need to have the ability to exclude certain patterns when doing backups.

Something like:
-e, --exclude pattern exclude a pattern (can be specified multiple times)

knoxite -r /tmp/knoxite -p pass store 47eded9b ~/src --exclude ~/src/github.com
knoxite -r /tmp/knoxite -p pass clone 347f7cad ~/src --exclude ~/src/github.com

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.