Giter VIP home page Giter VIP logo

keywhiz-fs's Introduction

Status

We plan to deprecate keywhiz-fs shortly. While this approach has served us well, we've decided the advantages of using FUSE do not outweigh the operational difficulty. A mostly drop-in replacement is https://github.com/square/keysync

Overview

license build coverage

KeywhizFs is a client for Keywhiz which represents accessible secrets as a userland filesystem (using FUSE). This client will mount a directory which contains files for each secret that is accessible.

Exposing secrets as a filesystem has many benefits.

  1. Consumers of secrets require no special libraries or code.
  2. Unix user and group permissions restrict which processes can read a secret.

Transparently, authentication is performed with a Keywhiz server using mutually-authenticated TLS. A client certificate, trusted by Keywhiz, is required and used to authenticate KeywhizFs. Refer to the Keywhiz documentation for generating and managing client access.

Directory structure

KeywhizFs will display all secrets under the top level directory of the mountpoint. Secrets may not begin with the '.' character, which is reserved for special control "files".

Control files

  • .running
  • This "file" contains the PID of the owner process.
  • .clear_cache
  • Deleting this empty "file" will cause the internal cache of KeywhizFs to be cleared. This should seldom be necessary in practice but has been useful at times.
  • .json/
  • This sub-directory mimics the REST API of Keywhiz. Reading files will directly communicate with the backend server and display the unparsed JSON response.

Building

Run make keywhiz-fs to build a binary and make test to run tests.

We use glide to manage vendored dependencies.

Running

/etc/fuse.conf

In order to allow KeywhizFs to expose its filesystems to other users besides the owner of the process, fuse must be configured with the 'user_allow_other' option. Put the following snippet in /etc/fuse.conf.

# The following line was added for keywhiz-fs
user_allow_other

fusermount setuid permissions

The fusermount progam is used within the go-fuse library. Generally, it is installed setuid root, with group read/execute permissions for group 'fuse'. For KeywhizFs to work, the running user must be a member of the 'fuse' group.

mlockall / CAP_IPC_LOCK capability

To prevent secrets from ending up in swap, KeywhizFs will attempt to mlockall memory. This is not required, but is beneficial. To disable this behavior, pass --disable-mlock to keywhiz-fs on startup. Disabling mlockall means that secrets may end up in swap.

If you want to mlockall memory, you will need to make sure the KeywhizFs binary has the CAP_IPC_LOCK capability. On Linux, set the proper capability on the KeywhizFs binary so memory can be locked without running as root. Example assumes your binary is at /sbin/keywhiz-fs.

setcap 'cap_ipc_lock=+ep' /sbin/keywhiz-fs

Usage

usage: keywhiz-fs --key=FILE --ca=FILE [<flags>] <url> <mountpoint>

A FUSE based file-system client for Keywhiz.

Flags:
  --help                   Show context-sensitive help (also try --help-long and --help-man).
  --cert=FILE              PEM-encoded certificate file
  --key=FILE               PEM-encoded private key file
  --ca=FILE                PEM-encoded CA certificates file
  --asuser="keywhiz"       Default user to own files
  --group="keywhiz"        Default group to own files
  --debug                  Enable debugging output
  --timeout=20s            Timeout for communication with server
  --metrics-url=URL        Collect metrics and POST them periodically to the given URL (via HTTP/JSON).
  --metrics-prefix=PREFIX  Override the default metrics prefix used for reporting metrics.
  --syslog                 Send logs to syslog instead of stderr.
  --disable-mlock          Do not call mlockall on process memory.
  --version                Show application version.

Args:
  <url>         server url
  <mountpoint>  mountpoint

The --cert option may be omitted if the --key option contains both a PEM-encoded certificate and key.

Running in Docker

We have included a Dockerfile so you can easily build and run KeywhizFs with all of its dependencies. To build a kewhizfs Docker image run the following command:

docker build --rm -t square/keywhiz-fs .

After building, you can run the newly built image by running:

docker run --device /dev/fuse:/dev/fuse --cap-add MKNOD --cap-add IPC_LOCK --cap-add SYS_ADMIN --security-opt apparmor:unconfined square/keywhiz-fs --debug --ca=/go/src/github.com/square/keywhiz-fs/fixtures/cacert.crt --key=/go/src/github.com/square/keywhiz-fs/fixtures/client.pem https://localhost:443 /secrets/kwfs

Note that we have to pass --device /dev/fuse:/dev/fuse to mount the fuse device into the container, and give SYS_ADMIN capabilities to the container, so it can mount fuse-fs filesystems.

This build mounts the KeywhizFs filesystem at /secrets/kwfs/.

Contributing

Please contribute! And, please see CONTRIBUTING.md.

keywhiz-fs's People

Contributors

alokmenghrajani avatar csstaub avatar deinspanjer avatar dgalling avatar diogomonica avatar gyulaweber avatar homebysix avatar jameswald avatar mcpherrinm avatar mortonfox avatar stfinney avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

keywhiz-fs's Issues

Some situation leads to 100% cpu on dev machine

I've had something happen two or three times recently where I've had a keywhiz process using > 100% cpu on my mac. I'm not sure what it is -- but both times it happened when i'd closed the terminal I was running it in. I'll attempt a proper repro & profile.

For development purposes, can I run the keywhiz server and keywhiz-fs client on my mac?

I have the keywhiz server running locally, and now trying to get the keywhiz-fs client mounted to it so I can get a better understanding as to how this all works and how we might be able to use it for our current solution. I'm getting this error when trying to mount however:

mlockall() not implemented on this system
panic: open client.crt: no such file or directory

goroutine 1 [running]:
panic(0x446d980, 0xc8200f2090)
/usr/local/Cellar/go/1.6/libexec/src/runtime/panic.go:464 +0x3e6
main.panicOnError(0x4cbb5d8, 0xc8200f2090)
/Users/dsgrant/git/src/keywhiz-fs/main.go:163 +0x4b
main.NewClient(0x453cdf0, 0xa, 0x453ce00, 0xa, 0x453c870, 0xa, 0xc820084600, 0x4a817c800, 0x4612000, 0x7fff5fbffb44, ...)
/Users/dsgrant/git/src/keywhiz-fs/client.go:83 +0x207
main.main()
/Users/dsgrant/git/src/keywhiz-fs/main.go:76 +0x671

fusermount failed

I just clone the project and build the docker image as described in the readme. when i run the container, i get this.

docker run -it --device /dev/fuse:/dev/fuse --cap-add=IPC_LOCK --cap-add=SYS_ADMIN keywhizfs -debug=true -ca=/go/src/github.com/square/keywhiz-fs/fixtures/cacert.crt -key=/go/src/github.com/square/keywhiz-fs/fixtures/client.pem https://localhost:443 /secrets/kwfs
ERROR kwfs_main[/secrets/kwfs]: 2015/08/04 05:53:41 Error starting syslog logging, continuing: Unix syslog delivery error
DEBUG kwfs_main[/secrets/kwfs]: 2015/08/04 05:53:41 Certificate file not specified, assuming certificate also in /go/src/github.com/square/keywhiz-fs/fixtures/client.pem
ERROR kwfs_client[/secrets/kwfs]: 2015/08/04 05:53:41 Error starting syslog logging, continuing: Unix syslog delivery error
ERROR kwfs[/secrets/kwfs]: 2015/08/04 05:53:41 Error starting syslog logging, continuing: Unix syslog delivery error
ERROR kwfs_cache[/secrets/kwfs]: 2015/08/04 05:53:41 Error starting syslog logging, continuing: Unix syslog delivery error
/bin/fusermount: mount failed: Permission denied
2015/08/04 05:53:41 Mount fail: fusermount exited with code 256

i'm running

uname -a
Linux -- 3.19.0-21-generic #21-Ubuntu SMP Sun Jun 14 18:31:11 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

docker info
Containers: 17
Images: 255
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 289
Dirperm1 Supported: true
Execution Driver: native-0.2
Kernel Version: 3.19.0-21-generic
Operating System: Ubuntu 15.04
CPUs: 8
Total Memory: 15.45 GiB
Registry: [https://index.docker.io/v1/]
WARNING: No swap limit support

also, i have uncommented user_allow_other in /etc/fuse.conf and added the current user to fuse group. i can view the /etc/fuse.conf with the user starting docker.

Thanks.

Fix Dockerfile to use glide for vendoring

I'm trying to build the docker daemon and getting this error.
I looked at the call in main.go and it has the following signature:

return sqmetrics.NewMetrics(*metricsURL, prefix, metrics.DefaultRegistry)

But https://github.com/square/go-sq-metrics/blob/master/metrics.go shows the following definition:

func NewMetrics(metricsURL, metricsPrefix string, interval time.Duration, registry metrics.Registry) *SquareMetrics {

There aren't any docs in either project to indicate whether it is used or how, but using git blame, I can see that the new parameter was added a week ago in square/go-sq-metrics@d2f04d2 by @csstaub.

Looks like this project just needs to be updated as well?

Invalid link count

Some piece of software is complaining one of its secrets has an invalid link count, 0 != 1.

Seems like it's getting that from fstat. We should make sure link count is 1.

Docker example isn't working

Found two more issues, and I'm going to give up at this point, but I wanted to offer a fix for as far as I got.

  1. The docker_kwfs.sh script has two bugs:
    • The executable name should be /go/bin/keywhiz-fs as that is what is built
    • the asuser and group arguments are long options and need two hyphens rather than one.
  2. The README.md instructs the user to run the container passing an argument '--debug=true'. This is an error and will cause the argument parser to mess up during parsing giving the error "keywhiz-fs: error: unexpected /secrets/kwfs, try --help"

Support StatFs

The StatFs system call should be implemented. Nicer output would then be available when running df.

Allow trailing slash for the URL

./main https://localhost:4444/ random_dir ends up failing with

<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
<title>Error 404 Not Found</title>
</head>
<body><h2>HTTP ERROR 404</h2>
<p>Problem accessing //secrets. Reason:
<pre>    Not Found</pre></p><hr><i><small>Powered by Jetty://</small></i><hr/>

</body>
</html>```

We should fix this in the server or client (or both?).

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.