The goals of the mounter are:

• Maintain per volume mount information
• Reconstruct mount information up startup

I think a simple volume which resides on local host can be useful for new / first time user of osd.

A btrfs or nfs needs some configuration, but a file/folder based volume will be helpful while trying the osd, which require zero configuration.

function io(..) in drivers/test/driver.go has an incorrect diff test causing make test failure.

This has relatively undefined behavior within the context of this test package:

• if volume.VolumeDriver is a pointer, the pointer reference will be copied, ie modifications to the VolumeDriver will be made for both contexts
• if volume.VolumeDriver is a struct, the struct will be copied, i.e. modifications to the VolumeDriver will not be made for the original context
• same reasoning for api.VolumeID, which theoretically could be:

type VolumeID interface {
  fmt.Stringer
}

func NewVolumeID() VolumeID {
  return &volumeID{uuid.NewV4().String()}
}

type volumeID struct {
  value string
}

func (v *volumeID) String() string {
  return v.value
}

func (v *volumeID) SomeStatefulModificationThatWeAddedWhenSleepy() {
  if whyDidWeDoThis() {
    becauseWeWereSleepySadFace()
  }
}

Following instructions but still got below errors:
Got below errors when I am trying to give a try to openstorage

go get -d github.com/libopenstorage/openstorage/...

cd /home/jupiturliu/tmp/src/github.com/libopenstorage/openstorage; git pull --ff-only

From https://github.com/libopenstorage/openstorage

• [new branch] master -> origin/master
• [new branch] proto -> origin/proto
  There is no tracking information for the current branch.
  Please specify which branch you want to merge with.
  See git-pull(1) for details

OpenStorage should support the CoprHD controller (https://coprhd.github.io/)

This will allow OpenStorage to work with EMC (and other) storage arrays.

Some functions, such as attach, detach, format are specific to block storage systems.

For a filesystem that wants to integrate with open storage, these functions may not make sense.

The volume.Register function should allow for a provider to specific the type of driver being registered and therefore implement a partial set of functionality.

it is very slow

On typing primitives, my experience: leave dynamic values untyped, static values (ie enums) typed.

I used to type strings like api.VolumeID the same way, but my experience has been that it just gets in the way and doesn't add any protection. Engs still end up casting whenever something doesn't work, and generally its just a bunch of annoying casting.

For enums, using strings (as I mention in #71) leads to insonsistent usage, as some engs will just use the freeform value, even though another eng might look at the list of values and think this is all that is used. We already see this with Filesystem - btrfs is used as a Filesystem, but is not defined. There were more to on the branch proto: https://github.com/libopenstorage/openstorage/blob/proto/api/api.proto#L11

Thoughts?

Hi
Is there any benchmark can run for certainer applications like NoSql database or Sql database?

Regards,
James

I tried to do osd -d -f config_btrfs.yaml, which of course does not work with out all the normal btrfs fun times :) I think that the btrfs driver should be the good getting started case, as it does not require any external dependencies from your local (linux) machine like nfs and aws do (thoughts on that?).

I have this https://github.com/peter-edge/dotfiles/blob/master/bin/setup_docker_storage.sh which I can adapt for osd, what does everyone think?

I'm seeing this whenever I run docker volume ls or docker volume rm:

WARN[0009] 404 Failed to locate volume: Cannot locate volume test  Driver=nfs ID=test Request=remove

I think I can fairly reproduce this issue:

• Launch osd with: /usr/local/bin/osd -d -f /usr/local/etc/osd.yml
• Create a new volume: docker volume create --driver=nfs --name=test1
• Stop the daemon with Ctrl+C
• Relaunch osd with:/usr/local/bin/osd -d -f /usr/local/etc/osd.yml
• Run docker volume ls or docker volume rm test1

I'm running:

$ osd --version
osd version 0.3

$ docker version
Client:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.2
 Git commit:   a34a1d5
 Built:        Fri Nov 20 13:25:01 UTC 2015
 OS/Arch:      linux/amd64

Server:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.2
 Git commit:   a34a1d5
 Built:        Fri Nov 20 13:25:01 UTC 2015
 OS/Arch:      linux/amd64

Config file is:

osd:
  drivers:
      nfs:
        server: "10.1.8.43"
        path: "/home/sites/configurations"

In order for the UI to hit OSD it needs a port to hit. currently we only provide a SOCK.

github.com/libopenstorage/openstorage/vendor/github.com/docker/docker/daemon/graphdriver/btrfs

vendor/github.com/docker/docker/daemon/graphdriver/btrfs/btrfs.go:8:25: fatal error: btrfs/ioctl.h: No such file or directory
#include <btrfs/ioctl.h>
^
compilation terminated.
make: *** [install] Error 2

Right now, labels can be passed down to the drivers. Having special or specific labels that mean specific things will help standardize all drivers and provide a known way to pass properties down to the drivers while provisioning volumes.

For example, having a specific label such as one for a class of service will prevent different drivers from having different labels for the same purpose.

Hi,

I am really interested in this project, and would like to contribute.
But I cant see any discussion on this forum. It will be great to have design & development discussion on github, so anyone interested from outside team can easily pickup.

sudo /home/jupiturliu/work/bin/osd -d -f config.yaml
INFO[0000] Starting volume driver: nfs
INFO[0000] NFS driver initializing with 127.0.0.1:/nfs
INFO[0000] Unable to mount 127.0.0.1:/nfs at /var/lib/openstorage/nfs/ (permission denied)
WARN[0000] Unable to start volume driver: nfs, permission denied
jupiturliu@jupiturliu:~/work/src/github.com/libopenstorage/openstorage$ sudo /home/jupiturliu/work/bin/osd -d -f config.yaml
INFO[0000] Starting volume driver: buse
INFO[0000] BUSE initializing in clustered mode
INFO[0000] Adding cluster event listener: buse
INFO[0000] BUSE initialized and driver mounted at: /var/lib/openstorage/buse/
INFO[0000] Starting REST service on /var/lib/osd/driver/buse.sock
INFO[0000] Starting REST service on /run/docker/plugins/buse.sock
INFO[0000] Starting volume driver: nfs
INFO[0000] NFS driver initializing with 127.0.0.1:/nfs
INFO[0000] Unable to mount 127.0.0.1:/nfs at /var/lib/openstorage/nfs/ (permission denied)
WARN[0000] Unable to start volume driver: nfs, permission denied

Also typo at

Same file, what is line 39 for?

Hi,

If any changes in struct or API are done, It should be communicated.

I think a simple process may be followed such changes.

• A issues can be created for such changes, explaining need of change or just a reference point.
• After checkin, Update the issue with what changes result into other codebase.

This is with reference of adding checkin e74adf7

Thanks,
Kunal

Examples:

import (
  log "github.com.Sirupsen/logrus"
  graph "github.com/docker/docker/daemon/graphdriver"
)

It makes the code difficult to follow in terms of what packages are actually being used - only use if there is a conflict

It's defined and used in some places, but in others you see stuff like:

ctx.Filesystem = "btrfs"

volume/volume.go:124:// BlockDriver needs to be implemented by block volume drivers.  Filesystem volume
volume/drivers/test/driver.go:29:   Filesystem string
volume/drivers/test/driver.go:39:       Filesystem:   string(""),
volume/drivers/test/driver.go:92:           Format:  api.Filesystem(ctx.Filesystem),
volume/drivers/aws/aws_test.go:24:  ctx.Filesystem = "ext4"
volume/drivers/btrfs/btrfs.go:80:       return api.BadVolumeID, fmt.Errorf("Filesystem format (%v) must be %v",
volume/drivers/nfs/nfs_test.go:30:  ctx.Filesystem = "nfs"
volume/drivers/btrfs/btrfs_test.go:57:  ctx.Filesystem = "btrfs"
api/types.go:85:// Filesystem supported filesystems
api/types.go:86:type Filesystem string
api/types.go:89:    FsNone Filesystem = "none"
api/types.go:90:    FsExt4 Filesystem = "ext4"
api/types.go:91:    FsXfs  Filesystem = "xfs"
api/types.go:92:    FsZfs  Filesystem = "zfs"
api/types.go:93:    FsNfs  Filesystem = "nfs"
api/types.go:100:   SpecFilesystem       = "format"
api/types.go:115:   Format Filesystem
api/types.go:156:   // Format Filesystem type if any
api/types.go:157:   Format Filesystem
api/client/client_test.go:50:   ctx.Filesystem = string("btrfs")
api/server/docker.go:139:       case types.SpecFilesystem:
api/server/docker.go:140:           spec.Format = types.Filesystem(v)
cli/volumes.go:86:      Format:           api.Filesystem(context.String("fs")),

Docker 1.9 will have a new volumes API - add openstorage support for moby/moby#14242

It should reference a null driver and not an actual driver to avoid driver dependancies in the top level framework.

Then you can just do go install ./... :)

A lot of customers I talk to want to provision a volume to a container but stage the volume with contents from various sources, like an NFS server, github etc.

So can we have a spec file that is hopefully in YAML format that specified the initialization steps, such as

- format: ext4
- create:
  - /mnt
  - /stage/1
  - /stage/2
- stage: 
  - nfs://localhost/s1:/stage/1
  - nfs://externalhost/s2:/stage/2

I found this strange issue on my centos machine.
While docker container is created with volume of osd, mount API is not being invoked.
The same code runs well on Ubuntu.

Steps to reproduce

1. Create a volume with osd.
2. Pass the volume in docker run

Both machine have same code of osd and docker version (tested with 1.8.2 and 1.9.0-rc1)

Here are dump of messages of socket from centos machine

$ sudo socat -v UNIX-LISTEN:/run/docker/plugins/myvfs.sock UNIX-CONNECT:/run/docker/plugins/vfs.sock
[sudo] password for kunal:
> 2015/10/05 11:14:24.180673  length=208 from=0 to=207
POST /VolumeDriver.Create HTTP/1.1\r
Host: /run/docker/plugins/myvfs.sock\r
User-Agent: Go 1.1 package http\r
Content-Length: 29\r
Accept: application/vnd.docker.plugins.v1.1+json\r
\r
{"Name":"test222","Opts":{}}
< 2015/10/05 11:14:24.181600  length=130 from=0 to=129
HTTP/1.1 200 OK\r
Date: Mon, 05 Oct 2015 05:44:24 GMT\r
Content-Length: 13\r
Content-Type: text/plain; charset=utf-8\r
\r
{"Err":null}
> 2015/10/05 11:14:24.182740  length=196 from=208 to=403
POST /VolumeDriver.Path HTTP/1.1\r
Host: /run/docker/plugins/myvfs.sock\r
User-Agent: Go 1.1 package http\r
Content-Length: 19\r
Accept: application/vnd.docker.plugins.v1.1+json\r
\r
{"Name":"test222"}
< 2015/10/05 11:14:24.183292  length=144 from=130 to=273
HTTP/1.1 200 OK\r
Date: Mon, 05 Oct 2015 05:44:24 GMT\r
Content-Length: 27\r
Content-Type: text/plain; charset=utf-8\r
\r
{"Mountpoint":"","Err":{}}
> 2015/10/05 11:14:36.600748  length=196 from=404 to=599
POST /VolumeDriver.Path HTTP/1.1\r
Host: /run/docker/plugins/myvfs.sock\r
User-Agent: Go 1.1 package http\r
Content-Length: 19\r
Accept: application/vnd.docker.plugins.v1.1+json\r
\r
{"Name":"test222"}
< 2015/10/05 11:14:36.601803  length=144 from=274 to=417
HTTP/1.1 200 OK\r
Date: Mon, 05 Oct 2015 05:44:36 GMT\r
Content-Length: 27\r
Content-Type: text/plain; charset=utf-8\r
\r
{"Mountpoint":"","Err":{}}

Dump from ubuntu machine.

$ sudo socat -v UNIX-LISTEN:/run/docker/plugins/myvfs.sock UNIX-CONNECT:/run/docker/plugins/vfs.sock
> 2015/10/19 18:15:57.832352  length=179 from=0 to=178
POST /Plugin.Activate HTTP/1.1\r
Host: /run/docker/plugins/myvfs.sock\r
User-Agent: Go 1.1 package http\r
Content-Length: 5\r
Accept: application/vnd.docker.plugins.v1.1+json\r
\r
null
< 2015/10/19 18:15:57.833967  length=149 from=0 to=148
HTTP/1.1 200 OK\r
Date: Mon, 19 Oct 2015 09:15:57 GMT\r
Content-Length: 32\r
Content-Type: text/plain; charset=utf-8\r
\r
{"Implements":["VolumeDriver"]}
> 2015/10/19 18:15:57.834478  length=207 from=179 to=385
POST /VolumeDriver.Create HTTP/1.1\r
Host: /run/docker/plugins/myvfs.sock\r
User-Agent: Go 1.1 package http\r
Content-Length: 28\r
Accept: application/vnd.docker.plugins.v1.1+json\r
\r
{"Name":"vol9","Opts":null}
< 2015/10/19 18:15:57.835367  length=130 from=149 to=278
HTTP/1.1 200 OK\r
Date: Mon, 19 Oct 2015 09:15:57 GMT\r
Content-Length: 13\r
Content-Type: text/plain; charset=utf-8\r
\r
{"Err":null}
> 2015/10/19 18:15:57.835723  length=193 from=386 to=578
POST /VolumeDriver.Path HTTP/1.1\r
Host: /run/docker/plugins/myvfs.sock\r
User-Agent: Go 1.1 package http\r
Content-Length: 16\r
Accept: application/vnd.docker.plugins.v1.1+json\r
\r
{"Name":"vol9"}
< 2015/10/19 18:15:57.836346  length=144 from=279 to=422
HTTP/1.1 200 OK\r
Date: Mon, 19 Oct 2015 09:15:57 GMT\r
Content-Length: 27\r
Content-Type: text/plain; charset=utf-8\r
\r
{"Mountpoint":"","Err":{}}
> 2015/10/19 18:15:57.891437  length=194 from=579 to=772
POST /VolumeDriver.Mount HTTP/1.1\r
Host: /run/docker/plugins/myvfs.sock\r
User-Agent: Go 1.1 package http\r
Content-Length: 16\r
Accept: application/vnd.docker.plugins.v1.1+json\r
\r
{"Name":"vol9"}
< 2015/10/19 18:15:57.895529  length=176 from=423 to=598
HTTP/1.1 200 OK\r
Date: Mon, 19 Oct 2015 09:15:57 GMT\r
Content-Length: 59\r
Content-Type: text/plain; charset=utf-8\r
\r
{"Mountpoint":"/var/lib/osd/mounts/vol9/.data","Err":null}
^C

Has anyone tried on CentOS?

Include parent volume ID if the volume was created from a snap or the source where the volume data was seeded from

To mount volume created by is osd, VolumeID is required for reference.
This can be fixed using lookup-function for volume-name which returns the volumeID from KVDB

Current Implementation.

$ sudo osd localfs create vol1
cfbb378d-1e0b-4321-8946-63cbb76ccd4d

$ docker run -it -v cfbb378d-1e0b-4321-8946-63cbb76ccd4d:/data --volume-driver localfs ubuntu bash

Expected behavior

$ sudo osd localfs create vol1
cfbb378d-1e0b-4321-8946-63cbb76ccd4d

$ docker run -it -v vol1:/data --volume-driver localfs ubuntu bash

Apcera @earlruby would like an interface to be able to query the underlying node capabilities in terms of storage performance so that the container placement decisions can be influenced.

make test doesn't pass either, but I care less about that.

In the future, we should always have code checked in at master passing make docker-test. This is what I consider the golden state, and what is replicable for OSS developers. It's also what our CI should be based on.

I'm going to get this fixed now.

create
inspect
update
enumerate
attach
mount
io
unmount
detach
delete
create
attach
mount
create
attach
unmount
detach
delete
detach
delete
shutdown
snap
create
attach
snapInspect
snapEnumerate
snapDiff
snapDelete
detach
delete
shutdown
--- FAIL: TestAll (6.97s)
        Error Trace:    driver.go:220
            driver.go:56
            driver.go:61
            btrfs_test.go:55
    Error:      Expected value not to be nil.
    Messages:   An error is expected but got nil. Mount of different devices to same path must fail

        Error Trace:    driver.go:281
            driver.go:224
            driver.go:56
            driver.go:61
            btrfs_test.go:55
    Error:      Received unexpected error "Failed to destroy btrfs snapshot /var/test_dir/volumes/subvolumes for cd208762-cbf3-4f75-97d8-c9b21d0f0cdc: device or resource busy"
    Messages:   Delete failed

FAIL

https://golang.org/s/go1.4-syscall

Detaching a block device may need to flush pending I/O before the device is detached and can be re-attached (on the same node or a different node). We need to represent this state in the cluster database as well

Should there be some authentication mechanism in place to connect docker with docker volumes providers/Drivers?

• If not using storage which requires some kind of auth key etc (aws s3, ceph etc), volumes can be accessed by anyone, who knows the driver/storage providers available.

All PRs are shown as failing :)

When the openstorage process restarts, it should re-instate mounts in-use by running containers. A request to mount an already mounted path should not result in a failure.

• Fix 'go vet' reported problems
• Include 'go vet' to be run with builds

Fetching https://bazil.org/fuse?go-get=1
https fetch failed.
import "bazil.org/fuse": https fetch: Get https://bazil.org/fuse?go-get=1: dial tcp: lookup bazil.org on 127.0.1.1:53: no such host
package bazil.org/fuse: unrecognized import path "bazil.org/fuse"
Fetching https://bazil.org/fuse/fs?go-get=1
https fetch failed.
import "bazil.org/fuse/fs": https fetch: Get https://bazil.org/fuse/fs?go-get=1: dial tcp: lookup bazil.org on 127.0.1.1:53: no such host
package bazil.org/fuse/fs: unrecognized import path "bazil.org/fuse/fs"

The format option is implicit in create. The spec contains the required format.

Hi,I use osd on ubuntu14.04 64bit with go version go1.5.1 linux/amd64.
And I build nfs on a remote machine with " /home/pugna/share 10.6.0.0/16(rw,sync,no_subtree_check)" in /etc/exports, I run osd on a docker host machine.
osd daemon:
root@pugna-docker:/home/pugna/osd.conf.d# /home/pugna/go_work/bin/osd -d -f config.yaml
INFO[0000] Starting volume driver: nfs
INFO[0000] NFS driver initializing with 10.6.8.141:/home/pugna/share
INFO[0000] NFS initialized and driver mounted at: /var/lib/openstorage/nfs/
INFO[0000] Starting REST service on /var/lib/osd/driver/nfs.sock
INFO[0000] Starting REST service on /run/docker/plugins/nfs.sock
INFO[0118] mkdir /var/lib/openstorage/nfs/5d2a03cd-d2c0-4899-b00c-e3c53d939a9b: permission denied
INFO[0118] Driver=nfs ID= Request=create

osd nfs create:
root@pugna-docker:/home/pugna# ./go_work/bin/osd nfs create vol1
create: mkdir /var/lib/openstorage/nfs/5d2a03cd-d2c0-4899-b00c-e3c53d939a9b: permission denied

If I just mount nfs by using fstab , it works well(could write and read).
Could somebody help me solve this problem?

I might be missing something. That version (3.12 for Ubuntu 14.04) works just fine for docker's btrfs package (which this go code is using). It would save a lot of pain in the Dockerfile.

Hi Guys, this is cool. Can we get an NFS backend driver? We use Netapp and would like to use this interface for our Docker roll out.

Hi,I use osd on ubuntu14.04 64bit with go version go1.5.1 linux/amd64.
And I build nfs on a remote machine with " /home/pugna/share 10.6.0.0/16(rw,sync,no_subtree_check)" in /etc/exports, I run osd on a docker host machine.
osd daemon:
root@pugna-docker:/home/pugna/osd.conf.d# /home/pugna/go_work/bin/osd -d -f config.yaml
INFO[0000] Starting volume driver: nfs
INFO[0000] NFS driver initializing with 10.6.8.141:/home/pugna/share
INFO[0000] NFS initialized and driver mounted at: /var/lib/openstorage/nfs/
INFO[0000] Starting REST service on /var/lib/osd/driver/nfs.sock
INFO[0000] Starting REST service on /run/docker/plugins/nfs.sock
INFO[0118] mkdir /var/lib/openstorage/nfs/5d2a03cd-d2c0-4899-b00c-e3c53d939a9b: permission denied
INFO[0118] Driver=nfs ID= Request=create

osd nfs create:
root@pugna-docker:/home/pugna# ./go_work/bin/osd nfs create vol1
create: mkdir /var/lib/openstorage/nfs/5d2a03cd-d2c0-4899-b00c-e3c53d939a9b: permission denied

If I just mount nfs by using fstab , it works well(could write and read).
Could somebody help me solve this problem?

docker run
--privileged
-v /home/jupiturliu/work/src/github.com/libopenstorage/openstorage:/etc
-v /usr/share/docker/plugins:/usr/share/docker/plugins
-v /var/lib/osd/driver:/var/lib/osd/driver
-v /mnt:/mnt
openstorage/osd -d -f /etc/config.yaml
no such file or directory
Error response from daemon: Cannot start container 346e697283af3b4104e7cce0b1aea71c21798c64717167065c8fd81b8fe5f639: [8] System error: no such file or directory
make: *** [launch] Error 1

http://emma.openstorage.org/job/Openstorage/371/console

Error response from daemon: client is newer than server (client API version: 1.21, server API version: 1.20)
make: *** [docker-build-osd-internal] Error 1

I upgraded docker in the Dockerfile to 1.9.1 from 1.8.2.

Current code has compiling errors.

$ make build
.
.
.
.
# github.com/kunalkushwaha/openstorage/cluster
cluster/manager.go:115: cannot assign []docker.APIContainers to node.Containers (type []dockerclient.Container) in multiple assignment
# github.com/kunalkushwaha/openstorage/vendor/github.com/Sirupsen/logrus/examples/hook
vendor/github.com/Sirupsen/logrus/examples/hook/hook.go:12: cannot use airbrake.NewHook(123, "xyz", "development") (type *airbrake.airbrakeHook) as type "github.com/kunalkushwaha/openstorage/vendor/github.com/Sirupsen/logrus".Hook in argument to log.Hooks.Add:
        *airbrake.airbrakeHook does not implement "github.com/kunalkushwaha/openstorage/vendor/github.com/Sirupsen/logrus".Hook (wrong type for Fire method)
                have Fire(*"github.com/Sirupsen/logrus".Entry) error
                want Fire(*"github.com/kunalkushwaha/openstorage/vendor/github.com/Sirupsen/logrus".Entry) error
# github.com/kunalkushwaha/openstorage/cmd/osd
cmd/osd/main.go:71: too many arguments in call to cluster.New
cmd/osd/main.go:71: multiple-value cluster.New() in single-value context
# github.com/kunalkushwaha/openstorage/vendor/github.com/samalba/dockerclient/mockclient
vendor/github.com/samalba/dockerclient/mockclient/mock.go:114: undefined: AuthConfig
make: *** [build] Error 2

OSD aws driver in docker doesn't work:
ubuntu@ip-172-31-17-72:~/go/src/github.com/libopenstorage/openstorage$ docker run --privileged -v $PWD:/etc -v /usr/share/docker/plugins:/usr/share/docker/plugins -v /var/lib/osd/driver:/var/lib/osd/driver -v /mnt:/mnt osd /bin/osd -d -f /etc/config.yaml
Starting volume driver: aws
time="2015-10-08T14:16:58Z" level=info msg="AWS instance i-75ecbeb5 zone us-west-1b"
Unable to start volume driver: aws RequestError: send request failed
caused by: Post https://ec2.us-west-1.amazonaws.com/: x509: failed to load system roots and no roots provided

https://golang.org/s/go15vendor

The Dockerfile already uses golang 1.5, and vendoring will be this standard going forward, so since this is a newish project (and Godeps has a ton of recursive problems as it is, ie if I use openstorage as a library), maybe enforce this?

If GO15VENDOREXPERIMENT is set in golang 1.5, then godep respects it.

We do this in github.com/pachyderm/pachyderm.