NFS mounts fail with 1.7.0 and 1.7.1 on Mavericks about boot2docker-vagrant-box HOT 15 CLOSED

FYI, I'm noticing the same things with dduportal/boot2docker-vagrant-box running on virtualbox , meaning NFS works in 1.6.0 and not in 1.7.1 (very similar symptoms as described above).

UPDATE: I'm experience similar failures with dduportal/boot2docker-vagrant-box 1.6.0 as well.

from boot2docker-vagrant-box.

@posita I've just found that it could be worked around with specifying "nolock" mount option:

config.vm.synced_folder(mnt_path, mnt_path, nfs: true, mount_options: ["nolock"])

It was discussed in dduportal/boot2docker-vagrant-box#10 (comment)

@dduportal Have you figured out why it doesn't work without "nolock" option?

from boot2docker-vagrant-box.

Hi !

Nope, i have to admit that I completely trusted contributors in that.
I started to read a bit of doc on the implication to the nolock option, but it appears to be used in 2 cases :

• Production performances issues,
• Older NFS server compatibility.

Considering that :

• We are in development use case (or you just do not use vagrant at all if production is targeted)
• Files in the shared folder should not been written with concurrency considering Docker paradigms (use data volumes, with single shot copying, or build you images by sending data thru the Docker daemon API inside images)

So i figured out to accept an Oob working solution, and if problems would have benne appeared with this option, i had to

• Implement a test around that problem
• Understand the implications helped by community
• Correct that, even if choice has to be made.

Does the use of this option causes problem for you for now ? In the future (i don't have your insights in mind, maybe i miss use cases :) ) ?

from boot2docker-vagrant-box.

TL;DR: ideally nolock should be set by the administrator on a case-by-case basis, it shouldn't be a blanket requirement if at all possible.

@legal90, thanks for the work-around! 😀 Unfortunately, as @dduportal points out, nolock should likely be an option, not a requirement (I don't think supporting ancient NFS implementations applies here). Even where you need more performance, you still have to make sure you won't need FS-level locking, which I'm not sure you can guarantee with all host mounts.

That being said, many applications seem to have their own locking mechanisms that accommodate NFS's nolock. MySQL, for example, has the --external-locking and --skip-external-locking options (external locking has apparently been turned off by default since 4.0). Running two bitcoind instances syncing blockchain data on a host-mounted file system seems to do the right thing (the later one refuses to start since it can't get a lock on the data directory, although I didn't start them anywhere near the same time, so I'm not confident everything works in all cases).

from boot2docker-vagrant-box.

@posita that's right in a platform mind, we are here in a development environment, given that you should not run applications based on the data which is on this filesystem.
If you want to simulate a production ready environment, use a container for that : you'll be full controlling the NFS server configuration of this container .

I really think that shared folder should be used in one shot mode, so the nolock should not be a problem.

I'm a really big fan of using rsync for shared folder : it will make your workflow a "push" one and will avoid the sharing problems (exemple symlink with a Windows host, user rights with Mac OS which have strange UIDs, etc.).

from boot2docker-vagrant-box.

Looks like I spoke too soon regarding a work-around. 😞 When using a 1.7.1 release, despite using NFS with nolock on all mounts, after some number of repetitions of vagrant halt ; vagrant up, I am running into Level/levelup#222. 😕

So there's something wonky about mounts after 1.6.0 that I don't think is addressed via nolock (in other words, nolock may be a red herring). I'm trying to create a small testcase to reproduce, but it's difficult, since the symptoms don't appear right away.

@dduportal, with respect to your comments regarding dev environments, best practices, etc. I'm less confident that your proposed limitations accommodate legitimate use cases. Here's one example:

1. Developer runs OS X on a laptop.
2. Production runs bitcoind in Docker containers.
3. As of this writing, bitcoind requires downloading 50+ GB of blockchain data (which can take days) before being usable.
4. Developer uses Vagrant and boot2docker for development and testing bitcoind container.
5. Developer wants blockchain data to persist beyond Vagrant life cycle (i.e., vagrant destroy is a necessary part of the development process).
6. bitcoind must have write access to blockchain data (i.e., it is incapable of operating in read-only mode).

Constraints:

1. No hardware (either new machines or accessories) may be added to the Developer's setup.

The (perhaps naive) solution that comes to mind is store the blockchain data on the native file system, mount it via Vagrant in boot2docker and then use Docker data containers to mount the boot2docker mount points inside the appropriate container. (By the way, I'm not the only person to think of this. This seems to be the recommended method when using boot2docker on a Mac, whether or not Vagrant is involved. That being said, Vagrant affords some administrative conveniences that I'm personally not yet ready to abandon.)

In the above example, my naive solution works without issue in 1.6.0. It fails after a number of Vagrant cycles starting in 1.7.0 (again, after some playing, I think the nolock thing probably has nothing to do with the failure).

from boot2docker-vagrant-box.

Okay, setting aside the nolock debate for a moment, I created a convoluted test-ish thingy to explore various combinations of versions and settings. What I found was that Parallels/boot2docker-vagrant-box differs substantially from dduportal/boot2docker-vagrant-box in what it can accommodate.

Three main observations:

1. Parallels/boot2docker-vagrant-box 1.7.1 with nolock has permission problems when writing to NFS mounts (see this). UPDATE: this observation was due to Parallel's default mounting of /Users via prl_fs which hid my NFS mount. NFS in Vagrant uses mapall by default, but prl_fs preserves permissions. See my comment below.
2. Parallels/boot2docker-vagrant-box 1.7.1 is installed alongside 1.6.0, and you use 1.6.0 explicitly (e.g., via config.vm.box_version), mounts break when restarting the VM until you uninstall 1.7.1 (see this). UPDATE: this is not accurate. The observed behavior comes from user error. See my comment below.
3. dduportal/boot2docker-vagrant-box with nolock pretty much just works.

To be clear, I was mistaken about any intermittent reproducibility of the symptoms I observed. I was confused by a combination of the problems I mentioned in my previous comment.

from boot2docker-vagrant-box.

1. Parallels/boot2docker-vagrant-box 1.7.1 is installed alongside 1.6.0, and you use 1.6.0 explicitly (e.g., via config.vm.box_version), mounts break when restarting the VM until you uninstall 1.7.1 (see this).

TL;DR: After switching the box version you should recreate a VM from the scratch (vagrant destroy && vagrant up)

On the second step of your test you have switched the box version from "1.6.0" to "1.7.1". Each box version has its own Vagrantfile, bundled with the box. Vagrant always parsed it according to the load order(https://docs.vagrantup.com/v2/vagrantfile/index.html).

So, Vagrantfiles in parallels/boot2docker v1.6.0 and v1.7.1 are different:
v1.6.0: https://github.com/Parallels/boot2docker-vagrant-box/blob/parallels/v1.6.0/vagrantfile.tpl
v1.7.1: https://github.com/Parallels/boot2docker-vagrant-box/blob/parallels/v1.7.1/vagrantfile.tpl

In the v1.7.1 we are mounting /Users via Parallels Shared Folders (prl_fs), not via NFS. In v1.6.0 prl_fs is not supported.
Yet another important thing: starting since v1.7.0 the boot2docker.iso is dumped to the virtual machine HDD. It simplifies the box (we don't have to lookup and attach an iso), but it means that if we switch the box version for the existing VM, then we have to destroy&up this VM to take an effect (like any other guests for Vagrant).

from boot2docker-vagrant-box.

TL;DR: After switching the box version you should recreate a VM from the scratch (vagrant destroy && vagrant up)

This was my mistake. I was operating under the misunderstanding that a Vagrant instance was pegged to the box+version, not just the box. I have fixed this in posita/vagrant-b2d-nfs-test@fd94077, and so long as I explicitly set config.vm.box_version in the Vagrantfile, it doesn't matter which version(s) are installed.

from boot2docker-vagrant-box.

Parallels/boot2docker-vagrant-box 1.7.1 with nolock has permission problems when writing to NFS mounts (see this).

In the v1.7.1 we are mounting /Users via Parallels Shared Folders (prl_fs), not via NFS. In v1.6.0 prl_fs is not supported.

If I explicitly disable the prl_fs /Users mount, my test works. It looks like this is because shared folders are mounted after NFS shared folders, so even if ${HOME} is mounted via NFS, it will be hidden when /Users is later mounted over it via prl_fs.

While this is different from @dduportal's VirtualBox behavior, I'm no longer confident this should be classified as a bug. The default behaviors in Parallels/boot2docker-vagrant-box caused problems (or rather, behavioral discrepancies) for my particular configuration, but those can be addressed by additional settings in one's Vagrantfile.

Apologies for any inconvenience.

from boot2docker-vagrant-box.

Hello !

Thanks for sharing your work and feedback, I understand better use cases and the problem you ran trough. It was tricky !

To let you know, i am thinking on moving to docker-machine since the 0.4.0(.*) seems to bring a lot of new useful features (daemon config, KVP for env like http_proxy), Windows support. The only "problem" left is compose on Windows.

In my company, we have forked the dduportal/boot2docker to address hundred of developers, and we are seriously thinking doing the move on docker-machine to leverage the maintaining effort. I think that it would be the same for the image.

What is your thought about that ?

I would be interested to draw some kind of matrix to help people choose between vagrant stack / docker machine stack to see pros ans cons, would you be interested on that ?

from boot2docker-vagrant-box.

@dduportal Docker Machine is really cute and it is a much more convenient way to manage Docker hosts rather than Vagrant. But it will not have a Parallels support in v0.4.0 due to our PR has been declined (docker/machine#939 (comment)).

Seems like we have to support this vagrant box until Docker Machine v0.5.0 with the new plugin model is released. We will also publish the custom docker-machine_darwin-amd64 binary with Parallels driver support, but not all customers will be satisfied by this solution (docker/machine#939 (comment)).

from boot2docker-vagrant-box.

Just read that, yeah it's not easy food, but i think you're in the right direction !

Thank for feedback !

from boot2docker-vagrant-box.

I'd love to be able to move to machine, but likely won't until Docker provides some avenue to use Parallels. I like the flexibility of Vagrant from a dev/admin perspective (complexities can be encapsulated in a Vagrantfile and devs can run vagrant up to get a working environment). There will probably be ways to do the same thing with Docker Machine (wrapper scripts?), but I won't likely look into them until that team gets its act together. (At least) twice in past few weeks, different teams within Docker have ripped the carpet out from under their users without any warning. I hope those are just growing pains, and not indicative of a more engrained culture.... 😞

from boot2docker-vagrant-box.

Without jumping in the debates which "local optimum or global optimum", I think your point is right given that vagrant as 5 years of life at least, since docker-machine is in 0.4.0-release candidate :)

The target is 2 months at least, given they make a monthly version update, and you need 0.5.0 with the new plugin architecure that will allow your code to be merged.

Keep in mind that Vagrant is dev. oriented, so you create a dev/ops wall (but maybe it will fit to your organisation so it would not be a problem !), but Docker try to address a "no-wall" world : provisionning your infra is basically docker run or docker-compose up in their ideas : high level reproductiblity so on.

So you get it right : the workflow may be slightly different. If it does not feet, do not hesitate to replace Docker (I used packer + vagrant since a long tiime and it was working), given Docker may add much more complexity :)

Whatever, as @legal90 stated, for the next months, vagrant box should be maintained while docker-machien reach its goals.

Thank you for the feedbacks !

from boot2docker-vagrant-box.