Filesystem Sharing about lima HOT 56 OPEN

Docker Desktop Edge has a VirtioFS option that is looking very good.

Looks like that dropped to stable! https://docs.docker.com/desktop/mac/release-notes/#docker-desktop-460

Yes, it is available a an experimental feature and will be replace the gRPC FUSE implementation (hopefully). Instead of the hypervisor.framework it requires the new virtualization.framework which is available Big Sur. But VirtioFS in Docker Desktop for Mac is incredibly fast. With that feature the experience with developing inside containers is very comfortable. Would be very nice to have it in lima (perhaps optionally with additional libs / runtime options).

from lima.

The 9p patch is now merged in the master branch of QEMU qemu/qemu@f45cc81

from lima.

Just so you know: There is a patch for Darwin 9p support developed within the Nixpkgs project. I have adapted it to current QEMU from a patchset originally by Keno Fischer.

The people over at Nixpkgs would certainly be interested in any effort to accelerate file sharing using VirtioFS, which is why I’m now subscribed here.

from lima.

In parallel, we can consider supporting mutagen integration as well. IIUC it is used by Docker for Mac as well.

https://mutagen.io/

from lima.

I am preparing to propose the revised patch upstream. Have not had sufficient time to push this forward, but I hope to do this next week. It would certainly be good to have this upstream.

Regarding binaries: Nix has a binary cache, but the packages do not run standalone (they depend on other Nix packages). The patch however should apply to the vanilla QEMU sources, so you should be able to just recompile QEMU 6 with it.

I’m not aware of vsock developments. Unfortunately I’m not familiar with QEMU development internals at all. I just happened to have a need for 9p and worked on this patch.

from lima.

My two cents as the developer of Mutagen:

Mutagen works well if you have access to its full configuration and you can tweak it to your needs (e.g. via manual usage or Mutagen Compose) or if you're using it as part of some higher-level tooling that configures it in a way where it works for 99% of users automatically (e.g. the way ddev does).

It doesn't work well as a general purpose caching mechanism, and that was what really hampered the Docker for Mac integration (i.e. the simple on/off switch design and inability to see what's going on internally).

The inotify issue is something I'm working on addressing via use of fanotify and its enhancements in Linux 5.1+, but it's probably going to be restricted to very controlled environments (e.g. the sidecar container used by Mutagen Compose) because it requires CAP_SYS_ADMIN and CAP_DAC_READ_SEARCH and has somewhat odd path handling behavior with bind mounts.

Conflicts and handling of problematic content have been significantly improved in Mutagen v0.12, so hopefully that's less of an issue.

Anyway, I won't interrupt the discussion further, but feel free to reach out via email ([email protected]) or Slack if you have any questions.

from lima.

Samba is now available on homebrew (https://github.com/Homebrew/homebrew-core/blob/master/Formula/samba.rb), so I'm planning to replace sshfs with samba soon.

Samba will be executed with slirp_add_exec: https://github.com/qemu/qemu/blob/4cc10cae64c51e17844dc4358481c393d7bf1ed4/net/slirp.c#L964

So Samba will not listen on any actual TCP port.

cc @jandubois

from lima.

I added a "mountType" selector. Theoretically it could also include something like NFS.

• #726

mountType: "reverse-sshfs" (sshocker)

mountType: "9p" (virtfs)

from lima.

The 9p patch is now also merged in brew (qemu 6.2.0_1), so hopefully that will be enough for most Mac users too...

from lima.

Docker Desktop Edge has a VirtioFS option that is looking very good.

VirtioFS is still unavailable in the macOS version of QEMU.

Apple's Virtualization.framework supports VirtioFS, but it is not the best fit for Lima, as it lacks support for firmware, qcow2, slirp, and a lot of features of QEMU. And yet it is proprietary.

from lima.

Simple enough, though.

1. Set up a ssh "host"
   limactl show-ssh --format config default >>~/.ssh/config

$ ssh lima-default uname
Linux
$ ssh lima-default grep PRETTY /etc/os-release
PRETTY_NAME="Ubuntu 21.10"

2. Set up a ssh "sync"
   mutagen sync create /tmp/test lima-default:/tmp/test

$ mutagen sync list
--------------------------------------------------------------------------------
Identifier: sync_WxCidKXaUaHGgkXXkRWZp5UBANGKpZlJ8JmJ0VgKepQ
Labels: None
Alpha:
	URL: /tmp/test
	Connection state: Connected
Beta:
	URL: lima-default:/tmp/test
	Connection state: Connected
Status: Watching for changes
--------------------------------------------------------------------------------

Works as advertised.

So if it was included, the YAML file would look something like:

mounts:
  - location: "~"
    writable: false
  - location: "/tmp/lima"
    writable: true
mutagen:
  - location: "/tmp/test"
    mode: "two-way-safe"

anders@lima-default:/tmp$ findmnt -T lima
TARGET    SOURCE     FSTYPE     OPTIONS
/tmp/lima :/tmp/lima fuse.sshfs rw,nosuid,nodev,relatime,user_id=1000,group_id=1000,allow_other
anders@lima-default:/tmp$ findmnt -T test
TARGET SOURCE    FSTYPE OPTIONS
/      /dev/vda1 ext4   rw,relatime,discard,errors=remount-ro

With some arbitrary mapping of the mutagen YAML configuration:

https://mutagen.io/documentation/introduction/configuration#configuration-files

$ mutagen sync list --long
--------------------------------------------------------------------------------
Identifier: sync_WxCidKXaUaHGgkXXkRWZp5UBANGKpZlJ8JmJ0VgKepQ
Labels: None
Configuration:
	Synchronization mode: Default (Two Way Safe)
	Maximum allowed entry count: Default (2⁶⁴−1)
	Maximum staging file size: Default (18 EB)
	Symbolic link mode: Default (Portable)
	Ignore VCS mode: Default (Propagate)
	Ignores: None
Alpha configuration:
	URL: /tmp/test
	Watch mode: Default (Portable)
	Watch polling interval: Default (10 seconds)
	Probe mode: Default (Probe)
	Scan mode: Default (Accelerated)
	Stage mode: Default (Mutagen Data Directory)
	File mode: Default (0600)
	Directory mode: Default (0700)
	Default file/directory owner: Default
	Default file/directory group: Default
Beta configuration:
	URL: lima-default:/tmp/test
	Watch mode: Default (Portable)
	Watch polling interval: Default (10 seconds)
	Probe mode: Default (Probe)
	Scan mode: Default (Accelerated)
	Stage mode: Default (Mutagen Data Directory)
	File mode: Default (0600)
	Directory mode: Default (0700)
	Default file/directory owner: Default
	Default file/directory group: Default
Alpha:
	Connection state: Connected
Beta:
	Connection state: Connected
Status: Watching for changes
--------------------------------------------------------------------------------

from lima.

Note that it is the 9p for Darwin that is "new", the 9p for Linux has been there for ages

Yeah, I guess most of us are interested in the Darwin version as we're trying to mount @ MacOS for replacing Docker Desktop for good.

from lima.

It is still enroute to be merged for 7.0, and still available as a patch for 6.x - so no real news from last year, except that more people are actually trying it now

from lima.

After the 9p patch (https://lists.gnu.org/archive/html/qemu-devel/2022-03/msg02090.html) gets merged we can switch the default from reverse sshfs to 9p, however, probably we will have to continue maintain reverse sshfs for EL8 family 😞

[root@lima-rocky ~]# grep -i 9p /boot/config-4.18.0-348.el8.0.2.x86_64 
# CONFIG_NET_9P is not set

from lima.

That's right -- the 9p patch has been backported to 6.2 in both brew and nixpkgs at this point and there's no reason why both will have any other issues making it to 7.0 regardless, so I'd consider it widely available.

from lima.

Thanks!

from lima.

@AkihiroSuda That is awesome to hear. From the benchmarks i have seen so far, it is supposedly 2x faster with most workloads.

What would it take for us to support inotify events?

from lima.

What would be needed for integrating mutagen ? It seems like it should work out-of-the-box, over the ssh connection ?

For docker-machine, the support was more "hands-on". The user had to run the scp and rsync and sshfs themselves...

from lima.

This could be a separate "mount" type, perhaps ? Similar to the current shorthand for setting up sshocker (reverse sshfs)

Didn't understand the part of the paths needing to the same on both sides, but will have to try it myself. (EDIT: it doesn't)

from lima.

My idea was to leave it as opt-in, just wanted it easier to do...
I don't think it should be bundled, more treated as a tool (like rsync)

rsync -av /tmp/test/ lima-default:/tmp/test/
rsync -av lima-default:/tmp/test/ /tmp/test/

scp /tmp/test/foo lima-default:/tmp/test/foo
scp lima-default:/tmp/test/bar /tmp/test/bar

3. Just sync filesystem state between VM and host.

The two biggest lima improvements are probably 1) acknowledging that the VM even exists 2) adding it as a ssh "host"

That opens up for a lot of different use cases, where the "built-in" mounts fall short. As all of them will do, eventually.

from lima.

This seems like good news NixOS/nixpkgs#122420 (comment)

from lima.

Looks like there has been some progress/success with 9p and Qemu containers/podman#8016 (comment).

from lima.

My current plan:

• Step 1: Use 9p over virtio serial: #1

• Step 2: Implement vsock support in QEMU for Darwin, and switch away from virtio serial to vsock

• Step 3: Implement virtiofs support in QEMU for Darwin

from lima.

Step 3: Implement virtiofs support in QEMU for Darwin

Do you think this is possible? It requires to be able to share mmap'ped memory pages between host and guest. It this is possible, it would be cool.

from lima.

Thanks @mroi !

• The patch seems proposed to the upstream in 2018, but what's the current status of upstreaming?
• Is there / will there be a binary with the patch, that can be installed without nix? For both Intel and ARM. (Would it be possible to crosscompile ARM target on an Intel host?)

from lima.

@mroi Aside from 9p, do you know whether somebody is working on supporting vsock?
For reimplementing the even notification system that is currently implemented by running ssh -L /run/user/<UID>/lima-guestagent.sock:ga.sock.

from lima.

The patch seems to have issues 😞 NixOS/nixpkgs#122420 (comment)

from lima.

Until we can get virtio-9p-pci for macOS hosts in the QEMU upstream, I was planning to use 9P over virtserial, but it doesn't seem easy as expected 😞

diff --git a/pkg/qemu/qemu.go b/pkg/qemu/qemu.go
index d9b0778..19f0f75 100644
--- a/pkg/qemu/qemu.go
+++ b/pkg/qemu/qemu.go
@@ -154,7 +154,7 @@ func Cmdline(cfg Config) (string, []string, error) {
        // Parallel
        args = append(args, "-parallel", "none")
 
-       // Serial
+       // Legacy Serial
        serialSock := filepath.Join(cfg.InstanceDir, "serial.sock")
        if err := os.RemoveAll(serialSock); err != nil {
                return "", nil, err
@@ -167,7 +167,17 @@ func Cmdline(cfg Config) (string, []string, error) {
        args = append(args, "-chardev", fmt.Sprintf("socket,id=%s,path=%s,server,nowait,logfile=%s", serialChardev, serialSock, serialLog))
        args = append(args, "-serial", "chardev:"+serialChardev)
 
-       // We also want to enable vsock and virtfs here, but QEMU does not support vsock and virtfs for macOS hosts
+       // vport for 9p
+       vportSock := filepath.Join(cfg.InstanceDir, "vport.sock")
+       if err := os.RemoveAll(vportSock); err != nil {
+               return "", nil, err
+       }
+       const vportChardev = "char-vport"
+       args = append(args, "-device", "virtio-serial")
+       args = append(args, "-chardev", fmt.Sprintf("socket,id=%s,path=%s,server,nowait", vportChardev, vportSock))
+       args = append(args, "-device", fmt.Sprintf("virtserialport,chardev=%s,name=lima", vportChardev))
+
+       // TODO: use virtio-9p-pci when QEMU supports it for macOS hosts
 
        // QEMU process
        args = append(args, "-name", "lima-"+cfg.Name)

package main

import (
        "fmt"
        "os"
        "syscall"
)

func main() {
        if err := xmain(); err != nil {
                fmt.Fprintln(os.Stderr, err)
                os.Exit(1)
        }
}

func xmain() error {
        devPath := "/dev/vport5p1"
        mntPath := "/mnt/foo"

        devFd, err := syscall.Open(devPath, syscall.O_RDWR|syscall.O_NONBLOCK, 0600)
        if err != nil {
                return err
        }
        return syscall.Mount(
                "",
                mntPath,
                "9p",
                0,
                fmt.Sprintf("trans=fd,rfdno=%d,wfdno=%d", devFd, devFd),
        )
}

# go run main.go
invalid argument
exit status 1

# dmesg
...
[27353.823239] kernel write not supported for file /vport5p1 (pid: 2243 comm: kworker/3:1)

The error is happening because the vport device somehow wires up both write_iter and write: https://github.com/torvalds/linux/blob/v5.11/fs/read_write.c#L540-L545

I guess I have to inject some pipe as a "shim" fd, or use ssh instead of virtserial.

from lima.

virtserial seems slower than ssh :(

virtserial

qemu flags: -device virtio-serial -chardev socket,id=foo,path=foo.sock,server,nowait -device virtserialport,chardev=foo.sock,name=foo

[host]$ socat file:urandom-1G unix-connect:foo.sock 

[guest]$ time sha256sum /dev/virtio-ports/foo
real    0m10.022s

ssh

[host] $ cat urandom-1G | lima time sha256sum -
real    0m7.673s

from lima.

virtserial seems slower than ssh

Given that we seem to be stuck with ssh, at least for now, did you check if selecting a different cipher makes any difference for throughput? It seems like we are using the default:

debug1: kex: server->client cipher: [email protected] MAC: <implicit> compression: none
debug1: kex: client->server cipher: [email protected] MAC: <implicit> compression: none

I remember reading that aes128-ctr was twice as fast as [email protected] when running on a CPU with AES instructions, but not sure how much that affects throughput.

I'm somewhat confused though, as even when I add -c aes128-ctr to the ssh command, it still ends up using [email protected], so I don't know how to configure it.

from lima.

Given that we seem to be stuck with ssh, at least for now, did you check if selecting a different cipher makes any difference for throughput?

Haven't looked into. Help wanted 🙏

from lima.

Btw QEMU Samba seems roughly two times faster than sshfs

[root@lima-archlinux ~]# time sha256sum /mnt/smb/ubuntu-21.04-desktop-amd64.iso 
fa95fb748b34d470a7cfa5e3c1c8fa1163e2dc340cd5a60f7ece9dc963ecdf88  /mnt/smb/ubuntu-21.04-desktop-amd64.iso

real    0m15.578s
user    0m7.054s
sys     0m6.259s

[root@lima-archlinux ~]# time sha256sum /tmp/lima/ubuntu-21.04-desktop-amd64.iso 
fa95fb748b34d470a7cfa5e3c1c8fa1163e2dc340cd5a60f7ece9dc963ecdf88  /tmp/lima/ubuntu-21.04-desktop-amd64.iso

real    0m29.707s
user    0m9.244s
sys     0m6.370s

diff --git a/pkg/qemu/qemu.go b/pkg/qemu/qemu.go
index 796c6ee..1524a24 100644
--- a/pkg/qemu/qemu.go
+++ b/pkg/qemu/qemu.go
@@ -193,7 +193,7 @@ func Cmdline(cfg Config) (string, []string, error) {
        // CIDR is intentionally hardcoded to 192.168.5.0/24, as each of QEMU has its own independent slirp network.
        // TODO: enable bridge (with sudo?)
        args = append(args, "-net", "nic,model=virtio")
-       args = append(args, "-net", fmt.Sprintf("user,net=192.168.5.0/24,hostfwd=tcp:127.0.0.1:%d-:22", y.SSH.LocalPort))
+       args = append(args, "-net", fmt.Sprintf("user,net=192.168.5.0/24,hostfwd=tcp:127.0.0.1:%d-:22,smb=/tmp/lima", y.SSH.LocalPort))
 
        // virtio-rng-pci acceralates starting up the OS, according to https://wiki.gentoo.org/wiki/QEMU/Options
        args = append(args, "-device", "virtio-rng-pci")

Needs Homebrew/homebrew-core#80171 to be merged (which may take time)

from lima.

It would be interesting to see some benchmarks between sshfs and virtio-9p-pci and nfs and cifs/smb...

The virtfs authors seemed convinced: https://www.kernel.org/doc/ols/2010/ols2010-pages-109-120.pdf

from lima.

On Linux QEMU using virtfs is a lot faster than sshfs and even 9p. If anything is slowed down on macOS due to kernel lockdown/security don't take it as given for Linux users!

-–› Please always consider Linux users in mind since lima itself becomes more and more a tool for cross-platform development (macOS, Linux, WSL/Windows). Especially k8s/k3s is really nyce scenerio! Don't make your decisions based on macOS only! Thank you!

Docker for Desktop on Windows uses 9p (they switched away from SMB which they used before).

Docs for linux: https://www.linux-kvm.org/page/9p_virtio

Aslo check out how wimpy uses hardware-acceleration/virt*-kernel modules in Linux w/ qemu v6.x (yup 6.0+!):
https://github.com/wimpysworld/quickemu/blob/master/quickemu

from lima.

It's not Docker per se, but WSL2 that uses 9P for any WSL VM: https://devblogs.microsoft.com/commandline/a-deep-dive-into-how-wsl-allows-windows-to-access-linux-files/

I've been testing out WSL2 integration for other team members, and the filesystem performance is still not to a point where it's usable for large projects. For example, PHPStorm hangs and crashes trying to index a basic Drupal 9 codebase over the \\wsl$ share: ddev/ddev#3366

To date, mutagen (which is built into ddev) is giving our team the best experience. That work came about because Docker Desktop dropped their alpha mutagen support, and I think it would be really nice if support for it was at the lima (or colima) layer instead of at the next layer up.

I'm not sure it's worth the development effort to only get a relatively small performance improvement. If performance is any more than 20-50% slower than native IO, it's still too slow for most of our uses.

from lima.

Mutagen works, if you know your ssh and mutagen stuff.

I added an entry to ~/.ssh/config to disable host key check and specify the port, to make it easier to interact with rsync/mutagen:

HOST lima
Hostname localhost
Port 60022
StrictHostKeyChecking no
UserKnownHostsFile /dev/null

Then you can use a command like this to create a sync of the current working directory:

lima mkdir -p $PWD # create the needed folders
lima chmod 777 $PWD $(dirname $PWD) # mutagen will write a folder in the parent directory, so you need write access there too.
mutagen create \
  --sync-mode two-way-resolved \
  --stage-mode=neighboring \
  --ignore-vcs \
  --symlink-mode posix-raw \
  --default-file-mode 0666 \
  --default-directory-mode 0777 \
  --name "$(basename $PWD)" \
  $PWD lima:$PWD
mutagen monitor "$(basename $PWD)" # to watch your sync progress

Then stuff like lima nerdctl run -v $PWD:$PWD -w $PWD ubuntu ls will just run as expected. Just make sure your paths in the vm and on your machine are identical.

from lima.

It could but... I feel like it won't be intuitive or stable. There are issues with very large amounts of files (multiple projects) write privileges, the amount of inotify watches on linux and there is conflict resolution etc.
I can manually enable sync's as needed. But i'm not too sure it will be a great fit as a solution in the config file.

But maybe i'm also not the average user ;)

Docker for mac had mutagen once as well docker/for-mac#1592 (comment)
There implementation was pretty good, it was integrated into the ui and showed sync progress etc. But you were on your own if it failed and it used a lot of resources when you synced larger directories.

from lima.

At least you can use limactl show-ssh --format config default to set up the config, for host "lima-default" (etc).

To me it actually makes more sense to set up such syncs as needed, than to export your entire home directory...

Unfortunately it seems like mutagen doesn't have a ssh config file (yet?), so it needs to be in the user ssh_config.

• Issue mutagen-io/mutagen#301

from lima.

• Is there / will there be a binary with the patch, that can be installed without nix? For both Intel and ARM. (Would it be possible to crosscompile ARM target on an Intel host?)

@AkihiroSuda should we consider using go:embed to distribute static qemu-system-ARCH binaries?
Of course this would require a Workflow to maintain builds, but it would allow us to

• Remove the dependency on having qemu installed
• Apply the 9p support patch to Darwin builds

from lima.

I don't think we should use go:embed, but we can consider putting patched QEMU binaries under /usr/local/share/lima, if it is acceptable for Homebrew/MacPorts/nix.

from lima.

What is the difference between virtio-fs, virtio-9p, and virtio-blk? And which is simply called virtio?

from lima.

"virtio-fs" is a whole new project (using vsock), while virtio-9p is the standard one used for "virtfs"

• https://virtio-fs.gitlab.io/howto-qemu.html (mount -t virtiofs)

• https://wiki.qemu.org/Documentation/9psetup (mount -t 9p -o trans=virtio)

from lima.

• virtio-fs borrows the FUSE request model and uses a daemon running on the host to service file access requests. There is also a direct memory extension that requires host kernel support called dax (Linux only). Using virtio-fs with dax allows the host and the guest to share cache and offers a big performance boost.

• virtio-9p is just using 9p_net over virtio transport. It is a very old filesystem that has been stable in the kernel for a long time.

• virtio-blk is not a file access layer but rather exposes a block device.

EDIT: I realized the original post lays this all out already in a better fashion.

from lima.

They are waiting for virtiofs support (or nfs over vsock). Waiting, waiting, waiting. It's been years already.

https://access.redhat.com/discussions/1119043

We discussed it as a replacement for VirtualBox Shared Folders (vboxsf), but the outcome was: sshfs

• dhiltgen/docker-machine-kvm#2

from lima.

Note that it is the 9p for Darwin that is "new", the 9p for Linux has been there for ages

from lima.

The brew package is insanely large, since it includes some 30+ architectures. But that's another story... (One can compile and install just amd64/arm64 and their firmware, if needed)

from lima.

Docker Desktop Edge has a VirtioFS option that is looking very good.

from lima.

Docker Desktop Edge has a VirtioFS option that is looking very good.

Looks like that dropped to stable! https://docs.docker.com/desktop/mac/release-notes/#docker-desktop-460

from lima.

There is a new project which provides fuse via nfs, please follow these links for more info:

https://news.ycombinator.com/item?id=32726166
https://www.fuse-t.org/

IMHO, looks very promising. May be, it's would be very nice for lima ?

from lima.

It has some similar licensing issues as macfuse, it seems ?

from lima.

IMHO, looks very promising. May be, it's would be very nice for lima ?

It isn't open-source, and it isn't even free to use for commercial uses, so FUSE-T is not an option for Lima.

Also nfs is not really that simple to use on macOS: you need to be root to configure /etc/exports to secure the exported directories.

from lima.

Hello, some interesting conversation in this thread. I was wondering, looking at this from a pragmatic lens, if there was an immediate solution (maybe the 9p patch getting folded in) while the longer term solution is in the works (which sounds like VirtioFS)? Myself, and others, would be thrilled to use Rancher Desktop on a regular basis, which is a downstream package that uses lima, but there's one thing stopping us: rancher-sandbox/rancher-desktop#1209 (mounting volumes). Appreciate any thoughts or feedback.

from lima.

Looks like 6 days ago Lima documented using virtiofs c18ae23

from lima.

c18ae23

Yes, this is now available in v0.14.0-beta.0.

from lima.

Any updates to this? Looking for a way to solve the missing inotify events in docker-compose volume mounts in Colima and Lima on macOS.

from lima.

Any updates to this? Looking for a way to solve the missing inotify events in docker-compose volume mounts in Colima and Lima.

inotify is discussed here:

• #615

from lima.