winfsp / cgofuse Goto Github PK
View Code? Open in Web Editor NEWCross-platform FUSE library for Go - Works on Windows, macOS, Linux, FreeBSD, NetBSD, OpenBSD
Home Page: https://winfsp.dev
License: MIT License
Cross-platform FUSE library for Go - Works on Windows, macOS, Linux, FreeBSD, NetBSD, OpenBSD
Home Page: https://winfsp.dev
License: MIT License
I am testing hellofs example in Linux (CentOS 6). For very first time, I mounted successfully under "/root/adrive". When I tried next time to mount under same path, the mount failed without any error. However, it can succesfully mount again with new path E.g. "/root/adrive1".
Can you please help me on this?
Hi,
I would like to how to specify user defined value for Type field shown in Windows Explorer for Files/Folders instead of just showing "File folder" for Directory & "Text Document" for Text File.
My requirement is to set a user defined value for a Directory let's say to "My Folder Type" and for a Text File to "My Text File" in Type fields.
Please see the attachment for reference.
On Windows WinFsp and its API's are acquired dynamically using LoadLibrary
and GetProcAddress
. How should we handle the situation when it is not installed?
The easiest thing to do from C would be to print a message on stderr
and exit
. @ncw would this be acceptable for rclone?
Another approach is to somehow panic
from within C to give a chance to Go code to recover gracefully. It is not clear at this time how this can be accomplished.
I have a project with CGO as one of its dependencies. My Dockerfile looks like this:
COPY . /go/src/app
WORKDIR /go/src/app
RUN go get -v -d && \
go build .
WORKDIR /
COPY --from=0 /go/src/app/build .
EXPOSE 18088
CMD ["/app"]
I get this error message after trying to build the image:
...
github.com/billziss-gh/cgofuse/fuse
../github.com/billziss-gh/cgofuse/fuse/host.go:30:8: undefined: c_struct_fuse
The Dockerfile is incomplete and runs a single-stage build and is sub-optimal. However, optimizations is due for later. How do you properly build an image with CGO as a part of the build process?
I'm trying to track down a problem with my new mount code and rclone. I've changed the way the open flags are handled to (optionally) enable on disk caching which will make the file system a whole lot more compatible.
What seems to be happening is that when cgofuse calls the Create method, it calls it with some flags that rclone doesn't understand - in particular the 0x100 flag doesn't seem to match anything.
The flags which were supplied by the go program writing to the mount were from ioutil.WriteFile so should have been os.O_WRONLY|os.O_CREATE|os.O_TRUNC
. So it is as if O_TRUNC is being delivered as 0x100.
Can you confirm? Am I making a mistake assuming that the open flags are the same as the go flags?
I haven't managed to find definitions for these flags anywhere in your code so I'm not sure where the constants come from!
If you want to play with the code it is on the vfs-rw branch and I've been running in cmd/cmount
the command go test -v -tags cmount
.
Any help much appreciated - thanks!
2017/11/15 01:10:51 DEBUG : /a file: Create: flags=0x102, mode=0700
2017/11/15 01:10:51 DEBUG : a file: Open: flags=O_RDWR|0x100
2017/11/15 01:10:51 ERROR : a file: Can't open for read and write without cache
2017/11/15 01:10:51 DEBUG : a file: >Open: fd=<nil>, err=permission denied
2017/11/15 01:10:51 DEBUG : /a file: >Create: errc=-1, fh=0xFFFFFFFFFFFFFFFF
I'm using Windows 10 Pro (1903). If I take the memfs example, it first seems that I need to add a Statfs() method to make the filesystem writable. After that, however, I have an unknown problem with application compatibility. An MKV will open in "Windows Media Player" but not "Movies & TV". An MP3 will open with "iTunes" but not "Groove Music".
Unfortunately, I haven't been able to figure out anything more than this. Am I missing something?
Hi!
I'm trying to track down a performance issue with cgofuse vs bazil.org/fuse in rclone.
Here is cgofuse mounting a local disk on Linux.
$ dd if=/tmp/1G of=/mnt/tmp/1G bs=128k
8192+0 records in
8192+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 18.2636 s, 58.8 MB/s
And here is the mount command. Note the 4k writes
rclone cmount -vv /tmp/data /mnt/tmp/ 2>&1 | grep "Write: "
...
2021/03/24 16:13:04 DEBUG : /1G: >Write: n=4096
And here is bazil.org/fuse
$ time dd if=/tmp/1G of=/mnt/tmp/1G bs=128k
8192+0 records in
8192+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 4.67968 s, 229 MB/s
And its mount command - note the 128k blocks
rclone mount -vv /tmp/data /mnt/tmp/ 2>&1 | grep "Write: "
...
2021/03/24 16:14:42 DEBUG : &{1G (w)}: >Write: written=131072, err=<nil>
I'd say this difference is entirely down to the different sized write blocks, but I haven't been able to change it. I've tried loads of fuse parameters (max_write
,auto_cache
) and I haven't been able to budge cgofuse from its 4k blocks for write.
Fuse mysteriously says this in the docs for max_write
max_write=N
Set the maximum number of bytes in a single write operation. The default is 128kbytes. Note, that due to various limitations, the size of write requests can be much smaller (4kbytes). This limitation will be removed in the future.
This was originally noted with macOS on the rclone forum but it replicated with linux for me very easily.
Any help much appreciated - running short of hair to pull out ;-)
(Testing done with rclone master (built with go build -tags cmount
to include cgofuse support on Linux, and use --debug-fuse
to see the fuse debug).)
When I delete a Folder which contains one or more Files, the callback Rmdir is not called on Folder. Only the Unlink is called for all the files in that Folder.
Any idea or workaround for this issue?
This is probably me doing something stupid, so apologies in advance!
In the new rclone tests I'm writing, I'm running the tests multiple times with different cache modes.
This involves unmounting then remounting in the same process.
However I get this error The service cmount.test has failed to run (Status=c000010e)
which I don't understand! AFAICT it means "driver not loaded" which doesn't make things any clearer.
I tried putting a 5 second pause after the unmount before remounting but no joy.
If I try unmounting and remounting on the command line (with a new rclone process each time) it all seems to work fine.
So maybe cgofuse is unloading WinFSP on unmount, but isn't able to reload it again on mount for some reason?
Any ideas?
[successful test output snipped]
The service cmount.test has been stopped.
2017/11/17 13:19:04 DEBUG : host.Unmount succeeded
2017/11/17 13:19:09 Waiting for umount
2017/11/17 13:19:09 Finished test run with cache mode off
2017/11/17 13:19:09 Starting test run with cache mode minimal
2017/11/17 13:19:09 mount "Local file system at \\\\?\\C:\\Users\\Dev\\AppData\\Local\\Temp\\rclone540012982" "E:"
2017/11/17 13:19:09 DEBUG : Local file system at \\?\C:\Users\Dev\AppData\Local\Temp\rclone540012982: Mounting on "E:"
2017/11/17 13:19:09 DEBUG : vfs cache root is "C:\\Users\\Dev\\AppData\\Local\\rclone\\vfs\\local\\C\\Users\\Dev\\AppData\\Local\\Temp\\rclone540012982"
2017/11/17 13:19:09 DEBUG : Local file system at \\?\C:\Users\Dev\AppData\Local\Temp\rclone540012982: Mounting with options: ["-o" "fsname=local:\\\\?\\C:\\Users\\Dev\\AppData\\Local\\Temp\\rclone540012982" "-o" "subtype=rclone" "-o" "max_readahead=131072" "-o" "atomic_o_trunc" "-o" "uid=-1" "-o" "gid=-1" "--FileSystemName=rclone"]
The service cmount.test has failed to run (Status=c000010e).
2017/11/17 13:19:09 ERROR : Local file system at \\?\C:\Users\Dev\AppData\Local\Temp\rclone540012982: Mount failed
2017/11/17 13:19:09 mount failed: mount stopped before calling Init: mount failed
Is it possible the Init method is called slightly before the file system is ready?
My code is waiting for the Init() method to be called (wait for signal and signal), but I'm finding in the tests that sometimes the mount appears not to be ready then.
Bad
X:\go\src\github.com\ncw\rclone\cmd\cmount>go test -v -run Ls
2017/05/08 16:23:00 mount "Local file system at \\\\?\\C:\\Users\\Dev\\AppData\\
Local\\Temp\\rclone408473623" "E:"
2017/05/08 16:23:00 mount OK
=== RUN TestDirLs
--- FAIL: TestDirLs (0.00s)
Error Trace: fs.go:219
fs.go:254
dir.go:16
Error: Received unexpected error:
open E:/: The system cannot find the path specified.
The service cmount.test has been started. <=============== NB after the test runs
FAIL
2017/05/08 16:23:00 Unmounting "E:"
The service cmount.test has been stopped.
2017/05/08 16:23:00 Waiting for umount
exit status 1
FAIL github.com/ncw/rclone/cmd/cmount 0.075s
OK
X:\go\src\github.com\ncw\rclone\cmd\cmount>go test -v -run Ls
2017/05/08 16:23:46 mount "Local file system at \\\\?\\C:\\Users\\Dev\\AppData\\
Local\\Temp\\rclone968580219" "E:"
2017/05/08 16:23:46 mount OK
The service cmount.test has been started. <=============== NB before the test runs
=== RUN TestDirLs
--- FAIL: TestDirLs (0.00s)
Error Trace: fs.go:280
dir.go:18
Error: Received unexpected error:
mkdir E:/a directory: The handle is invalid. <=== I'm debugging this - it is an error in my code!
FAIL
2017/05/08 16:23:47 Unmounting "E:"
The service cmount.test has been stopped.
2017/05/08 16:23:47 Waiting for umount
exit status 1
FAIL github.com/ncw/rclone/cmd/cmount 1.077s
I can work around this with a small sleep, but it seems not to be working as advertised!
The instructions in your README get you this far:
go: GOPATH entry is relative; must be absolute path: "\\ext-go\\1".
For more details see: 'go help gopath'
Compiling for darwin-10.6/386...
go: GOPATH entry is relative; must be absolute path: "\\ext-go\\1".
For more details see: 'go help gopath'
2018/04/01 15:14:14 Failed to cross compile package: exit status 2.
Cgofuse currently automatically unmounts on SIGINT, SIGTERM and SIGHUP (on UNIX). I notice that rclone allows the user to send it the SIGHUP signal.
From rclone:
Alternatively, you can send a ` + "`SIGHUP`" + ` signal to rclone for
it to flush all directory caches, regardless of how old they are.
Assuming only one rclone instance is running, you can reset the cache
like this:
kill -SIGHUP $(pidof rclone)
Under cgofuse this would result in rclone mount
being terminated. Any thoughts?
Ping @ncw.
It is conceptually possible to eliminate the need for cgo under Windows. The intent is to allow building with either CGO_ENABLED=1
to build the (default) "cgo" version of cgofuse, or CGO_ENABLED=0
to build a "nocgo" version of cgofuse. This is accomplished with the help of syscall.DLL
, syscall.Proc
and syscall.NewCallbackCDecl
and a rewrite of all Windows related C code in Go.
This work is currently in the windll
branch.
Unfortunately this approach currently does not work, because of a hard bug in the Go runtime: golang/go#6751. According to that bug report callbacks created with syscall.NewCallback
will hang the runtime if they are invoked in a non-Go thread. This indeed happens when the FUSE init
operation is called in the "nocgo" cgofuse.
The information in golang/go#6751 suggests that doing an import "C"
fixes the issue. Unfortunately it also makes the program use cgo, which is what we wanted to avoid with "nocgo". Still it might be useful to allows us to complete testing the "nocgo" version in anticipation that the runtime bug is fixed in the future.
Additional information: ipfs/kubo#5003 (comment)
I have a mapped network drive (R:) which is pointing one of the folder on the network E.g. "\server-share\files". When I try to mount FUSE drive to let's say "R:\Fuse" I get below error "Cannot set WinFsp-FUSE file system mount point".
However, I am able to mount Fuse to Local Drive like "C:\Fuse". I have ensured "Fuse" folder is not present.
Any idea on this issue?
This is very related to #5 but a different need!
I'd like to be able to set direct_io on certain files. In particular I'd like to be able to set it on files where rclone doesn't know the length in advance (eg Google docs files).
Rclone reports the length of these files as 0 and without this flag 0 bytes is all that can be read. However setting this flag allows the files to be read even though their size reads as 0.
I imaging this is how the files in /proc
are implemented which all have 0 size but you can still cat
them.
This works very well with linux allowing these strange 0 size files to function a bit better.
It looks like OSXFUSE supports this too (well that is the mount option - I imagine the Open option is supported too).
Is this something which can be done in Windows file systems?
To implement this it would need the interface changing to the Open
and Create
calls. If you wanted to be 100% compatible then this could return a specific errc
meaning everything OK but set the direct io flag. Or we could make an extended Open call which is used in preference over Open.
I'd be happy to help with this if you think it is a good idea.
Is there any chance you could connect this repo to the docker hub to make automated builds of the container?
I did it recently with rclone and the process was quite straight forward :-)
It makes a build on every push to the repo and you can set it to build when upstream containers change too.
Thanks
Nick
Is there a way to make cgo build with static linking? We can not have a runtime dependencies on libfuse which may not be installed on the target platform. I noticed that when building for windows there is no runtime dependency as it opens the winfsp dll dynamically (it will just fail if its not installed). Is it possible to have the same thing on Linux?
This is a question rather than an issue: how do cgofuse manage concurrent invocations? Does each OS invocation run as a separate thread or are they queued on the same thread? For example, if I run multiple instances of echo "foo" >> file.txt
, how does the library handle such a case?
Cannot compile under Docker or any cross environment
--> windows/386 error: exit status 2
Stderr: # runtime/cgo
gcc_libinit_windows.c: In function 'x_cgo_sys_thread_create':
gcc_libinit_windows.c:56:12: error: implicit declaration of function '_beginthread' [-Werror=implicit-function-declaration]
thandle = _beginthread(func, 0, arg);
^~~~~~~~~~~~
gcc_libinit_windows.c:58:69: error: 'errno' undeclared (first use in this function)
fprintf(stderr, "runtime: failed to create new OS thread (%d)\n", errno);
^~~~~
gcc_libinit_windows.c:58:69: note: each undeclared identifier is reported only once for each function it appears in
cc1: all warnings being treated as errors
--> windows/amd64 error: exit status 2
Stderr: # github.com/billziss-gh/cgofuse/fuse
..\..\..\github.com\billziss-gh\cgofuse\fuse\fsop.go:28:2: error: #error platform not supported
#error platform not supported
^~~~~
--> linux/386 error: exit status 2
Stderr: # runtime/cgo
/usr/lib/gcc/x86_64-pc-msys/6.4.0/../../../../x86_64-pc-msys/bin/ld: cannot find -lpthread
/usr/lib/gcc/x86_64-pc-msys/6.4.0/../../../../x86_64-pc-msys/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-pc-msys/6.4.0/libgcc_s.dll.a when searching for -lgcc_s
/usr/lib/gcc/x86_64-pc-msys/6.4.0/../../../../x86_64-pc-msys/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-pc-msys/6.4.0/libgcc_s.dll.a when searching for -lgcc_s
/usr/lib/gcc/x86_64-pc-msys/6.4.0/../../../../x86_64-pc-msys/bin/ld: cannot find -lgcc_s
/usr/lib/gcc/x86_64-pc-msys/6.4.0/../../../../x86_64-pc-msys/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-pc-msys/6.4.0/libgcc.a when searching for -lgcc
/usr/lib/gcc/x86_64-pc-msys/6.4.0/../../../../x86_64-pc-msys/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-pc-msys/6.4.0/libgcc.a when searching for -lgcc
/usr/lib/gcc/x86_64-pc-msys/6.4.0/../../../../x86_64-pc-msys/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-pc-msys/6.4.0/libgcc.a when searching for -lgcc
/usr/lib/gcc/x86_64-pc-msys/6.4.0/../../../../x86_64-pc-msys/bin/ld: cannot find -lgcc
/usr/lib/gcc/x86_64-pc-msys/6.4.0/../../../../x86_64-pc-msys/bin/ld: cannot find -lmsys-2.0
/usr/lib/gcc/x86_64-pc-msys/6.4.0/../../../../x86_64-pc-msys/bin/ld: skipping incompatible /usr/lib/w32api/libadvapi32.a when searching for -ladvapi32
/usr/lib/gcc/x86_64-pc-msys/6.4.0/../../../../x86_64-pc-msys/bin/ld: skipping incompatible /usr/lib/w32api/libadvapi32.a when searching for -ladvapi32
/usr/lib/gcc/x86_64-pc-msys/6.4.0/../../../../x86_64-pc-msys/bin/ld: skipping incompatible /usr/lib/w32api/libadvapi32.a when searching for -ladvapi32
/usr/lib/gcc/x86_64-pc-msys/6.4.0/../../../../x86_64-pc-msys/bin/ld: cannot find -ladvapi32
/usr/lib/gcc/x86_64-pc-msys/6.4.0/../../../../x86_64-pc-msys/bin/ld: skipping incompatible /usr/lib/w32api/libshell32.a when searching for -lshell32
/usr/lib/gcc/x86_64-pc-msys/6.4.0/../../../../x86_64-pc-msys/bin/ld: skipping incompatible /usr/lib/w32api/libshell32.a when searching for -lshell32
/usr/lib/gcc/x86_64-pc-msys/6.4.0/../../../../x86_64-pc-msys/bin/ld: skipping incompatible /usr/lib/w32api/libshell32.a when searching for -lshell32
/usr/lib/gcc/x86_64-pc-msys/6.4.0/../../../../x86_64-pc-msys/bin/ld: cannot find -lshell32
/usr/lib/gcc/x86_64-pc-msys/6.4.0/../../../../x86_64-pc-msys/bin/ld: skipping incompatible /usr/lib/w32api/libuser32.a when searching for -luser32
/usr/lib/gcc/x86_64-pc-msys/6.4.0/../../../../x86_64-pc-msys/bin/ld: skipping incompatible /usr/lib/w32api/libuser32.a when searching for -luser32
/usr/lib/gcc/x86_64-pc-msys/6.4.0/../../../../x86_64-pc-msys/bin/ld: skipping incompatible /usr/lib/w32api/libuser32.a when searching for -luser32
/usr/lib/gcc/x86_64-pc-msys/6.4.0/../../../../x86_64-pc-msys/bin/ld: cannot find -luser32
/usr/lib/gcc/x86_64-pc-msys/6.4.0/../../../../x86_64-pc-msys/bin/ld: skipping incompatible /usr/lib/w32api/libkernel32.a when searching for -lkernel32
/usr/lib/gcc/x86_64-pc-msys/6.4.0/../../../../x86_64-pc-msys/bin/ld: skipping incompatible /usr/lib/w32api/libkernel32.a when searching for -lkernel32
/usr/lib/gcc/x86_64-pc-msys/6.4.0/../../../../x86_64-pc-msys/bin/ld: skipping incompatible /usr/lib/w32api/libkernel32.a when searching for -lkernel32
/usr/lib/gcc/x86_64-pc-msys/6.4.0/../../../../x86_64-pc-msys/bin/ld: cannot find -lkernel32
/usr/lib/gcc/x86_64-pc-msys/6.4.0/../../../../x86_64-pc-msys/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-pc-msys/6.4.0/libgcc_s.dll.a when searching for -lgcc_s
/usr/lib/gcc/x86_64-pc-msys/6.4.0/../../../../x86_64-pc-msys/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-pc-msys/6.4.0/libgcc_s.dll.a when searching for -lgcc_s
/usr/lib/gcc/x86_64-pc-msys/6.4.0/../../../../x86_64-pc-msys/bin/ld: cannot find -lgcc_s
/usr/lib/gcc/x86_64-pc-msys/6.4.0/../../../../x86_64-pc-msys/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-pc-msys/6.4.0/libgcc.a when searching for -lgcc
/usr/lib/gcc/x86_64-pc-msys/6.4.0/../../../../x86_64-pc-msys/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-pc-msys/6.4.0/libgcc.a when searching for -lgcc
/usr/lib/gcc/x86_64-pc-msys/6.4.0/../../../../x86_64-pc-msys/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-pc-msys/6.4.0/libgcc.a when searching for -lgcc
/usr/lib/gcc/x86_64-pc-msys/6.4.0/../../../../x86_64-pc-msys/bin/ld: cannot find -lgcc
collect2: error: ld returned 1 exit status
Stderr: # github.com/billziss-gh/cgofuse/fuse
..\..\..\github.com\billziss-gh\cgofuse\fuse\fsop.go:28:2: error: #error platform not supported
#error platform not supported
^~~~~```
When upgrading to 1.9.21096 (release) from 1.9.21033 (beta2) the virtio-fs disk does not show up anymore. Even tried to run "virtiofs.exe -d -1" in the command prompt (with Virtio-FS service disabled) without success.
Alpine is super popular for building containers. A common use case for cgofuse is implementing file systems to use as FlexVolumes in kubernetes.
The required dependencies are: fuse-dev gcc libc-dev
Two differences from deb-based Linux such as Ubuntu:
With MacFUSE i.e. OSXFuse 4.0, the location of the loadable library has changed to /usr/local/lib/libfuse.2.dylib
instead of /usr/local/lib/libosxfuse.2.dylib
. The older path is still present for backwards compatibility but it is probably better to use the newer path if available.
The include path also seems to have changed /usr/local/include/fuse
instead of /usr/local/include/osxfuse/fuse
.
Release Notes: https://github.com/osxfuse/osxfuse/releases/tag/macfuse-4.0.0
Hi,
WinFSP is a high quality project. But it has a restrictive license, so many project opt for DokanY instead.
Is it possible to choose between multiple backends on Windows?
Another cross-platform FUSE-alike is to host an FTP or WebDAV server, and then, each OS already has some way of mounting it. It might be interesting to support such as a platform-agnostic FUSE-alike backend.
A recent issue (winfsp/winfsp#114) in WinFsp asked for BSD flags support in the FUSE layer. I have added this functionality to WinFsp-FUSE and I am looking to add it to cgofuse.
The proposed changes to cgofuse are as follows:
Add a Flags uint32
field to struct Stat_t
: [link]. File systems that wish to support this functionality may fill the Flags
field during Getattr
.
Add a Chflags(path string, flags uint32) int
method to FileSystemInterface
: [link]
Add the BSD/OSX UF_HIDDEN, UF_READONLY, UF_SYSTEM, UF_ARCHIVE
constants.
Add func (host *FileSystemHost) SetCapBsdFlags(value bool)
to FileSystemHost
. When this capability is enabled, the file system is expected to fill the Flags
field of struct Stat_t
. It may also optionally implement Chflags
. QUESTION: do we really need this or should we assume that this capability is always enabled for cgofuse file systems?
This functionality will be supported on WinFsp-FUSE and OSXFUSE, as they both support the BSD flags functionality.
@ncw my main concern with this scheme is backward compatibility. Since releasing a new go binary requires recompilation these changes look backward compatible and safe to me. Since you have a lot more experience with go than me, I would appreciate your feedback.
Nice project, good API and happy that it is also available on Windows.
I'm trying to build a FUSE filesystem for OSX and Windows. Got a part working, but when testing on Microsoft Windows [Version 10.0.16299.125]
it doesn't work.
Also when testing with memfs.go it doesn't work well on Windows. When running in DEBUG mode I see that all files and directories are uppercase. Some issues that I'm having running memfs:
I'm using the latest WinFSP version.
Can you help me out what is going wrong here?
An rclone user has discovered that sometimes Windows tries to set times on files which are 1601-01-01 via Utimens
2019/08/22 18:57:39 DEBUG : /rclonereadme - Shortcut.lnk: Utimens: tmsp=[{Sec:-11644473600 Nsec:-100} {Sec:-11644473600 Nsec:-100}]
According to my reading, that time is the epoch on NTFS so that is effectively a zero time value.
Should cgofuse (or maybe WinFSP) be filtering these out? Or is it rclone's job to work out that these are invalid?
I'm not sure of the circumstances which these come through yet.
Any help much appreciated - thanks :-)
For background see: https://forum.rclone.org/t/io-error-googleapi-error-403-insufficient-permission-insufficientpermissions/11372/3
From host.go
// Mount unmounts a file system.
func (host *FileSystemHost) Unmount() {
if nil != host.fuse {
C.fuse_exit(host.fuse)
}
}
However according to the fuse docs fuse_exit just stops the event loop, it doesn't do an unmount.
So this method should probably be renamed "Stop" or something like that.
An Unmount method would be really useful though. Here is how bazil/fuse does it for linux and everyone else. I'm guessing a different procedure would be needed for Windows.
As part of testing #38 I noticed that you haven't made a release for a while.
In the new world of go modules, consumers of this repo will only use your latest tagged versions by default.
There aren't too many commits of significance here but CreateEx and OpenEx is one :-)
Thank you Bill and hope you are keeping well in these troubled times.
I'm trying to host several different file systems within a single program.
To do this, I construct a FileSystemHost
for each file system implementation, and then concurrently call their Mount
method.
Doing so on a fresh boot of the OS, or after installing the driver for the first time, causes cgofuse to log an error to the console (Cannot create WinFSP-FUSE file system: FSD not found.
), for all but 1 of the Mount
calls.
This does not happen on subsequent runs.
When run again, each call to Mount
works as expected under the same conditions.
Demonstration: https://www.youtube.com/watch?v=V0Z_abvSPNs
Modifying memfs's main function gives a simple reproducible:
func main() {
memfs1, memfs2, memfs3 := NewMemfs(), NewMemfs(), NewMemfs()
host1, host2, host3 := fuse.NewFileSystemHost(memfs1), fuse.NewFileSystemHost(memfs2), fuse.NewFileSystemHost(memfs3)
hostPath := `C:\whatever`
os.Mkdir(hostPath, 0777)
for i, hst := range []*fuse.FileSystemHost{host1, host2, host3} {
go func(index int, host *fuse.FileSystemHost) {
host.SetCapReaddirPlus(true)
hostTarget := filepath.Join(hostPath, strconv.Itoa(index))
fmt.Println("Mounting: ", hostTarget)
host.Mount(hostTarget, nil)
}(i, hst)
}
semaphore := make(chan struct{})
<-semaphore
}
I've experienced this on versions of Go 1.14 - 1.15 with WinFSP versions 2020 - 2020.2
And assume it's not intended behavior. Although I'm wondering if the client program is expected to gaurd against such a thing (prevent concurrent calls to Mount
even across different FileSystemHost
instances)
If I can help provide more context for this, let me know.
A call to mount should not panic when winfsp is not installed but return an error instead. This allows an application to handle this case gracefully.
As an alternative you could provide a public function that will allow to check winfsp/fuse availability before calling mount.
If I pass -o ro
to WinFsp then doesn't mark the file system as read only as it does on linux. I think libfuse passes unknown options on to mount.fuse
which says in its man page
Most of the generic mount options described in mount are supported (ro, rw, suid, nosuid, dev, nodev, exec, noexec, atime, noatime, sync, async, dirsync).
This brings to mind 3 options
-EROFS
in any methods which do writing?What is your preferred option?
Thanks
Ref: golang/go#34530
Thanks for a great library!
Perhaps it is obvious for others, but I was wondering about thread-safety and the different callback methods. Is it on the implementor of the FileSystemInterface
to protect data structures, or is it done by the library? In the Passthrough
example, nothing is protected while in Memfs
all methods are protected behind a single Mutex
. Is it because the thread-safety is guaranteed by the underlying filesystem in the Passthrough
case?
Would be nice to have some mention of this in the README/godoc reference.
Normally I'd read O_RDONLY from syscall, but I don't think that will work on Windows, so thsese constants need to be duplicated in cgofuse.
These are necessary when working out which mode to open a file in (read or write) when the flags just can't be passed on.
I am able to run the example/hellofs and mount it on my windows computer. However, when I try to share the drive, I get an error message "The device or directory does not exist". Any suggestions on how to get the mounted drive to be shared?
I am working towards freezing the public API for a 1.0.x release. Some issues for discussion:
The Mount
API currently accepts FUSE command line arguments. Perhaps this should be changed.
From @advanderveer:
The question would be if we want to present the reference C implementation to Go via this "lazy/high level" function? It makes a lot of assumptions about the environment and arguments, e.g the first argument must be the program name, and --options will output some help text here, It installs signal handlers etc.
There is currently no mechanism to request any of the capabilities advertised during fuse_operations::init
. [FIXED - see SetCapCaseInsensitive
and SetCapReaddirPlus
. We can add more capabilities if desired.]
The fuse_operations::lock
operation is not currently supported.
The FileSystemInterface.Getxattr
method had a very clunky interface. This was streamlined in commit cf3eb1b.
Any other issues.
When I try an xgo compile
xgo -image=billziss/xgo-cgofuse -targets=darwin/amd64 github.com/billziss-gh/cgofuse/fuse
I get this error
Checking docker installation...
Client:
Version: 19.03.8
API version: 1.40
Go version: go1.13.8
Git commit: afacb8b7f0
Built: Wed Mar 11 23:42:35 2020
OS/Arch: linux/amd64
Experimental: true
Server:
Engine:
Version: 19.03.8
API version: 1.40 (minimum version 1.12)
Go version: go1.13.8
Git commit: afacb8b7f0
Built: Wed Mar 11 22:48:33 2020
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.3.3-0ubuntu2
GitCommit:
runc:
Version: spec: 1.0.1-dev
GitCommit:
docker-init:
Version: 0.18.0
GitCommit:
Checking for required docker image billziss/xgo-cgofuse... found.
Cross compiling github.com/billziss-gh/cgofuse/fuse...
Fetching main repository github.com/billziss-gh/cgofuse/fuse...
github.com/billziss-gh/cgofuse (download)
Compiling for darwin-10.6/amd64...
# github.com/billziss-gh/cgofuse/fuse
./host_cgo.go:390:6: error: no member named 'nonseekable' in 'struct fuse_file_info'
fi->nonseekable = nonseekable;
~~ ^
1 error generated.
2020/06/29 11:55:41 Failed to cross compile package: exit status 2.
I bisected the problem and (unsuprisingly!) this was the commit that broke it 8e41a0c - this was new to v1.3.0
So I think the FUSE headers need to be updated somehow in the Docker file for darwin.
I note that 2.8.3 is a way behind the current 3.10.6
So I think either we need to do one of these
#ifdef
out that feature on osxI'm not sure which of those is the best way forward though!
I tried cross-compiling using the billziss/xgo-cgofuse Docker image, but it fails due to too old go version. I then tried using the Dockerfile
in the repo instead, and built the image locally - that worked. I think all that is missing is a new version of billziss/xgo-cgofuse pushed to the Docker registry.
I encountered an issue on macOS where under load, a ls
on a cgofuse-mounted filesystem would sometimes show files twice, or not show some files at all. My application is confirmed to call fill
exactly as many times with the same filenames when the issue shows as when it does not. To confirm this is an issue outside of my application, I created a simple test case: https://gist.github.com/sgielen/b229ded71f5946fc242bfb6e75e08da6
The issue seems to worsen when the Readdir
call takes longer, so my testcase Readdir implementation has a sleep between 0-200 ms in it. Furthermore, I've noticed that the lengths of filenames, and perhaps the hierarchy of the filesystem, must be reproduced in order to reproduce the bug.
I can reproduce with golang 1.16.3, cgofuse 1.5.0, macfuse 4.1.2, macOS Big Sur 11.2.3. I can't reproduce on Windows or on Linux.
I'm willing to investigate further, but could use your advice for next steps.
I've been experimenting trying to make rclone mount work with OpenBSD
Plus or minus a few unsupported options it seems to be working :-)
I've come across an oddity with Readdir
which I'm not sure of the cause - I think it may be an OpenBSD bug but it might be a cgofuse bug so I thought I'd ask you first.
I replicated this with the hellofs
example.
This is the result of mounting and doing ls
on the root directory (note that you can't mount FUSE fs as a user on OpenBSD).
openbsd$ sudo ./hellofs -d /mnt/tmp
Opcode: init
Opcode: getattr Inode: 1 /
Opcode: getattr Inode: 1 /
Opcode: opendir Inode: 1 /
Opcode: getattr Inode: 1 /
Opcode: readdir Inode: 1 Offset: 0 Size: 4096 /
Opcode: readdir Inode: 1 Offset: 96 Size: 4000 /
Opcode: readdir Inode: 1 Offset: 96 Size: 4096 / <----- unexpected
Opcode: releasedir Inode: 1 /
What is odd here is the 3rd call to readdir
.
I put a debug line into the Readdir function
diff --git a/examples/hellofs/hellofs.go b/examples/hellofs/hellofs.go
index d33566f..6d6eb39 100644
--- a/examples/hellofs/hellofs.go
+++ b/examples/hellofs/hellofs.go
@@ -14,6 +14,7 @@ package main
import (
"os"
+ "log"
"github.com/billziss-gh/cgofuse/fuse"
)
@@ -66,6 +67,7 @@ func (self *Hellofs) Readdir(path string,
fill func(name string, stat *fuse.Stat_t, ofst int64) bool,
ofst int64,
fh uint64) (errc int) {
+ log.Printf("\n*** Readdir(path=%q, ofst=%d, fh=0x%X) ***\n", path, ofst, fh)
fill(".", nil, 0)
fill("..", nil, 0)
fill(filename, nil, 0)
And it produces this on OpenBSD - note the Readdir
function is called 3 times rather than just once.
Opcode: init
Opcode: getattr Inode: 1 /
Opcode: getattr Inode: 1 /
Opcode: opendir Inode: 1 /
Opcode: getattr Inode: 1 /
Opcode: readdir Inode: 1 Offset: 0 Size: 4096 /2020/09/01 12:47:31
*** Readdir(path="/", ofst=0, fh=0xFFFFFFFFFFFFFFFF) ***
Opcode: readdir Inode: 1 Offset: 96 Size: 4000 /2020/09/01 12:47:31
*** Readdir(path="/", ofst=96, fh=0xFFFFFFFFFFFFFFFF) ***
Opcode: readdir Inode: 1 Offset: 96 Size: 4096 /2020/09/01 12:47:31
*** Readdir(path="/", ofst=96, fh=0xFFFFFFFFFFFFFFFF) ***
Opcode: releasedir Inode: 1 /
Whereas if I try the same code on Linux I get the Readdir
function being called just once as expected with two READDIR
fuse calls.
unique: 10, opcode: GETATTR (3), nodeid: 1, insize: 56, pid: 115481
getattr /
unique: 10, success, outsize: 120
unique: 12, opcode: OPENDIR (27), nodeid: 1, insize: 48, pid: 115481
opendir flags: 0x18800 /
opendir[-1] flags: 0x18800 /
unique: 12, success, outsize: 32
unique: 14, opcode: READDIR (28), nodeid: 1, insize: 80, pid: 115481
readdir[18446744073709551615] from 0
2020/09/01 11:50:41
*** Readdir(path="/", ofst=0, fh=0xFFFFFFFFFFFFFFFF) ***
unique: 14, success, outsize: 112
unique: 16, opcode: LOOKUP (1), nodeid: 1, insize: 46, pid: 115481
LOOKUP /hello
getattr /hello
NODEID: 2
unique: 16, success, outsize: 144
unique: 18, opcode: READDIR (28), nodeid: 1, insize: 80, pid: 115481
unique: 18, success, outsize: 16
unique: 20, opcode: RELEASEDIR (29), nodeid: 1, insize: 64, pid: 0
releasedir[18446744073709551615] flags: 0x0
unique: 20, success, outsize: 16
So I think this might be a bug in the directory filling routines in OpenBSD, whether that is in cgofuse, or OpenBSD libfuse I don't know!
This causes a problem for rclone since it checks to see that Readdir
is never called with a non-zero offset which it never should be as it always passes 0 as an offset to the fill
function. This is easy enough to work-around but I thought I'd report it in case it is indicative of an underlying problem.
I tested this on a OpenBSD 6.7 VM I installed from scratch using go version go1.13.9 openbsd/amd64
(which is what you get with pkg_add
. I can send you this VM if you would like.
It is understood that WinFsp and the rclone mount command will only work in Windows 7 and up. But cgofuse is causing rclone to fail in Windows XP SP2 32-bit with this error, even without mount:
"The procedure entry point RegGetValueW could not be located in the dynamic link library ADVAPI32.dll".
That's just running rclone with no command-line options. A bit of googling shows that RegGetValueW is only supported in 64-bit Windows XP. If rclone is going to continue supporting Windows XP (albeit without mount support), cgofuse should fail gracefully when RegGetValueW is not available.
See rclone/rclone#1481 (comment) for more context.
The alternative is to build rclone without cgofuse, which also fixes the problem. (e.g. special build rclone/rclone#1481 (comment) , which does run in Windows XP 32-bit).
For instance this causes an exception
rclone mount -vv C:\Users\Dev Q: --fuse-flag --FileSystemName=ddddddddddddd
Whereas this doesn't
rclone mount -vv C:\Users\Dev Q: --fuse-flag --FileSystemName=dddddddddddd
The exception I got was
Exception 0xc0000005 0x1 0x2633700 0x72c90007
PC=0x72c90007
signal arrived during external code execution
github.com/billziss-gh/cgofuse/fuse._Cfunc_hostMount(0x8f, 0x1245d680, 0x13195b0, 0x0)
github.com/billziss-gh/cgofuse/fuse/_obj/_cgo_gotypes.go:402 +0x3f
github.com/billziss-gh/cgofuse/fuse.(*FileSystemHost).Mount.func9(0x1245d680, 0x10, 0x10, 0xf, 0x1245d680, 0x13195b0, 0x0)
X:/go/src/github.com/billziss-gh/cgofuse/fuse/host.go:1110 +0xf3
github.com/billziss-gh/cgofuse/fuse.(*FileSystemHost).Mount(0x12451f20, 0x12318108, 0x2, 0x12456900, 0xc, 0xc, 0x0)
X:/go/src/github.com/billziss-gh/cgofuse/fuse/host.go:1110 +0x2e0
github.com/ncw/rclone/cmd/cmount.mount.func1(0x12451f20, 0x12318108, 0x2, 0x12456900, 0xc, 0xc, 0xf2d640, 0x123ada90, 0x1245d640)
X:/go/src/github.com/ncw/rclone/cmd/cmount/mount.go:253 +0x4d
created by github.com/ncw/rclone/cmd/cmount.mount
X:/go/src/github.com/ncw/rclone/cmd/cmount/mount.go:259 +0x326
I suspect this is probably a bug in WinFsp rather than cgofuse, but I'm not 100% certain!
You can find an rclone with the --fuse-flag
option on the cmount branch if you need to try it for yourself.
Using
rclone -vv cmount c:\Users\Dev Q:
I did this with python
>>> f = open("Q:/text.txt", "w")
>>> f.write("hello")
>>> f.close()
>>> import os
>>> os.unlink("Q:/text.txt")
>>> # note no error here
However I see this in the rclone log
2017/05/08 19:31:58 DEBUG : /text.txt: Unlink()
2017/05/08 19:31:58 DEBUG : text.txt: Dir.Remove
2017/05/08 19:31:58 ERROR : text.txt: Dir.Remove file error: remove \\?\c:\Users
\Dev\text.txt: The process cannot access the file because it is being used by an
other process.
2017/05/08 19:31:58 ERROR : IO error: remove \\?\c:\Users\Dev\text.txt: The proc
ess cannot access the file because it is being used by another process.
2017/05/08 19:31:58 DEBUG : /text.txt: Unlink() returning -5
2017/05/08 19:31:58 DEBUG : text.txt: ReadFileHandle.Flush
2017/05/08 19:31:58 DEBUG : text.txt: ReadFileHandle.Flush OK
2017/05/08 19:31:58 DEBUG : text.txt: ReadFileHandle.Release closing
2017/05/08 19:31:58 DEBUG : text.txt: ReadFileHandle.Release OK
And the underlying file hasn't been deleted
C:\Users\Dev>dir
Volume in drive C has no label.
Volume Serial Number is 5CD1-D250
Directory of C:\Users\Dev
08/05/2017 19:31 <DIR> .
08/05/2017 19:31 <DIR> ..
[snip]
08/05/2017 19:31 5 text.txt
C:\Users\Dev>
The in use error is caused by the delayed Flush/Close which comes some time after the close in userspace. This I think is a fuse oddity.
This error gets translated to -5 EIO, however the error from Unlink didn't get reported back to userspace - I would have expected the os.unlink
to raise an exception.
This is the definition of Unlink
I've been using for these tests
// Unlink removes a file.
func (fsys *FS) Unlink(filePath string) (errc int) {
fs.Debugf(filePath, "Unlink()")
defer func() {
fs.Debugf(filePath, "Unlink() returning %d", errc)
}()
leaf, parentDir, errc := fsys.lookupParentDir(filePath)
if errc != 0 {
return errc
}
return translateError(parentDir.Remove(leaf))
}
Any ideas?
Great work on getting such a clean interface for cross platform FUSE! When I was trying this out today I couldn't get the Filesystem to unmount cleanly though. For example, adapting the memfs example's main() like this causes the filesystem to hang around when the program exits:
func main() {
exitCh := make(chan os.Signal)
signal.Notify(exitCh, os.Interrupt)
memfs := NewMemfs()
host := fuse.NewFileSystemHost(memfs)
fmt.Println("mounting...")
go host.Mount(os.Args)
<-exitCh
fmt.Println("unmounting...")
host.Unmount()
}
on OSX/Linux the Unmount() doesn't seem to do anything, on Windows the library itself seems to unmount but not under control of my own program but rather somewhere inside cgofuse itself.
In Open I'd like to be able to set the nonseekable flag.
In rclone seeking is impossible on files that are open for write and difficult on files open for read (so the users may turn it off).
I can detect this in the Read/Write however I'm used to getting fuse to do this for me.
I guess some of the other flags in file_info might be useful too.
That said, if this interferes with winfsp, it isn't a big deal!
Looks like right now it's using fuse version 2.8, is there a plan to support fuse3?
Hi,
We are trying to create Application Shortcut on Windows where icons being referred from WinFsp Fuse Mounted Drive (memfs). Now, the shortcut is created successfully but the we are not able to see the icon instead a blank icon is shown. However, if we refer to same icon which is in Local Disk it works fine.
Can you please us to understand the issue and a possible workaround?
Hi @billziss-gh
I sent you an email about commercial support but didn't get a response.
Could you please contact me via [email protected]
Thanks
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.