rharder / fusepy Goto Github PK

I consolidated all the fuse*.py files into one, and modified the setup.py to 
allow the module to work in Python3.x with 2to3 (distribute flag).

My changes are technically linear, but I used --no-ff when merging the branch 
so I could describe what all the changes I made are doing.

When trying to learn how to use the software there are only a couple of 
complicated examples to try and learn from and there is no documentation

Almost all other python modules have some kind of tutorial and definitely have 
documentation.

It would be great if there were a way to set flag_nullpath_ok (defined in
fuse.h) from Python.

What steps will reproduce the problem?
1. Run example.py on 64-bit linux (Centos 5.2 x86_64)
2. do 'ls <dir>' where <dir> is the directory you have example.py mounted 
on 

What is the expected output? What do you see instead?

Expect normal directory listing, instead get this:

$ ls -l mydir
ls: mydir: Input/output error

What version of the product are you using? On what operating system?

latest trunk version (r6).  centos 5.2 x86_64

Please provide any additional information below.

I verified with logging statements that example.py's getattr() function is 
retuning c_stat object.  It's total speculation that this is related to 64-
bit, however I noticed that a lot of the sizes/types specified for the 
c_stat Structure in fuse.py appear to be different on 64-bit Linux (e.g., 
inodes are 8 bytes instead of 4, and there are no __pad fields, etc).  
However, that's a total guess, it could be something else entirely.

It would be nice if there was a way to explicitly call the filler()
function that is passed to readdir(). That would allow to return stat
information (most importantly st_ino) and work with offsets.

Currently, I am locally patching fuse.py as follows

    def readdir(self, path, buf, filler, offset, fi):
        def pyfiller(name, attrs, off):
            st = c_stat()
            if attrs != None:
                for key, val in attrs.items():
                    if key in ('st_atime', 'st_mtime', 'st_ctime'):
                        timespec = getattr(st, key + 'spec')
                        timespec.tv_sec = int(val)
                        timespec.tv_nsec = int((val - timespec.tv_sec) * 10
** 9)
                    elif hasattr(st, key):
                        setattr(st, key, val)

                filler(buf, name, st, off)
            else:
                filler(buf, name, None, off)

        self.operations("readdir", path, pyfiller, offset, fi.contents.fh)
        return 0

It would be great if similar changes could be incorporated (although right
now I don't see how to make this backwards compatible).

setuptools and easy_install refuse to install fusepy. Because of this, projects 
that depend on fusepy 
can't list fusepy in their "install_requires" section of the setup.py script.

Currently, the open() operation has no way to set the direct_io flag,
because it only gets the file handle rather than the whole struct
fuse_file_info.

I am trying to implement a simple filesytem that actually dispatch all
operations for a underline real filesystem. It will be the version version
of my filesystem. 

For example. If one mount:

/mnt

With my filesystem, it should create a directory inside the mountpoint as
the real root of this new filesystem:

/mnt/.data

then, if one create a file, the filesystem will change append the ".data/"
in the name and call the real filesystem to create the file inside ".data/".

Is it clear? My point is that I haven't understand what the open operation
must return? Should it be:

def open(self, path, flags):
   return open(self.realpath(path), flags)

If so, what is the last parameter of read operation? What object will I
need to pass to the read operation?

1. Is the "machine check code"(What I mean is listed below) of linux used 
for byte boundary alignment?
    _machine = machine()
    if _machine == 'i686':
If another cpu architechure was used such as ARM or PowerPC, that is I am 
going to port the fusepy to another platform, what I need to do is to fix 
the byte boundary alignment?

Thanks.

Currently, the return code of fuse_main_real is ignored. However, it
contains important information (e.g. if the mount failed because the
mountpoint does not exist). 

There are two possible solutions:

a) Check the return code if fuse_main_real and raise an exception if it is
non-zero. 

b) Move the call out of the constructor into a separate mount_and_serve()
(or something like that) method and return the errorcode to the caller as
the return value.

What steps will reproduce the problem?
1. Download memory3.py and fuse3.py into the same directory
2. Run memory3.py
3.

What is the expected output? What do you see instead?
Expected working FUSE memory filesystem. Got print_stack:
Traceback (most recent call last):
  File "./memory3.py", line 128, in <module>
    fuse = FUSE(Memory(), argv[1], foreground=True)
  File "/home/suicide16/test/fuse3.py", line 279, in __init__
    argv = (c_char_p * len(args))(*args)
TypeError: bytes or integer address expected instead of str instance

What version of the product are you using? On what operating system?
fuse3.py from Trunk(r65), Python 3.3.3 on x86_64 Gentoo Linux box

Please provide any additional information below.

What steps will reproduce the problem?
1. Download fuse.py and loopback.py.
2. Modify loopback.py so it's willing to run in the background, and so it
   does not chatter.  I'll attach a patch.  loopback.py is simple and 
   small, so I'll also attach this revised loopback.py.
3. Create directories "fromdir" and "todir" and a 30 megabyte file of 
   random noise in "fromdir" with the commands:
      mkdir fromdir
      mkdir todir
      head -c 31457280 /dev/urandom > fromdir/bits
4. Give the following command 20 times or so:
      python loopback.py fromdir todir; cmp fromdir/bits todir/bits;
fusermount -u todir
   On my system cmp will complain 2 to 4 times out of 20.  The 
   typical complaint is 
      fromdir/bits todir/bits differ: byte 2736129, line 10648
   with randomly different byte numbers and line numbers.

What is the expected output? 

cmp should say the loopback-mounted version of bits has the same bits as
the original version.

What do you see instead?

cmp says they differ.

What version of the product are you using? 

I'm not seeing version numbers here.  I fetched whatever was downloadable
on August 25, 2009. 

On what operating system?

Ubuntu 8.10

Please provide any additional information below.

When the files differ, the wrong bytes in todir/bits are blocks that appear
at some other location in fromdir/bits.  The problem is that there are
multiple threads reading the file, they all share the same filehandle, and
they non-atomically seek in the file and then read a block from the file. 
One thread seeks, the other thread seeks, then the first thread reads a
block from the wrong place.  I'll attach a patch that fixes the problem,
and the final loopback.py that has the bug fix and the chatter removed and
runs in the background.

It would be nice to have a unique namespace package `fusepy` (e.g.
`fusepy.FUSE`, `fusepy.Operations`) and an install script (`setuptools` or
`distutils`) in order to use your package as a library.

Having an entry in PyPI would be also great.

P. S. I could prepare a patch containing these changes for you.

statfs returns data that appears to be incorrectly aligned on a Core 2 DUO 
(64bit) Mac running MacFUSE 2.1.5 with the standard snow leopard python (2.6.1)

Changing the type defs as follows seemed to resolve the issue for me:

    c_fsblkcnt_t = c_uint32
    c_fsfilcnt_t = c_uint32

What steps will reproduce the problem?
1. sudo port -v install macfuse
2. svn checkout http://fusepy.googlecode.com/svn/trunk/ fusepy-read-only
3. cd fusepy-read-only

And than I tried with both macports python 2.5 and python from Mac OS:

4. /opt/local/bin/python2.5 example.py

visnet-109:HyFile arademaker$ /opt/local/bin/python2.5 example.py
Traceback (most recent call last):
  File "example.py", line 4, in <module>
    from fuse import Fuse
  File "/opt/local/lib/python2.5/site-packages/fuse.py", line 26, in <module>
    from fuseparts._fuse import main, FuseGetContext, FuseInvalidate
ImportError:
dlopen(/opt/local/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5
/site-packages/fuseparts/_fusemodule.so,
2): Symbol not found: _iconv
  Referenced from: /opt/local/lib/libfuse.0.dylib
  Expected in: flat namespace

OR

visnet-109:fusepy-read-only arademaker$ python example.py
Traceback (most recent call last):
  File "example.py", line 106, in <module>
    fuse = FUSE(MyOperations(), foreground=True, fsname="ExampleFS")
  File "fuse.py", line 253, in __init__
    libfuse.fuse_main_real(len(args), argv, pointer(fuse_ops),
sizeof(fuse_ops), None)
  File
"/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/ctypes/_
_init__.py",
line 338, in __getattr__
    func = self.__getitem__(name)
  File
"/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/ctypes/_
_init__.py",
line 343, in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
AttributeError: dlsym(RTLD_DEFAULT, fuse_main_real): symbol not found

When chown() is called with a value of -1 for uid or gid, the respective
property of the file should not be changed.

However, fuse.py (correctly) defines both uid_t and gid_t as unsigned
integers. The number -1 therefore becomes some large positive number.

The Python implementation of chown() that tries to compare against -1
therefore fails to detect it correctly and sets uid or gid to a wrong value.

Possible fix: Have fuse.py determine the unsigned value that corresponds to
-1 with some type casts and pass -1 to the Python implementation of chown()
if this value is encountered.

Using the standard Python 2.6 that comes with Snow Leopard, the context.py demo 
hangs for 
about a minute and then segfaults. Using Python 2.5 (again, the system-bundled 
version) works 
flawlessly.

The from __future__ import division is unneeded, as the division class is
never used.

This is not an issue but I didn't know where to put the request.

I would like to develope a MacFUSE filesystem with the help of this fusepy 
project. As stated on 
the main page it seems to be made for it. But I can't figure out concrete how 
get started. For 
example:

How are file request like .DS_Store and other Mac specific handled?
How to provide custom file system volume icon?
How to specify Read or Read/Write state?

In general all the options and parameters, how can they be accessed through 
this binding?

Thanks!

Hello!
I tried the sftp example. It failed on my system because a directory contained 
umlaut characters.
The method readdir did not expect a unicode string. The attached patch fixes 
this issue for me. But I am not sure if this should be better fixed elsewhere?

As you all know macfuse is discontinued so I was trying to use fuse4x with 
fusepy.
The problem seems to be fusepy doesn't support 64bit inodes that OSX≥10.6.X 
is using by default. 
The original discussion with fuse4x guys can be found here: 
https://groups.google.com/forum/#!topic/fuse4x/X1giX-ne-Gc


What steps will reproduce the problem?
1. OS X ≥ 10.6.X uses 64bit inodes by default so get  OSX� ≥ 10.6.X
2. Install fuse4x 
3. Try executing memory.py

Fuse will hang the forked process and won't mount the fs..

Here's a patch that describes the "stat" structure on (32-bit) PPC Linux.

The program I used to come up with the information is here: 
http://grep.ro/quickpub/fusepy-ppc/. 
I hope I got the arcane macro definitions right :) Anyway, the patch works for 
me, on a PoewerPC G4 
MacMini running Debian lenny.

Would it be possible to update the package in PyPI:
http://pypi.python.org/pypi/fusepy/

The one there is a little old. I've got another PyPI package that depends on it 
and I would like to avoid having to include the newer fuse.py with the package:
http://pypi.python.org/pypi/vofs/

Thanks

According to the front page of this site, "Development of fusepy has moved to 
github: https://github.com/terencehonles/fusepy".

In this case, it is confusing to keep hosting the old codebase here. (I wasted 
many hours working with this old codebase and issue tracker, because I missed 
that notice at the top of the front page.) Please delete all of the files in 
the SVN repository and replace the with a single "README" file that contains 
the text "Development of fusepy has moved to github: 
https://github.com/terencehonles/fusepy".

All of the history will still be accessible in the SVN history, but it will be 
clear to anybody who fetches the code that it is no longer canonically hosted 
at this location.

Currently, the _operation_wrapper treats all OSError exceptions as
"expected" in the sense that everything went fine in the filesystem itself
and that the exception was deliberately raised to encapsulate the return
value of the system call.

However, this is not necessarily true. If my filesystem tries to open some
internal files and receives an OSError, which it (errorneously) doesn't
catch, this exception should be handled as a real error in the filesystem
and not just passed on.

I therefore propose to introduce a new exception class FUSEError that is
meant exclusively for exceptions that encapsulate the return code for the
systemcalls. _operation_wrapper and LoggingMixIn should then catch only
exceptions of type FUSEError, so that OSErrors result in a full error
message with backtrace.

Patch is attached.

Hello,

Currently, if I run loopback.py and try to terminate it by pressing ctrl-c, 
(SIGINT) nothing happens. If I terminate it with ctrl-\ (SIGKILL), it stops but 
the mount point becomes unusable. The way to terminate it gracefully is to 
start another terminal and kill the process with SIGTERM.

I discovered that if the Python's handler for SIGINT is disabled, ctrl-c 
terminates the process gracefully. So I added to loopback.py the line:

signal.signal(signal.SIGINT, signal.SIG_DFL)

just before the last line which starts actually starts FUSE, and now it works 
fine.

I suggest either add this line (and "import signal" at the beginning) to the 
examples, or perhaps add it to fuse.py itself.

Thanks for the great library!
Noam

While working on a fuse FS in Linux, I switched from python-fuse to fusepy 
(much nicer, thanks).

In doing so, and poking about I made some changes (attached) that you may want 
to consider in your trunk.

Ignoring the debugging stuff which you'll likely remove, to summarise:

 * Add several missing fields in statvfs on Linux.
 * Possibility of hooking the --help option from FUSE.
 * Try minimize masking of non-present fields in some structs. I understand this was probably done due to OS differences in what fields are actually present, but it hid several bugs, perhaps there's a neater way to do this than just ignoring those fields that aren't present.
 * All the OSError's raised in Operations should use the default error message. Since the provided Operations struct never modifies the message, the default should be used.
 * Defaulting to EPERM on an unimplmented rmdir may be wiser than telling the caller the system is read-only. At least on Linux, EPERM can indicate that rmdir is unsupported (rmdir(2)).
 * Finally, it would be much easier for new users if the provided mixin used the wording "Unhandled Exception", as the present error message gives the impression that something went wrong inside fusepy, which is not the case.

I hope some of these may prove useful. Thanks for a good read, and a clean 
python fuse binding! If you have any interest in my FS (nothing spectacular): 
http://code.google.com/p/anacrolix/source/browse/#hg/projects/clfs

It would be great if the fuse init() function were made part of the
Operations class API.

When the Operations instance wants to start its own threads, this can only
be done in init(), since it is called after fuse has daemonized. __init__,
on the other hand, is called before fuse daemonizes, so any threads started
there are lost.

[deleted issue]

Consider memory.py from trunc with this change:

    def statfs(self, path):
        return dict(f_bsize=512, f_frsize=128, f_blocks=4096, f_bavail=2047,
                    f_bfree=2048, f_files=120, f_ffree=101, f_favail=100)

when the file system is mounted, stat reports:

$ stat -f tmp/
-> statfs / ()
<- statfs {'f_bsize': 512, 'f_bavail': 2047, 'f_favail': 100, 'f_files':
120, 'f_frsize': 128, 'f_blocks': 4096, 'f_ffree': 101, 'f_bfree': 2048}
  File: "tmp/"
    ID: 0        Namelen: 0       Type: UNKNOWN (0x65735546)
Block size: 512        Fundamental block size: 512
Blocks: Total: 4096       Free: 2048       Available: 2047
Inodes: Total: 120        Free: 101

The so fundamental block size is wrong, it should be 128 as specified in
statfs().

The code in fuse.FUSE for removexattr doesn't forward the call on to its
"operations" object:

@safe
    def removexattr(self, path, name):
        raise FuseError(EFAULT)

I think it should be

@safe
    def removexattr(self, path, name):
        return self.operations("removexattr", path, name)

The handling of stat structs in fusepy is a little easy to stuff up. 
Furthermore there's no way to be sure that the values one is returning are 
correct for the current filesystem (see set_st_attrs).

I propose that instead of returning a dict of 'st_*' keys in the hope they're 
matched correctly to the c_stat for the current platform, that the c_stat is 
passed into those functions that would normally require them as per the C API. 
This way the user code can apply values directly to the struct, getting 
feedback on the presence of attributes, and out of range values. It also means 
the user code can test for certain features in the platform's c_stat, via 
hasattr.

An example change with fgetattr is attached. If the idea is welcomed, I'll 
provide a full patch for fuse3 in a few days. I believe this change will 
significantly reduce the chance for accidental bugs in user code.

Looks like the order of the "position" and "options" arguments are swapped
in fusepy.  The man page for setxattr(2) says:

int setxattr(const char *path, const char *name, void *value, size_t size,
u_int32_t position, int options);

but the fusepy code says:

if _system == 'Darwin':
    setxattr_t = CFUNCTYPE(c_int, c_char_p, c_char_p, POINTER(c_byte),
c_size_t, c_int, c_uint32)

and then

    @safe
    def setxattr(self, path, name, value, size, options, *args):
        return self.operations('setxattr', path, name, value, options, *args)


Doesn't "size" have to be passed?  Can you get it back from "value"?  I
think this should be

if _system == 'Darwin':
    setxattr_t = CFUNCTYPE(c_int, c_char_p, c_char_p, POINTER(c_byte),
c_size_t, c_uint32, c_int)

and

    @safe
    def setxattr(self, path, name, value, size, position, options):
        return self.operations('setxattr', path, name, value, size,
options, position)

Please try to mount the attached filesystem. It is a version of loopback.py
that tries to synchronize calls to read() and counts how often a single
thread is calling read(). 

When the filesystem is mounted, try to concurrently read two files, e.g:

$ cat mountpoint/fuse.py & cat mountpoint/fuse.py

Here is the logging output that I get from loopback.py:

$ ./loopback.py  ~ ~/mountpoint/
5:40:17,859 -1210665280: [root] Starting
15:40:25,382 -1220179056: [root] read: Acquiring lock...
15:40:25,382 -1220179056: [root] read: Acquired lock. Reading..
15:40:25,386 -1228571760: [root] read: Acquiring lock...
15:40:30,388 -1220179056: [root] read: Released lock. Number of readings in
this thread: 1
15:40:30,388 -1236964464: [root] read: Acquiring lock...
15:40:30,389 -1236964464: [root] read: Acquired lock. Reading..
15:40:30,389 -1211786352: [root] read: Acquiring lock...
15:40:35,390 -1236964464: [root] read: Released lock. Number of readings in
this thread: 1
15:40:35,391 -1220179056: [root] read: Acquiring lock...
15:40:35,391 -1220179056: [root] read: Acquired lock. Reading..
15:40:40,396 -1220179056: [root] read: Released lock. Number of readings in
this thread: 1
15:40:40,397 -1211786352: [root] read: Acquired lock. Reading..
15:40:45,403 -1211786352: [root] read: Released lock. Number of readings in
this thread: 1
15:40:45,404 -1220179056: [root] read: Acquiring lock...
15:40:45,404 -1220179056: [root] read: Acquired lock. Reading..
15:40:45,404 -1236964464: [root] read: Acquiring lock...
15:40:50,406 -1220179056: [root] read: Released lock. Number of readings in
this thread: 1

etc.

One can see that whenever read() is called, the self.local instance has no
no attribute, even though it has been initialized before in the same thread
(note the thread ids in the second column). On the other hand, the
threading.local() object clearly works as expected, so it seems that Python
is quite aware of the different threads being at work here.

See attached patch.

After python 3.2 every call must encode/decode strings between C and python 
explicitly.

In order to support files with an unknown size in a FUSE file system
setting the direct_io flag seems to be needed. I.e. I do 
fuse = FUSE(..., direct_io=True)

With this flag enabled a read() call now needs to be able to signal EOF
when all file data has been read. Currently, fusepy does not seem to
support this as returning an empty string to signal EOF doesn't work, as
FUSE.read() returns the number of bytes requested to be read, instead of
the number of bytes actually read. By using the version below an EOF can be
signaled correctly.

class FUSE(object):
    ...
    def read(self, path, buf, size, offset, fi):
        ret = self.operations('read', path, size, offset, fi.contents.fh)
        ret_size = len(ret)
        if ret_size > 0:
            memmove(buf, create_string_buffer(ret), ret_size)
        return ret_size

support OSXFUSE, a successor of MacFUSE

Shouldn't almost all default methods in `Operations` class return `-ENOSYS`?
According to the comment `/usr/include/fuse/fuse.h`:

> If the operation is not defined, they return -ENOSYS, with the
> exception of fuse_fs_open, fuse_fs_release, fuse_fs_opendir,
> fuse_fs_releasedir and fuse_fs_statfs, which return 0.

I guess you wanted to provide more useful default implementations. Is it
really better to return `-EACCESS` than `-ENOSYS` if a filesystem doesn't
support many features like creation of new files, symbolic links, etc.?

Fuse deals with UTF-8 paths. Since those paths arrive from C code, at least in 
fuse3, they should be encoded to unicode strings before arriving at user code.

I can provide a patch for this if my assumptions are correct.

What steps will reproduce the problem?
1. Have fuse4x installed (or a library occurs after the first library import 
attempt, i.e. osxfuse)
2. Open Python
3. import fuse

What is the expected output? What do you see instead?
The expected output is nothing, fuse should import correctly.
Instead, there's an error about the first library searched not existing (i.e. 
osxfuse).
I've got fuse4x, which is supported, but a failure on the first search keeps 
the next from happening.

What version of the product are you using? On what operating system?
r65, OS X 10.8


Please provide any additional information below.
While this may not be the optimal solution, it seems to fit with your 
approach.. the find_library function from ctypes.util seems to raise an 
exception when a library isn't failed -- maybe some versions return None? 
Anyway, I edited my copy of fuse.py, and here's the 'patch' for getting things 
to work for me:

if _system == 'Darwin':
    _libfuse_path = None
    try:
        _libfuse_path = find_library('osxfuse')
    except:
        try:
            _libfuse_path =  find_library('fuse4x')
        except:
            try:
                _libfuse_path = find_library('fuse')
            except:
                pass