nathanhi / pyfatfs Goto Github PK

View Code? Open in Web Editor NEW

28.0 4.0 13.0 321 KB

Python based FAT12/FAT16/FAT32 implementation with VFAT support

Home Page: https://pypi.org/project/pyfatfs/

License: MIT License

Python 98.85% Makefile 1.15%

python3 filesystem-library pyfilesystem2 fat12 fat16 fat32 vfat 8dot3 dosfs fat

pyfatfs's Introduction

pyfatfs

pyfatfs is a filesystem module for use with PyFilesystem2 for anyone who needs to access or modify files on a FAT filesystem. It also provides a low-level API that allows direct interaction with a FAT filesystem without PyFilesystem2 abstraction.

pyfatfs supports FAT12/FAT16/FAT32 as well as the VFAT extension (long file names).

Installation

pyfatfs is available via PyPI as pyfatfs, so just execute the following command to install the package for your project:

$ pip install pyfatfs

Usage

The easiest way to get started with pyfatfs is to use it in conjunction with PyFilesystem2:

PyFilesystem2

Use fs.open_fs to open a filesystem with a FAT FS URL:

import fs
my_fs = fs.open_fs("fat:///dev/sda1")

Parameters

It is possible to supply query parameters to the URI of the PyFilesystem2 opener to influence certain behavior; it can be compared to mount options. Multiple parameters can be supplied by separating them via ampersand (&).

encoding

pyfatfs offers an encoding parameter to allow overriding the default encoding of ibm437 for file names, which was mainly used by DOS and still is the default on Linux.

Any encoding known by Python can be used as value for this parameter, but keep in mind that this might affect interoperability with other systems, especially when the selected encoding/codepage is not native or supported.

Please note that this only affects encoding of the 8DOT3 short file names, not long file names of the VFAT extension, as LFN are always stored as UTF-16-LE.

import fs
my_fs = fs.open_fs("fat:///dev/sda1?encoding=cp1252")

offset

Specify an offset in bytes to skip when accessing the file. That way even complete disk images can be read if the location of the partition is known:

import fs
my_fs = fs.open_fs("fat:///dev/sda?offset=32256")

preserve_case

Preserve case when creating files. This will force LFN entries for all created files that do not match the 8DOT3 rules. This defaults to true but can be disabled by setting preserve_case to false:

import fs
my_fs = fs.open_fs("fat:///dev/sda1?preserve_case=false")

read_only

Open filesystem in read-only mode and thus don't allow writes/modifications. This defaults to false but can be enabled by setting read_only to true:

import fs
my_fs = fs.open_fs("fat:///dev/sda1?read_only=true")

utc

Create all timestamps on the filesystem in UTC time rather than local time. Affects all directory entries' creation, modification and access times.

import fs
my_fs = fs.open_fs("fat:///dev/sda1?utc=true")

lazy_load

If set to true (default), the directory entries are loaded only when accessed to increase performance with larger filesystems and resilience against recursion / directory loops.

import fs
my_fs = fs.open_fs("fat:///dev/sda1?lazy_load=false")

Testing

Tests are located at the tests directory. In order to test your new contribution to pyfatfs just run

$ python setup.py test

from your shell.

pyfatfs's People

Contributors

Stargazers

Watchers

Forkers

akx matthewfcarlson koolkdev dseeley abrasive hashbrowncipher docbobo zurcher beckerben danny8376 tanyjin123 anonymous941 swissbit-eis

pyfatfs's Issues

Is `PyFatFS.setinfo()` "Not yet properly implemented" docstring outdated?

PyFatFS.setinfo() is documented as "Not yet properly implemented", but at least for setting the modified time, it seems to work apart from issue #26 meaning that the value read back can be wrong if minutes > 31. Is that docstring outdated, or are you able to provide more information about what problems I should expect if I try to use it?

Opening file for write without append fails to truncate it, old data left at EOF

With pyfatfs checked out from the bugfix/dentry_ordering branch as at 96ae6bc, with an MS-DOS 5 floppy image (probably the filesystem contents don't matter though), if I create a file and then overwrite it with less data, the old content remains at the end of the file:

>>> fs.writetext("/testfile", "1234567890")
>>> fs.writetext("/testfile", "abcd")
>>> fs.readtext("/testfile")
'abcd567890'

This test passes with fs.memoryfs.MemoryFS.

I assume that any case where a file is opened for write but not for append can trigger this bug.

I suspect that pyfatfs.FatIO.FatIO.__init__() needs to call truncate() when self.mode.truncate is True like fs.memoryfs.MemoryFile.__init__() does.

File sizes >4GB are not handled gracefully

If a file with >4GB in size is created struct.pack throws an exception while trying to pack the size:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/nathanhi/src/pyfatfs/.venv/lib/python3.11/site-packages/fs/copy.py", line 144, in copy_file
    copy_file_if(
  File "/home/nathanhi/src/pyfatfs/.venv/lib/python3.11/site-packages/fs/copy.py", line 223, in copy_file_if
    copy_file_internal(
  File "/home/nathanhi/src/pyfatfs/.venv/lib/python3.11/site-packages/fs/copy.py", line 279, in copy_file_internal
    _copy_locked()
  File "/home/nathanhi/src/pyfatfs/.venv/lib/python3.11/site-packages/fs/copy.py", line 272, in _copy_locked
    dst_fs.upload(dst_path, read_file)
  File "/home/nathanhi/src/pyfatfs/.venv/lib/python3.11/site-packages/fs/base.py", line 1417, in upload
    tools.copy_file_data(file, dst_file, chunk_size=chunk_size)
  File "/home/nathanhi/src/pyfatfs/.venv/lib/python3.11/site-packages/fs/tools.py", line 54, in copy_file_data
    write(chunk)
  File "/home/nathanhi/src/pyfatfs/pyfatfs/FatIO.py", line 200, in write
    self.fs.update_directory_entry(self.dir_entry.get_parent_dir())
  File "/home/nathanhi/src/pyfatfs/pyfatfs/__init__.py", line 25, in _wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/nathanhi/src/pyfatfs/pyfatfs/PyFat.py", line 34, in _wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/nathanhi/src/pyfatfs/pyfatfs/PyFat.py", line 595, in update_directory_entry
    dir_entries += bytes(d)
                   ^^^^^^^^
  File "/home/nathanhi/src/pyfatfs/pyfatfs/FATDirectoryEntry.py", line 247, in __bytes__
    entry += struct.pack(self.FAT_DIRECTORY_LAYOUT,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
struct.error: 'L' format requires 0 <= number <= 4294967295```

Instead, a PyFatException with errno `EFBIG` should be raised

Create filesystem if it does not exist and 'create' flag is set in open_fs

open_fs function takes in the create optional argument, which when set, will create the filesystem instead raising an exception on opening a (non-existent) disk image file.
Currently, pyfatfs does not make use of this argument and raises a pyfatfs._exceptions.PyFATException when opening a .img file.

For example:

>>> import fs
>>> fatfile = fs.opener.open_fs('fat://test.img', create=True) 
Traceback (most recent call last):
  File "C:\Users\[removed]\AppData\Local\Programs\Python\Python39\lib\site-packages\pyfatfs\PyFat.py", line 241, in open
    self.__set_fp(open(filename, mode=mode))
FileNotFoundError: [Errno 2] No such file or directory: 'test.img'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\[removed]\AppData\Local\Programs\Python\Python39\lib\site-packages\fs\opener\registry.py", line 229, in open_fs
    _fs, _path = self.open(
  File "C:\Users\[removed]\AppData\Local\Programs\Python\Python39\lib\site-packages\fs\opener\registry.py", line 186, in open
    open_fs = opener.open_fs(fs_url, parse_result, writeable, create, cwd)
  File "C:\Users\[removed]\AppData\Local\Programs\Python\Python39\lib\site-packages\pyfatfs\PyFatFSOpener.py", line 36, in open_fs
    fs = PyFatFS(filename=parse_result.resource,
  File "C:\Users\[removed]\AppData\Local\Programs\Python\Python39\lib\site-packages\pyfatfs\PyFatFS.py", line 53, in __init__
    self.fs.open(filename, read_only=read_only)
  File "C:\Users\[removed]\AppData\Local\Programs\Python\Python39\lib\site-packages\pyfatfs\PyFat.py", line 243, in open
    raise PyFATException(f"Cannot open given file \'{filename}\'.",
pyfatfs._exceptions.PyFATException: Cannot open given file 'test.img'.

But for a different file, it works as expected:

>>> zipfile = fs.opener.open_fs('zip://test.zip', create=True)
>>> print(zipfile) 
<zipfs-write 'test.zip'>

So an option to create a blank filesystem would be nice

Google Colab Error: "UnsupportedProtocol: protocol 'fat' is not supported"

I am trying to use pyfatfs in Google Colab.

See:
https://colab.research.google.com/drive/1vV77OL3aEq3liEAihNOB8AE17GbhZ1dI?usp=sharing

I can get the tests to run (and pass, mostly -- i assume due to uimplemented features). But when I try to execute example:

!pip install pyfatfs
!pip install fs

import fs
my_fs = fs.open_fs("fat:///dev/core")

I get an error:

UnsupportedProtocol Traceback (most recent call last)
in ()
1 import fs
----> 2 my_fs = fs.open_fs("fat:///dev/core")

2 frames
/usr/local/lib/python3.6/dist-packages/fs/opener/registry.py in get_opener(self, protocol)
126 else:
127 raise UnsupportedProtocol(
--> 128 "protocol '{}' is not supported".format(protocol)
129 )
130

UnsupportedProtocol: protocol 'fat' is not supported

(the baseline google colab file system which I've used for this example doesn't have any FAT file systems, so I'm expecting an error. But this error seems to occur much before that -- not even recognizing the 'fat:///' prefix. I get the same error message in a private environment pointing to a FAT16 file image. Seems like I'm missing something obvious :(

Files have correct date but no time

Hi, thanks for the fantastic tool!

I'm having a bit of trouble with __combine_dosdatetime in FATDirectoryEntry.py. It combines the date and time, but it doesn't return the result of the .combine() call, instead returning the (unmodified) dt.

I can fix this for my use case by changing it to

        return dt.combine(dt, DosDateTime.deserialize_time(tm))

but I think this makes it return a datetime.datetime instead of a DosDateTime; what do you think is the correct fix?

release new version

The last released version (v1.0.5) is from April 2022. Would you mind releasing a new version and pushing it to pypi?

We need the fix of b83f27c (which interestingly is not part of v1.0.5, because the release apparently happened on a different branch), so we pin the first commit after merging the v1.0.5 branch back into master (d5f123d).

However, it would be great if we could could pin to an official version instead of a random commit.

Removal of last entry in directory leaves remnants

If the last directory entry in a directory is removed via PyFatFS.remove(), then this entry is not properly overwritten in the filesystem, leaving a broken directory entry in place:

import fs
with fs.open_fs("/dev/sda1") as my_fs:
    my_fs.removetree("/EFI")

$ sudo mount /dev/sda1 /mnt; sudo ls -laR /mnt; sudo umount /mnt
/mnt:
ls: cannot access '/mnt/EFI': Input/output error
total 20
drwxr-xr-x 3 root root 4096 Jan  1  1970 .
drwxr-xr-x 1 root root  276 Dez 19 09:46 ..
d????????? ? ?    ?       ?            ? EFI
ls: cannot open directory '/mnt/EFI': Input/output error

Suspected cause is that with an empty byte_repr data size the erase function does not properly work and cannot align to the next biggest byte size of the directory cluster and thus does not erase the actual directory entry.

makedir() reorders files, moving system files, making MS-DOS 5 non-bootable

With pyfatfs 1.0.3, if I open a bootable MS-DOS 5 floppy as fs and then invoke:

fs.makedir("KILLER")
fs.close()

it is no longer possible to boot from the floppy. This is what Central Point PC Tools 9's Disk Editor shows after the above:

NAME     EXT     SIZE     DATE      TIME     CLU#    ARC R/O SYS HID DIR VOL
KILLER  .             0  4/09/22   08:43p    1398                    Dir
CONFIG  .SYS         63  9/15/13   04:25p    1397    Arc
FDISK   .EXE      57224  4/09/91   05:00a     996    Arc
EGA     .CPI      58873  4/09/91   05:00a     390    Arc
MEM     .EXE      39818  4/09/91   05:00a    1108    Arc
HIMEM   .SYS      11552  4/09/91   05:00a     505    Arc
IO      .SYS      33430  4/09/91   05:00a       2            Sys Hid
MIRROR  .COM      18169  4/09/91   05:00a    1186    Arc
KEYB    .COM      14986  4/09/91   05:00a     528    Arc
MSDOS   .SYS      37394  4/09/91   05:00a      68            Sys Hid
RAMDRIVE.SYS       5873  4/09/91   05:00a    1222    Arc
KEYBOARD.SYS      34697  4/09/91   05:00a     558    Arc
SHARE   .EXE      10912  4/09/91   05:00a    1234    Arc
MODE    .COM      23537  4/09/91   05:00a     626    Arc
SMARTDRV.SYS       8335  4/09/91   05:00a    1256    Arc
SETVER  .EXE      12007  4/09/91   05:00a     672    Arc
COMMAND .COM      47845  4/09/91   05:00a     142    Arc
SYS     .COM      13440  4/09/91   05:00a    1273    Arc
ANSI    .SYS       9029  4/09/91   05:00a     696    Arc
EGA     .SYS       4885  4/09/91   05:00a     236    Arc
UNDELETE.EXE      13924  4/09/91   05:00a    1300    Arc
DEBUG   .EXE      20634  4/09/91   05:00a     714    Arc
FORMAT  .COM      32911  4/09/91   05:00a     246    Arc
UNFORMAT.COM      18576  4/09/91   05:00a    1328    Arc
DOSKEY  .COM       5883  4/09/91   05:00a     755    Arc
NLSFUNC .EXE       7052  4/09/91   05:00a     311    Arc
XCOPY   .EXE      15804  4/09/91   05:00a    1365    Arc
EDLIN   .EXE      12642  4/09/91   05:00a     767    Arc
COUNTRY .SYS      17069  4/09/91   05:00a     325    Arc
AUTOEXEC.BAT         24  9/15/13   04:25p    1396    Arc
EMM386  .EXE      91742  4/09/91   05:00a     792    Arc
DISPLAY .SYS      15792  4/09/91   05:00a     359    Arc
FASTOPEN.EXE      12050  4/09/91   05:00a     972    Arc
STARTUP .             0  9/15/13   04:25p       0                        Vol
             Unused
               Unused

Note that the new directory appears first, and the files are listed in some unknown order.

This is what it looked like prior to adding the directory:

NAME     EXT     SIZE     DATE      TIME     CLU#    ARC R/O SYS HID DIR VOL
IO      .SYS      33430  4/09/91   05:00a       2            Sys Hid
MSDOS   .SYS      37394  4/09/91   05:00a      68            Sys Hid
STARTUP .             0  9/15/13   04:25p       0                        Vol
COMMAND .COM      47845  4/09/91   05:00a     142    Arc
EGA     .SYS       4885  4/09/91   05:00a     236    Arc
FORMAT  .COM      32911  4/09/91   05:00a     246    Arc
NLSFUNC .EXE       7052  4/09/91   05:00a     311    Arc
COUNTRY .SYS      17069  4/09/91   05:00a     325    Arc
DISPLAY .SYS      15792  4/09/91   05:00a     359    Arc
EGA     .CPI      58873  4/09/91   05:00a     390    Arc
HIMEM   .SYS      11552  4/09/91   05:00a     505    Arc
KEYB    .COM      14986  4/09/91   05:00a     528    Arc
KEYBOARD.SYS      34697  4/09/91   05:00a     558    Arc
MODE    .COM      23537  4/09/91   05:00a     626    Arc
SETVER  .EXE      12007  4/09/91   05:00a     672    Arc
ANSI    .SYS       9029  4/09/91   05:00a     696    Arc
DEBUG   .EXE      20634  4/09/91   05:00a     714    Arc
DOSKEY  .COM       5883  4/09/91   05:00a     755    Arc
EDLIN   .EXE      12642  4/09/91   05:00a     767    Arc
EMM386  .EXE      91742  4/09/91   05:00a     792    Arc
FASTOPEN.EXE      12050  4/09/91   05:00a     972    Arc
FDISK   .EXE      57224  4/09/91   05:00a     996    Arc
MEM     .EXE      39818  4/09/91   05:00a    1108    Arc
MIRROR  .COM      18169  4/09/91   05:00a    1186    Arc
RAMDRIVE.SYS       5873  4/09/91   05:00a    1222    Arc
SHARE   .EXE      10912  4/09/91   05:00a    1234    Arc
SMARTDRV.SYS       8335  4/09/91   05:00a    1256    Arc
SYS     .COM      13440  4/09/91   05:00a    1273    Arc
UNDELETE.EXE      13924  4/09/91   05:00a    1300    Arc
UNFORMAT.COM      18576  4/09/91   05:00a    1328    Arc
XCOPY   .EXE      15804  4/09/91   05:00a    1365    Arc
AUTOEXEC.BAT         24  9/15/13   04:25p    1396    Arc
CONFIG  .SYS         63  9/15/13   04:25p    1397    Arc
           Unused
             Unused

Note that the files are listed in some different, unknown order, but that IO.SYS and MSDOS.SYS are the first two entries. https://en.wikipedia.org/wiki/IO.SYS notes that:

The two first entries of the root directory must be allocated by IO.SYS and MSDOS.SYS, in that order.
I'm not sure if that restriction still holds in Windows 95 but it certainly seems to affect MS-DOS 5.

It would be safer to mimic what I think DOS does and just use the first available directory entry for the entry being added rather than rewriting the entire directory. I noticed that adding files one by one to a directory seems a bit slow compared to shelling out to mtools to copy all of the files in one batch, and I wonder if that is because the entire directory is being rewritten each time?

Open in-memory FS as RO?

4bbf04e adds support for in-memory, io.BytesIO-backed FS to be opened via the new PyFatBytesIOFS(..) constructor.

Would it be possible to allow opening such FS in read-only mode as well, similar to how the regular, file-backed PyFatFS allows that?

I'm using pyfatfs for forensic-like work, and being able to prevent any-and-all writes to images would be very handy to have.

[FAT32] FSInfo not updated on write

According to the specification, FSInfo may contain a valid count of free clusters on the volume. pyfatfs should ideally update the value on writes.

PyFat.mkfs() - volume label does not appear to be applied

Attempting to create a volume using the new mkfs() function:

temp_file = open(file="test_mkfs.fat12", mode="w+")
temp_pyfat = PyFat()
temp_pyfat.mkfs(filename=temp_file.name, fat_type=PyFat.FAT_TYPE_FAT12, label="TSTLABEL", size=(1024 * 256))

It appears that on mounting, the volume label is not set:

$ sudo losetup /dev/loop0 ./test_mkfs.fat12
$ lsblk -o name,label,size,uuid
NAME   LABEL  SIZE UUID
loop0         256K 5444-8E7B

I get the same results with FAT16/32

A similarly created file using mkfs yields a label:

$ mkfs.fat -C -F12 -n "TSTLABEL" test_mkfs.fat12 256
$ sudo losetup /dev/loop0 ./test_mkfs.fat12
$ lsblk -o name,label,size,uuid
NAME   LABEL     SIZE UUID
loop0  TSTLABEL  256K A562-5D4B

The DosDateTime.deserialize_time function produce ValueError: second must be in 0..59

I'm finding at the top of every minute, it is possible to get the value error in the deserialize_time function.

You can produce the error with:

#Function from DasDateTime
def deserialize_time(tm: int) -> time:
        """Convert a DOS time format to a Python object."""
        second = (tm & (1 << 5) - 1) * 2
        minute = (tm >> 5) & ((1 << 6) - 1)
        hour = (tm >> 11) & ((1 << 5) - 1)
        return time(hour, minute, second)

#Code to show the out of range second error
python_time = deserialize_time(0x001E)
print(python_time)

Time deserialization loses high bit of minutes field

With pyfatfs checked out from the bugfix/dentry_ordering branch as at 96ae6bc, I found that PyFatFS.setinfo(path, {"details": {"modified": value}}) seemed to work despite the "Not yet properly implemented." docstring, but only partly.

While investigating that, I found this, which I think is the root cause of the issue I was seeing:

>>> pyfatfs.DosDateTime.DosDateTime.deserialize_time(31 << 5)
datetime.time(0, 31)
>>> pyfatfs.DosDateTime.DosDateTime.deserialize_time(32 << 5)
datetime.time(0, 0)

The most significant bit of the minutes field is lost. https://wiki.osdev.org/FAT says that the minutes field is stored in 6 bits, so the bit mask used is wrong, and this trivial fix seems to help:

-        minute = (tm >> 5) & ((1 << 5) - 1)
+        minute = (tm >> 5) & ((1 << 6) - 1)