cosmos72 / fstransform Goto Github PK

The ext4 logical partition made of three hard disks, the data is about 20T, and the conversion to xfs is wrong. How to solve this problem?

10:04:13 dd: Unable to target output file”/tmp/fstransform.mount.120513/.fstransform.loop.120513 35994497712127 Truncate at byte: file is too large

On openSUSE Tumbleweed convertation hangs at this part of code:

srv:/ # lsof /home
srv:/ # fstransform /dev/sda4 ext4
fstransform: starting version 0.9.3, checking environment
fstransform: checking for which... '/usr/bin/which'
fstransform: checking for expr... '/usr/bin/expr'
fstransform: checking for id... '/usr/bin/id'
fstransform: parsing command line arguments
fstransform: checking for stat... '/usr/bin/stat'
fstransform: checking for mkfifo... '/usr/bin/mkfifo'
fstransform: checking for blockdev... '/sbin/blockdev'
fstransform: checking for losetup... '/sbin/losetup'
fstransform: checking for fsck... '/sbin/fsck'
fstransform: checking for mkfs... '/sbin/mkfs'
fstransform: checking for mount... '/usr/bin/mount'
fstransform: checking for umount... '/usr/bin/umount'
fstransform: checking for mkdir... '/usr/bin/mkdir'
fstransform: checking for rmdir... '/usr/bin/rmdir'
fstransform: checking for rm... '/usr/bin/rm'
fstransform: checking for dd... '/usr/bin/dd'
fstransform: checking for sync... '/usr/bin/sync'
fstransform: checking for fsmove... '/usr/sbin/fsmove'
fstransform: checking for fsremap... '/usr/sbin/fsremap'
fstransform: checking for fsck(source file-system)... '/sbin/fsck'
fstransform: checking for fsck(target file-system)... '/sbin/fsck'
fstransform: looking for optional commands
fstransform: checking for sleep... '/usr/bin/sleep'
fstransform: checking for date... '/usr/bin/date'
14:48:03 fstransform: environment check passed.
14:48:03 fstransform: saving output of this execution into /var/tmp/fstransform/fstransform.log.6868
14:48:03 fstransform: preparing to transform device '/dev/sda4' to file-system type 'ext4'
14:48:03 fstransform: device is mounted at '/home' with file-system type 'xfs'
14:48:03 fstransform: device raw size = 194209906688 bytes
14:48:03 fstransform: creating sparse loop file '/home/.fstransform.loop.6868' inside device '/dev/sda4'...
14:48:03 dd: 1+0 records in
14:48:03 dd: 1+0 records out
14:48:03 dd: 1 byte copied, 0.000159346 s, 6.3 kB/s
^Csrv:/ # stat -c %o /home/.fstransform.loop.6868
4096
srv:/ #

Basically, having read the warnings, I am attempting to proceed with converting a 6TB NTFS drive to EXT4.

Everything looks fine up until this point:

06:33:49 fsremap: ERROR: ff_posix_fibmap(): error, dev_block_count = 11720777728, file_block_count = 11720777728 overflow type (int): File too large
06:33:49 fsremap: ERROR: failed to list file blocks with ioctl(FS_IOC_FIEMAP): Operation not supported
         
06:33:49 ERROR! fstransform: command '/usr/bin/fsremap -q -n -- /dev/sdf2 /tmp/fstransform.mount.1718/.fstransform.loop.1718' failed (exit status 229)
                this is potentially a problem.
                you can either quit now by pressing ENTER or CTRL+C,

I assume I should be concerned, so I am wondering what the best course of action would be here.

System is a custom PC with Manjaro, Ubuntu and Windows 10. I'm trying to fix my corrupted ubuntu home partition of 1.8TB with 1.1TB of data.

I ran

$ sudo fsck.ext4 -vy /dev/sdc2

This fixed most of the issues, all of my data was still there. I was able to mount it in my manjaro and read it. But the file system was not recognizing it as a ext4 partition so I ran,

$ sudo fstransform /dev/sdc2 ext4

fstransform hangs on job.3. I see a huge fstransform.loop.4143 file which I assume is where my data is at.

This screenshot below is from my Manjaro where I ran fstransform from.

I tried resuming multiple times and each time my system hangs with no keyboard or mouse response exactly at this point.

Hi!

I'd love to package this project for GNU Guix, but hit a snag.

All of fstransform is free software but two files, e4attr.cc and e4attr.hh, are released under a GPL2-only licence that would be incompatible with the majority of other code.

I don't think this currently causes legal problems, because code never happens to be combined in an incompatible way (…yet).

However, I'm not vouching for that either, and anyway this looks more like an oversight than a deliberate decision.

@cosmos72, would you be open to replacing the licence of GPL2-only files with one used elsewhere (such as this GPL2-or-later one)?

17:18:46 ERROR! fstransform: command '/sbin/fsmove -- /tmp/fstransform.mount.12641 /tmp/fstransform.loop.12641 --exclude /tmp/fstransform.mount.12641/.fstransform.loop.12641' failed (exit status 240)
this is potentially a problem.
you can either quit now by pressing ENTER or CTRL+C,

            or, if you know what went wrong, you can fix it yourself,
            then manually run the command '/sbin/fsmove -- /tmp/fstransform.mount.12641 /tmp/fstransform.loop.12641 --exclude /tmp/fstransform.mount.12641/.fstransform.loop.12641'
            (or something equivalent)

I am trying to transform /dev/mapper/centos-home from xfs to ext4. Here is where fstransform appears to fail .. relevant part log file below:

2016-01-04 00:14:27 fsmove: progress: 90.0% done, 933.1 megabytes still to move, estimated 1 minute and 40 seconds left
2016-01-04 00:14:46 fsmove: skipped `/home/.fstransform.loop.21612', matches exclude list
2016-01-04 00:14:46 fsmove: job completed.
2016-01-04 00:14:46 fstransform: unmounting and running '/usr/sbin/fsck' (disk check) on loop file '/home/.fstransform.loop.21612'
2016-01-04 00:14:47 fsck: fsck from util-linux 2.23.2
2016-01-04 00:14:47 fsck: /dev/loop0 is mounted.
2016-01-04 00:14:47 fsck: e2fsck: Cannot continue, aborting.
2016-01-04 00:14:47 fsck:
2016-01-04 00:14:47 fsck:

2016-01-04 00:14:47 ERROR! fstransform: command '/usr/sbin/fsck -p -f /dev/loop0' failed (exit status 8)
this is potentially a problem.
you can either quit now by pressing ENTER or CTRL+C,

                      or, if you know what went wrong, you can fix it yourself,
                      then manually run the command '/usr/sbin/fsck -p -f /dev/loop0'
                      (or something equivalent)
                      and finally resume this script by typing CONTINUE and pressing ENTER: 

2016-01-04 00:17:35 fstransform: exiting.

summary: I'd appreciate your assistance with

1. identifying the cause of a problem encountered after using fstransform and then manually editing my fstab as described in this issue

and either (or both)
<ignore this text: it's only here to force GFM to not renumber my list/>
2. (preferred) fixing whatever in your code is causing this
3. (acceptable) telling the user what s/he will need to do to fix the problem (if the fix cannot or should not be automated)

details:

As detailed here, I recently encountered (on a box up-to-date with Debian Jessie) boot-time problems after

1. installing fstransform (from Debian package name=fstransform version=0.9.3-1)
2. running fstransform to update the filesystem type of the box's /boot
3. rebooting

I got the following message immediately after the GRUB2 screen (in the white-text-on-black-screen boot phase):

error: no such device: 91998f3d-7138-42ce-acfd-4e9946071dd9
Loading Linux <kernel version/> ...
Loading initial ramdisk ...
Press any key to continue...

followed by

fsck.ext2: Unable to resolve 'UUID=91998f3d-7138-42ce-acfd-4e9946071dd9'
<big red FAIL in here somewhere/>
fsck exited with status code 8

I was able to fix the second message by [editing my /etc/fstab' to change the UUID](https://github.com/cosmos72/fstransform/issues/5)--at least, that second, fsck`-related message has not reappeared after a few reboots. However the first message continues to display at every reboot.

Unfortunately I have no clue what is causing this message: if you know, please enlighten me. I would be even more appreciative if you could

1. (preferred) fix whatever in your code is causing that message
2. (acceptable) tell the user what s/he will need to do to fix the problem (if the fix cannot or should not be automated)

TIA, Tom Roche [email protected]

F2FS file system is designed for NAND-based flash memory equipped with FTL. This technology is used in all flash memory with a SCSI/SATA/PCIe/NVMe interface. F2FS aims to increase the lifespan and performance of such storage devices which is why it's growing in popularity. Proper conversion support for this filesystem would be really useful

I've tried to convert a 64G usb disk from F2FS to ext4 (yep, I know not supposed to work)
It fails with:

ERROR  loop-file has an odd-sized last block (2828 bytes long) that exceeds device rounded length.
ERROR  Exiting, please shrink loop-file to 64155025408 bytes or less before running fsremap again.

I'd like to shrink the loop file, but I was unable to locate it. not in /tmp, nor in /var/tmp
any pointers ?

fstransform version 0.9.3 - I did a conversion of reiserfs to ext4 on a Debian 10 system. After the conversion it tried to do fsck:

12:43:24 fstransform: running again '/sbin/fsck' (disk check) on device '/dev/sda3'
12:43:24 fsck: fsck from util-linux 2.33.1
12:43:24 fsck: Failed to open the device '/dev/sda3': Unknown code er3k 127
12:43:24 fsck:
12:43:24 fsck:

12:43:24 ERROR! fstransform: command '/sbin/fsck -p -f /dev/sda3' failed (exit status 8)
                this is potentially a problem.
                you can either quit now by pressing ENTER or CTRL+C,

                or, if you know what went wrong, you can fix it yourself,
                then manually run the command '/sbin/fsck -p -f /dev/sda3'
                (or something equivalent)
                and finally resume this script by typing CONTINUE and pressing ENTER:

The problem was that the filesystem was still contained in /etc/fstab and listed as reiserfs there, so fsck used the reiser specific filesystem checking tool for checking.

It would be better if the conversion tool also adjusted the fstab and changed reiserfs to ext3 (because the previous value is now obviously wrong) and didn't fail here.

Or use the filesystem specific way to check for the filesystem like e2fsck in my case.

How can I resume if fstransform had an error and exited during fsmove?

2017-12-28 22:12:16 ERROR! fstransform: command '/usr/sbin/fsmove -- /tmp/fstransform.mount.2405 /tmp/fstransform.loop.2405 --exclude /tmp/fstransform.mount.2405/.fstransform.loop.2405' failed (exit status 251)
this is potentially a problem.
 you can either quit now by pressing ENTER or CTRL+C,
                          
or, if you know what went wrong, you can fix it yourself,
 then manually run the command '/usr/sbin/fsmove -- /tmp/fstransform.mount.2405 /tmp/fstransform.loop.2405 --exclude /tmp/fstransform.mount.2405/.fstransform.loop.2405'
(or something equivalent)
and finally resume this script by typing CONTINUE and pressing ENTER: 
2017-12-28 22:31:15 fstransform: exiting.

Steps to reproduce

1. # apt-get install -y fstransform reiserfsprogs

Repeat the following steps:

2. Format a partition into ext2 (4 GB is enough).
3. Mount the partition.
4. Create a couple random files on the partition:

# for i in `seq 1 2`; do head -c 1G /dev/urandom > "random_$i.txt"; done

5. # fstransform /path/to/partition reiserfs
6. # fstransform /path/to/partition ext2

The issue doesn't reproduce every time (I'd say approximately 1/30 tries), so the steps need to be repeated.

Actual result

fstransform exits with an error, because a program it calls exits with "Segmentation fault" or "Killed":

fstransform: starting version 0.9.4, checking environment
fstransform: checking for which...         '/usr/bin/which'
[...]
12:02:32 fstransform: environment check passed.
12:02:32 fstransform: saving output of this execution into /var/tmp/fstransform/fstransform.log.15019
12:02:32 fstransform: preparing to transform device '/dev/sdb1' to file-system type 'ext2'
12:02:32 fstransform: device is mounted at '/test-mount' with file-system type 'reiserfs'
12:02:32 fstransform: device raw size = 4292870144 bytes
12:02:32 fstransform: creating sparse loop file '/test-mount/.fstransform.loop.15019' inside device '/dev/sdb1'...
Killed

12:02:32 ERROR! fstransform: failed to create or truncate '/test-mount/.fstransform.loop.15019' to zero bytes
                maybe device '/dev/sdb1' is full or mounted read-only?

The program can exit at different stages, e.g. at moving '/dev/sdb1' contents into the loop file:

fstransform: starting version 0.9.4, checking environment
fstransform: checking for which...         '/usr/bin/which'
fstransform: checking for expr...         '/usr/bin/expr'
fstransform: checking for id...         '/usr/bin/id'
fstransform: parsing command line arguments
fstransform: checking for stat...         '/usr/bin/stat'
fstransform: checking for mkfifo...         '/bin/mkfifo'
fstransform: checking for blockdev...         '/sbin/blockdev'
fstransform: checking for losetup...         '/sbin/losetup'
fstransform: checking for fsck...         '/sbin/fsck'
fstransform: checking for mkfs...         '/sbin/mkfs'
fstransform: checking for mount...         '/bin/mount'
fstransform: checking for umount...         '/bin/umount'
fstransform: checking for mkdir...         '/bin/mkdir'
fstransform: checking for rmdir...         '/bin/rmdir'
fstransform: checking for rm...         '/bin/rm'
fstransform: checking for dd...         '/bin/dd'
fstransform: checking for sync...         '/bin/sync'
fstransform: checking for fsmove...         '/usr/sbin/fsmove'
fstransform: checking for fsmount_kernel...         '/usr/sbin/fsmount_kernel'
fstransform: checking for fsremap...         '/usr/sbin/fsremap'
fstransform: checking for fsck(source file-system)...        '/sbin/fsck'
fstransform: checking for fsck(target file-system)...        '/sbin/fsck'
fstransform: looking for optional commands
fstransform: checking for sleep...         '/bin/sleep'
fstransform: checking for date...         '/bin/date'
15:38:02 fstransform: environment check passed.
15:38:02 fstransform: saving output of this execution into /var/tmp/fstransform/fstransform.log.6911
15:38:02 fstransform: preparing to transform device '/dev/sdb1' to file-system type 'reiserfs'
15:38:02 fstransform: device is mounted at '/test-mount' with file-system type 'ext2'
15:38:02 fstransform: device raw size = 4292870144 bytes
15:38:02 fstransform: creating sparse loop file '/test-mount/.fstransform.loop.6911' inside device '/dev/sdb1'...
15:38:02 dd: 1+0 records sent
15:38:02 dd: 1+0 records received
15:38:02 dd: 1 byte copied, 0,000177744 s, 5,6 kB/s
15:38:02 fstransform: device file-system block size = 4096 bytes
15:38:02 fstransform: device usable size = 4292870144 bytes
15:38:02 dd: 1+0 records sent
15:38:02 dd: 1+0 records received
15:38:02 dd: 1 byte copied, 0,000110167 s, 9,1 kB/s
15:38:02 fstransform: unmounting device '/dev/sdb1' and remounting it read-only using kernel driver
15:38:02 fstransform: launching '/usr/sbin/fsremap' in simulated mode for pre-validation
15:38:02 fsremap: setting log level to NOTICE
15:38:02 fsremap: starting job 2, persistence data and logs are in '/var/tmp/fstransform/fsremap.job.2'
15:38:02 fsremap: analysis completed: 8.00 kilobytes must be relocated
15:38:03 fsremap: allocated 8.00 kilobytes RAM as memory buffer
15:38:03 fsremap: primary-storage is 1.00 megabytes, initialized and mmapped() to contiguous RAM
15:38:03 fsremap: (simulated) starting in-place remapping. this may take a LONG time ...
15:38:03 fsremap: (simulated) progress: 50.0% done,   8.0 kilobytes still to remap
15:38:03 fsremap: (simulated) clearing 4.00 gigabytes free-space from device ...
15:38:03 fsremap: (simulated) job completed.
15:38:03 fstransform: unmounting device '/dev/sdb1' and remounting it read-write
15:38:03 fstransform: connected loop device '/dev/loop0' to file '/test-mount/.fstransform.loop.6911'
15:38:03 fstransform: formatting loop device '/dev/loop0' with file-system type 'reiserfs'...
15:38:03 mkfs: mkfs.reiserfs 3.6.27
15:38:03 mkfs:
15:38:03 fstransform: mounting loop device '/dev/loop0' on '/tmp/fstransform.loop.6911' ...
15:38:03 fstransform: loop device '/dev/loop0' mounted successfully.
15:38:03 fstransform: preliminary steps completed, now comes the delicate part:
15:38:03 fstransform: fstransform will move '/dev/sdb1' contents into the loop file.

15:38:03 fstransform: WARNING: THIS IS IMPORTANT! if either the original device '/dev/sdb1'
                      or the loop device '/dev/loop0' become FULL,

                       YOU  WILL  LOSE  YOUR  DATA !

                      fstransform checks for enough available space,
                      in any case it is recommended to open another terminal, type
                        watch df /dev/sdb1 /dev/loop0
                      and check that both the original device '/dev/sdb1'
                      and the loop device '/dev/loop0' are NOT becoming full.
                      if one of them is becoming full (or both),
                      you MUST stop fstransform with CTRL+C or equivalent.

15:38:03 fstransform: moving '/dev/sdb1' contents into the loop file.
15:38:03 fstransform: this may take a long time, please be patient...
Killed

15:38:03 ERROR! fstransform: command '/usr/sbin/fsmove -- /test-mount /tmp/fstransform.loop.6911 --exclude /test-mount/.fstransform.loop.6911' failed (exit status 137)
                this is potentially a problem.
                you can either quit now by pressing ENTER or CTRL+C,

                or, if you know what went wrong, you can fix it yourself,
                then manually run the command '/usr/sbin/fsmove -- /test-mount /tmp/fstransform.loop.6911 --exclude /test-mount/.fstransform.loop.6911'
                (or something equivalent)

This results in the following in journalctl:

kernel: BUG: unable to handle page fault for address: ffffffffd60b47cb
kernel: #PF: supervisor read access in kernel mode
kernel: #PF: error_code(0x0000) - not-present page

Full: journalctl-1.log

Or:

kernel: kernel BUG at fs/reiserfs/journal.c:3039!
kernel: invalid opcode: 0000 [#1] PREEMPT SMP PTI

Full: journalctl-2.log

Expected result

No errors, including kernel errors in journalctl.

Extra information

It's possible this can be reproduced with filesystems other than ext2.

Reproducibility

Tested on ALT Linux Sisyphus and p10; tested on both VMs and real hardware.

It seems this is reproducible just on 6.1 kernels:

• 5.10.194 no
• 6.1.52 reproducible
• 6.1.53 reproducible
• 6.4.16 no

Package versions:

fstransform-0.9.4-alt2_11.x86_64

libreiserfsprogs-3.6.27-alt1.x86_64
reiserfsprogs-3.6.27-alt1.x86_64

Reproducible regardless of whether the following packages are installed:

libprogsreiserfs-0.3.0.5-alt5.x86_64
libreiser4-1.2.1-alt3.x86_64

I am running fstransform to convert a XFS LVM group to EXT4 (in order to shrink it later)
I was running fstransform with nohup, cause I run it on a headless server. When the error occurred the process got abwartet and I could not enter the CONTINUE, as it was not interactive.

During the run the following error occurred:

2022-09-05 18:42:25 fsmove: progress: 99.2% done,  18.4 gigabytes still to move, estimated 15 minutes left
2022-09-05 18:45:04 fsmove: using backward copy and truncate for file `/tmp/fstransform.loop.11454/.fstransform.loop.9759': less than 8.19 terabytes free space left
2022-09-06 00:05:41 fsmove: using backward copy and truncate for file `/tmp/fstransform.loop.11454/.fstransform.loop.11249': less than 8.19 terabytes free space left
2022-09-06 05:31:44 fsmove: skipped `/srv/single/.fstransform.loop.11454', matches exclude list
2022-09-06 05:31:44 fsmove: job completed.
2022-09-06 05:31:44 fstransform: unmounting and running '/usr/sbin/fsck' (disk check) on loop file '/srv/single/.fstransform.loop.11454'
2022-09-06 05:31:45 fsck: fsck from util-linux 2.37.2
2022-09-06 05:31:53 fsck: /dev/loop19: Inode 78578626 extent tree (at level 1) could be shorter.  IGNORED.
2022-09-06 05:32:02 fsck: /dev/loop19: Inode 197985782 extent tree (at level 1) could be shorter.  IGNORED.
2022-09-06 05:32:02 fsck: /dev/loop19: Inode 197987857 extent tree (at level 1) could be shorter.  IGNORED.
2022-09-06 05:32:02 fsck: /dev/loop19: Inode 198050450 extent tree (at level 1) could be shorter.  IGNORED.
2022-09-06 05:32:02 fsck: /dev/loop19: Inode 198050740 extent tree (at level 1) could be shorter.  IGNORED.
2022-09-06 05:32:26 fsck: /dev/loop19: 196609/274714624 files (11.9% non-contiguous), 592666591/2197694464 blocks
2022-09-06 05:32:26 fstransform: disconnected loop device '/dev/loop19' from file '/srv/single/.fstransform.loop.11454'
2022-09-06 05:32:26 fstransform: unmounting device '/dev/mapper/groupCD-unmirrored' before disk check
2022-09-06 05:32:26 umount: /srv/single: target is busy.
2022-09-06 05:32:26 ERROR! fstransform: command '/usr/bin/umount /dev/mapper/groupCD-unmirrored' failed (exit status 32)
                          this is potentially a problem.
                          you can either quit now by pressing ENTER or CTRL+C,

                          or, if you know what went wrong, you can fix it yourself,
                          then manually run the command '/usr/bin/umount /dev/mapper/groupCD-unmirrored'
                          (or something equivalent)
                          and finally resume this script by typing CONTINUE and pressing ENTER:
2022-09-06 05:32:26 fstransform: exiting.

I can manually unmount the partition and run xfs_repair without error. The file /srv/single/.fstransform.loop.11454 is there and also fsck.ext4 it without error.
Can I continue/restart the process or do I need to restart and run for another 30 hours?

hello after compiled i get error
dash not fount after that i change dash to bash

https://github.com/cosmos72/fstransform/blob/master/fstransform/fstransform-sh

From what I have read on it, the major issue supporting NTFS has been the NTFS-3G FUSE driver. That said, at least in my tests, fstransform does work both ways (albeit I needed to manually specify source file system). Now that Ubuntu 22.04 will use kernel 5.15 (which has a new in-kernel driver NTFS3), I think its support will be super useful

Trying to convert +3TB NTFS to EXT4. Initially the problem was fsconvert ERROR: error opening loop-file '(null)': Bad address trying to contnue a job that was interrupted by a system restart.

Following the guide I got stuck at point 12. The error shown in console is:

sudo /usr/sbin/fsremap -n -q --resume-job=1 --cmd-umount=/bin/umount -- /mnt/sdf2/.fstransform.loop.10941 fsremap: resuming job 1, persistence data and logs are in '/var/tmp/fstransform/fsremap.job.1' ERROR: failed device fstat('/mnt/sdf2/.fstransform.loop.10941'), **re-run with option '-f' if you want to continue anyway (AT YOUR OWN RISK): Block device required**

adding -f:

sudo /usr/sbin/fsremap -n -q -f --resume-job=1 --cmd-umount=/bin/umount -- /mnt/sdf2/.fstransform.loop.10941 fsremap: resuming job 1, persistence data and logs are in '/var/tmp/fstransform/fsremap.job.1' WARN: WARNING: failed device fstat('/mnt/sdf2/.fstransform.loop.10941'), continuing AT YOUR OWN RISK due to '-f': Block device required **ERROR: error in device ioctl('/mnt/sdf2/.fstransform.loop.10941', BLKGETSIZE64): Invalid argument**

Following an unexpected system restart I had a restart and tried to continue from the point logs noted, but as the loop file wasn't given in the options it didn't run:

fsremap: you CAN RESUME it with: /usr/sbin/fsremap -n -q --resume-job=1 -- /dev/sdf2
fsconvert ERROR: error opening loop-file '(null)': Bad address

Any help please?

I'm trying to compile this software under SLES, but this error is displayed:
.
.
.
checking for waitpid... yes
checking whether C++ library implements std::unordered_map<K,V>... no
checking whether C++ library implements std::tr1::unordered_map<K,V>... yes
checking for error_message in -lcom_err... no
checking for ext2fs_extent_replace in -lext2fs... no
./configure: line 413: test: too many arguments
configure: error:
./configure: line 302: return: missing: numeric argument required
./configure: line 312: exit: missing: numeric argument required

Do you know how can I solve it?

So I left the fsmove process run for ~25 hours, it did its thing, but it complained that I removed an old file from that drive while the move was in progress. It was an old .fstransform file that I didn't let go very long (hence didn't successfully copy a single file - I checked by mounting it).

Upon completion of the fsmove, it errored with:

2019-05-01 03:02:17 fsmove: progress: 98.4% done,  50.6 gigabytes still to move, estimated 25 minutes left
2019-05-01 03:08:02 fsmove: ERROR: failed to lstat() `/tmp/fstransform.mount.1097/.fstransform.loop.13621': No such file or directory
                   
2019-05-01 03:08:02 ERROR! fstransform: command '/usr/sbin/fsmove -- /tmp/fstransform.mount.1097 /tmp/fstransform.loop.1097 --exclude /tmp/fstransform.mount.1097/.fstransform.loop.1097' failed (exit status 254)
                          this is potentially a problem.
                          you can either quit now by pressing ENTER or CTRL+C,
                          
                          or, if you know what went wrong, you can fix it yourself,
                          then manually run the command '/usr/sbin/fsmove -- /tmp/fstransform.mount.1097 /tmp/fstransform.loop.1097 --exclude /tmp/fstransform.mount.1097/.fstransform.loop.1097'
                          (or something equivalent)
                          and finally resume this script by typing CONTINUE and pressing ENTER: 
2019-05-01 03:08:02 fstransform: exiting.

As I'd hit ENTER at some point in that terminal over the last 25 hours, it immediately quit.

I've mounted both the mount and loop mountpoints again, and I can confirm that all files except the excluded loop file have been moved to the loop filesystem.

So how to continue?

(and on another note, can we drop just a plain ENTER as being an ABORT?! especially if it may have been entered many, many hours prior and sat in a buffer since....

I was testing the NTFS to XFS/F2FS conversion, but couldn't get it to even start, it gives me a No Such Device error, not sure why since I made sure to follow the Requirements.

NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
sda           8:0    0 931.5G  0 disk
└─sda1        8:1    0 931.4G  0 part

fstransform: starting version 0.9.4, checking environment
fstransform: checking for which...         '/usr/bin/which'
fstransform: checking for expr...         '/usr/bin/expr'
fstransform: checking for id...         '/usr/bin/id'
fstransform: parsing command line arguments
fstransform: device initial (current) file-system type 'ntfs' specified on command line
fstransform: forcing trasformation of untested file systems (DANGEROUS). '--force-untested-file-systems' bytes specified on command line
fstransform: checking for stat...         '/usr/bin/stat'
fstransform: checking for mkfifo...         '/usr/bin/mkfifo'
fstransform: checking for blockdev...         '/usr/bin/blockdev'
fstransform: checking for losetup...         '/usr/bin/losetup'
fstransform: checking for fsck...         '/usr/bin/fsck'
fstransform: checking for mkfs...         '/usr/bin/mkfs'
fstransform: checking for mount...         '/usr/bin/mount'
fstransform: checking for umount...         '/usr/bin/umount'
fstransform: checking for mkdir...         '/usr/bin/mkdir'
fstransform: checking for rmdir...         '/usr/bin/rmdir'
fstransform: checking for rm...         '/usr/bin/rm'
fstransform: checking for dd...         '/usr/bin/dd'
fstransform: checking for sync...         '/usr/bin/sync'
fstransform: checking for fsmove...         '/usr/bin/fsmove'
fstransform: checking for fsmount_kernel...         '/usr/bin/fsmount_kernel'
fstransform: checking for fsremap...         '/usr/bin/fsremap'
fstransform: checking for fsck(source file-system)...        '/usr/bin/fsck'
fstransform: checking for fsck(target file-system)...        '/usr/bin/fsck'
fstransform: looking for optional commands
fstransform: checking for sleep...         '/usr/bin/sleep'
fstransform: checking for date...         '/usr/bin/date'
03:26:05 fstransform: applying special options for file-system type 'ntfs'
03:26:05 fstransform: checking for fsck(source file-system)...        '/usr/bin/ntfsresize' ('ntfsresize' was specified)
03:26:05 fstransform: environment check passed.
03:26:05 fstransform: saving output of this execution into /var/tmp/fstransform/fstransform.log.21879
03:26:05 fstransform: preparing to transform device '/dev/sda1' to file-system type 'f2fs'
03:26:05 fstransform: device '/dev/sda1' not found in the output of command /usr/bin/mount, assuming it is not mounted
03:26:06 fstransform: device is now mounted at '/tmp/fstransform.mount.21879' with file-system type 'fuseblk'
03:26:06 fstransform: file-system type 'fuseblk' is a placeholder name for FUSE... ignoring it (user specified type 'ntfs')
         
03:26:06 fstransform: WARNING: this program is UNTESTED on device current file system 'ntfs' !
         
03:26:06 fstransform: WARNING: this program is UNTESTED on target file system 'f2fs' !
                      this program is tested ONLY on file systems: ext2 ext3 ext4 reiserfs jfs xfs
                      continuing anyway due to option '--force-untested-file-systems' (DANGEROUS)
03:26:06 fstransform: device raw size = 1000068874240 bytes
03:26:06 fstransform: creating sparse loop file '/tmp/fstransform.mount.21879/.fstransform.loop.21879' inside device '/dev/sda1'...
03:26:06 dd: 1+0 records in
03:26:06 dd: 1+0 records out
03:26:06 dd: 1 byte copied, 6.3639e-05 s, 15.7 kB/s
03:26:06 fstransform: device file-system block size = 4096 bytes
03:26:06 fstransform: device usable size = 1000068874240 bytes
03:26:06 dd: 1+0 records in
03:26:06 dd: 1+0 records out
03:26:06 dd: 1 byte copied, 0.000386415 s, 2.6 kB/s
03:26:06 fstransform: unmounting device '/dev/sda1' and remounting it read-only using kernel driver
03:26:06 fsmount_kernel: ERROR: syscall mount(source='/dev/sda1', target='/tmp/fstransform.mount.21879', fstype='ntfs', mountflags=0x1, options='') failed: No such device
         
03:26:06 ERROR! fstransform: command '/usr/bin/fsmount_kernel /dev/sda1 /tmp/fstransform.mount.21879 -o ro -t ntfs' failed (exit status 237)
                this is potentially a problem.
                you can either quit now by pressing ENTER or CTRL+C,
                
                or, if you know what went wrong, you can fix it yourself,
                then manually run the command '/usr/bin/fsmount_kernel /dev/sda1 /tmp/fstransform.mount.21879 -o ro -t ntfs'
                (or something equivalent)
                and finally resume this script by typing CONTINUE and pressing ENTER: 03:26:07 fstransform: exiting.

Hi there

Sorry for the support request, but hopefully you can help me out.

I am in the process of converting a drive from jfs to ext4 and sadly after a few days of it chugging along nicely fsremap ended up encountering bad blocks and not getting over it. I retried a few times but it always aborts around the same sector.

I'm aware that the drive might be close to dying but at this point I am more curious if I can get anything out of this operation than really caring about the data (-:

Is there some option or way to get fsremap to skip the operations regarding the bad blocks or range of bad blocks? I don't mind if it would end up with a few hundred MB of unreadable data.

I thought of trying to forge a line in fsremap.persist to make fsremap think its further along but I'm not sure if that's a good idea. Also ideally I'd skip over only a few dozen MB and not GB.

Here's the error as reported by fsremap
2018-07-01 15:26:17 ERROR [io/io_posix.flush_copy_bytes(1089)] I/O error while copying from device to RAM, read({fd = 6, offset = 597347381248}, address + 44683264, length = 552960): Input/output error

And here's how the errors are reported in dmesg:

sd 32:0:1:0: [sdb] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
sd 32:0:1:0: [sdb] Sense Key : Medium Error [current]
sd 32:0:1:0: [sdb] Add. Sense: Unrecovered read error
sd 32:0:1:0: [sdb] CDB: 
Read(10): 28 00 45 8a 57 27 00 00 08 00
blk_update_request: critical medium error, dev sdb, sector 1166694183
Buffer I/O error on dev sdb1, logical block 145836765, async page read

Here are the full application logs:
fstransform.log
fsremap.log
Would the other files (loop_extents.txt, free_space_extents.txt, fsremap.persist) be of any use here?

Thanks again for the tool and your time!

First of all, thank you! Very handy tool. I am attempting to convert from XFS to ext4 (to allow shrinking) and noticed that xattrs are not preserved. What are your thoughts on this matter?

Could this tool be used to convert a filesystem from 512 byte sectors to 4k?

For example:
If one replaces or adds a 4K formatted disk to lvm or md raid it will transform the lvm volume/md raid to a 4K devices which some filesystems (xfs at least) struggle with.
XFS for example will give the error xfs device supports 4096 byte sectors (not 512) and refuse to mount again until you remove the 4K device (which tranforms the device back to a 512 byte one).

Ubuntu 17.10 on a remote 10 y/o asus laptop
fstransform 0.9.3
/dev/SDB is and 1TB external USB 3.0 spinning HDD

I launched these two commands in two tmux terminals about a week ago:
fstransform --old-file-system=ntfs --force-untested-file-systems /dev/sdb1 ext4
(the command proposed to monitor the loop file size - it never exceeded 60%)
the conversion went to 100% and looked like it would succeed yesterday but then it threw these errors:
02:55:14 fstransform: disconnected loop device '/dev/loop1' from file '/tmp/fstransform.mount.7857/.fstransform.loop.8107'
02:55:14 fstransform: unmounting device '/dev/sdb1' before disk check
02:55:14 umount: /tmp/fstransform.mount.7857: target is busy.

02:55:14 ERROR! fstransform: command '/bin/umount /dev/sdb1' failed (exit status 32)
this is potentially a problem.
you can either quit now by pressing ENTER or CTRL+C,

            or, if you know what went wrong, you can fix it yourself,
            then manually run the command '/bin/umount /dev/sdb1'
            (or something equivalent)
            and finally resume this script by typing CONTINUE and pressing ENTER: CONTINUE

12:47:23 fstransform: running '/sbin/fsck' (disk check) on device '/dev/sdb1'
12:47:23 fsck: fsck from util-linux 2.31.1
12:47:23 fstransform: mounting again device '/dev/sdb1' read-only
12:47:24 mount: ntfs-3g-mount: mount failed: Device or resource busy

12:47:24 ERROR! fstransform: command '/bin/mount /dev/sdb1 /tmp/fstransform.mount.7857 -o ro' failed (exit status 21)
this is potentially a problem.
you can either quit now by pressing ENTER or CTRL+C,

            or, if you know what went wrong, you can fix it yourself,
            then manually run the command '/bin/mount /dev/sdb1 /tmp/fstransform.mount.7857 -o ro'
            (or something equivalent)
            and finally resume this script by typing CONTINUE and pressing ENTER: CONTINUE

16:22:09 fstransform: launching '/usr/sbin/fsremap' in simulated mode
16:22:09 fsremap: starting job 1, persistence data and logs are in '/var/tmp/fstransform/fsremap.job.1'
16:22:09 fsremap: if this job is interrupted, for example by a power failure,
16:22:09 fsremap: you CAN RESUME it with: /usr/sbin/fsremap -n -q --resume-job=1 -- /dev/sdb1
16:22:09 fsremap: ERROR: error opening loop-file '/tmp/fstransform.mount.7857/.fstransform.loop.8107': No such file or directory

16:22:09 ERROR! fstransform: command '/usr/sbin/fsremap -n -q -- /dev/sdb1 /tmp/fstransform.mount.7857/.fstransform.loop.8107' failed (exit status 254)
this is potentially a problem.
you can either quit now by pressing ENTER or CTRL+C,

            or, if you know what went wrong, you can fix it yourself,
            then manually run the command '/usr/sbin/fsremap -n -q -- /dev/sdb1 /tmp/fstransform.mount.7857/.fstransform.loop.8107'
            (or something equivalent)
            and finally resume this script by typing CONTINUE and pressing ENTER: CONTINUE

16:39:36 fstransform: launching '/usr/sbin/fsremap' in REAL mode to perform in-place remapping.
16:39:36 fsremap: starting job 2, persistence data and logs are in '/var/tmp/fstransform/fsremap.job.2'
16:39:36 fsremap: if this job is interrupted, for example by a power failure,
16:39:36 fsremap: you CAN RESUME it with: /usr/sbin/fsremap -q --resume-job=2 -- /dev/sdb1
16:39:36 fsremap: ERROR: error opening loop-file '/tmp/fstransform.mount.7857/.fstransform.loop.8107': Transport endpoint is not connected

16:39:36 ERROR! fstransform: command '/usr/sbin/fsremap -q --cmd-umount=/bin/umount -- /dev/sdb1 /tmp/fstransform.mount.7857/.fstransform.loop.8107' failed (exit status 149)
this is potentially a problem.
you can either quit now by pressing ENTER or CTRL+C,

            or, if you know what went wrong, you can fix it yourself,
            then manually run the command '/usr/sbin/fsremap -q --cmd-umount=/bin/umount -- /dev/sdb1 /tmp/fstransform.mount.7857/.fstransform.loop.8107'
            (or something equivalent)
            and finally resume this script by typing CONTINUE and pressing ENTER:"

I'll admit I'm lost but these are the steps I attempted to fix it:
-killed anything I thought could be keeping /dev/sdb1 busy
-after running out of ideas, forced the umount with -l (lazy)
-problem still wasn't fixed with that
-I ended up force remounting the drive with -o force (probably a stupid idea)
-commented out entries in fstab that mount that drive
-rebooted (another stupid idea probably)
-now any attempts to sudo fsremap -q --resume-job=1|2|3 /dev/sdb1 fail with
ERROR: error opening loop-file '(null)': Bad address
I'm asking for help here now to try to avoid making things worse (maybe too late?)

What happened and can you halp?!

I am looking to convert a volume that is ~32TB in allocated size. Currently, the volume is 75% utilized. The attempted conversion is from ext4 to xfs on a Rocky Linux 8 machine. Any guidance on how to proceed with be helpful.

$ sudo fstransform /dev/sdd xfs
fstransform: starting version 0.9.3, checking environment
fstransform: checking for which... 	'/bin/which'
fstransform: checking for expr... 	'/bin/expr'
fstransform: checking for id... 	'/bin/id'
fstransform: parsing command line arguments
fstransform: checking for stat... 	'/bin/stat'
fstransform: checking for mkfifo... 	'/bin/mkfifo'
fstransform: checking for blockdev... 	'/sbin/blockdev'
fstransform: checking for losetup... 	'/sbin/losetup'
fstransform: checking for fsck... 	'/sbin/fsck'
fstransform: checking for mkfs... 	'/sbin/mkfs'
fstransform: checking for mount... 	'/bin/mount'
fstransform: checking for umount... 	'/bin/umount'
fstransform: checking for mkdir... 	'/bin/mkdir'
fstransform: checking for rmdir... 	'/bin/rmdir'
fstransform: checking for rm... 	'/bin/rm'
fstransform: checking for dd... 	'/bin/dd'
fstransform: checking for sync... 	'/bin/sync'
fstransform: checking for fsmove... 	'/sbin/fsmove'
fstransform: checking for fsremap... 	'/sbin/fsremap'
fstransform: checking for fsck(source file-system)...	'/sbin/fsck'
fstransform: checking for fsck(target file-system)...	'/sbin/fsck'
fstransform: looking for optional commands
fstransform: checking for sleep... 	'/bin/sleep'
fstransform: checking for date... 	'/bin/date'
17:45:36 fstransform: environment check passed.
17:45:36 fstransform: saving output of this execution into /var/tmp/fstransform/fstransform.log.308145
17:45:36 fstransform: preparing to transform device '/dev/sdd' to file-system type 'xfs'
17:45:36 fstransform: device is mounted at '/virtual_0' with file-system type 'ext4'
17:45:36 fstransform: device raw size = 32635174780928 bytes
17:45:36 fstransform: creating sparse loop file '/virtual_0/.fstransform.loop.308145' inside device '/dev/sdd'...
17:45:36 dd: 1+0 records in
17:45:36 dd: 1+0 records out
17:45:36 dd: 1 byte copied, 5.2134e-05 s, 19.2 kB/s
17:45:36 fstransform: device file-system block size = 4096 bytes
17:45:36 fstransform: device usable size = 32635174780928 bytes
17:45:36 dd: failed to truncate to 32635174780927 bytes in output file '/virtual_0/.fstransform.loop.308145': File too large
         
17:45:36 ERROR! fstransform: command '/bin/dd if=/dev/zero of=/virtual_0/.fstransform.loop.308145 bs=1 count=1 seek=32635174780927' failed (exit status 1)
                this is potentially a problem.
                you can either quit now by pressing ENTER or CTRL+C,
                
                or, if you know what went wrong, you can fix it yourself,
                then manually run the command '/bin/dd if=/dev/zero of=/virtual_0/.fstransform.loop.308145 bs=1 count=1 seek=32635174780927'
                (or something equivalent)
                and finally resume this script by typing CONTINUE and pressing ENTER:

When given "-m something_too_low", fsremap corrupts data. The precise value of "too low" is dependent on the data to be remapped, I have seen a case when 64 MB is not enough.

Here is a test case that relies only on downloadable files.

wget https://downloads.lede-project.org/releases/17.01.2/targets/x86/64/lede-17.01.2-x86-64-rootfs-ext4.img.gz
zcat lede-17.01.2-x86-64-rootfs-ext4.img.gz > lede-17.01.2-x86-64-rootfs-ext4.img
losetup /dev/loop7 lede-17.01.2-x86-64-rootfs-ext4.img
fstransform --opts-fsremap="-m 262144" /dev/loop7 xfs
xfs_repair -n /dev/loop7
# lots of corruption found

I am a Fedora user and dash is not usually part of a typical fedora install. If I am correct, only Ubuntu and its derivatives install these packages by default.

I've changed /bin/dash to /bin/bash before make install in the fstransform script and it seems to work for my case (transforming a xfs partition to ext4). I am still running the process (10% now) but so far no errors.

As far as I know, Ubuntu symlinks /bin/bash to /bin/dash for compatibility reasons so there should be no issue in changing that.

fstransform version 0.9.3 - conversion of reiserfs to ext3 on Debian 10. Somewhere at the beginning I get:

12:03:32 fsmove: progress: 88.0% done,   2.4 gigabytes still to move, estimated 15 seconds left
12:03:37 fsmove: progress: 90.0% done,   2.0 gigabytes still to move, estimated 20 seconds left
12:03:37 fsmove: job completed.
12:03:37 fstransform: unmounting and running '/sbin/fsck' (disk check) on loop file '/tmp/fstransform.mount.17211/.fstransform.loop.17211'
12:03:46 fsck: fsck from util-linux 2.33.1
12:03:47 fsck: /dev/loop0: 182167/2436448 Dateien (0.3% nicht zusammenhängend), 5160437/9744640 Blöcke
12:03:47 fstransform: disconnected loop device '/dev/loop0' from file '/tmp/fstransform.mount.17211/.fstransform.loop.17211'
12:03:47 fstransform: unmounting device '/dev/sda3' before disk check
12:03:47 umount: /tmp/fstransform.mount.17211: target is busy.

12:03:47 ERROR! fstransform: command '/bin/umount /dev/sda3' failed (exit status 32)
                this is potentially a problem.
                you can either quit now by pressing ENTER or CTRL+C,

                or, if you know what went wrong, you can fix it yourself,
                then manually run the command '/bin/umount /dev/sda3'
                (or something equivalent)
                and finally resume this script by typing CONTINUE and pressing ENTER:

Went into another shell, did the umount manually, and it worked without problems. The target wasn't busy anymore. Maybe one should add a small wait loop here or wait until the previously started processes operating on the filesystem are really stopped.

In a test VM, I got fsremap OOM-killed. I think it is a good idea to add protection against this, by having fsremap (or maybe even the fstransform script?) write -1000 into its own oom_score_adj file.

Hi,

When transforming an ntfs partition to ext2, a lot of the time only one core hits 100% while the rest are basically idle, a sign that the program is cpu bound due to a lack of parallelization.
Is there something that can be done about this? Or is it the underlying tools that are single threaded?

Edit: I also noticed that the program remounts ntfs without the big_writes parameter, which significantly slows down the procedure. I have a 450GB ntfs partition with an estimated completion time of 1 day and 20 hours.

Hi,

I try to transform a logical volume (lvm) from ext4 to xfs.
./fstransform /dev/mapper/data--elastic-lvdata xfs

All goes smoothly except:
The last fsck didn't pass because it's invoking fsck.ext4 instead of fsck.xfs
But the volume is in good condition when i run xfs_repair

12:40:07 fstransform: running again '/sbin/fsck' (disk check) on device '/dev/mapper/data--elastic-lvdata'
12:40:07 fsck: fsck from util-linux 2.27.1
12:40:07 fsck: fsck.ext4: Bad magic number in super-block while trying to open /dev/mapper/data--elastic-lvdata
12:40:07 fsck: /dev/mapper/data--elastic-lvdata:
12:40:07 fsck: The superblock could not be read or does not describe a valid ext2/ext3/ext4
12:40:07 fsck: filesystem. If the device is valid and it really contains an ext2/ext3/ext4
12:40:07 fsck: filesystem (and not swap or ufs or something else), then the superblock
12:40:07 fsck: is corrupt, and you might try running e2fsck with an alternate superblock:
12:40:07 fsck: e2fsck -b 8193
12:40:07 fsck: or
12:40:07 fsck: e2fsck -b 32768
12:40:07 fsck:

12:40:07 ERROR! fstransform: command '/sbin/fsck -p -f /dev/mapper/data--elastic-lvdata' failed (exit status 8)
this is potentially a problem.
you can either quit now by pressing ENTER or CTRL+C,

            or, if you know what went wrong, you can fix it yourself,
            then manually run the command '/sbin/fsck -p -f /dev/mapper/data--elastic-lvdata'
            (or something equivalent)
            and finally resume this script by typing CONTINUE and pressing ENTER: packet_write_wait: Connection to 10.181.80.107 port 22: Broken pipe

summary: If you have not already, please make fstransform (either as an option or by default) edit the user's fstab when appropriate: e.g., when fstransform changes a UUID or filesystem type listed in the fstab. Minimally, fstransform should warn the user that s/he might need to edit their fstab.

details:

I recently encountered a problem on a box running Debian Jessie: its /boot partition had filesystem type=ext2, blocking GRUB2 upgrade. For the first part of the fix I installed and used fstransform and was mostly impressed with its ease of use: see session transcript here. However I encountered (with fstransform)

1. one problem that caused considerable anxiety at first, but which I was able to fix manually as detailed below
2. another problem that continues to annoy

so I'm putting 2 issues. This one is about the first issue, which I believe I have solved manually (though perhaps suboptimally), a fix for which I hope you will {incorporate into, automate with} your code. (If you haven't already--again, note that I was using debian package=fstransform version=0.9.3-1, not a fresh repo clone.)

After running fstransform, I rebooted (to test the filesystem upgrade), and got the following message immediately after the GRUB2 screen (in the white-text-on-black-screen boot phase):

error: no such device: 91998f3d-7138-42ce-acfd-4e9946071dd9
Loading Linux <kernel version/> ...
Loading initial ramdisk ...
Press any key to continue...

I'm not a sysadmin (except on my own boxes) but recognized that the first line ended in a UUID, which reappeared later in the boot process (still in the white-text-on-black-screen boot phase, but this time also including a red FAIL):

fsck.ext2: Unable to resolve 'UUID=91998f3d-7138-42ce-acfd-4e9946071dd9'
...
fsck exited with status code 8
File system check failed. A log is being saved in /var/log/fsck/checkfs ...

/var/log/fsck/checkfs was not enlightening, but I guessed that the error message's UUID

• corresponded to my old /boot
• had been set in my fstab
• had not been updated by fstransform

This appears correct:

$ date ; fgrep -e '/boot' /etc/fstab
Thu Feb  4 18:48:54 MST 2016
UUID=91998f3d-71e8-42ce-acfd-4e9946071dd9 /boot ext2 relatime 0 2

I didn't know how to find the new UUID, but after a bit of DuckDuckGo, learned to do

$ date ; sudo blkid | fgrep -e 'sda3'
Thu Feb  4 18:55:49 MST 2016
/dev/sda3: UUID="aadfc68a-9724-481a-a402-393a0c75815d" TYPE="ext4" PARTUUID="ce0b2a49-03"

I do know enough bash and sed to edit my fstab with

OLD_UUID='91998f3d-71e8-42ce-acfd-4e9946071dd9'
OLD_FSTYPE='ext2'
NEW_UUID='aadfc68a-9724-481a-a402-393a0c75815d'
NEW_FSTYPE='ext4'
sudo sed -i "s/${OLD_UUID}/${NEW_UUID}/g" /etc/fstab
sudo sed -i "s/${OLD_FSTYPE}/${NEW_FSTYPE}/g" /etc/fstab
fgrep -e '/boot' /etc/fstab

the last line of which now produces (as expected) the output

UUID=aadfc68a-9724-481a-a402-393a0c75815d /boot ext4 relatime 0 2

Upon rebooting, I still get the other error (i.e., the first message above), about which I made a separate issue, but at least now I don't get a scary red FAIL in my boot screen :-) So I'd appreciate it if you could (if you haven't already) make your code either

1. (preferred) if appropriate, automate editing the user's fstab
2. (acceptable) warn the user that they may need to edit their fstab

to avoid causing unnecessary distress to less experienced users.

TIA, Tom Roche [email protected]

Hi and thank you for this project.

When possible (partition occupation below 50%), would you rather recommend shrinking the source partition, adding a target one, moving the files and expanding the target afterward?
Which method do you reckon is the fastest/the safest?

xfs -> ext4 conversion does not transfer SELinux contexts

Doing this on RHEL7.

edit:
Apparantly it does not transfer anything in the xattr field.

Plz, pretty please add this feature.

Hi there,
I shall report a bug similar to #19. After a file system conversion from reiserfs to ext4 and subsequent remapping were successful, the tool tried to fsck again the device, but as a reiserfs file system; therefore throwing an error.

Message like "reiserfs_open: the reiserfs superblock cannot be found on /dev/mapper/...." or

ERROR! fstransform: command '/sbin/fsck -p -f /dev/mapper/......' failed (exit status 8)
this is potentially a problem.
you can either quit now by pressing ENTER or CTRL+C,
Appeared .

Version is fstransform-0.9.3-1 on CentOS 7.x.

Thank you for writing such a great tool! Is there roadmap for supporting ntfs/vfat/exfat?

I have an xfs partition created when the ftype default was 0, but I now need it to be set to 1. What is the easiest way of transforming it? Can I do xfs > xfs or do I have to do something like xfs > ext4 > xfs?

Please also flag this as a question rather than an issue.

Hi, this is a question rather than a problem. I have a (degraded as one disk is missing) 54TB RAID5 array (was 4 x 18TB) formatted as XFS. I don't want to replace the missing disk, but just shrink the filesystem to allow me to convert the array to a 3 disk RAID5. There is a bit less than 10TB of data on the array. Are the conversion times dependent on actual disk usage or partition size? I am trying to gauge how long it may take. 36 days at 16h/1TB based on the partition size I can't do but I can think about 160h based on the data size.

Software version: master

Error Message:

depbase=`echo ../src/e4attr.o | sed 's|[^/]*$|.deps/&|;s|\.o$||'`;\
g++ -DHAVE_CONFIG_H -I. -I../../../fstransform/fsattr/build -I../../fsremap/src     -g -O2 -Wall -Wextra -W -Wno-missing-field-initializers -MT ../src/e4attr.o -MD -MP -MF $depbase.Tpo -c -o ../src/e4attr.o ../../../fstransform/fsattr/build/../src/e4attr.cc &&\
mv -f $depbase.Tpo $depbase.Po
depbase=`echo ../src/log.o | sed 's|[^/]*$|.deps/&|;s|\.o$||'`;\
g++ -DHAVE_CONFIG_H -I. -I../../../fstransform/fsattr/build -I../../fsremap/src     -g -O2 -Wall -Wextra -W -Wno-missing-field-initializers -MT ../src/log.o -MD -MP -MF $depbase.Tpo -c -o ../src/log.o ../../../fstransform/fsattr/build/../src/log.cc &&\
mv -f $depbase.Tpo $depbase.Po
depbase=`echo ../src/main.o | sed 's|[^/]*$|.deps/&|;s|\.o$||'`;\
g++ -DHAVE_CONFIG_H -I. -I../../../fstransform/fsattr/build -I../../fsremap/src     -g -O2 -Wall -Wextra -W -Wno-missing-field-initializers -MT ../src/main.o -MD -MP -MF $depbase.Tpo -c -o ../src/main.o ../../../fstransform/fsattr/build/../src/main.cc &&\
mv -f $depbase.Tpo $depbase.Po
depbase=`echo ../src/mstring.o | sed 's|[^/]*$|.deps/&|;s|\.o$||'`;\
g++ -DHAVE_CONFIG_H -I. -I../../../fstransform/fsattr/build -I../../fsremap/src     -g -O2 -Wall -Wextra -W -Wno-missing-field-initializers -MT ../src/mstring.o -MD -MP -MF $depbase.Tpo -c -o ../src/mstring.o ../../../fstransform/fsattr/build/../src/mstring.cc &&\
mv -f $depbase.Tpo $depbase.Po
In file included from ../../../fstransform/fsattr/build/../src/log.cc:25:
../../../fstransform/fsattr/build/../src/first.hh:51:10: fatal error: autoconf.hh: No such file or directory
   51 | #include "autoconf.hh"
      |          ^~~~~~~~~~~~~
compilation terminated.
In file included from ../../../fstransform/fsattr/build/../src/main.cc:23:
../../../fstransform/fsattr/build/../src/first.hh:51:10: fatal error: autoconf.hh: No such file or directory
   51 | #include "autoconf.hh"
      |          ^~~~~~~~~~~~~
compilation terminated.
make[1]: *** [Makefile:383: ../src/log.o] Error 1
make[1]: *** Waiting for unfinished jobs....
make[1]: *** [Makefile:383: ../src/main.o] Error 1
In file included from ../../../fstransform/fsattr/build/../src/e4attr.cc:23:
../../../fstransform/fsattr/build/../src/first.hh:51:10: fatal error: autoconf.hh: No such file or directory
   51 | #include "autoconf.hh"
      |          ^~~~~~~~~~~~~
compilation terminated.
make[1]: *** [Makefile:383: ../src/e4attr.o] Error 1
In file included from ../../../fstransform/fsattr/build/../src/mstring.cc:26:
../../../fstransform/fsattr/build/../src/first.hh:51:10: fatal error: autoconf.hh: No such file or directory
   51 | #include "autoconf.hh"
      |          ^~~~~~~~~~~~~
compilation terminated.
make[1]: *** [Makefile:383: ../src/mstring.o] Error 1
make[1]: Leaving directory '/xxx/fstransform-build/fsattr/build'
make: *** [Makefile:359: all-recursive] Error 1

../src/arch/mem_linux.cc:70]: (error) Width 256 given in format string (no. 1) is larger than destination buffer 'label[256]', use %255s to prevent overflowing it.

../src/arch/mem_linux.cc:70]: (error) Width 8 given in format string (no. 3) is larger than destination buffer 'unit[8]', use %7s to prevent overflowing it.

macOS 10.13 converts HFS+ volumes to APFS automatically, by default, in-place, and quickly during installation. My understanding is that it does this by rewriting the filesystem data, without actually changing the files' positions on-disk. I'm not sure if this is practical for all possible FS switches (differences in block size, alignment requirements, fragmentation capabilities, etc), but I would expect it should be at least for some, and that the approach should improve performance on almost any filesystem if applied per-file.

This approach has the disadvantage of requiring much more filesystem-specific handling (possibly enough to make the endeavor impractical), but the advantage of being much faster, and potentially safer: since only the filesystem structure is modified, it can be backed up ahead of time using little space, and restored if necessary.

Hi,

As it is clear from the title, I am looking to convert my home XFS partition to something else, that can be shrunk without data loss. The choice has fallen to BTRFS so I would like to ask if this is supported or if it is intended to be, at some point in the future.

As a workaround, perhaps it might be possible to do two rounds, i.e. XFS -> EXT4 and then EXT4 -> BTRFS. I believe I have seen something about online conversion of EXT4 to BTRFS.

Edit: btrfs-convert is available for the potential workaround of converting ext4 to btrfs.

It seems like for sure it is not officially supported, but some people seem to think it would work.

Can y'all explicitly clarify this?

Thanks for the awesome project - I hope to use it (I didn't realize before getting into zfs that it would be impossible to downsize!)

Hi,

I tried to use fsremap to convert the loop file into the device, but I got the error showing in the following screenshot. Any idea about how to solve it. Thanks!

I probably need to install something but this is beyond my erudition:

In file included from ../src/e4attr.cc:23:
../src/first.hh:1:1: error: expected unqualified-id before ‘.’ token
1 | ../../fsremap/src/first.hh
| ^
../src/e4attr.cc:306:1: error: ‘FT_NAMESPACE_END’ does not name a type
306 | FT_NAMESPACE_END
| ^~~~~~~~~~~~~~~~
make[1]: *** [Makefile:389: ../src/e4attr.o] Error 1
make[1]: Leaving directory '/home/dch/Downloads/fstransform-master/fsattr/build'
make: *** [Makefile:360: all-recursive] Error 1

I tried converting my 9.1TB JFS filesystem to EXT4 (so I could shrink it) and after three days of churning it finally got to the fsremap part which promptly crashed:

17:55:22 fstransform: mounting again device '/dev/md0' read-only
17:55:22 fstransform: launching '/usr/sbin/fsremap' in simulated mode
17:55:22 fsremap: starting job 1, persistence data and logs are in '/var/tmp/fstransform/fsremap.job.1'
17:55:22 fsremap: if this job is interrupted, for example by a power failure,
17:55:22 fsremap: you CAN RESUME it with: /usr/sbin/fsremap -n -q --resume-job=1 -- /dev/md0
17:55:23 fsremap: ERROR: ff_posix_fibmap(): error, dev_block_count = 7, file_block_count = 2441889920 overflow type (int): File too large
17:55:23 fsremap: ERROR: failed to list file blocks with ioctl(FS_IOC_FIEMAP): Operation not supported

fstransform resolves which to the alias output (which which) instead of it's actual filesystem location (/usr/bin/which) and fails to run.

Overriding this path in the fstransform startup script fixes the problem. Setting --cmd-which=/usr/bin/which, as per the error output, did not have an effect.

# fstransform  /dev/vdb1 ext4                                                                           [1755/1810]
fstransform: starting version 0.9.4, checking environment
fstransform: checking for which...         found 'which ()
{ 
    ( alias;
    eval ${which_declare} ) | /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot $@
}', but is NOT executable by you!
fstransform: checking for expr...         '/bin/expr'
fstransform: checking for id...         '/bin/id'

ERROR! fstransform: environment check failed.
       Please install the commands 'which' before running fstransform
       If these commands are already installed, add them to your $PATH
       or tell their location with the option --cmd-COMMAND=/path/to/your/command

# which which
which ()
{ 
    ( alias;
    eval ${which_declare} ) | /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot $@
}

Hi!
Maybe we'll try to work together to get fiemap merged in fuse at last? :-)
// I'm Vitaliy Filippov, the author of that fiemap patch

static analyser cppcheck says:

mem_linux.cc:70:20: error: Width 256 given in format string (no. 1) is larger than destination buffer 'label[256]', use %255s to prevent overflowing it. [invalidScanfFormatWidth]

and

mem_linux.cc:70:20: error: Width 8 given in format string (no. 3) is larger than destination buffer 'unit[8]', use %7s to prevent overflowing it. [invalidScanfFormatWidth]

Source code is

    if ((err = fscanf(f, "%256s %" FT_ULL " %8s\n", label, & n_ull, unit)) <= 0) {