This would take a new command-line option as well.

Consider a progress meter? http://linuxcommando.blogspot.com/2008/06/show-progress-during-dd-copy.html

Greetings,

thanks for the script. Made my life much easier, due to handling with different operating systems.

I have a raspberry pi running with raspbian jessie. I formated multiple USB pen drives with success but had a misfortune with my beloved external SSD in an USB enclosure driven by an AsMedia 1351 chipset.

When the script started the format process I could hear a buzz coming from the drive. Sometimes later the procedure stopped with an I/O timeout error. Plugging out and plugging it back unfortunately didn't bring back the drive. After attaching it to a Windows PC I tried to force a Firmware update via Crucial's Storage Executive to no avail. Sadly enough I can't read any SMART data with various tools depending on the mood of the drive. Sometimes it manages to give some data (with healthy data), sometimes its garbage. I even tried to issue an ATA Secure Erase which sometimes proceeds as it should but without any results.

Since the drive still gets recognized, I thought I might fiddle with hdparm aswell. So if anyone has a magic up their sleeves with hdparm I'm ready to be a guinea pig.

Last week I got hands on a Samsung 850 Pro. I followed exactly the same procedure in the same enclosure with the rpi with positive results. I do now have quite a fast external drive with UDF, although with some casualties on the way.

PS: both drives report(-ed) 512 for physical and logical block size

Hi,
Is it still impossible to create a partition (not entire drive) that can be mounted rw in both windows and linux? I see #24, but as I understand, he was not able to achieve that goal.
What are related caveats? I have GPT partition table.

method

I tested with dd id=/dev/zero of=test count=200000. The device has 512 block size. Tested in Debian testing and macOS 10.15. Tried with formatting in Debian, and formatting with macOS. Repeated the test 3-5 times in each OS. Debian was installed in Macbook Pro 9,2 and macOS was installed in Macbook 11,3. A 32 GB SanDisk cruiser SDCZ36-032G is used for all tests. None of the USB ports on the MacBook Pros look blue (not USB 3). The SanDisk USB port look red (USB 2?).

result

In all cases, Debian wrote around 60 MB/s but macOS wrote around 8 MB/s.

Any idea why that is the case? I'm wondering if that is due to macOS maybe implementing udf support in userspace.

The format-udf script makes the assumptions that (1) it is run as unprivileged user and (ii) sudo is available. I wished that format-udf also works without sudo, run as root.

Hi,

I'm trying to tackle the (frankly glaring) lack of any repair-enabled UDF fsck under macOS, by (shallow) porting of the still beta (and to-be-merged upstream) fsck, as part of (Linux) udftools (pali/udftools#13).

You can find more details here:

argorain/udftools#1

as well as another contribution of mine (a Homebrew formula):

https://github.com/gmerlino/homebrew-udffsck

so that it is readily a tap-and-install away.

I opened this issue just as a heads up for you (and your community), in case you want to update your wiki:

https://github.com/JElchison/format-udf/wiki/fsck-tools-for-UDF

with some of the aforementioned pointers, and help with testing (the fsck tool) as well.

Thanks for your help (and your great formatter).

The script as is failed in Ach linux, using udftools 2.0-2 ; the error was :

mkudffs: Error: Option --media-type must be specified before option --udfrev

So I just edited the script ; I moved the --media-type option before the --udfrev option and retried. It then worked fine. I'm sending the diff attached.

Thank you for this great script anyway !
format-udf.sh.diff.zip

Thank you for your super duper program! I used it with success, but I had to comment out lines 309 and 334 (validate device/parent device identifier) because the validation was a little too strict for my OS - my device ID was mmcblk0 and I think that may not have satisfied the regex. I don't know how this would affect other systems but if it won't mess things up, I would suggest loosening the validation a little.

Pop!_OS, System76 Darter Pro, using format-udf on an SD card

method

I tested with dd id=/dev/zero of=test count=200000. The device has 512 block size. Tested in Debian testing and macOS 10.15. Used the script with default parameters in the master branch. Tried with formatting in Debian. Also tried formatting in macOS. Repeated the test 3-5 times in each OS. Debian was installed in Macbook Pro 9,2 and macOS was installed in Macbook 11,3. A 32 GB SanDisk cruiser SDCZ36-032G is used for all tests. None of the USB ports on the MacBook Pros look blue (not USB 3). The SanDisk USB port look red (USB 2?).

result

In all cases, Debian wrote around 60 MB/s but macOS wrote around 8 MB/s.

Any idea why that is the case? I'm wondering if that is due to macOS maybe implementing udf support in userspace.

Formatted USB stick under raspberry pi running Ubuntu MATE 15.04. Formatting goes fine. When USB stick is inserted to Mac OS X 10.10 (yosemite), it does not contain filesystem. If USB stick is formatted on Mac, Ubuntu will recognize filesystem but shows label "MAC OS X ...". UDF is fine filesystem but there seems to be some inconsistency with tools and how labels are handled.

I ran format-udf on a flash drive of mine, and loaded some data onto it. It reads fine on my Linux system (64bit Manjaro Linux), but on my Windows system (64bit Windows 8.1 Pro), it does not get automatically mounted on bootup or when I remove and re-insert the flash drive.

Running gdisk -l on my flash drive returns the following output:

GPT fdisk (gdisk) version 1.0.5

Partition table scan:
  MBR: MBR only
  BSD: not present
  APM: not present
  GPT: not present


***************************************************************
Found invalid GPT and valid MBR; converting MBR to GPT format
in memory. 
***************************************************************

Warning! Main partition table overlaps the first partition by 34 blocks!
You will need to delete this partition or resize it in another utility.

Warning! Secondary partition table overlaps the last partition by
33 blocks!
You will need to delete this partition or resize it in another utility.
Disk /dev/sde: 485228544 sectors, 231.4 GiB
Model: Ultra           
Sector size (logical/physical): 512/512 bytes
Disk identifier (GUID): 410064B0-8EA4-406E-AD44-504336157E5E
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 485228510
Partitions will be aligned on 2048-sector boundaries
Total free space is 0 sectors (0 bytes)

Number  Start (sector)    End (sector)  Size       Code  Name

I was looking to use UDF to format a shared Linux/Win/OSX data partition on laptops.

Is this approach suitable for this use case?

I tried formatting the thing using format-udf in both linux and osx. Everything works fine between these two OSes, but when I connect the usb stick on Windows 10, it doesn't recognize it and wants to format it.

See comments in code: "TODO not sure why the following is required on bash 3.2.51(1) on OS X (doesn't exit with false even with set -e)"

Example:

# verify that DEVICE doesn't have partition number on end, or that it's in OS X format
(echo "$DEVICE" | egrep -q '^([hs]d[a-z]|disk[0-9]+)$') || (echo "[-] <device> is of invalid form" >&2 && false)
# TODO not sure why the following is required on bash 3.2.51(1) on OS X (doesn't exit with `false` even with 'set -e')
RET=$?; if [[ $RET -ne 0 ]]; then
exit $RET
fi

There are several occurrences of this logic in the code (all marked with TODO).

All third-party utilites just only works with cd/dvd, not usb flash.
If there is any utility that can work with usb flash, give me a link :)

Hi Jonathan,

FYI, I have posted a new comment on the old WordPress blog post that brought me in this repository. I think it needs a little edit for posterity but maybe it's me who's missing the mark. :-)

References:

• https://apple.stackexchange.com/questions/9284/does-mac-have-something-similar-to-a-linux-loop-device
• https://stackoverflow.com/questions/4549310/loopback-mount-in-osx
• https://serverfault.com/questions/174909/mount-block-file-on-osx

Suggested by @lucky62 in #24:

Please consider to allow /dev/ prefix for the device name. So both formats will be possible:

• without /dev/ - e.g. sdb or sdb2
• with /dev/ - e.g. /dev/sdb or /dev/sdb2

I see that /dev/$DEVICE pattern is used many times in your script so I think better is to check at the beginning if has the /dev/ prefix and add it if not. Then you can use simple $DEVICE instead of /dev/$DEVICE.

I'm trying to format a veracrypt encrypted device with format-udf but it fails because it only searches for devices in the /dev/ directory. I changed every appearance of /dev/$Device to /dev/mapper/$DEVICE but the script complains about:

[-] <device> is of invalid form

Please allow virtual block devices to be formatted with format-udf.

https://github.com/pali/udftools

Right now, Travis CI runs unit-test.sh in an environment that contains whatever version of udftools that is in the repo at that time. Would be nice to test against the latest/greatest udftools (master branch) for early warning of issues like #40 and #29, before they hit the repos.

How about creating a partition table with one partition, starting on the 0th sector? Like this script. Also, some comments on why is this needed might help.

I never personally tested my udf disk on all those systems, I only use linux, but it seems like a cool idea anyway.

Currently the script uses blockdev --getpbsz to detect the physical block size. However, this is incorrect for Advanced Format drives using SCSI-to-ATA Translation (SAT; includes USB enclosures and such), as detailed here: http://nunix.fr/index.php/linux/7-astuces/65-too-hard-to-be-above-2tb

The correct way to determine it is by using either hdparm -I or smartctl -d sat -a. Since the former doesn't need explicit specification that it's a SAT drive, it's probably better to use that, like so:

BLOCK_SIZE=$(sudo hdparm -I /dev/$DEVICE | grep "Physical Sector size" | awk '{print $(NF-1)}')

USB disks formatted with this utility were working fine with Windows 10 21H2 OS Build 19043, but now Windows pops up a message saying "Please insert a disk into USB Drive (X:)".
UDF support is still working, since drives with "format X: /FS:UDF /R:2.01 /q" still does the job.

We have hundreds of pendrives formatted this way, any idea if we can patch Windows registry or something to make them work again?

Thx

Thanks for the handy script. 8-)

I'm having a few problems getting the drive recognised by Windows 7 tho. I just attached my 64GB USB Flash drive to a VirtualBox guest running CentOS 7. I ran:

$ sudo ./format-udf.sh sdb "Gordon"
[+] Testing dependencies...
[+] Looking for drive listing tool...which: no diskutil in (/sbin:/bin:/usr/sbin:/usr/bin)
 using /sbin/blockdev
[+] Looking for unmount tool...which: no diskutil in (/sbin:/bin:/usr/sbin:/usr/bin)
 using /bin/umount
[+] Looking for UDF tool...which: no newfs_udf in (/sbin:/bin:/usr/sbin:/usr/bin)
 using /bin/mkudffs
[+] Validating arguments...
[+] Gathering drive information...
Cruzer Edge     
RO    RA   SSZ   BSZ   StartSec            Size   Device
rw   256   512   512          0     64016220160   /dev/sdb
The above-listed drive (and partitions, if any) will be completely erased.
Type 'yes' if this is what you intend:  yes
[+] Detecting native sector size...
[+] Validating detected sector size...
[+] Unmounting drive...
[+] Zeroing out any existing partition table on drive...
4096+0 records in
4096+0 records out
2097152 bytes (2.1 MB) copied, 5.91336 s, 355 kB/s
[+] Formatting /dev/sdb ...
start=0, blocks=64, type=RESERVED 
start=64, blocks=12, type=VRS 
start=76, blocks=180, type=USPACE 
start=256, blocks=1, type=ANCHOR 
start=257, blocks=16, type=PVDS 
start=273, blocks=1, type=LVID 
start=274, blocks=125031149, type=PSPACE 
start=125031423, blocks=1, type=ANCHOR 
start=125031424, blocks=239, type=USPACE 
start=125031663, blocks=16, type=RVDS 
start=125031679, blocks=1, type=ANCHOR 
[*] Successfully formatted /dev/sdb: LABEL="Gordon" TYPE="udf" 
Please disconnect/reconnect your drive now.
[ben@bentos7 format-udf]$ 

The CentOS 7 guest itself seems happy enough with things - if I disconnect and reconnect the drive, it recognises it and I can view, create folders and files, etc.

However, I cannot get Windows to use the drive. I safely remove the drive from CentOS and disconnect it from the VirtualBox guest. In Windows I then go to "Computer Management" and "Disk Management" and a pop-up window tells me "You must initialize a disk before Logical Disk Manager" can access it. I assume this is because of the zeroing of the first part of the disk? How do I go about attaching a drive letter to the newly created disk?

Cheers, Ben

I'm on Mac (in fact hackintosh and macbook air) and I succeed to format a first USB flash drive and a internal hard drive in UDF, they were both USB2. But the process fails with a USB3 flash drive. I don't know why.
https://www.7dayshop.com/products/grixx-usb-3-0-flash-drive-90m-s-64gb-WH2-GRUSB64GBALUU3
This is a copy text of the terminal :

The above-listed drive (and partitions, if any) will be completely erased.
Type 'yes' if this is what you intend:  yes
[+] Detecting total size...
[*] Using total size of 62440340992
[+] Validating detected total size...
[+] Detecting physical block size...
[*] Using block size of 512
[+] Validating detected block size...
[+] Unmounting drive...
Unmount of all volumes on disk7 was successful
[+] Zeroing out any existing partition table on drive...
4096+0 records in
4096+0 records out
2097152 bytes transferred in 1.334498 secs (1571491 bytes/sec)
[+] Formatting /dev/disk7 ...
write: Device not configured
[-] Format failed!

But otherwise it's working perfectly with USB2 devices on Linux, OS X and Windows ! Thanks for that !

sudo mount -a doesn't work

Window 7 is indeed able to read the UDF 2.01 file system created with format-udf (mkudffs from udftools 1.3).

Whenever I run CHKDSK on Windows 7, however, it "finds" some errors and "fixes" them:

The next time I run CHKDSK against the same file system, it finds the very same issues.

I understand this most probably can't be easily fixed, but probably the issue should be mentioned in the documentation/release notes.

Linux Mint 17.1 Cinnamon sees the fake partition but not the UDF.
Interestingly my Windows 7 installation says that the disk is not initialised.
Let me know what diagnostics you need.

It would be good to have the option to format single partition on partitioned disk.
I know that this will not work with OS X. But it will be useful for peoples using only Linux and Windows.

Hello,
I am having a problem after formatting with your script. I am running the following commands:

`

    sudo mkdir /media/usb
    sudo apt-get install -y udftools coreutils vim-common
    cd /home/pi/Teleonome/build/format-udf/
    sudo chmod +x format-udf.sh
    ./format-udf.sh -f sda "MyDrive"
    echo "Mouting USB Key to /media/usb

`
I then take the usb out and plug it back in. Then I execute:

sudo mount -t udf /dev/sda /media/usb

that produces the following error:

mount: unknown filesystem type 'udf'

Do you have any suggestions?

Thanks

I would like you to change text like "Example: ./format-udf.sh sda "My External Drive" to avoid "sda" references :)

Add smarter command line options/argument parsing.

Then, add -f (--force) option to force non-interactive run of script (with warning about its danger)

Thanks for writing this script, it is really quite useful.

One very minor suggestion. For Linux, the lshw utility is not always installed, and is also very slow. (The slowness is due to fetching all sorts of non-block-device-related info, which is then filtered out in this script by grep /dev.) The "approved" alternative would be blockdev, which (at least on Debian and derivatives thereof like Ubuntu) is in an essential package and therefore guaranteed present. It is also about 200x faster.

$ sudo time --format 'real %e' blockdev --report > /dev/null
real 0.01

$ sudo time --format 'real %e' lshw > /dev/null
real 1.83                 

You can get a list of all available block devices with blockdev --report and you can get the physical sector size of a single device with, e.g., blockdev --getpbsz /dev/sda.

GNOME Files doesn't see the resulting drive of format-udf.sh as mountable.

In comparison: running mkudffs --media-type=hd --vid=Morrow --lvid=Morrow /dev/sdc produces a drive I can mount and read and write to in Linux.

NAME="openSUSE Tumbleweed"

VERSION="20161003"

udftools version is 1.0.0b3-161.4

Here's the result of:
./format-udf.sh sdc Morrow

[+] Testing dependencies...
[+] Looking for drive listing tool...which: no diskutil in (/usr/local/bin:/usr/bin:/bin:/usr/games:/sbin)
 using /sbin/blockdev
[+] Looking for unmount tool...which: no diskutil in (/usr/local/bin:/usr/bin:/bin:/usr/games:/sbin)
 using /usr/bin/umount
[+] Looking for UDF tool...which: no newfs_udf in (/usr/local/bin:/usr/bin:/bin:/usr/games:/sbin)
 using /usr/bin/mkudffs
[+] Parsing options...
[+] Validating arguments...
[+] Gathering drive information...
BUP Fast HDD    
RO    RA   SSZ   BSZ   StartSec            Size   Device
rw  1024   512   512          0   4000797859328   /dev/sdc
The above-listed device (and partitions, if any) will be completely erased.
Type 'yes' if this is what you intend:  yes
[+] Detecting total size...
[*] Using total size of 4000797859328
[+] Validating detected total size...
[+] Detecting physical block size...
[*] Using block size of 4096
[+] Validating detected block size...
[+] Unmounting device...
umount: /dev/sdc: not mounted
[+] Zeroing out first chunk of device...
4096+0 records in
4096+0 records out
16777216 bytes (17 MB, 16 MiB) copied, 0.848001 s, 19.8 MB/s
[+] Formatting /dev/sdc ...
start=0, blocks=8, type=RESERVED 
start=8, blocks=3, type=VRS 
start=11, blocks=245, type=USPACE 
start=256, blocks=1, type=ANCHOR 
start=257, blocks=16, type=PVDS 
start=273, blocks=1, type=LVID 
start=274, blocks=976756758, type=PSPACE 
start=976757032, blocks=1, type=ANCHOR 
start=976757033, blocks=239, type=USPACE 
start=976757272, blocks=16, type=RVDS 
start=976757288, blocks=1, type=ANCHOR 
[+] Writing fake MBR...
16+0 records in
16+0 records out
16 bytes copied, 0.000415576 s, 38.5 kB/s
2+0 records in
2+0 records out
2 bytes copied, 0.0398515 s, 0.1 kB/s
[+] Successfully formatted /dev/sdc: PTTYPE="dos"
Please disconnect/reconnect your drive now.

taohansen@nohostname:~/Documents> ./format-udf.sh sdc Morrow
[+] Testing dependencies...
[+] Looking for drive listing tool...which: no diskutil in (/usr/local/bin:/usr/bin:/bin:/usr/games:/sbin)
 using /sbin/blockdev
[+] Looking for unmount tool...which: no diskutil in (/usr/local/bin:/usr/bin:/bin:/usr/games:/sbin)
 using /usr/bin/umount
[+] Looking for UDF tool...which: no newfs_udf in (/usr/local/bin:/usr/bin:/bin:/usr/games:/sbin)
 using /usr/bin/mkudffs
[+] Parsing options...
[+] Validating arguments...
[+] Gathering drive information...
/dev/sdc: PTTYPE="dos"
BUP Fast HDD    
RO    RA   SSZ   BSZ   StartSec            Size   Device
rw  1024   512   512          0   4000797859328   /dev/sdc
rw  1024   512   512          0    500099731968   /dev/sdc1
The above-listed device (and partitions, if any) will be completely erased.
Type 'yes' if this is what you intend:  yes
[+] Detecting total size...
[*] Using total size of 4000797859328
[+] Validating detected total size...
[+] Detecting physical block size...
[*] Using block size of 4096
[+] Validating detected block size...
[+] Unmounting device...
umount: /dev/sdc: not mounted
[+] Zeroing out first chunk of device...
4096+0 records in
4096+0 records out
16777216 bytes (17 MB, 16 MiB) copied, 0.872517 s, 19.2 MB/s
[+] Formatting /dev/sdc ...
start=0, blocks=8, type=RESERVED 
start=8, blocks=3, type=VRS 
start=11, blocks=245, type=USPACE 
start=256, blocks=1, type=ANCHOR 
start=257, blocks=16, type=PVDS 
start=273, blocks=1, type=LVID 
start=274, blocks=976756758, type=PSPACE 
start=976757032, blocks=1, type=ANCHOR 
start=976757033, blocks=239, type=USPACE 
start=976757272, blocks=16, type=RVDS 
start=976757288, blocks=1, type=ANCHOR 
[+] Writing fake MBR...
16+0 records in
16+0 records out
16 bytes copied, 0.000392997 s, 40.7 kB/s
2+0 records in
2+0 records out
2 bytes copied, 0.000148946 s, 13.4 kB/s
[+] Successfully formatted /dev/sdc: PTTYPE="dos"
Please disconnect/reconnect your drive now.

Tried running this script in msys2 on Windows 10. Can't find diskutil or blockdev. Where can I find any of these two?

Sorry if this question/topic has been solved before or if I'm just not smart enough to get the syntax right with the script, but I have not located a clear answer after hours of scouring...

I have a new Seagate 5TB drive that has the 4k (Advanced Format) architecture on it with 512 emulation (512e), fdisk -l shows "Units: sectors of 1 * 512 = 512 bytes" "Sector size (logical/physical): 512 bytes / 4096 bytes" "I-O size (minimum/optimal): 4096 bytes / 33553920 bytes".. I want to obviously use the entire 5TB drive when formatting udf and not limit the capacity to 2TB; however, using the -b flag spits out the arguments help output and the README doesn't seem to provide clear language about how to run this argument. I guess my main problem is figuring out how to properly syntax the command to run the tool and configure the drive properly. Can someone please help by clearly spelling out the command in a Linux system? Many thanks.

This is maybe something for the Readme.md. Recently I tried to install a fresh copy of Windows 10 May 2004 on a nvme disk that was formated as UDF using "format-udf" and noticed that the installer wasn't able to detect a disk, even not a option for reformating/repartitioning. The same UDF formated nvme was previously working fine as a data disk (not boot nor system) in Windows 1909.

It will probably be available by Windows shell cli tools "diskpart", but the GUI installer hides UDF nvme and probably regular UDF disk probably because they mimic readonly optical disc media and those are also not a valid destinations for a Windows installation.

By running wipefs -a on ubuntu 20.04 I was able to get the nvme disk detected again by the Windows 10 installer.

Debian stable still uses 1.0.0b3-14.3, but Ubuntu has picked up 1.2-1build1 as of Yakkety.

It would be good to test format-udf.sh with udftools 1.2 per @pali here.

When create on OS X, the label shows up in Ubuntu as something like:

MacOS X UDF 2015-02-20 1304.44�

While for the most part it's better to maximise compatibility, there may be cases (and probably increasingly more cases) where the disk in question is larger than 2 TiB. The MBR solution won't be able to handle anything past that size. (UDF at 512 block size seems to not be able to handle it either, so this only matters when the block size is higher than that.) So it would be nice if there was a way to use GPT instead of MBR.

On that note, the MBR trick, while interesting, seems to confuse some tools, such as GPartEd. Maybe there should be a switch controlling whether to use it or not? In my case, the drive won't be used in OS X, so the trick isn't needed.

This may be a bit out of scope for this project, but I'm thinking that a place documenting the available UDF fsck tools would be nice. One of the main problems why UDF isn't used more on Linux is the lack of a filesystem check tool; the one provided by udf-tools is basically /bin/true.

There's apparently also udf_test from something called Philips UDF Conformance Tool; but I can't actually find it and I'm not sure if it actually fixes problems, or just finds them. Looking at Google results, the last release of it might have been in 2007, which would mean it would cover all UDF revisions.

I see that Windows does have the ability to check UDF volumes; in fact, it seems that my Win10 installation claims to have repaired the volume I created with format-udf, even if I didn't actually put any files on it. I wonder what it changes... (Hopefully not the MBR?)

Apparently there's also a check tool for Solaris, which perhaps could be ported to Linux too: https://bugs.launchpad.net/ubuntu/+source/udftools/+bug/384156#3

Does OS X have a UDF check tool to begin with?