List compression status of file about czicompress HOT 6 CLOSED

Hi @m-ringler !

Thanks a lot for the fantastic and very informative explanation. This sounds exactly like what we need. I will give it a try with the command line tool.

Thank you!!!

from czicompress.

@m-ringler Super, thank you so much for the stellar support!

Unfortunately (or rather fortunately) I'm using linux so I've had to compile myself. Same for czicompress. But all went good, compilation was successful and the tool works perfect. Very happy that now we can scan our entire storage and free up a few hundred terabytes.....

The only issue so far is that some files give us some strange errors involving the metadata, but I think one of my colleagues was already in touch and sent some example files. But if not, happy to provide them!

CZIcmd -c PrintInformation -i AllSubblocks -s 05b_limb_bud.czi 
FATAL ERROR: std::exception caught
 -> Illegal data detected at offset 9920 -> Invalid MetadataSegment-magic

Thanks a lot!!

from czicompress.

Hi @diego-rt, thanks for your interest in our CZI tools. Strictly speaking, a CZI file does not necessarily have a single compression status. The pixel data in a CZI file is organized into units called subblocks and each of these can be compressed (with one of various methods - one of which is zstd) or uncompressed. That said, most of the time, all subblocks have the same compression 'status'.

With 'the tool' are you referring to CZI Shrink (the UI tool), or would it be acceptable for you to use a command line tool? If so, you could use CZIcmd from the libczi repo. The basic usage is

CZIcmd -c PrintInformation -i AllSubblocks -s Your.czi

which will print something like

Complete list of sub-blocks
---------------------------

#0:  M=0 logical=(0,0,100,100) phys.=(100,100) pixeltype=gray16 compression=zstd1
#1:  M=1 logical=(100,0,100,100) phys.=(100,100) pixeltype=gray16 compression=zstd1

if your CZI is zstd-compressed,
and something like

Complete list of sub-blocks
---------------------------

#0:  M=1 logical=(100,0,100,100) phys.=(100,100) pixeltype=gray16 compression=uncompressed
#1:  M=0 logical=(0,0,100,100) phys.=(100,100) pixeltype=gray16 compression=uncompressed

if your CZI is uncompressed.

With this, we can build a script that finds CZI files with at least one uncompressed subblock. Here's how you could do it for the bash shell (which is also available for windows; you can install it e. g. with the git installer):

• Build libczi with CZIcmd (-DLIBCZI_BUILD_CZICMD=ON)
• Find the CZIcmd executable and put it on your path, for example like this:

path_to_czicmd=$(find $(pwd -P) \( -name CZIcmd -or -name CZIcmd.exe \) -type f -executable | head -n 1)
dir_of_czicmd=$(dirname "$path_to_czicmd")
PATH="$PATH:$dir_of_czicmd"

• Alternatively, just use the full path of the CZIcmd(.exe) file where I've written CZIcmd below.
• Check that you can actually run CZIcmd

CZIcmd --version

• Create a short script file called has_an_uncompressed_subblock.sh with the following contents:

#/bin/bash
file="$1"
if CZIcmd -c PrintInformation -i AllSubblocks -s "$file" | grep -q -m 1 'compression=uncompressed'
then
  echo $file
fi

• Now you can run

find /folder/to/search -iname '*.czi' -exec bash /path/to/has_an_uncompressed_subblock.sh '{}' \;

and it will print a list of files in /folder/to/search that have at least one uncompressed subblock. You can redirect the output to a file using > list_of_uncompressed_czis.txt.

from czicompress.

If you have difficulties building CZIcmd please let us know (with the information which operating system you want to run it on). We can probably provide a compiled binary.

from czicompress.

@diego-rt In my fork of the libczi repo, i've modified the github build a bit so that you can download compiled CZIcmd binaries from e. g. https://github.com/m-ringler/libczi/actions/runs/7046393353

If you are on Windows, either of the 'win-x64' artifacts should work for you, I'd suggest that you try CZIcmd-StreamsOFF-win-x64.

If you are on Linux, building yourself is probably the better option because

• it is unlikely that the libraries used on the github build runner are the same as on your machine
• It's relatively straight-forward to make your own build on linux.

from czicompress.

At first glance, this output is indicating that the CZI-file is broken in some way or another. In this case it seems that we had trouble reading the metadata-segment (c.f. maybe here at the bottom).
You might want to check out czicheck which might give a more readable and thorough assessment of the well-formedness/correctness of the CZI-file.

from czicompress.