pmqs / zipdetails Goto Github PK

zipdetails - Display details about the internal structure of Zip files.

Perl 90.05% Java 3.68% Python 4.18% Batchfile 2.09%

zip zip64 metadata zipfiles appnote pkzip winzip winzip-aes

zipdetails's Issues

weak encryption + streaming does not set local CRC value to zero

appnote section 4.4.4 says this about the CRC value when streaming is enabled

    Bit 3: If this bit is set, the fields crc-32, compressed 
           size and uncompressed size are set to zero in the 
           local header.  The correct values are put in the 
           data descriptor immediately following the compressed
           data.  (Note: PKZIP version 2.04g for DOS only 
           recognizes this bit for method 8 compression, newer 
           versions of PKZIP recognize this bit for any 
           compression method.)

In practice, if weak encryption is enabled along with streaming, the CRC value is not set to set to zero. Below is from Info-ZIP source code where it sets the CRC value when weak encryption is enabled

        /* Traditional encryption with an extended header implies that
          * we use (the low 16 bits of) the MS-DOS modification time
          * instead of the real (unknown) CRC as the "CRC" for the
          * pseudo-random seed datum.  (The high 16 bits of the "CRC"
          * are used in the Traditional encryption header.)
          *
          * This use of file time as the CRC and then use of a data
          * descriptor to hold the CRC is not standard, but is readable
          * by various utilities out there.  We do it this way to avoid
          * reading a file more than once and to support streaming.

The zipdetails code currently expects the appnote behaviour. That needs to change so that it uses the de-facto standard that is used by the implementations.

Compression methods & version needed to extract

Check version needed to extract against compression method. See 4.4.3.2 of APPNOTE

t/files/0010-apache-commons-compress/input.bin is too large

The huge input.bin (> 2GB) is part of the App-zipdetails-2.101 distribution --- I assume this is an accident and it should not be included?

Error decoding `Xceed Unicode extra field` (0x0x554e)

1. Overview

Although I haven't actually run it, I found an error in the way his subroutine decode_Xceed_unicode in the script zipdetails was handled, so I would like to report it.

2. Extra field `0x554e` format

As a result of experiments and analysis, it appears that the format of the extra field 0x0x554e is as follows.

2,1 For central directory headers

offset (bytes)	length (bytes)	value
0	4	signature (0x5843554e)
4	2	Half the number of bytes in the entry name encoded in UTF-16
6	2	Half the number of bytes in the comment encoded in UTF-16
8	(4th byte value) * 2	byte array of entry name encoded in UTF-16
8 + (4th byte value) * 2	(6th byte value) * 2	Byte array of comment encoded in UTF-16

2.2 For local headers

offset (bytes)	length (bytes)	value
0	4	signature (0x5843554e)
4	2	Half the number of bytes in the entry name encoded in UTF-16
6	(4th byte value) * 2	byte array of entry name encoded in UTF-16

3. About the contents of script `zipdetails`

The comment of the subroutine decode_Xceed_unicode in the script zipdetails/bin/zipdetails says Found the Null prefix.
My guess is that the reason why there appears to be a 2-byte null at the beginning of the entry name is probably because the comment length field in the 8th byte in "2.1 For central directory headers" happens to be 0.

I don't understand the perl language very well, so I don't know how to fix it. sorry.

Invalid unicode codepoint warning

When displaying the contents of the zip file at https://github.com/ByteHackr/unzip_poc/blob/main/CVE-2022-0529/testcase this warning gets output

Operation "pattern match (m//)" returns its argument for non-Unicode code point 0x2020276 at zipdetails line 2446.

Fix is to use "utf-8-strict" when decoding in the code. That traps invalid code points.

cpantesters smokes failing on Windows

The GitHub windows pipeline tests do not fail, but the cpantesters smokes are all failing the 002-main.t tests.

Example is http://cpantesters.org/cpan/report/81e1b37c-6bf3-1014-a333-e3f4c4025101

The output from these tests looks fine, so likely an EOL issue.

Add support for OpenVMS zip files

Sample OpenVMS zip file available at https://ftp.zx.net.nz/pub/archive/ftp.hp.com/pub/openvms/opensource/

Add more test for directory entries

Check for trailing "/" when External Attributes has directory flag(s) set.
Check if directory bit(s) set in External Attributes when training "/" is present.
Check that directory entry does not have uncompressed payload (APPNOTE 6.3.10, sec 4.3.8)
Check if compressed payload present & uncompresses to zero bytes.
Check that Extract Version is set to 2.0 or greater (APPNOTE 6.3.10, sec 4.4.3.2)

Notes for behaviour of unzip executables when dealing with directories without the trailing "/"

On Linux Info-ZIP 6.0 will extract it to an empty file
On Linux 7z & bsdtar (aka libarchive) extract it as a directory if the Unix bit is set, but not if the only the dos flag is set

Removal of 32 bit support break core perl i386 test t/porting/utils.t

Current core fails test on 32 bit architectures with:

# Failed test 83 - utils/zipdetails compiles at porting/utils.t line 85
#      got "Integer overflow in hexadecimal number at utils/zipdetails line 1432.
Integer overflow in hexadecimal number at utils/zipdetails line 2247.
Integer overflow in hexadecimal number at utils/zipdetails line 2248.
Integer overflow in hexadecimal number at utils/zipdetails line 2249.
Integer overflow in hexadecimal number at utils/zipdetails line 2250.
Integer overflow in hexadecimal number at utils/zipdetails line 2251.
Integer overflow in hexadecimal number at utils/zipdetails line 2252.
Integer overflow in hexadecimal number at utils/zipdetails line 2253.
Integer overflow in hexadecimal number at utils/zipdetails line 2254.
utils/zipdetails syntax OK
"
# expected "utils/zipdetails syntax OK
"
t/porting/utils .................................................. FAILED at test 83

I created Perl/perl5#19618 which detects the warning produced by this patch.

Decoding of DOS Attributes in External Attributes is wrong & incomplete

First 16 bits of the decoded External Attributes in Central header should map to DOS attributes. Decoding of this field uses an incorrect and incomplete bitmask

Code uses 0x0100 for Offline, 0x200 for Not Indexed & and 0x400 for Encrypted. Values should be 0x1000, 0x2000 and 0x4000 respectively.

The field names for the 0x0100, 0x0200, 0x0400 and 0x0800 value should be Temporary, Sparse, Reparse Point and Compressed.

In practice the 0x4000 and 0x8000 bits are used for a different purpose.

7z/p7zip uses the 0x8000 bit to signal that the high 16-bits are Unix attributes.

Mac/iOS uses 0x4000 bit for some unknown purpose.

References

iOS ipa files

These are zip file. Some use compression method 99 to store a payload compressed with LZFSE

Method 99 is already registered in APPNOTE for AES encryption.

May be able to infer LZFSE if it has a signature.

Modifying header

Hi
Do you plan to add the possibility of modifying the header? Like atime, ctime, mtime etc.
Thanks.

NTFS Timestamps displayed in wrong order

ZIPDETAILS 6.3.9 section 4.5.3 says the order should be modification, access, creation

         Tag        Size       Description
         -----      ----       -----------
         0x0001     2 bytes    Tag for attribute #1 
         Size1      2 bytes    Size of attribute #1, in bytes
         Mtime      8 bytes    File last modification time
         Atime      8 bytes    File last access time
         Ctime      8 bytes    File creation time

the code fetches & displays the timestamps in order modification, creation, access.

Unzip detects overlap with APK file

test-services-1.1.0.apk sourced from https://issues.apache.org/jira/browse/COMPRESS-562

unzip doesn't like it

$ unzip -t !$
unzip -t test-services-1.1.0.apk
Archive:  test-services-1.1.0.apk
error [test-services-1.1.0.apk]:  missing 237 bytes in zipfile
  (attempting to process anyway)
error: invalid zip file with overlapped components (possible zip bomb)

$ zipdetails -v test-services-1.1.0.apk


00000 00004 50 4B 03 04 LOCAL HEADER #1       04034B50
Can't use an undefined value as an ARRAY reference at /media/paul/Linux-Shared/base/perl/ext/zipdetails/main/bin/zipdetails line 831.

Extended Timestamp [UT] says Change Time, should be Creation Time

Currently displaying "change" but should be "creation".

Probably set to "change" because that is the what *nix uses.

Needs investigation.

Uninitialized values warnings

Seeing these warnings with zip file at https://github.com/ByteHackr/unzip_poc/blob/main/CVE-2022-0530/testcase

Use of uninitialized value $centralValue in string ne at zipdetails line 1726.
Use of uninitialized value $ver in division (/) at zipdetails line 2795.
Use of uninitialized value $ver in modulus (%) at zipdetails line 2796.

add new Info-ZIP extra fields


          Value           Size       Description
          -----           ----       -----------
 (Stream) 0x6C78          2 bytes    Tag for this extra block type ("xl")
          Size            2 bytes    Data size for this block
          Bitmap          m bytes    Determines which fields below this
                                     point are included
          Version Made By 2 bytes    As in Central Directory File Header
          Int File Attrs  2 bytes    As in Central Directory File Header
          Ext File Attrs  4 bytes    As in Central Directory File Header

          This extra block is used to include information in the local file
          header that previously has only been included in fields in the
          central directory file header for this entry.  This extra field
          is intended for use only in the local header, not the central
          directory.

          The information in this local extra field must match the information
          in the central directory header for this entry, or this extra field
          should be considered invalid.  The purpose of this local extra field
          is to provide in the local header a copy of central directory
          information required for the proper extraction of entries so that
          entries can be extracted as they appear in the stream.

          If this streaming extra field is present in the central directory
          header, the information must match the information already in the
          central directory header as well as information in the local header
          streaming extra field, if present.


          The bitmap identifies which fields actually appear in this block.
          Fields always appear in the exact order listed in the bitmap,
          starting from byte zero/bit zero.  If fields are added to this extra
          block in the future, they will appear at the location (in the order)
          indicated by the respective bit in the bitmap.  A 1 at a bit position
          in the bitmap indicates presence of the corresponding field block,
          while a 0 indicates the field block is absent.  This allows for
          removing obsolete fields in the future.  A map bit may map to
          multiple fields, where a 1 means those fields are present.  The size
          of each field block must be determined, either as a fixed size field
          or by including a length count at the start of the field block.

          The bitmap consists of as many bytes as needed to define the
          contents of this extra block.  For each byte in the bitmap, bit 7 is
          1 if and only if there is a following byte in the bitmap.  Bit 7
          will be 0 in the last byte of the bitmap.  Currently only one map
          byte is used, so bit 7 of that byte is 0.  Bitmap bytes are stored
          in order, starting with Byte 0 immediately following Size.  For
          example, for a three byte BitMap:

            +--+--+--+--+--+--+--+--+
            |  Tag                  |
            +--+--+--+--+--+--+--+--+
            |  Tag                  |
            +--+--+--+--+--+--+--+--+
            |  Size                 |
            +--+--+--+--+--+--+--+--+
            |  Size                 |
            +--+--+--+--+--+--+--+--+
            Bitmap (showing locations of bit numbers 0, 1, and 2):
            +--+--+--+--+--+--+--+--+
            | 1|  |  |  |  | 2| 1| 0|  Byte 0
            +--+--+--+--+--+--+--+--+
            | 1|  |  |  |  |  |  |  |  Byte 1
            +--+--+--+--+--+--+--+--+
            | 0|  |  |  |  |  |  |  |  Byte 2
            +--+--+--+--+--+--+--+--+
              ^
              |
              +-------- Bit 7

            +--+--+--+--+--+--+--+--+
            |  Version Made By      |
            +--+--+--+--+--+--+--+--+
            ...

          The current field blocks and bitmap mappings are shown below.
          (MB = Map Byte (numbered starting from 0), BN = Bit Number,
          MV = Mask Value and FBS = Field Block Size = number of bytes in
          this field block when present.)  Currently only one byte is
          needed for the Bitmap.  This one byte map consists of the
          following bits ORed together:

            MB  BN   MV  FBS Description
            --  --   --  --- ------------
             0   0    1   2  "version made by" field is included
             0   1    2   2  "internal file attributes" field is included
             0   2    4   4  "external file attributes" field is included

          The bitmap to include all these fields would be the one byte (bit 0
          on the right):

            00000111

          A user of this extra block should verify that this is a correctly
          formatted block by summing the expected sizes (FBS) of each present
          field block, adding to this the size of the bitmap in bytes, and
          comparing to Size.  If they do not match, this extra block should
          not be used.



         -Info-ZIP Placeholder Extra Field:
          ================================

          This extra field holds no data.  It is only used to reserve space
          in the local header extra field block for possible use by another
          extra field.  Currently this is only used to reserve space for the
          Zip64 local extra field.  No data should ever be stored in this
          extra field and it should always be ignored.
          (Last Revision 20150605)

          Value         Size        Description
          -----         ----        -----------
  (PHold) 0x4850        Short       tag for this extra block type ("PH")
          TSize         Short       total data size for this block
          Data          Variable    see below

          The size of this extra field is selected to match the size of the
          extra field it is reserving space for.  The purpose of this extra
          field is to reserve space in the extra field block so that another
          extra field can use the space later.

          The main use currently is to reserve space for the Zip64 local
          header extra field when the size of the input data is not known
          (for instance, when the input is a stream).  If, once the data is
          read, Zip64 is needed, this Placeholder extra field can be replaced
          by the Zip64 local extra field.  If not, the Placeholder remains
          and should be ignored.  This allows rewriting the local header
          after data is read without it changing size.

          Though the contents of this extra field should never be used and so
          does not need to be specified, it is recommended that the Data field
          be written with all zero (character code 0x00) bytes.

Complains about length of `0x5855` data field.

Why is zipdetails happy with a length 12 0x5855 extensible data field in the local header:

000000025 Extra ID #0001        5855 'UX: Unix Extra type 1'
000000027   Length              000C
000000029   Access Time         62DF1FC7 'Mon Jul 25 15:57:11 2022'
00000002D   Mod Time            62DF0738 'Mon Jul 25 14:12:24 2022'
000000031   UID                 01F5
000000033   GID                 0014

but then complains about the exact same field in the central directory?

CC90F4A05 Extra ID #0001        5855 'UX: Unix Extra type 1'
CC90F4A07   Length              000C
# ERROR: Offset 0xCC90F4A07: 'Length' field in 'Extra ID' 0x5855 (Unix Extra type 1) invalid: expected 0x8, got 0xC

Fails if timezone is not UTC

It seems that there are timezone-related test failures in App-zipdetails-2.105. E.g. with TZ=Europe/Berlin there are differences in the "Last Mod Time" output:

# 000A Last Mod Time         5277983D 'Tue Mar 23 18:01:58 2021'

# 000A Last Mod Time         5277983D 'Tue Mar 23 19:01:58 2021'

pmqs / zipdetails Goto Github PK

zipdetails's Issues

1. Overview

2. Extra field 0x554e format

2,1 For central directory headers

2.2 For local headers

3. About the contents of script zipdetails

References

Recommend Projects

Recommend Topics

Recommend Org

2. Extra field `0x554e` format

3. About the contents of script `zipdetails`