The installer for focal tries to install openjdk-14-jdk but no such package is available. openjdk-11-jdk or openjdk-13-jdk should work, though.

The srec files unpacked by the generic carver often contain additional data at the end which leads to the plugin failing with the error

'utf-8' codec can't decode byte 0xf4 in position 148638: invalid continuation byte

Stopping at the point where this error occurs should be an easy fix

For btrfs.img in this repo:

virt-filesystems -a ./fact_extractor/plugins/unpacking/generic_fs/test/data/btrfs.img
guestmount -m /dev/sda -a  ./fact_extractor/plugins/unpacking/generic_fs/test/data/btrfs.img ~/mnt

I haven't looked in depth what filesystems are supported and how it works

In the latest version, binwalk seems to need an additional parameter --run-as=root in order to unpack files.

Extractor Exception: Binwalk extraction uses many third party utilities, which may not be secure. If you wish to have extraction utilities executed as the current user, use '--run-as=root' (binwalk itself must be run as root).
----------------------------------------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/binwalk-2.3.3+fa0c0bd-py3.6.egg/binwalk/core/module.py", line 258, in __init__
    self.load()
  File "/usr/local/lib/python3.6/dist-packages/binwalk-2.3.3+fa0c0bd-py3.6.egg/binwalk/modules/extractor.py", line 147, in load
    raise ModuleException("Binwalk extraction uses many third party utilities, which may not be secure. If you wish to have extraction utilities executed as the current user, use '--run-as=%s' (binwalk itself must be run as root)." % user_info.pw_name)
binwalk.core.exceptions.ModuleException: Binwalk extraction uses many third party utilities, which may not be secure. If you wish to have extraction utilities executed as the current user, use '--run-as=root' (binwalk itself must be run as root).
----------------------------------------------------------------------------------------------------

The unpack-kernel tool which gets called in avm_kernel_image.py tries to call unlzma but can't find it:

ERROR: this script requires unlzma tool which is not found on your system, expected location of the tool is "/opt/app/fact extractor/bin/unlzma"

unlzma seems to be installed but the tool expects it in the same directory.

Fix: adding a symlink to the bin directory should fix this

In certain use cases, I will get less files than expected from a particular firmware. According tot he container logs, the unpacker process ( whichever it might be -- binwalk, for example ) has run the machine out of memory and been killed before it completed. Because the logic for successful unpack is whether or not there are files resultant, if binwalk runs and doesn't finish but does manage to produce files, the unpack will have been deemed 'successful' by the plugin, but in reality it was not successful.

Now, it is noted that the plugins are calling execute_shell_command from common_helper_process. execute_shell_command seems to be dropping exit codes on the floor. Could it be possible to instead use execute_shell_command_get_return_code instead, and handle the case in which a process returns something non-zero?

By the way the plugins return data, it seems to be a dictionary, so adding the return code into the dictionary along with the output shouldn't break anything. Perhaps something can be done in unpack.py to handle this? I am willing to help in this effort.

Currently I have a raw disk with multiple partitions. However one partition doesn't contain a valid fs (EG, if there is a start of a valid partition, but the file is truncated). This causes kpartx to only enumerate the valid partition but still throw an error. So the genericFS plugin doesn't process the correct partitions at all and just exits. This then results in binwalk being unleashed, which of course ends badly ;)

sample:

add map loop0p1 (253:0): 0 4096 linear 7:0 2048
add map loop0p2 (253:1): 0 6144 linear 7:0 6144
add map loop0p3 (253:2): 0 4096 linear 7:0 12288
device-mapper: reload ioctl on loop0p4  failed: Invalid argument
create/reload failed on loop0p4

Creating this Issue so that there is a record of it. I have updated mbr.img file to test for the issue. I will fix the code to ensure the existing tests pass and submit a PR shortly.

As of now binwalk is installed using a fork. The fix from the fork has made it to master for a while now, so we should be able to switch back again.

The FACT extraction process ignores empty files and duplicates such that after extraction, all such files are dropped before they are moved to our destination folder. This can lead to multiple unwanted side effects. To preserve a full extraction fact_extractor should introduce parameters that makes keeping all files optional through docker wrapper and direct execution.

This should be the critical code switch that has to be parametrized.

Currently there is no unpacker for base64 encoded data streams

Running fact_extractor with 0.zip gives me wired results.
Here is the output of tree in the respective extraction directory:

.
├── files
│   └── 0.zip
├── input
│   └── 0.zip
└── reports
    └── meta.json

The report tells us that one file was extracted which is the file itself.
They even have the same hashes.

What happened here?

HP PJL extraction writes the raw PJL commands to the output. These raw commands can contain non-printable characters, and is stored as a bytes object. The fact_extractor unpacker writes the output of each extractor as serialized JSON to the meta.json file, and therefore when HP-PJL extraction results are written an exception of 'TypeError: Object of type 'bytes' is not JSON serializable' occurs.

To reproduce this, try to unpack any file which goes through the HP PJL extractor via the fact_extractor container.

In the README, there is an example on how to run it using privileged mode, sharing the /dev folder.

Is this required? I mean, in case fact_extractor is not run with privileges, there would be limitations?

I have not inspected the problem very thoroughly, but in short it does not work because something in the freetz-ng installation fails.
To get the freetz unpacker working without the fact_extractor you have to do:

# These variables instruct freetz to use the system versions of lzma and sfk
export UNLZMA=$(which lzma) 
# http://stahlworks.com/dev/swiss-file-knife.html
export SFK=$(which sfk)
tools/unpack-kernel -u kernel.img

Currently the first thing we do is copy the whole directory in the docker image.
So when changing anything the whole thing is rebuild from scratch.

A use case would be working on an unpacker. I don't want to wait 10 minutes just to text if it works.
There are two solutions I'd propose.

Making the image more mount firendly

A workaround would be to mount the fact_extractor directory in the container when developing.
This does not work for the following reasons.

Bin dir

Binaries are installed in the repo. Easy to change. I'd suggest /usr/local/bin.

venv

The venv is installed in the repo. Easy to change. I'd siggest /venv for the docker image.

Uncoupeling the installer

The installer is tightly coupled to the rest of the code.
If we could execute it separately then we could execute the installer first (which takes most of the time)
and then copy the rest of the directory.
Of course if you change sth in the plugin installers the image would not work which is not ideal.

application/x-arj is supported through patool by applying the arj executable.

arj throws an error if the given file does not end on .arj. As FACT files generally have the filename [length]_[sha256], this leads to errors.

Thus arj has to be handled seperately.

This docker container provides a useful capability, but it is very limited in that it only extracts a single file and does not attempt to recursively extract all additional files. It would be much more useful if a '--recursive' flag was added that enabled extraction to attempt to unpack an entire nested firmware using the same overall methodology that FACT_core uses.

Extracting US_W312AV1.0BR_V2.0.0.3(1330)_EN_TDE.bin (attached in ZIP archive: US_W312AV1.0BR_V2.0.0.3(1330)_EN_TDE.zip) leads to an UnicodeDecodeError in the uboot plugin.

Output:

$ ./extract.py US_W312AV1.0BR_V2.0.0.3\(1330\)_EN_TDE.bin 
[2021-02-06 08:49:10][unpack][WARNING]: Uboot could not extract any file from /tmp/extractor/input/US_W312AV1.0BR_V2.0.0.3(1330)_EN_TDE.bin -> generic carver fallback
[2021-02-06 09:49:11][extract][WARNING]: Now taking ownership of the files. You may need to enter your password.
{
    "plugin_used": "generic_carver",
    "plugin_version": "0.8",
    "error": "<class 'UnicodeDecodeError'>: 'utf-8' codec can't decode byte 0x80 in position 6: invalid start byte",
    "analysis_date": 1612601350.8576572,
    "number_of_excluded_files": 0,
    "0_FALLBACK_Uboot": "Uboot (failed) -> generic/carver (fallback)",
    "output": "/bin/sh: 1: Syntax error: \"(\" unexpected\n",
    "filter_log": "",
    "number_of_unpacked_files": 0,
    "number_of_unpacked_directories": 0,
    "summary": [
        "packed"
    ],
    "entropy": 0.892847092329587
}

I will provide a PR which addresses this issue.

/usr/bin/fakeroot: 175: /usr/bin/fakeroot: /opt/app/fact extractor/plugins/unpacking/squashFS/code/../bin/unsquashfs4-avm-be: not found

any avm firmware should reproduce this

• the binwalk/raw extractor prepends a _filename.extracted/ directory which can be safely removed (because it does not add any additional information)

ascii85 was found in samples (e.g. cafc32c77a1d7a0fc9ec939219be8d097cf84104f8382fa0425159f46f402192_37834625) and currently there is no unpacker

Tektronix extended HEX is a binary hex format and was found in samples (e.g. db52848359d4152f2f60b155e1e32dd7d38f75079da897ba1130604b5adcedba_383837).
Currently, we have no unpacker for the format, but it could be realized easily using either objcopy or the SRecord tool.

What I'm finding in a bunch of cases is a need for unpack plugins to have access to multiple files at the same time in order to complete a full unpack. Here is an example... I ported the following go code to python:
packsparseimg.go

The idea with this is that there is a rawprogram*.xml file that holds information about how to unpack the img files that go with it. So for instance there can be several userdata_x.img files, and the xml file contains the offsets of where to write the individual files in order to resemble the outer image.

This isn't a problem if the outer container format is known. For instance, if it's a tar file I could add functionality to the patool plugin to check for rawprogram*.xml files and process them. But now if the outer container is a 7z file, then I have to duplicate that functionality to the 7z plugin. I can put this functionality in a helper class that's available from several plugins, but that doesn't feel very extensible.

What I'd like to see is a different type of plugin that registers a file pattern to look for in extracted files, if found, it calls this plugin with the directory that contains the extracted files so that it can re-assemble the .img files into something another plugin can then unpack in isolation.

In my opinion this makes it more module and extensible, but would like your opinion about this. Are you open to this plan? If I implement this in a fork will you consider a PR with this functionality? If this doesn't sound appealing or you have objections I'd love to hear them and I'll adjust accordingly.

I have an idea on how to fix this that I'm going to push soon, but here are the details of the problem.

This is the code that determines if a password was required to unpack the archive:

        if 'Wrong password' not in output:
            if 'AES' in output:
                meta['password'] = password
            break

if 'AES' in output is the source of both the false positives and false negatives.

False Positives

On my machine, AES is in all outputs of 7z because these lines are always at the top:

7-Zip [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,64 bits,4 CPUs Intel(R) Core(TM) i7-10875H CPU @ 2.30GHz (A0652),ASM,AES-NI)

False Negatives

If 7z extracts an encrypted zip, there is nothing in the output to indicate that the archive required a password (unlike when an encrypted 7zip file is extracted). Here's some example output:

$ 7z x -ppassword archive.zip 

7-Zip [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,64 bits,4 CPUs Intel(R) Core(TM) i7-10875H CPU @ 2.30GHz (A0652),ASM,AES-NI)

Scanning the drive for archives:
1 file, 433 bytes (1 KiB)

Extracting archive: archive.zip
--
Path = archive.zip
Type = zip
Physical Size = 433

Everything is Ok

Folders: 2
Files: 1
Size:       3
Compressed: 433

stuffit archives extraction has long been done by unar now, but unstuff is still installed. This can be dropped.

File already knows the signature for bzImage or other vmlinux files, only a mime is missing.
Also there is an official script to do that (duh).

Adding this would make file trees of linux based firmware look a lot nicer.

the header of srec files seems to be optional so this should be considered during unpacking
i.e. 40ba21d58b0e5d51517b9462945a4a65341c63b0ff45f468de9fe9b8d96155bc_799985

python-lzma packege is needed to run the extractor

Running Archlinux with kernel 6.5.5-arch1-1 the docker image is extremely slow because of the usage of fakeroot.

References: moby/moby#45436

sorry, i not use --privileged launch docker

Filesystems can contain FF padding, leading to the RAW unpacker tearing the FS apart.
The resulting blocks will not be extracted correctly afterwards.

There are some HP printers that use pjl for updates but function different from the supported ones. One example is [1].
The compession seems to be different from the truncated lzma streams support already.
See discussion in #7 for more information.

[1] ftp://ftp.hp.com/pub/softlib/software13/printers/ojp6970/1910a/OJP6970_1910A.exe

FIles that are dropped might be owned to uid 0. Besides other consequences this enforces tools that integrate extraction (e.g. FACT) to have some root privileges to process them. One could add some parameter that is passed to docker at run time that contains the uid of the spawning user (e.g. 1000, 1001 ..).
WIth this, the extractor could change ownership of files to this to remove the necessity for privileged processing.

It would be nice to extend extract.py to recursively extract all files until a certain termination criterion is met (e.g. no more extractions possible, maximum depth reached, ...).

When using fact_extractor as a docker image, the only way to reconfigure it is to create a custom image. It would be nice if you could reconfigure it on the fly, both when using it stand-alone and in FACT_core.

I'm a bit unsure, whether this is the correct repository for this issue - while the description states:

Wraps FACT unpack plugins into standalone utility.

it seems at the same time the only repository, which contains code dedicated to unpack the various firmware formats.

Nonetheless I'll try to show/explain here, why your attempts to unpack/analyze the firmware for AVM's model 4020 got failed.

If a device model by AVM uses a "combined image" for the firmware, it consists of a kernel image, immediately followed by a filesystem image using SquashFS format. For version 4 of SquashFS, AVM has changed the official format (which uses only "little endian" byte order anymore) to an own, where some data is still stored with "big endian" byte order, if the platform uses BE storage order.

If the platform of the device is a MIPS processor, the SquashFS image isn't stored as one continuous data stream - it contains a gap at an offset, that will be loaded mapped to physical memory address 0xC00000, where the NMI vectors will be looked up in a "very basic state" of processor initialization. The size of this gap varies with the processor and its architecture. If the loader size in flash memory is 0x20000 (and the kernel/filesystem partition starts after the loader partition), this gap will be found in the (single) file kernel.image at offset 0x00BE0000. (EDIT: The load address doesn't really matter, see my earlier post here: https://www.ip-phone-forum.de/threads/%C3%9Cbersicht-von-fritz-boxen-mit-junk-bytes-im-squashfs-image.286318/)

According to your file avm_kernel_image.py (https://github.com/fkie-cad/fact_extractor/blob/master/fact_extractor/plugins/unpacking/avm_kernel_image/code/avm_kernel_image.py#L26) you're trying to split these images into the kernel part and take the whole rest as filesystem image (that's how find-squashfs works). If the filesystem part doesn't contain the NMI vector gap, everything works as expected - but if the SquashFS image contains this gap, the (later) extraction process for the SquashFS image will fail.

There are two options to handle this case correctly ... either you let remove the NMI vector gap from the extracted filesystem image (see this shell script from Freetz project: https://github.com/Freetz/freetz/blob/master/tools/remove-nmi-vector) or you use an extension to the unsquashfs binary (you're using the proper sources already and the files copied from your Freetz container during installation support these extensions) and unpack the SquashFS data directly from the kernel.image file:

peh@vidar:~> mkdir /tmp/FB4020
peh@vidar:~> cd /tmp/FB4020
peh@vidar:/tmp/FB4020> git clone https://github.com/PeterPawn/yf_bin
Cloning into 'yf_bin'...
remote: Enumerating objects: 99, done.
remote: Counting objects: 100% (99/99), done.
remote: Compressing objects: 100% (78/78), done.
remote: Total 926 (delta 22), reused 90 (delta 19), pack-reused 827
Receiving objects: 100% (926/926), 75.25 MiB | 3.54 MiB/s, done.
Resolving deltas: 100% (180/180), done.
Updating files: 100% (535/535), done.
peh@vidar:/tmp/FB4020> wget -q -O - https://ftp.avm.de/fritzbox/fritzbox-4020/deutschland/fritz.os/FRITZ.Box_4020.de-en-es-it-fr-pl.147.07.01.image | tar -x -O ./var/tmp/kernel.image > avm_kernel_image.image
peh@vidar:/tmp/FB4020> yf_bin/squashfs/unsquashfs4-be -stat -scan avm_kernel_image.image
Found a valid superblock at offset 0x001C5F00 while scanning avm_kernel_image.image.
NMI vector found at 0x00BE0000, size=4096
Found TI checksum (0xD5030BF4) at the end of the image.
Found a valid big endian SQUASHFS 4:0 superblock on avm_kernel_image.image.
Creation or last append time is not available because of modified AVM-format (mkfs_time == bytes_used)
Filesystem size 13360.20 Kbytes (13.05 Mbytes)
Compression xz
Block size 65536
Filesystem is exportable via NFS
Inodes are compressed
Data is compressed
Fragments are compressed
Always-use-fragments option is not specified
Xattrs are not stored
Duplicates are removed
Number of fragments 254
Number of inodes 2691
Number of ids 1
peh@vidar:/tmp/FB4020> sudo yf_bin/squashfs/unsquashfs4-be -scan -no-progress avm_kernel_image.image
Found a valid superblock at offset 0x001C5F00 while scanning avm_kernel_image.image.
NMI vector found at 0x00BE0000, size=4096
Found TI checksum (0xD5030BF4) at the end of the image.
Filesystem on avm_kernel_image.image is xz compressed (4:0)
Parallel unsquashfs: Using 2 processors
2508 inodes (2971 blocks) to write

created 2042 files
created 183 directories
created 454 symlinks
created 12 devices
created 0 fifos
peh@vidar:/tmp/FB4020>

Because the new option -scan does not affect any "pure" SquashFS image, it doesn't matter, whether it's always used to search for the SquashFS superblock - you may unpack "plain" SquashFS images, too, while using this option.

Using the option -scan, the superblock offset is determined first and then the existence of the NMI vector gap is checked. If the NMI vector gap is present, it will be skipped while reading/unpacking files from this image. But this is checked/done only, if the new option was specified while calling the tool.

Even if the Freetz implementation (Freetz/freetz@ba45d88) differs slightly from my own (PeterPawn/YourFreetz@9f89c49), both serve the same result - and you should be able now to unpack the 4020 firmware, too.

And by the way ... this is the same procedure for all FRITZ!Box models, which are using a MIPS architecture and this "single image format" for its software - usually these devices have NOR or SPI flash only, because with NAND flash the firmware structure is a different one.

The 4020 was the only model with this structure in your portfolio - otherwise you would have problems unpacking the firmware for other AVM models, too. Try the 7390 firmware (it's still using SquashFS3 format) or 7360v2 (this is a SquashFS4 image with BE byte order) as other examples of these MIPS-devices with NMI vector gap ... if you want to enhance/verify/test your unpacker.

I'm not providing a patch for your file(s), because the if-then construct in avm_kernel_image.py looks odd to me, too. I can't see, why you want to unpack the contained kernel only, if the present file does not contain a SquashFS image - this makes only sense, if you're calling this function recursively and the splitted kernel image from the first call is unpacked with a second call later. This makes the logic a bit obscurely to me - so I'll better let you rule, which changes are needed.

And looking into squash_fs.py (https://github.com/fkie-cad/fact_extractor/blob/master/fact_extractor/plugins/unpacking/squashFS/code/squash_fs.py), the needed changes seem to be more expansive ... currently there aren't different command line options (per tool) while "probing" the right tool to unpack data.

As long as the oldest SquashFS image to process uses SquashFS3 format (and not an earlier one), the tools for SquashFS4 format will be able to unpack this, too, and another try with unsquashfs3-multi should not change the results anymore, if the v4 tools were unable to unpack a file.

Does fact_extractor have the function of extract recursively?

What's needed AFAIK:

• Use py3.12 in Dockerfile, so py3.12 compatibility
• replace simg2img with android-sdk-libsparse-utils
• fkie-cad/entropython#1