Giter VIP home page Giter VIP logo

avbroot's Introduction

Signing keys

All signed commits, tags, and artifacts I create are signed with one of the following keys. Starting in 2023, unless GPG is required, most things will be signed with the SSH key.

This repo is pushed to Codeberg, Gitlab, and Github. All three repos should be identical and show commits signed with the keys below.

SSH signing key

  • Public key: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDOe6/tBnO7xZhAWXRj3ApUYgn+XZ0wnQiXM8B7tPgv4

GPG signing key

  • Fingerprint: 2233C479609BDCEC43BE9232F6A3B19090EFF32C
  • Signing subkey: 4E8D2BD62F5B7EFC4264CE03A923B14FECFFB81E
  • Public key: gpg.asc

avbroot's People

Contributors

agentoak avatar bugreportion avatar chenxiaolong avatar pascallj avatar schnatterer avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

avbroot's Issues

Implement FEC generation and validation

Currently, avbroot's AVB logic can only verify and update the hash tree portion of hash tree descriptors. The FEC (forward error correction) data is left untouched. We currently don't modify dm-verity partitions, but if we need to in the future, broken/invalid FEC data could cause the user's device to crash or soft brick in the event of a bit flip that trips dm-verity.

(My original understanding of how things worked)

AVB FEC uses RS(255, 253) (Reed-Solomon codewords with 253 bytes of data and 2 bytes of parity) over GF(256) with the 0x11d primitive polynomial. The data fed into RS is not the raw data, but instead, an interleaving pattern. The interleaving is determined by 3 variables:

  • FEC block size. This is always 4096.
  • RS K value. The K component of RS(N, K), ie. the number of data bytes per codeword. AOSP defaults to 253, which is the only value avbroot will support (because the best Rust library for these calculations uses code generation that requires the K value at compile time). EDIT: avbroot will support everything in [231, 253] (everything allowed by dm-verity).
  • "rounds" value. It's undocumented in AOSP, but is equal to ceil(file size / FEC block size / RS K value).

Interleaving pattern:
* Input offset 0 == output offset 0.
* Each time the input offset is incremented by 1, the output offset is incremented by this value times the block size.
~* Once the input offset cross a multiple of rs_k, the output offset "wraps around" and starts again at the beginning again.`

For example, with file_size=2048000 and rs_k=253 (and thus, rounds=2):

offset 0   -> offset 0       (= 0 + 0   * 2 * 4096)
offset 1   -> offset 8192    (= 0 + 1   * 2 * 4096)
offset 2   -> offset 16384   (= 0 + 2   * 2 * 4096)
...
offset 252 -> offset 2064384 (= 0 + 252 * 2 * 4096)
# rs_k chunk boundary.
offset 253 -> offset 1       (= 1 + 0   * 2 * 4096)
offset 254 -> offset 8193    (= 1 + 1   * 2 * 4096)
offset 255 -> offset 16385   (= 1 + 2   * 2 * 4096)
...
# Pattern repeats until EOF.

EDIT: Better documented in the implementation:

/// The interleaving access pattern can be visualized by placing the file
/// offsets in a two-dimensional grid. For example, when reading a 2072576-byte
/// file for calculating RS(255, 253):
///
/// ```text
/// | <-------- Round 0 --------> | <-------- Round 1 --------> |
/// |-----------------------------|-----------------------------|
/// ^ | 0 1 ... 4095 | 4096 4097 ... 8191 |
/// | | 8192 8192 ... 12287 | 12288 12289 ... 16383 |
/// rs_k | 16384 16385 ... 20479 | 20480 20481 ... 24575 |
/// | | ....... ....... ... ....... | ....... ....... ... ....... |
/// v | 2064384 2064385 ... 2068479 | 2068480 2068481 ... 2072575 |
/// ```
///
/// A regular sequential read of the file is traversing the grid row-by-row,
/// while an interleaving read is traversing the grid column-by-column. Each
/// column is always `rs_k` items tall, so each column forms the data portion of
/// an RS codeword. The number of columns is always a multiple of the FEC block
/// size. Since RS operates on fixed-size codewords and a file size might not
/// always fill the grid completely, out-of-bounds offsets are treated as if
/// they contain a `\0` byte.
///
/// All operations are multithreaded with I/O operations parallelized at the
/// "round" level and RS operations parallelized at the column level.

When calculating this FEC data, avbroot will probably default to reading everything into memory with an option to use mmap for memory-constrained systems. The access pattern is probably going to cause a whole lot of thrashing with mmap due to it needing to read one byte at a time all over the file. EDIT: avbroot now uses a much smarter way of doing the interleaved reads, inspired by cryptsetup's veritysetup.

Once implemented (already have a working proof of concept), avbroot avb verify and avbroot ota verify will be updated to check the validity of the FEC data.

Cannot run script, Missing dependencies or something

Sorry to bother you with this, I have been trying to get this script to work for a while now but I am not sure what steps I need to take in order to have all the files required for the patching step

I have built GrapheneOS from source for my Pixel 6 Pro
I am able to successfully build signed packages with my own keys
But I don't know how to sign the boot with a custom magisk patched image

I am able to use the avbtool that I found in my graphene build files
But when I get to the step of patching the OTA, I am missing lots of different files that it's looking for
Many of them I was able to locate from here https://android.googlesource.com/platform/build/+/master/tools/releasetools
After adding all the dependencies and missing files I could find, I am still unable to run the script without errors of missing things

Just an explanation of how you setup your build environment or something so that you were able to run the script
I am just not able to replicate it and I don't know what you have done to get all those required files
Any help would be greatly appreciated

I believe if I can just insert my own custom patched image earlier in the build process of Graphene then I wouldn't even need this script but I can't figure it out, either the image is unpatched or it's inserted too late and not signed

Problem with signing OTA.zip

Hello! I came here because I have a problem with signing the OTA image. I have tried using Ubuntu 22.04 and macOS Monterey but they either spit out an error or seem to work but also fail. I am using the December 2022 OTA zip directly from Google.

Ubuntu shows:

***** Skipping META-INF/com/android/metadata *****
***** Skipping META-INF/com/android/metadata.pb *****
***** Copying apex_info.pb *****
***** Copying care_map.pb *****
***** Patching payload.bin *****
Traceback (most recent call last):
  File "/media/COMPUTERNAME/USB/avbroot/avbroot.py", line 308, in <module>
    main()
  File "/media/COMPUTERNAME/USB/avbroot/avbroot.py", line 300, in main
    patch_subcommand(args)
  File "/media/COMPUTERNAME/USB/avbroot/avbroot.py", line 234, in patch_subcommand
    metadata = patch_ota_zip(
  File "/media/COMPUTERNAME/USB/avbroot/avbroot.py", line 181, in patch_ota_zip
    properties = patch_ota_payload(
  File "/media/COMPUTERNAME/USB/avbroot/avbroot.py", line 63, in patch_ota_payload
    version, manifest, blob_offset = ota.parse_payload(f_in)
  File "/media/COMPUTERNAME/USB/avbroot/avbroot/ota.py", line 50, in parse_payload
    manifest = update_metadata_pb2.DeltaArchiveManifest()
AttributeError: module 'update_metadata_pb2' has no attribute 'DeltaArchiveManifest'

macOS simply shows:

File "avbroot.py", line 128
zipfile.ZipFile(f_zip_in, 'r') as z_in
                                         ^
SyntaxError: invalid syntax

I have no idea if this script works between operating systems or if I was supposed to use the factory image instead of the OTA zip. Both have java, python3, and all of the other dependencies that are required to run this script. Both the keys and the Magisk APK were both placed inside the avbroot folder and I have also made sure to download the repo through git.

Skip passphrase prompt

Is it somehow possible to skip the passphrase prompt?
I am trying to make an automatic update script for my ROM of choice, and it almost works, but I have to manually press every time once it comes to the actual patching.

[Question] Replace with custom kernel

I'm trying to apply a custom kernel on my Pixel 7 Pro, which contains boot.img,dtbo.img,vendor_dlkm.img and vendor_kernel_boot.img.

In the first trial, I patch the kernel with Magisk only with the following command:

python avbroot.py \
    patch \
    --input $ota_path \
    --privkey-avb $avbkey_home/avb.key \
    --privkey-ota $avbkey_home/ota.key \
    --cert-ota $avbkey_home/ota.crt \
    --magisk $magisk_path \
    --magisk-preinit-device persist \
    --output cheetah-ota-tq3a.230805.001-310bce47.magisk.zip

and everything works fines.

Then I try to add a custom kernel using avbtool and --replace:

avbtool add_hash_footer --image kernel/dtbo.img --partition_name dtbo --partition_size $(stat --format=%s original/dtbo.img)
avbtool add_hashtree_footer --image kernel/vendor_dlkm.img --partition_name vendor_dlkm --do_not_generate_fec
avbtool add_hash_footer --image kernel/vendor_kernel_boot.img --partition_name vendor_kernel_boot --partition_size $(stat --format=%s original/vendor_kernel_boot.img)

python avbroot.py \
    patch \
    --input $ota_path \
    --replace boot kernel/boot.img \
    --replace dtbo kernel/dtbo.img \
    --replace vendor_dlkm kernel/vendor_dlkm.img \
    --replace vendor_kernel_boot kernel/vendor_kernel_boot.img \
    --privkey-avb $avbkey_home/avb.key \
    --privkey-ota $avbkey_home/ota.key \
    --cert-ota $avbkey_home/ota.crt \
    --magisk $magisk_path \
    --magisk-preinit-device persist \
    --output cheetah-ota-tq3a.230805.001-310bce47.custom.magisk.zip

But it doesn't boot anymore and shows No valid operating system could be found. and Your device is corrupt. Did I do anything wrong?

Issues with libfs_avb

Hi!, I'm trying to use this to patch an unofficial Lineage OS 20 build for my Pixel 3a XL, but I'm having some issues.
Everything seems to work fine but after booting in and check dmesg with root I'm seeing this:

[    1.245460] init: [libfs_avb]Invalid hash size: 
[    1.245470] init: [libfs_avb]Failed to verify vbmeta digest
[    1.245480] init: [libfs_avb]Returning avb_handle with status: VerificationDisabled
[    1.245492] init: [libfs_avb]AVB HASHTREE disabled on: /system
[    1.249671] init: [libfs_avb]AVB HASHTREE disabled on: /system_ext
[    1.251388] init: [libfs_avb]AVB HASHTREE disabled on: /vendor
[    1.253754] init: [libfs_avb]AVB HASHTREE disabled on: /product

I tried forcing openssl 1.1 incase that was the issue but that didn't fix anything, and I can't find an existing issue abt this so I don't know where I should go from here.. could I get some guidance? thanks,,

Android 14 support

I'm planning on day 1 support for Android 14 in both avbroot and Custota. It's very likely the current code will just work as-is. Unlike the Java parts of Android, Google doesn't withhold changes to AOSP repos for the OTA-related components. I haven't seen anything in the master branches that would cause trouble.

But I'll open this issue to keep track of problems in case any come up.

Would a honest to god homebrew OTA server be feasible? [feature request, understood to be unlikely]

The general idea is to make one's own home-hosted OTA server instead of native one.
I realize it is way too much work probably but with root (and magisk "strategically" substituting some files) it seems like it might be conceptually possible.

I understand chenxiaolong is probably busy enough just keeping all other projects up to date :-) so a whole custom OTA server + OTA agent may bee too much work, but maybe someone else will get inspired to write and maintain that as a separate project (at least for Pixels which have pretty documented and "standardized" process of updating)

Magisk asks for Additional setup after fresh patched OTA sideload

Device: Pixel 4 XL(coral)
Magisk version: v26.1
LineageOS build: lineage-20.0-20230709-nightly-coral-signed.zip
avbroot version: lastest

I installed patched LineageOS OTA on both A and B slots with these steps:

  1. Unlock bootloader
> fastboot flashing unlock
  1. Flash factory image
> ./flash-all.sh
  1. Erase userdata partition
> fastboot -w
  1. Flash avb_custom_key partition with my custom avb key
> fastboot erase avb_custom_key
> fastboot flash avb_custom_key signing-keys/avb_pkmd.bin
  1. Flash boot partition with extracted boot image from extracted patched OTA
> fastboot flash boot_a lineage-20.0-20230709-nightly-coral-signed_Magisk-v26.1_PATCHED.extracted/boot.img
> fastboot --set-active=a
  1. Reboot to recovery from bootloader

  2. Sideload patched OTA(Current boot slot:a)

> adb sideload output/lineage-20.0-20230709-nightly-coral-signed_Magisk-v26.1_PATCHED.zip
  1. Reboot to recovery from recovery

Sideload patched OTA on the other slot(Current boot slot:b)

> adb sideload output/lineage-20.0-20230709-nightly-coral-signed_Magisk-v26.1_PATCHED.zip
  1. Reboot to system from recovery

Using both methods: patching with --magisk-preinit-device metadata(which is what Magisk uses when I patch stock LineageOS boot image using Magisk app) and --prepatched(with patch LineageOS boot image)
After doing all these step Magisk still asks for Additional setup. Root works fine with apps but Magisk modules can't be installed(results in errors) without Additional setup. If I let Magisk patch boot partition, it would break verified boot.
I speculate it's avbroot's issue. If any additional log is needed, please let me know.

Lost root / Magisk doesn't detect root

I have a Pixel 7 running GrapheneOS with root using this tool. After some updates, the Magisk app stopped working (I had used the hide app feature), so I deleted it. I had initially used an older version of the tool to install the OS and root, and it worked well for a long time. But when I forgot to update the tool first and patched the new OTA with new Magisk versions, the Magisk app no longer worked on the phone, but all the previously set up root apps continued working. I re-patched a new OTA properly, setting the Magisk preinit device, flashed it, and reinstalled the Magisk app on phone. While it opens, it does not seem to detect root. Other root apps continue to work. How to fix this?

Add subcommands for unpacking/packing AVB images

With #146, avbroot's AVB functionality now has 100% feature parity** with AOSP's avbtool. We should expose that to users.

Similar to how there are avbroot boot {unpack,pack,repack,info} commands for boot images, we should also have avbroot avb {unpack,pack,repack,info} for AVB images. The AVB header, footer, and image size fields would be unpacked into avb.toml and the raw image (if any) would be unpacked into raw.img. When packing again, the hashes, hash trees, and FEC data are automatically recomputed and the new image can be signed with the user's key.

As with almost all of avbroot's commands, the goal is to help with an unpack-edit-repack workflow. Creating AVB images from scratch is not a goal, though it's possible by manually creating an avb.toml. I'm not yet committing to a stable data representation inside avb.toml so it may change in future avbroot versions.

An example of how things would work: if someone wanted to modify a signed boot image:

avbroot avb unpack -i boot.img
avbroot boot unpack -i raw.img

# Edit ramdisk or whatever...

avbroot boot pack -o raw.img
avbroot avb pack -o boot.img --key avb.key

(**) Besides RSA8192 keys due to RustCrypto's limitations, but no bootloader I'm aware of supports that anyway.

Magisk "Requires Additional Setup" after sideloading upgrade

Hey,

so far been using CalyxOS (~June I think) + Magisk v26.1 signed with python avbroot on my Pixel 4a (sunfish).
Now I upgraded by sideloading CalyxOS 23412000 + Magisk v26.3 signed with avbroot 2.0.3:

$ ./avbroot ota patch --input sunfish-ota_update-23412000.zip --privkey-avb ~/avbroot/avb.key \
    --privkey-ota ~/avbroot/ota.key --cert-ota ~/avbroot/ota.crt --magisk Magisk.v26.3.apk --magisk-preinit-device metadata
[*] Replacing zip entry: META-INF/com/android/otacert
[*] Copying zip entry: apex_info.pb
[*] Copying zip entry: care_map.pb
[*] Patching zip entry: payload.bin
[*] Extracting from original payload: boot
[*] Extracting from original payload: vbmeta_system
[*] Extracting from original payload: vbmeta
[*] Patching boot images: boot
[*] Patching vbmeta images: vbmeta
[*] vbmeta signature algorithm type changed to Sha256Rsa4096
[*] Compressing replacement images: boot, vbmeta
[*] Generating new OTA payload
[*] Patching zip entry: payload_properties.txt
[*] Generating new OTA metadata
[*] Verifying metadata offsets
[*] Completed after 19.1s
$ adb reboot sideload
$ adb sideload sunfish-ota_update-23412000.zip.patched
Total xfer: 2.00x

Upgrade worked fine and Magisk app had been replaced with the stub that asks to download and install the latest Magisk APK.
However, when agreeing some activity flashes up for a split second, then nothing happens. After several tries I uninstalled the Magisk app and installed Magisk.v26.3.apk manually.

It detects that Magisk is installed (26300 / Zygisk: Yes / Ramdisk: Yes) and lists all my root apps and modules, but tells me Magisk "Requires Additional Setup" and wants me to proceed and reboot. Simply rebooting the device does not help, the prompt returns. Is this fine or will this modify my boot partition?

As far as I can tell my modules and root apps work fine. I also checked that the --magisk-preinit-device is definitely correct. Using the Magisk v26.3 app to patch a fresh boot.img results in PREINITDEVICE=metadata.

Not sure if I did anything wrong or should I just ignore the prompt?

prepatch full GKI fail.

hi, so lets explain my problems I have build full gki witch kernelsu (and other mod) its result 4 img generated : boot.img , vendor_boot.img , vendor_dlkm.img , dtbo.img . flashing only boot.img will no boot regarding to change I made.

I see avbroot support only --prepatched for boot. could you look into it ?

Question: Why it is recommended to compile from source with Magisk for from non-stock firmware?

Is there any benefit gained from from compiling from source instead of downloading and patching non-stock firmware OTA(in my case: LineageOS with MicroG)? or because it might cause unexpected bugs?
Though LineageOS has documentation about signing builds with my own key, I can't find documentation to include Magisk to the building progress, so I think it is more convenient to use avbroot directly on LineageOS with MicroG OTA.
edit: Is it because dm-verity is usually only available on stock ROMs so applying avbroot on non-stock ROMs doesn't help with the security model?(crossed out as AVB 2.0 uses vbmeta for system partition verification)

dmesg doesn't contain "libfs_avb"

I completed first 8 step, but when I try to verify using sudo dmesg | grep libfs_avb , console returned nothing. My device is Pixel 6 (oriole), and I used GrapheneOS, for detailed explanation I'm uploading the output of sudo dmesg command in console

dmesg.log

Ramdisk size incorrect for bluejay images

Since #37, bluejay (Pixel 6a) images are not patched correctly. Tested both January and February images. I get the following error:

***** Copying apex_info.pb *****
***** Copying care_map.pb *****
***** Patching payload.bin *****
***** Extracting vbmeta, boot, vendor_boot from the payload *****
***** Patching boot, vendor_boot *****
Traceback (most recent call last):
  File "/home/pascal/avbroot_home/avbroot/avbroot.py", line 352, in <module>
    main()
  File "/home/pascal/avbroot_home/avbroot/avbroot.py", line 344, in main
    patch_subcommand(args)
  File "/home/pascal/avbroot_home/avbroot/avbroot.py", line 274, in patch_subcommand
    metadata = patch_ota_zip(
  File "/home/pascal/avbroot_home/avbroot/avbroot.py", line 212, in patch_ota_zip
    properties = patch_ota_payload(
  File "/home/pascal/avbroot_home/avbroot/avbroot.py", line 101, in patch_ota_payload
    future.result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result
    return self.__get_result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/pascal/avbroot_home/avbroot/avbroot.py", line 87, in apply_patches
    boot.patch_boot(
  File "/home/pascal/avbroot_home/avbroot/avbroot/boot.py", line 231, in patch_boot
    patch_func(f.name)
  File "/home/pascal/avbroot_home/avbroot/avbroot/boot.py", line 39, in __call__
    self.patch(image_file, boot_image)
  File "/home/pascal/avbroot_home/avbroot/avbroot/boot.py", line 60, in patch
    self._patch(image_file, boot_image, zip)
  File "/home/pascal/avbroot_home/avbroot/avbroot/boot.py", line 68, in _patch
    entries, ramdisk_format = _load_ramdisk(boot_image.ramdisks[0])
IndexError: list index out of range

This does not happen for cheetah.

boot_image contains the following:

Boot image v4 header:
- Kernel size:    24884567
- Ramdisk size:   0
- OS version:     0x0
- Reserved:       00000000000000000000000000000000
- Kernel cmdline: b''

It seems there is something wrong with the sizes, because instead of ramdisk size, it reports kernel size. And the size doesn't seem legit anyway. It's much bigger than you would expect.

FileNotFoundError: [WinError 2] System can not find file

Passphrase for .\avb.key: Traceback (most recent call last): File "D:\avbroot\avbroot.py", line 6, in <module> main.main() File "D:\avbroot\avbroot\main.py", line 494, in main patch_subcommand(args) File "D:\avbroot\avbroot\main.py", line 322, in patch_subcommand passphrase_avb = openssl.prompt_passphrase(args.privkey_avb) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\avbroot\avbroot\openssl.py", line 207, in prompt_passphrase subprocess.check_output(['openssl', 'pkey', '-in', pkey, '-noout']) File "C:\Python311\Lib\subprocess.py", line 466, in check_output return run(*popenargs, stdout=PIPE, timeout=timeout, check=True, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Python311\Lib\subprocess.py", line 548, in run with Popen(*popenargs, **kwargs) as process: ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Python311\Lib\unittest\mock.py", line 1124, in __call__ return self._mock_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Python311\Lib\unittest\mock.py", line 1128, in _mock_call return self._execute_mock_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Python311\Lib\unittest\mock.py", line 1189, in _execute_mock_call result = effect(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^ File "D:\avbroot\avbroot\openssl.py", line 81, in __call__ return self.orig_popen(new_cmd, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Python311\Lib\subprocess.py", line 1026, in __init__ self._execute_child(args, executable, preexec_fn, close_fds, File "C:\Python311\Lib\subprocess.py", line 1538, in _execute_child hp, ht, pid, tid = _winapi.CreateProcess(executable, args, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ FileNotFoundError: [WinError 2] Das System kann die angegebene Datei nicht finden

I sadly have absolutly no idea what is going wrong here, can someone help me? Passphrase is 100% right.

Detecting pre-init device on an already rooted phone (no spare specimen, "main" phone in use) problem/concern

Hi! long story short I don't have a spare device for tests and am a bit wary updating the magisk app without proper OTA update

BUT I can't update to latest magisk "boot image wise" because it wants me to specify the pre-init partition

which I do not know and in order to find out I'd need to update magisk.apk (without new boot.img) then perform a test patching (as specified by readme) and then do proper OTA patch and update

Needless to say this spooks me immensely because it feels like something important may get broken

Is installing latest magisk with a "slightly :) " out of date boot.img known-safe?

Getting a TypeError from bootimage.py, process fails

Hi!
The total output looks like :

`
File "/home/user/androidstuff/avbroot/avbroot.py", line 3, in
from avbroot import main
File "/home/user/androidstuff/avbroot/avbroot/main.py", line 13, in
from . import boot
File "/home/user/androidstuff/avbroot/avbroot/boot.py", line 12, in
from .formats import bootimage
File "/home/user/androidstuff/avbroot/avbroot/formats/bootimage.py", line 115, in
class BootImage:
File "/home/user/androidstuff/avbroot/avbroot/formats/bootimage.py", line 118, in BootImage
f: None | typing.BinaryIO = None,
TypeError: unsupported operand type(s) for |: 'NoneType' and 'type'

`

Is this something broken on my end (python 3.9.2) or not?

Nothing happens when I try to patch OTA zip

I have produced key files, Now patching the ota.zip with:
python avbroot.py \ patch \ --input ota.zip \ --privkey-avb avb.key \ --privkey-ota ota.key \ --cert-ota ota.crt \

It doesn't give any error; looks like it is working but it doesnt do anything, I have tried waiting for half an hour. Is it supposed to take more time?

the contents of ota.zip are boot.img, dtbo.img, and many other img files.
I also tried patching the ota.zip which has payload.bin files. Same issue.
Please check!

old Pixels do not have vendor_boot.img, use boot.img instead

First of all, many thanks for your script! It is much easier than building GrapheneOS from scratch (and it does not build anyway lol if you follow instructions on their official website)

However I had to modify your script to make it work with Pixel 4a, please see a diff attached. You might consider adding some checks to determine the phone model automatically.

avbroot_sunfish.txt

Attempting to patch an OTA ZIP fails with - "The prepatched boot image is not compatible" error

Hi,

I'm attempting to patch an OTA Update ZIP file, and with a KernelSU boot img, and I'm getting the below error:

File "C:\tools\avbroot\avbroot\boot.py", line 371, in patch raise ValueError('The prepatched boot image is not compatible ' ValueError: The prepatched boot image is not compatible with the original: - Kernel module interface version changed: 5.10-android13-4 -> None

I'm using the below command:

python avbroot\avbroot.py patch --input ota.zip --privkey-avb avb.key --privkey-ota ota.key --cert-ota ota.crt --prepatched kernelsu.img --boot-partition @gki_kernel

There's a chance that I'm using an updated KernelSU image than one flashed to my device initially during the "Initial Setup", however, I believe that should be unrelated. Do I need to extract the update, and change something, before doing the step-5 in https://github.com/chenxiaolong/avbroot#usage ?

Full Error:
C:\tools>python avbroot\avbroot.py patch --input ota.zip --privkey-avb avb.key --privkey-ota ota.key --cert-ota ota.crt --prepatched kernelsu.img --boot-partition @gki_kernel
Passphrase for avb.key:
Passphrase for ota.key:
***** Copying apex_info.pb *****
***** Copying care_map.pb *****
***** Patching payload.bin *****
***** Extracting boot, vbmeta, vbmeta_system, vbmeta_vendor, vendor_boot from the payload *****
***** Patching boot, vendor_boot *****
Traceback (most recent call last):
File "C:\tools\avbroot\avbroot\main.py", line 196, in patch_ota_payload
future.result()
File "C:\Users\user\AppData\Local\Programs\Python\Python311\Lib\concurrent\futures_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "C:\Users\user\AppData\Local\Programs\Python\Python311\Lib\concurrent\futures_base.py", line 401, in __get_result
raise self._exception
File "C:\Users\user\AppData\Local\Programs\Python\Python311\Lib\concurrent\futures\thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\tools\avbroot\avbroot\main.py", line 180, in apply_patches
boot.patch_boot(
File "C:\tools\avbroot\avbroot\boot.py", line 444, in patch_boot
patch_func(f.name)
File "C:\tools\avbroot\avbroot\boot.py", line 39, in call
boot_image = self.patch(image_file, boot_image)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\tools\avbroot\avbroot\boot.py", line 371, in patch
raise ValueError('The prepatched boot image is not compatible '
ValueError: The prepatched boot image is not compatible with the original:

  • Kernel module interface version changed: 5.10-android13-4 -> None

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Users\user\AppData\Local\Programs\Python\Python311\Lib\shutil.py", line 620, in _rmtree_unsafe
os.unlink(fullname)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\Users\user\AppData\Local\Temp\tmpj_zczwus\extract\boot.img'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Users\user\AppData\Local\Programs\Python\Python311\Lib\tempfile.py", line 878, in onerror
_os.unlink(path)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\Users\user\AppData\Local\Temp\tmpj_zczwus\extract\boot.img'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\tools\avbroot\avbroot.py", line 6, in
main.main()
File "C:\tools\avbroot\avbroot\main.py", line 725, in main
patch_subcommand(args)
File "C:\tools\avbroot\avbroot\main.py", line 474, in patch_subcommand
metadata = patch_ota_zip(args.input, temp, context)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\tools\avbroot\avbroot\main.py", line 375, in patch_ota_zip
properties = patch_ota_payload(
^^^^^^^^^^^^^^^^^^
File "C:\tools\avbroot\avbroot\main.py", line 128, in patch_ota_payload
with tempfile.TemporaryDirectory() as temp_dir:
File "C:\Users\user\AppData\Local\Programs\Python\Python311\Lib\tempfile.py", line 904, in exit
self.cleanup()
File "C:\Users\user\AppData\Local\Programs\Python\Python311\Lib\tempfile.py", line 908, in cleanup
self._rmtree(self.name, ignore_errors=self._ignore_cleanup_errors)
File "C:\Users\user\AppData\Local\Programs\Python\Python311\Lib\tempfile.py", line 890, in _rmtree
_shutil.rmtree(name, onerror=onerror)
File "C:\Users\user\AppData\Local\Programs\Python\Python311\Lib\shutil.py", line 759, in rmtree
return _rmtree_unsafe(path, onerror)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\user\AppData\Local\Programs\Python\Python311\Lib\shutil.py", line 617, in _rmtree_unsafe
_rmtree_unsafe(fullname, onerror)
File "C:\Users\user\AppData\Local\Programs\Python\Python311\Lib\shutil.py", line 622, in _rmtree_unsafe
onerror(os.unlink, fullname, sys.exc_info())
File "C:\Users\user\AppData\Local\Programs\Python\Python311\Lib\tempfile.py", line 881, in onerror
cls._rmtree(path, ignore_errors=ignore_errors)
File "C:\Users\user\AppData\Local\Programs\Python\Python311\Lib\tempfile.py", line 890, in _rmtree
_shutil.rmtree(name, onerror=onerror)
File "C:\Users\user\AppData\Local\Programs\Python\Python311\Lib\shutil.py", line 759, in rmtree
return _rmtree_unsafe(path, onerror)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\user\AppData\Local\Programs\Python\Python311\Lib\shutil.py", line 603, in _rmtree_unsafe
onerror(os.scandir, path, sys.exc_info())
File "C:\Users\user\AppData\Local\Programs\Python\Python311\Lib\shutil.py", line 600, in _rmtree_unsafe
with os.scandir(path) as scandir_it:
^^^^^^^^^^^^^^^^
NotADirectoryError: [WinError 267] The directory name is invalid: 'C:\Users\user\AppData\Local\Temp\tmpj_zczwus\extract\boot.img'

C:\tools>

Thank you!

error when patching..

Maanz79@DESKTOP-A8ONTUU MINGW64 ~/avbroot (master)
$ python avbroot.py
patch
--input C:/Users/Maanz79/Downloads/ota.zip
--privkey-avb C:/Users/Maanz79/avbroot/avb.key
--privkey-ota C:/Users/Maanz79/avbroot/ota.key
--cert-ota C:/Users/Maanz79/avbroot/ota.crt
--magisk C:/Users/Maanz79/Downloads/magisk.apk

Traceback (most recent call last):
File "C:\Users\Maanz79\avbroot\avbroot.py", line 316, in
main()
File "C:\Users\Maanz79\avbroot\avbroot.py", line 308, in main
patch_subcommand(args)
File "C:\Users\Maanz79\avbroot\avbroot.py", line 217, in patch_subcommand
with tempfile.TemporaryDirectory(dir=util.tmpfs_path()) as key_dir:
^^^^^^^^^^^^^^^^^
File "C:\Users\Maanz79\avbroot\avbroot\util.py", line 38, in tmpfs_path
if os.uname().sysname == 'Linux':
^^^^^^^^
AttributeError: module 'os' has no attribute 'uname'. Did you mean: 'name'?



Hi. I received this some kind of error when patching. Do you know how can i solve this?
Thanks..

Patch Disable OEM Unlocking

Is there a way to patch the disable OEM unlocking UI button? It seems super risky to use software where one click of a button can render my $700 phone useless.

[Question] What are the limitations?

Quick Intro:

I want to use this on my daily driver Bluejay. Due to lack of time to experiment with stuff, I want to get it right the first / second time itself.
I'm currently using GrapheneOS with bootloader locked.

I prefer applying patches to the Official GrapheneOS over building from scratch and planning to use KernelSU given that it provides better hiding features compared to Magisk.

Questions

  • Once if I use AVBRoot to patch and get KernelSU working on my GrapheneOS with bootloader, to what extent can I go with customization?
    • If I modify hosts file, or install modules, or restore backups from apps like Swift, or anything as such, is there a possibility of getting the device bricked given that the bootloader is locked?
    • I personally want Pixel Launcher's recents menu on my phone and there exist a module for that as well, installing such modules would result in brick?

I'm still calculating the things since I cannot take holidays to fix my phone if anything goes wrong...

issue with patching

I tried it on windows and linux and also with different python versions but depending on the python version I get the following:

Traceback (most recent call last): File "C:\avbroot\avbroot.py", line 14, in <module> from avbroot import ota File "C:\avbroot\avbroot\ota.py", line 16, in <module> import ota_from_target_files File "C:\avbroot\avbroot\..\external\build\tools\releasetools\ota_from_target_files.py", line 263, in <module> import care_map_pb2 File "C:\avbroot\avbroot\..\external\build\tools\releasetools\care_map_pb2.py", line 7, in <module> from google.protobuf import descriptor as _descriptor ModuleNotFoundError: No module named 'google'

or

File "avbroot.py", line 127 zipfile.ZipFile(f_zip_in, 'r') as z_in,

Any fix/help would be appreciated.

Unable to build OEMUnlock module in Powershell

Getting the following error message when trying to build the OEMUnlockOnBoot module. I am running Windows 11 with the latest version of Python installed (3.11.4) and the latest Android Studio version installed with the ANDROID_HOME variable set to the correct SDK folder:

PS C:\Users\BNGEE\Desktop\Files\avbroot> python .\modules\build.py
Built module C:\Users\BNGEE\Desktop\Files\avbroot\modules\dist\clearotacerts-1.0.zip
Traceback (most recent call last):
File "C:\Users\BNGEE\Desktop\Files\avbroot\modules\build.py", line 164, in
main()
File "C:\Users\BNGEE\Desktop\Files\avbroot\modules\build.py", line 150, in main
'data': build_dex([os.path.join(module_dir, 'Main.java')]),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\BNGEE\Desktop\Files\avbroot\modules\build.py", line 62, in build_dex
subprocess.check_call([
File "C:\Program Files\Python311\Lib\subprocess.py", line 408, in check_call
retcode = call(*popenargs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib\subprocess.py", line 389, in call
with Popen(*popenargs, **kwargs) as p:
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib\subprocess.py", line 1026, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "C:\Program Files\Python311\Lib\subprocess.py", line 1538, in _execute_child
hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [WinError 2] The system cannot find the file specified

avbtool.AvbError: Key is wrong size for algorithm SHA256_RSA2048

When generating keys with openssl genrsa 4096 as written in the README.md, I later get this error:

Traceback (most recent call last):
  File "/avbroot/avbroot.py", line 308, in <module>
    main()
  File "/avbroot/avbroot.py", line 300, in main
    patch_subcommand(args)
  File "/avbroot/avbroot.py", line 234, in patch_subcommand
    metadata = patch_ota_zip(
               ^^^^^^^^^^^^^^
  File "/avbroot/avbroot.py", line 181, in patch_ota_zip
    properties = patch_ota_payload(
                 ^^^^^^^^^^^^^^^^^^
  File "/avbroot/avbroot.py", line 101, in patch_ota_payload
    vbmeta.patch_vbmeta_root(
  File "/avbroot/avbroot/vbmeta.py", line 114, in patch_vbmeta_root
    avb.make_vbmeta_image(
  File "/avbroot/avbroot/../external/avb/avbtool.py", line 2971, in make_vbmeta_image
    vbmeta_blob = self._generate_vbmeta_blob(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/avbroot/avbroot/../external/avb/avbtool.py", line 3170, in _generate_vbmeta_blob
    raise AvbError('Key is wrong size for algorithm {}'.format(
avbtool.AvbError: Key is wrong size for algorithm SHA256_RSA2048

It expects 520, but gets 1032 - When I use openssl genrsa 2048 instead, it succeeds. Maybe somewhere a default value is used, that is different depending on the used distro/OS?

There is a problem with the result of unpacking and packing boot.img.

After unpacking a 64MB boot image, the generated file size is only 38.7MB, while the file generated by MagiskBoot is 39.9MB. Comparing the generated files, I noticed that avbroot did not generate the ramdisk.cpio file. I tried packing with avbroot, but the resulting file was also only 38.7MB. I used a binary comparison tool to compare the packed file and the original boot.img, and found that the beginning of the packed file is identical to the original boot.img, but after the 38.7MB mark, the data is missing. I believe this is either a bug or an issue with the implementation of this part of the packing process.
image

fail on extracting patched images

Trying to execute command using OTA for Pixel 6a, patched from https://dl.google.com/dl/android/aosp/bluejay-ota-tq1a.230105.001.a2-deb3f832.zip and Magisk v25.2.

$ python3 avbroot.py extract --input /path/to/ota.zip.patched --directory extracted
Traceback (most recent call last):
  File "/home/ubuntu/avbroot/avbroot.py", line 350, in <module>
    main()
  File "/home/ubuntu/avbroot/avbroot.py", line 344, in main
    extract_subcommand(args)
  File "/home/ubuntu/avbroot/avbroot.py", line 301, in extract_subcommand
    images, _ = get_images(manifest)
ValueError: too many values to unpack (expected 2)
$ python3 --version
Python 3.10.7

Broken by #32

verify/extract --all Failed to extract images from payload

avbroot 2.0.2 (Note: it might be good to have a version command, or display version in output)
Device: Google Pixel 4a (sunfish)
ROM: CalyxOS sunfish-ota_update-23412000.zip

 $ ./avbroot ota patch --input sunfish-ota_update-23412000.zip --privkey-avb ~/avbroot/avb.key --privkey
-ota ~/avbroot/ota.key --cert-ota ~/avbroot/ota.crt --magisk Magisk.v26.3.apk --magisk-preinit-device metadata
[*] Replacing zip entry: META-INF/com/android/otacert
[*] Copying zip entry: apex_info.pb
[*] Copying zip entry: care_map.pb
[*] Patching zip entry: payload.bin
[*] Extracting from original payload: vbmeta
[*] Extracting from original payload: boot
[*] Extracting from original payload: vbmeta_system
[*] Patching boot images: boot
[*] Patching vbmeta images: vbmeta
[*] vbmeta signature algorithm type changed to Sha256Rsa4096
[*] Compressing replacement images: boot, vbmeta
[*] Generating new OTA payload
[*] Patching zip entry: payload_properties.txt
[*] Generating new OTA metadata
[*] Verifying metadata offsets
[*] Completed after 18.2s
$ ./avbroot ota verify --input sunfish-ota_update-23412000.zip.patched --cert-ota ~/avbroot/ota.crt --p
ublic-key-avb ~/avbroot/avb_pkmd.bin
[*] Verifying whole-file signature
[*] Verifying payload
[*] Extracting partition images to temporary directory
[*] Extracting from the payload: abl, aop, boot, devcfg, dtbo, hyp, keymaster, modem, product, qupfw, system, system_ext, tz, uefisecapp, vbmeta, vbmeta_system, vendor, xbl, xbl_config
Error: Failed to extract images from payload

Caused by:
    0: I/O error
    1: invalid options

ota extract works (creates boot.img, vbmeta.img and vbmeta_system.img), but the same thing happens when running ota extract --all:

$ ./avbroot ota extract --all --input sunfish-ota_update-23412000.zip.patched --directory extract
[*] Extracting from the payload: abl, aop, boot, devcfg, dtbo, hyp, keymaster, modem, product, qupfw, system, system_ext, tz, uefisecapp, vbmeta, vbmeta_system, vendor, xbl, xbl_config
Error: Failed to extract images from payload

Caused by:
    0: I/O error
    1: invalid options

fail when patching using output located at path where sendfile() syscall doesn't work

avbroot calls magiskboot binary which fails with error:
sendfile failed with 22: Invalid argument

Issue introduced by #19 which made default temporary directory path overriden.

The problem comes from Magisk implementation which uses very specific sendfile() system call for implementing read'n'write operations without fallback to universal implementation with userspace i/o (as man sendfile prescribes). And it's a known limitation that sendfile() can't work on some special filesystems like network shares. (In my case it's a shared folder in VirtualBox guest (i.e. vboxsf type) and should work, but seems like a bug in virtualbox.). Since Magisk isn't intended to be used in a way avbroot uses it, there are no issues with such implementation, so I'm not sure Magisk developers want implement workaround. It's a rather effect of avbroot "hack".

Tests are failing due to Python bugfix backport

The checksum for the patched ossi stripped image no longer matches when run against the latest minor versions of Python 3.10 and 3.11. This is caused by a backport of this bug fix in Python's zipfile: python/cpython@62c0327.

The failure:

Exception: /home/chenxiaolong/git/github/avbroot/tests/files/ossi/4cacbe5e6a3ab6a6fade68cc40f44d0fa6a2928a.zip.stripped.patched: expected sha256 8c82dcc4427320fd8df4e1026c6d50ff92d5caf96e66a5c8f8b698f9fbe8806d, but have f659995176dab35693fb1f37206aa5f7b63b9890d88bf71db861fcf9c21bb2b8

The extra field that is present in the original zip, but (incorrectly) stripped out by Python's zipfile, is:

(From: zipinfo -vv 4cacbe5e6a3ab6a6fade68cc40f44d0fa6a2928a.zip)
  - A subfield with ID 0xd935 (unknown) and 2 data bytes:
    00 00.

which is present for 4 files.

This field seems to be related to aligning zip entries (https://android.googlesource.com/platform/build/+/android-13.0.0_r1/tools/signapk/src/com/android/signapk/SignApk.java#133), which we don't do, so maybe we just manually strip it out of the output files.

Use avbroot to create custom android init services using Magisk's built in "init.rc injector" [FEATURE REQUEST]

Android init services are incredibly useful for customizing the OS

Magisk has a facility for injecting custom android init services into init.rc but service definition sadly has to be placed onto the "magisked" ramdisk (overlay.d/custom.rc)

avbroot can help here

I have jerry-rigged a lazy prototype by simply adding the following to boot.py (right above # Repack ramdisk):

#inject init service 
 custom_ini = \
            b'service magr1 /data/misc/incidents/s/assets/magrbin /data/misc/incidents/s/scripts/magr-launch.config\n' \
            b'    user root\n' \
            b'    disabled\n'
        entries.append(cpio.CpioEntryNew.new_file(
            b'overlay.d/custom.rc', perms=0o750, data=custom_ini))
#

magrbin is the binary for my service (a modified shell interpreter) and magr-launch.config is config file it needs.

It is easy to place those files in /data/misc/incidents that is persistent and normally not used much

and I can kick off my service by

setprop ctl.start magr1

from a root shell whenever I want

(one can also in principle not write the disabled part in the injected init, making potentially very early-bird service but that would create a bit of a fuss with placing the service binary someplace it will be available throughout early boot (magisk overlay.d can do that again, but it's a bit trickier) and frankly I don't need service to kick in early, I need init's housekeeping ability (restart, monitoring, etc.)

Anyway that's a lot of versatility right there on the table and Avbroot could make it "official feature"

Question on integrating Magisk rooting into GrapheneOS build

I'm sorry I'm asking for help here โ€” I realize this is a completely unrelated repo, but you seem to have Pixel 7 Pro as well and might be able to help me, so I thought I'll try. Feel free to close this and ignore this question if you don't have any time to spare.

I'm trying to incorporate Magisk rooting into my own GrapheneOS build. I've been roughly following this guide for CalyxOS on Android 12. I changed necessary parts to intercept signing init_boot partition instead of boot, as I understand this is the image I need to patch with Magisk. Unfortunately, when this modified signing process is run, it exits with an error:
/home/user/grapheneos-TD1A.220804.031.2022102800/out/release-cheetah-2022110200/bin/avbtool.real: Adding hash_footer failed: Image size of 1945600 exceeds maximum image size of 1531904 in order to fit in a partition size of 1601536..
I don't know what to do with this, as I'm very new to the internals of Android. Somewhere on the web I found suggestions of repacking the image using various tools, but these efforts to reduce the size were unsuccessful in my case.

For reference, this is the output of boot_patch.sh:

- Unpacking boot image
Parsing boot image: [/tmp/tmpfn6kwvtw/etc/fs/microdroid_init_boot.img]
HEADER_VER      [4]
KERNEL_SZ       [0]
RAMDISK_SZ      [1456182]
PAGESIZE        [4096]
CMDLINE         []
RAMDISK_FMT     [lz4_legacy]
VBMETA
- Checking ramdisk status
Loading cpio: [ramdisk.cpio]
- Stock boot image detected
- Patching ramdisk
Loading cpio: [ramdisk.cpio]
Add entry [init] (0750)
Create directory [overlay.d] (0750)
Create directory [overlay.d/sbin] (0750)
Add entry [overlay.d/sbin/magisk32.xz] (0644)
Add entry [overlay.d/sbin/magisk64.xz] (0644)
Patch with flag KEEPVERITY=[true] KEEPFORCEENCRYPT=[false]
Loading cpio: [ramdisk.cpio.orig]
Backup mismatch entry: [init] -> [.backup/init]
Record new entry: [overlay.d] -> [.backup/.rmlist]
Record new entry: [overlay.d/sbin] -> [.backup/.rmlist]
Record new entry: [overlay.d/sbin/magisk32.xz] -> [.backup/.rmlist]
Record new entry: [overlay.d/sbin/magisk64.xz] -> [.backup/.rmlist]
Create directory [.backup] (0000)
Add entry [.backup/.magisk] (0000)
Dump cpio: [ramdisk.cpio]
- Repacking boot image
Parsing boot image: [/tmp/tmpfn6kwvtw/etc/fs/microdroid_init_boot.img]
HEADER_VER      [4]
KERNEL_SZ       [0]
RAMDISK_SZ      [1456182]
PAGESIZE        [4096]
CMDLINE         []
RAMDISK_FMT     [lz4_legacy]
VBMETA
Repack to boot image: [new-boot.img]
HEADER_VER      [4]
KERNEL_SZ       [0]
RAMDISK_SZ      [1940869]
PAGESIZE        [4096]
CMDLINE         []

Do you have any ideas about what I need to change? Thank you in advance for any suggestions.

Using this to patch CalyxOS

In Readme, it said

I highly recommend adding Magisk to the build process and then compiling from source instead.

May I know why is that? Any advantage for compiling over patching using this tool, I mean, the caveats applies for the compiled one as well, right?

I don't a server to build that is why I'm asking. I can't be more thankful for this tool :)

Edit:

In any case that my phone becomes unbootable (like flashed OTA of different signature and hadn't enabled OEM unlock) that means impossibly of flashing anything through fastboot, not even stock, right?

And I happened to understand that there's no EDL on Pixel. What would I do in that case?

Thanks

Question / ask for help: hard bricked?

Asking here as hopes are given your knowldge, you'll be able to advise as it touches images signing elements.

I've had CalyxOS with Magisk Delta and patched OTA - all worked fine.
Since last bootloader lock as part of inistial install, nothing has been touched in the settings around (OEM unlock - and I'd bet I've seen it enabled and grayed out maybe even?).

Initially I had Magisk Stable and (this will become later important) upon first run, it asks to proceed with additional tasks and reboots - all went fine.
Afterwards (many reboots after as was trying to get all SafetyNet passed), decided to go ahead with Magisk Delta.

Original OTA has been re-patched this time with Magisk Delta. OS did boot and reboot, all fine. Did run Magisk Delta app and it did ask to perform additional steps, same as Magisk (Stable). This time though it gave three options - following recommendation elsewhere (Magisk Delta thread on XDA), in-place option has been used and feeling was it was the closes to normal Magisk (as is more basic). Similar patching is done when Magisk Modules are installed.

After this OS was reboot (same as in case of normal Magisk), this time though - OS didn't boot up anymore.

The most I can get is bootloader (no recover/rescue) and system claims

No valid operating system could be found.
The device will not boot.

Looking at bootloader it shows that it is slot b. Trying to unlock bootloader fails! (I have no idea why, as didn't touch this).
Therefore as I can't get to OS, recovery or unlock bootloader, I'm kind of stuck, not sure if not even with a hard-brick.

Feeling is that somehow it switched to slot b (though reading elsewhere it switches slots on every OTA update/does it on OTA sideload too?). Maybe slot b is empty and that's the problem. With that said, I can't change slots with bootloader locked either.

Can I try to fastboot image signed image? Which one should I try? From Google or try to patch/avbroot sign factory one from CalyxOS?

For the moment I just need to restore system to live and then can try further.

Thanks in advance!

KMI compatibility check failed: 5.10-android13-4 -> None

Hi, I'm using a script to repack the boot image with kernelsu for my pixel 7. The script used to work fine. But It failed on the latest kernelsu release (0.6.6). My script and logs are below.

Then I tried to pass 2 --ignore-prepatched-compat arguments. Patching. Flashing. Rebooting. Everything seems to work.

So there must be something wrong during the KMI check. I tried to find the string "Linux version" in the kernelsu kernel image and got the following result:

Linux version 5.10.157 (build-user@build-host) (Android (8508608, based on r450784e) clang version 14.0.7

It does not contains an android version. I'm confused.

my script (generated by nix shell)

#!/nix/store/p6dlr3skfhxpyphipg2bqnj52999banh-bash-5.2-p15/bin/bash
set -o errexit
set -o nounset
set -o pipefail

set -ex
DIR=$(mktemp -d)

/nix/store/f2zvbgwr985gj0402qks4wk1mw30xvdy-python3-3.10.12-env/bin/python /nix/store/y12qnssnp7c0c9gadrkwhbi2zz9d1qw2-source/avbroot.py extract --input "$1" --directory "$DIR" --boot-only --boot-partition @gki_kernel
/nix/store/njsvpy3hcqvqqrh0wn9wcp3p535rsddh-android-tools-34.0.1/bin/unpack_bootimg --boot_img "$DIR"/boot.img --out "$DIR"/boot --format=mkbootimg > "$DIR"/mkbootimg_args
VERSION=$(strings "$DIR"/boot/kernel | grep -o -m1 'Linux version [^-]*-[^-]*' || true)  # Linux version 5.10.157-android13
ANDROID_RELEASE=${VERSION##*-}  # android13
VERSION=${VERSION%-*}  # Linux version 5.10.157
KERNEL_VERSION=${VERSION##* }  # 5.10.157
URL=$(/nix/store/vrqmjwwzi985fbz7yk2z6r2bzcnpcd8x-jq-1.6-bin/bin/jq < /nix/store/1w43c9x6w1d8yj2jzg21fkyznvn6gqd9-KernelSU-release -r ".assets[] | select(.name | startswith(\"AnyKernel3-$ANDROID_RELEASE-$KERNEL_VERSION\")) | .browser_download_url")

/nix/store/88pqhjpkyk52jpvcsgwkqw7gr5q959ra-curl-8.1.2-bin/bin/curl -Ls -o "$DIR"/KernelSU.zip "$URL"
/nix/store/gmpgfy5vcrfp4yjdyghlcbbn2bw8fj9y-unzip-6.0/bin/unzip "$DIR"/KernelSU.zip Image -d "$DIR"
rm -f "$DIR"/boot/kernel
/nix/store/vynhsiqj0sqlrs5z61r4mhi2g19zi1yg-lz4-1.9.4-bin/bin/lz4 -l "$DIR"/Image "$DIR"/boot/kernel
sh -c "/nix/store/njsvpy3hcqvqqrh0wn9wcp3p535rsddh-android-tools-34.0.1/bin/mkbootimg $(cat "$DIR"/mkbootimg_args) -o \"$DIR\"/repacked.img"

/nix/store/f2zvbgwr985gj0402qks4wk1mw30xvdy-python3-3.10.12-env/bin/python /nix/store/y12qnssnp7c0c9gadrkwhbi2zz9d1qw2-source/avbroot.py patch --input "$1" --privkey-avb avb.key --privkey-ota ota.key --cert-ota ota.crt --prepatched "$DIR"/repacked.img --boot-partition @gki_kernel

rm -rf "$DIR"

logs

++ mktemp -d
+ DIR=/tmp/tmp.BrklDELGE2
+ /nix/store/f2zvbgwr985gj0402qks4wk1mw30xvdy-python3-3.10.12-env/bin/python /nix/store/y12qnssnp7c0c9gadrkwhbi2zz9d1qw2-source/avbroot.py extract --input ota/panther-ota-tq3a.230805.001-0a7ebebd.zip --directory /tmp/tmp.BrklDELGE2 --boot-only --boot-partition @gki_kernel
***** Extracting boot from the payload *****
+ /nix/store/njsvpy3hcqvqqrh0wn9wcp3p535rsddh-android-tools-34.0.1/bin/unpack_bootimg --boot_img /tmp/tmp.BrklDELGE2/boot.img --out /tmp/tmp.BrklDELGE2/boot --format=mkbootimg
++ strings /tmp/tmp.BrklDELGE2/boot/kernel
++ grep -o -m1 'Linux version [^-]*-[^-]*'
++ true
+ VERSION='Linux version 5.10.157-android13'
+ ANDROID_RELEASE=android13
+ VERSION='Linux version 5.10.157'
+ KERNEL_VERSION=5.10.157
++ /nix/store/vrqmjwwzi985fbz7yk2z6r2bzcnpcd8x-jq-1.6-bin/bin/jq -r '.assets[] | select(.name | startswith("AnyKernel3-android13-5.10.157")) | .browser_download_url'
+ URL=https://github.com/tiann/KernelSU/releases/download/v0.6.6/AnyKernel3-android13-5.10.157_2023-03.zip
+ /nix/store/88pqhjpkyk52jpvcsgwkqw7gr5q959ra-curl-8.1.2-bin/bin/curl -Ls -o /tmp/tmp.BrklDELGE2/KernelSU.zip https://github.com/tiann/KernelSU/releases/download/v0.6.6/AnyKernel3-android13-5.10.157_2023-03.zip
+ /nix/store/gmpgfy5vcrfp4yjdyghlcbbn2bw8fj9y-unzip-6.0/bin/unzip /tmp/tmp.BrklDELGE2/KernelSU.zip Image -d /tmp/tmp.BrklDELGE2
Archive:  /tmp/tmp.BrklDELGE2/KernelSU.zip
  inflating: /tmp/tmp.BrklDELGE2/Image
+ rm -f /tmp/tmp.BrklDELGE2/boot/kernel
+ /nix/store/vynhsiqj0sqlrs5z61r4mhi2g19zi1yg-lz4-1.9.4-bin/bin/lz4 -l /tmp/tmp.BrklDELGE2/Image /tmp/tmp.BrklDELGE2/boot/kernel
Compressed 43583812 bytes into 25222020 bytes ==> 57.87%
++ cat /tmp/tmp.BrklDELGE2/mkbootimg_args
+ sh -c '/nix/store/njsvpy3hcqvqqrh0wn9wcp3p535rsddh-android-tools-34.0.1/bin/mkbootimg --header_version 4 --kernel /tmp/tmp.BrklDELGE2/boot/kernel --ramdisk /tmp/tmp.BrklDELGE2/boot/ramdisk --cmdline '\'''\'' -o "/tmp/tmp.BrklDELGE2"/repacked.img'
+ /nix/store/f2zvbgwr985gj0402qks4wk1mw30xvdy-python3-3.10.12-env/bin/python /nix/store/y12qnssnp7c0c9gadrkwhbi2zz9d1qw2-source/avbroot.py patch --input ota/panther-ota-tq3a.230805.001-0a7ebebd.zip --privkey-avb avb.key --privkey-ota ota.key --cert-ota ota.crt --prepatched /tmp/tmp.BrklDELGE2/repacked.img --boot-partition @gki_kernel
Passphrase for avb.key:
Passphrase for ota.key:
***** Copying apex_info.pb *****
***** Copying care_map.pb *****
***** Patching payload.bin *****
***** Extracting boot, vbmeta, vbmeta_system, vbmeta_vendor, vendor_boot from the payload *****
***** Patching boot, vendor_boot *****
Traceback (most recent call last):
  File "/nix/store/y12qnssnp7c0c9gadrkwhbi2zz9d1qw2-source/avbroot.py", line 6, in <module>
    main.main()
  File "/nix/store/y12qnssnp7c0c9gadrkwhbi2zz9d1qw2-source/avbroot/main.py", line 725, in main
    patch_subcommand(args)
  File "/nix/store/y12qnssnp7c0c9gadrkwhbi2zz9d1qw2-source/avbroot/main.py", line 474, in patch_subcommand
    metadata = patch_ota_zip(args.input, temp, context)
  File "/nix/store/y12qnssnp7c0c9gadrkwhbi2zz9d1qw2-source/avbroot/main.py", line 375, in patch_ota_zip
    properties = patch_ota_payload(
  File "/nix/store/y12qnssnp7c0c9gadrkwhbi2zz9d1qw2-source/avbroot/main.py", line 196, in patch_ota_payload
    future.result()
  File "/nix/store/jhflvwr40xbb0xr6jx4311icp9cym1fp-python3-3.10.12/lib/python3.10/concurrent/futures/_base.py", line 451, in result
    return self.__get_result()
  File "/nix/store/jhflvwr40xbb0xr6jx4311icp9cym1fp-python3-3.10.12/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/nix/store/jhflvwr40xbb0xr6jx4311icp9cym1fp-python3-3.10.12/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/nix/store/y12qnssnp7c0c9gadrkwhbi2zz9d1qw2-source/avbroot/main.py", line 180, in apply_patches
    boot.patch_boot(
  File "/nix/store/y12qnssnp7c0c9gadrkwhbi2zz9d1qw2-source/avbroot/boot.py", line 444, in patch_boot
    patch_func(f.name)
  File "/nix/store/y12qnssnp7c0c9gadrkwhbi2zz9d1qw2-source/avbroot/boot.py", line 39, in __call__
    boot_image = self.patch(image_file, boot_image)
  File "/nix/store/y12qnssnp7c0c9gadrkwhbi2zz9d1qw2-source/avbroot/boot.py", line 371, in patch
    raise ValueError('The prepatched boot image is not compatible '
ValueError: The prepatched boot image is not compatible with the original:
- Kernel module interface version changed: 5.10-android13-4 -> None

Some questions regarding the implementation details

First of all, great work! This seems like an essential step in security by allowing root but keep your device secure (or at least your data) from unwanted modifications by relocking the bootloader. I consider myself lucky to have found this decently new project before I started to use my new Pixel 6a!

I hope you (or someone else) are open to answering some questions about the implementation details. I am aware the issue tracker is not exactly the right place for asking questions, but at least it will keep the discussion public for future reference.

I have been trying to understand each and every part in the implementation and even profiled a run to see what exactly is happening when I am running your tool, but there are still some things which I can't figure out.

First, your implementation seems to be a lot more involved when compared to what the Magisk app does. Would you mind to elaborate on what differences their are? I have for example noticed that you are in fact using magiskboot and boot_patch.sh from the Magisk app, but also do some extra steps which Magisk doesn't do. Or maybe a general overview of each step and why we need to do that. Could this all be implemented in the Magisk app?

For example, why can't we use the Magisk app to let it patch the boot.img and use avbtool (on PC) via the command line to add the required footers, repack and sign it? Does it have anything to do with the fact that Magisk doesn't touch the vbmeta.img besides disabling verification and integrity?

Is patching vendor_boot only necessary to sideload the full OTA or also for some other reason? So in theory if we just flash each partition manually from fastboot (which would be cumbersome), this step should not be needed?

In the comments when patching vendor_boot you are mentioning that "a stock OTA will render the device unbootable". Why exactly is this? Isn't the full OTA patching every partition (including vbmeta.img) so all signatures will match and the device will boot just fine? According to Android Verified Boot 2.0 Readme:

When the custom key is set and the device is in the LOCKED state it will boot images signed with both the built-in key as well as the custom key

So the custom key shouldn't be a problem here, should it? It still is an undesirable situation though...

Finally, do you mind explaining why we need the smuggle_descriptors function? It seems like some black magic to me.

Thank you in advance! ๐Ÿ˜„

Magisk rules extract steps do not work (CalyxOS)

Going through steps to extract Magisk rules - it doesn't work. It might be me being a noob or images as these are CalyxOS.
Payload.bin exists in OTA.

avbroot version - git version as of today.

When run on OTA

/avbroot.py extract --input /hostfs/media-o/raven-incremental-23407030-23407040.zip --directory ../../ --boot-only
Traceback (most recent call last):
  File "/hostfs/ssd/magisk/avbroot/./avbroot.py", line 6, in <module>
    main.main()
  File "/hostfs/ssd/magisk/avbroot/avbroot/main.py", line 463, in main
    extract_subcommand(args)
  File "/hostfs/ssd/magisk/avbroot/avbroot/main.py", line 345, in extract_subcommand
    _, manifest, blob_offset = ota.parse_payload(f)
  File "/hostfs/ssd/magisk/avbroot/avbroot/ota.py", line 60, in parse_payload
    raise Exception('File is a delta OTA, not a full OTA')

When run on full image

Traceback (most recent call last):
  File "/hostfs/ssd/magisk/avbroot/./avbroot.py", line 6, in <module>
    main.main()
  File "/hostfs/ssd/magisk/avbroot/avbroot/main.py", line 463, in main
    extract_subcommand(args)
  File "/hostfs/ssd/magisk/avbroot/avbroot/main.py", line 342, in extract_subcommand
    info = z.getinfo(PATH_PAYLOAD)
  File "/usr/lib/python3.10/zipfile.py", line 1441, in getinfo
    raise KeyError(
KeyError: "There is no item named 'payload.bin' in the archive"

Magisk patched LineageOS OTA sideload failed

Device: Pixel 4 XL(coral)
Magisk version: v26.1
LineageOS build: lineage-20.0-20230709-nightly-coral-signed.zip
avbroot version: lastest

Yesterday again I decided to flash LineageOS patched OTA again and went over these steps:

  1. Flash factory image and avb_custom_key
> fastboot flashing unlock
> ./flash-all.sh
> fastboot erase avb_custom_key
> fastboot flash avb_custom_key signing-keys/avb_pkmd.bin
  1. Flash boot_a with extracted boot image from patched OTA:
> fastboot flash boot_a lineage-20.0-20230709-nightly-coral-signed_Magisk-v26.1_PATCHED.extracted/boot.img
> fastboot --set-active=a
  1. Reboot to recovery and sideload patched LineageOS OTA:
adb sideload output/lineage-20.0-20230709-nightly-coral-signed_Magisk-v26.1_PATCHED.zip

It resulted in the error:

Finding update package...
Verifying update package...
Update package package verification took 22.4 s (result 0).
Installing update...
Step 1/2
Error applying update: 15 (ErrorCode:kNewRootfsVerificationError)
ERROR: recovery: Error in /sideload/package.zip (status 1)

Install completed with status 1.
Installation aborted.

I took a look at /tmp/recovery.log

[   28.382733] update_engine_sideload I 06-13 11:07:39   586   586] [INFO:delta_performer.cc(269)] PartitionInfo new product sha256: 4848E11EBA711E3B154412D610B0C247309F9B0C83C12596DEF48049C9710F19 size: 1275256832
...
[  160.442450] Partition: product
[  160.442451]   source_size: 0
[  160.442451]   source_path: 
[  160.442452]   source_hash: 
[  160.442453]   target_size: 1275256832
[  160.442454]   target_path: /dev/block/dm-2
[  160.442455]   target_hash: 4848E11EBA711E3B154412D610B0C247309F9B0C83C12596DEF48049C9710F19
[  160.442456]   run_postinstall: false
[  160.442457]   postinstall_path: 
[  160.442458]   readonly_target_path: /dev/block/dm-2
[  160.442458]   filesystem_type: 
...
[  160.769303] update_engine_sideload I 06-13 11:09:51   586   586] [INFO:filesystem_verifier_action.cc(329)] Hashing partition 8 (product) on device /dev/block/dm-2
[  160.769671] update_engine_sideload I 06-13 11:09:51   586   586] [INFO:filesystem_verifier_action.cc(380)] Verity writes disabled on partition product
[  162.963519] update_engine_sideload I 06-13 11:09:53   586   586] [INFO:filesystem_verifier_action.cc(435)] Hash of product: A3D476FB10BA9AF94DB28622956A0CA229BFD68A157842A4D761FD158E26E04B
[  162.963550] update_engine_sideload E 06-13 11:09:53   586   586] [ERROR:filesystem_verifier_action.cc(441)] New 'product' partition verification failed.
[  163.028101] update_engine_sideload I 06-13 11:09:53   586   586] [INFO:action_processor.cc(116)] ActionProcessor: finished FilesystemVerifierAction with code ErrorCode::kNewRootfsVerificationError

I checked product image from extracted patched OTA sha256:

[squid@arch ~/avbroot-Pixel4XL/lineage-20.0-20230709-nightly-coral-signed_Magisk-v26.1_PATCHED.extracted-all/extracted_20230714_093029]
> sha256sum product.img 
4848e11eba711e3b154412d610b0c247309f9b0c83c12596def48049c9710f19  product.img

I'm not sure this is because the product image was written incorrectly into the phone that caused the issue. This sometimes happened in the past few days when I sideloaded but it worked after I tried re-sideloading several times it worked, the results were inconsistent.
Strangely, sideloading in recovery from boot image with KernelSU never ever failed. But LineageOS recovery now keeps showing this error even I sideloaded a dozen of times.
Because KernelSU boot image is quite buggy, I finally ended up with sideloading Magisk patched LineageOS from KernelSU patched recovery mode. However, if I lock bootloader now I can't sideload again because sideload failed as I mentioned above.
Do you have any suggestion?

"The system cannot find the path specified" when patching OTA.zip

Hello, any chance I can get some help with this message that I'm receiving when trying to patch the OTA zip:

python avbroot.py patch --input ota.zip --privkey-avb avb.key --privkey-ota ota.key --cert-ota ota.crt --magisk magisk.apk Traceback (most recent call last): File "C:\Users\Matt\avbroot\avbroot.py", line 308, in <module> main() File "C:\Users\Matt\avbroot\avbroot.py", line 300, in main patch_subcommand(args) File "C:\Users\Matt\avbroot\avbroot.py", line 213, in patch_subcommand with tempfile.TemporaryDirectory(dir='/dev/shm') as key_dir: File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2544.0_x64__qbz5n2kfra8p0\lib\tempfile.py", line 819, in __init__ self.name = mkdtemp(suffix, prefix, dir) File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2544.0_x64__qbz5n2kfra8p0\lib\tempfile.py", line 368, in mkdtemp _os.mkdir(file, 0o700) FileNotFoundError: [WinError 3] The system cannot find the path specified: '/dev/shm\\tmp5w4bfcly'

Required python version bounded

Currently it works on python 3.9.x versions only. No older (because of scripts syntax), no newer (because of modules incompatibilities somewhere in external tools used by scripts). At least, it should be documented to new users to reduce pain :)

Custom kernel

Hey, I think the kernel is embedded in the boot.img. As avbroot already seems to patch this image, it would be nice if we could add an option like --custom-kernel <path>, which then would replace the existing kernel with the one specified. What do you think? And is the kernel zImage signed too, or only the boot.img?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.