Please tag version 3.6.1 for reproducibility verification about airgap-vault HOT 12 CLOSED

Thanks for taking another look at this, and sorry for the additional work this caused. I'm still trying to reproduce your failed reproduction locally so I can investigate it.

In the future, we'll ping you whenever we upload an APK to the store for review.

from airgap-vault.

Done https://github.com/airgap-it/airgap-vault/releases/tag/v3.6.1

from airgap-vault.

I just ran your script again locally, and like last time, my result was reproducible. See previous issue: #32 (comment)

I followed the same steps as I described. Because I had an issue with my old VM where I did the test, I had to start a new one from scratch. So the following test results, and also the last test of 3.5.1, were both done on completely fresh systems.

Could it be that the issue is that some things are cached between runs, which causes a diff if the version is updated in between? When I have some time, I'll try to reproduce 3.5.1 and after that 3.6.1 to see if I can see any kind of diff. I will also sporadically run the same script for 3.6.1 to see if something changes over time, which obviously shouldn't happen but I'm also not expecting it to happen.

Results:
appId:          it.airgap.vault
signer:         486381324d8669c80ca9b8c79d383dc972ec284227d65ebfe9e31cad5fd3f342
apkVersionName: 3.6.1
apkVersionCode: 26485
apkHash:        6068c88b2dbbc0033531f0237c77ea08b1d73d9fae5ea699ea7f551ae51a1920

Diff:
Files /tmp/fromPlay_it.airgap.vault_26485/apktool.yml and /tmp/fromBuild_it.airgap.vault_26485/apktool.yml differ
Files /tmp/fromPlay_it.airgap.vault_26485/original/META-INF/MANIFEST.MF and /tmp/fromBuild_it.airgap.vault_26485/original/META-INF/MANIFEST.MF differ
Only in /tmp/fromPlay_it.airgap.vault_26485/original/META-INF: PAPERS.RSA
Only in /tmp/fromPlay_it.airgap.vault_26485/original/META-INF: PAPERS.SF

Revision, tag (and its signature):
object 5767fd3c74b810c06f91cdc4c6a18f9f30d0ae6c
type commit
tag v3.6.1
tagger Andreas Gassmann <[email protected]> 1613650888 +0000

AirGap Vault v3.6.1

Run a full
diff --recursive /tmp/fromPlay_it.airgap.vault_26485 /tmp/fromBuild_it.airgap.vault_26485
meld /tmp/fromPlay_it.airgap.vault_26485 /tmp/fromBuild_it.airgap.vault_26485
for more details.

from airgap-vault.

Yes, last time there was an issue, too. The altered license though ... doesn't look like a toolchain issue. I had run the test again and it reproduced.

As to stale code being the culprit, I think the test script does clean up stuff before and after the run. It git clones the code to a new folder, even creates the docker containers anew, ... If the script misses to delete anything, please let me know or create a merge request.

from airgap-vault.

I ran the test again and got the same diff as before. I'm open to suggestions. I literally run the known test script on the apk I got from Google. I could spin up a server to try there but not immediately. In 6h maybe.

from airgap-vault.

I did some digging and found out some things, but there are still some open questions.

What I first did was verify version 3.5.0 in the same VM. This resulted in a reproduced build, as expected. Then I again built version 3.6.1, and surprisingly enough, I got this:

Results:
appId:          it.airgap.vault
signer:         486381324d8669c80ca9b8c79d383dc972ec284227d65ebfe9e31cad5fd3f342
apkVersionName: 3.6.1
apkVersionCode: 26485
apkHash:        6068c88b2dbbc0033531f0237c77ea08b1d73d9fae5ea699ea7f551ae51a1920

Diff:
Files /tmp/fromPlay_it.airgap.vault_26485/apktool.yml and /tmp/fromBuild_it.airgap.vault_26485/apktool.yml differ
Files /tmp/fromPlay_it.airgap.vault_26485/assets/public/3rdpartylicenses.txt and /tmp/fromBuild_it.airgap.vault_26485/assets/public/3rdpartylicenses.txt differ
Files /tmp/fromPlay_it.airgap.vault_26485/assets/public/index.html and /tmp/fromBuild_it.airgap.vault_26485/assets/public/index.html differ
Only in /tmp/fromBuild_it.airgap.vault_26485/assets/public: main.30370a138a2aeee7f14f.js
Only in /tmp/fromPlay_it.airgap.vault_26485/assets/public: main.b45c2c54936a9801f503.js
Files /tmp/fromPlay_it.airgap.vault_26485/original/META-INF/MANIFEST.MF and /tmp/fromBuild_it.airgap.vault_26485/original/META-INF/MANIFEST.MF differ
Only in /tmp/fromPlay_it.airgap.vault_26485/original/META-INF: PAPERS.RSA
Only in /tmp/fromPlay_it.airgap.vault_26485/original/META-INF: PAPERS.SF

Revision, tag (and its signature):
object 5767fd3c74b810c06f91cdc4c6a18f9f30d0ae6c
type commit
tag v3.6.1
tagger Andreas Gassmann <[email protected]> 1613650888 +0000

AirGap Vault v3.6.1

Run a full
diff --recursive /tmp/fromPlay_it.airgap.vault_26485 /tmp/fromBuild_it.airgap.vault_26485
meld /tmp/fromPlay_it.airgap.vault_26485 /tmp/fromBuild_it.airgap.vault_26485
for more details.

The interesting part here is the filename. It matches your "failed reproduction" exactly. So it seems that there are 2 deterministic states that appear seemingly at random.

My first thought was again that there must be some kind of cache / state persisted between the different versions, so I went ahead and did a docker system prune -a and deleted the folders in /tmp/ manually. Sadly, the next run run again resulted in the same "failed reproduction". I would like to try with a new VM again, but didn't do that for now.

I then took a look at the diff of the "beautified" version of the 2 javascript files. They had a lot of changes, but upon closer inspection it turns out that it's mostly just the order that was different, and because of the different order the variable names also didn't match.

When looking deeper at the changes between 3.5.0 and 3.5.1, which was the first version that had reproducibility issues, I noticed that we updated the major version of angular, which meant we were also using the new "Ivy" compiler.

In the build logs, I found this:

Warning: Entry point '@airgap/angular-core' contains deep imports into '@airgap/coinlib-core', [...]. This is probably not a problem, but may cause the compilation of entry points to be out of order.

I didn't find much about what that warning means exactly, but my best guess is that those "deep imports" are causing the entrypoints to be random, meaning that the code can be "out of order", which is exactly what I saw in the diff.

So it looks like the version 3.6.1 build is not deterministic and can end up in one of 2 states at random.

I assume you will not mark this release as verifiable. It would be nice if you could add a warning to the page that people should hold off downloading this update, but we are aware of the issue and are working on it. If this is indeed the issue, we will hopefully have a new, reproducible version out in a few days.

from airgap-vault.

Great you find that log line. Indeed telling! At Mycelium we went with disorderfs and from the verifier point of view it wouldn't bother me much if you also added disorderfs to the test.sh as it just makes test run slower but otherwise doesn't result in more work for me. I am not happy though to compile the app 12 times to see if I randomly get the right results eventually.

At some point I hope disorderfs can be considered not ok but that I would keep for when there are at least some 10 reproducible apps.

from airgap-vault.

We just tagged version 3.6.2, which fixes the deep imports and should make the build deterministic again.

The update is currently in the Google Play Store review process, but you can use the following APK to test it: AirGap Vault 3.6.2

I ran the script locally (twice) and successfully verified it:

Results:
appId:          it.airgap.vault
signer:         486381324d8669c80ca9b8c79d383dc972ec284227d65ebfe9e31cad5fd3f342
apkVersionName: 3.6.2
apkVersionCode: 26847
apkHash:        f3ff145265859f45da2c7a310ac6c94183c61910fe2bf2fba0da7dbf8e56e626

Diff:
Files /tmp/fromPlay_it.airgap.vault_26847/apktool.yml and /tmp/fromBuild_it.airgap.vault_26847/apktool.yml differ
Files /tmp/fromPlay_it.airgap.vault_26847/original/META-INF/MANIFEST.MF and /tmp/fromBuild_it.airgap.vault_26847/original/META-INF/MANIFEST.MF differ
Only in /tmp/fromPlay_it.airgap.vault_26847/original/META-INF: PAPERS.RSA
Only in /tmp/fromPlay_it.airgap.vault_26847/original/META-INF: PAPERS.SF

Revision, tag (and its signature):
object d6a4c99d48039454663d5e8a93fcf62fbfd51fc9
type commit
tag v3.6.2
tagger Andreas Gassmann <[email protected]> 1614355245 +0000

AirGap Vault v3.6.2

To make this process more transparent, I also created a small Github repo (https://github.com/AndreasGassmann/walletscrutiny-build) where we can upload our new releases before we submit them to the Play Store. The commit will trigger a Github Action, which will execute your walletscrutiny test.sh script to verify the APK we committed. If the verification is successful, you (or someone else) can just confirm that the hashes match once the Play Store version is available. I ran the action ran twice, here are the results:

https://github.com/AndreasGassmann/walletscrutiny-build/runs/1988859046?check_suite_focus=true
https://github.com/AndreasGassmann/walletscrutiny-build/runs/1989304277?check_suite_focus=true

I feel like running the verification script on a public CI is a great way to make the verifiable builds even more accessible to walletscrutiny users. So maybe we could think about a walletscrutiny group/repo on Github and Gitlab to run the script publicly on both platforms.

Note: The script Github Action currently doesn't fail if verification fails. It will always be green as long as there is no error. Maybe we could add that in the future to make failures more visible.

from airgap-vault.

@AndreasGassmann thanks for your effort! Yes, a CI sounds like the right approach! Ideally two would run that CI as neither you nor me are neutral with respect to all wallets but then the CI could be triggered by updates to the respective wallet repos in the following way:

1. Wallet X loads a new apk to their pre-release repo.
2. CI runs test.sh on it.
3. If test fails, we might change the test script and manually run the CI test again.

I tried to reproduce your apk but with no success on the first try:

Results:
appId:          it.airgap.vault
signer:         486381324d8669c80ca9b8c79d383dc972ec284227d65ebfe9e31cad5fd3f342
apkVersionName: 3.6.2
apkVersionCode: 26847
apkHash:        f3ff145265859f45da2c7a310ac6c94183c61910fe2bf2fba0da7dbf8e56e626

Diff:
Files /tmp/fromPlay_it.airgap.vault_26847/apktool.yml and /tmp/fromBuild_it.airgap.vault_26847/apktool.yml differ
Files /tmp/fromPlay_it.airgap.vault_26847/assets/public/index.html and /tmp/fromBuild_it.airgap.vault_26847/assets/public/index.html differ
Only in /tmp/fromBuild_it.airgap.vault_26847/assets/public: main.bb0b03ddacd85cd53711.js
Only in /tmp/fromPlay_it.airgap.vault_26847/assets/public: main.ed0732ee00ddda311cc0.js
Files /tmp/fromPlay_it.airgap.vault_26847/original/META-INF/MANIFEST.MF and /tmp/fromBuild_it.airgap.vault_26847/original/META-INF/MANIFEST.MF differ
Only in /tmp/fromPlay_it.airgap.vault_26847/original/META-INF: PAPERS.RSA
Only in /tmp/fromPlay_it.airgap.vault_26847/original/META-INF: PAPERS.SF

Revision, tag (and its signature):
object d6a4c99d48039454663d5e8a93fcf62fbfd51fc9
type commit
tag v3.6.2
tagger Andreas Gassmann <[email protected]> 1614355245 +0000

AirGap Vault v3.6.2

Checking if second run comes to a different result ...

from airgap-vault.

So the good news is, the second run looked better:

Results:
appId:          it.airgap.vault
signer:         486381324d8669c80ca9b8c79d383dc972ec284227d65ebfe9e31cad5fd3f342
apkVersionName: 3.6.2
apkVersionCode: 26847
apkHash:        f3ff145265859f45da2c7a310ac6c94183c61910fe2bf2fba0da7dbf8e56e626

Diff:
Files /tmp/fromPlay_it.airgap.vault_26847/apktool.yml and /tmp/fromBuild_it.airgap.vault_26847/apktool.yml differ
Files /tmp/fromPlay_it.airgap.vault_26847/original/META-INF/MANIFEST.MF and /tmp/fromBuild_it.airgap.vault_26847/original/META-INF/MANIFEST.MF differ
Only in /tmp/fromPlay_it.airgap.vault_26847/original/META-INF: PAPERS.RSA
Only in /tmp/fromPlay_it.airgap.vault_26847/original/META-INF: PAPERS.SF

Revision, tag (and its signature):
object d6a4c99d48039454663d5e8a93fcf62fbfd51fc9
type commit
tag v3.6.2
tagger Andreas Gassmann <[email protected]> 1614355245 +0000

AirGap Vault v3.6.2

but the bad news is that it is not deterministic? Doing a third run ...

from airgap-vault.

That's strange. I triggered the CI build again a couple of times with reproducible results. I now pushed a change so it will first run 3.6.1 and then 3.6.2 in the same script (to rule out any issues with "leftover" files). 3.6.1 was not reproducible, but 3.6.2 was.

I also ran the build locally in my VM about 10 times, all of them reproducible.

Would you be able to send me the apk of the failed reproduction, or more specifically the js file that was generated? I would like to take a look at the diff in that file.

from airgap-vault.

Would you be able to send me the apk of the failed reproduction, or more specifically the js file that was generated?

Not the first time I regret my script deletes build artifacts instead of creating new folders in /tmp at least.

I built three more times and got only good results. Guess instead of wasting time I just don't delete the apk the next time there is an issue.

AirGap Vault is back in the top category since yesterday. Please ping me with the next release to make sure it goes smooth. And please help me to configure the CI.

from airgap-vault.