Checksum collision with completely different photos about immich HOT 9 OPEN

I checked as well, it's definitely not a hash collision.

from immich.

Copying over immich data was the issue: we need to keep a mapping between asset ID and its file hash (otherwise we'd need to calculate all hashes on app start again and again). This mapping is wrong when moving the data to a new phone or when restoring a backup. We have this as an open issue somewhere.... To detect this case and ask the user to delete app data.

from immich.

I think this is one of them #4939

from immich.

@fyfrey I'm having a look at the app's hashing code and there's a fair bit of array index juggling. Nothing stands out to me immediately, but how likely do you think it is that there's some edge case where indexes get misaligned or such?

@miahi can you pull the complete logs from the app? I'm particularly interested to see what got logged at the time it calculated all the hashes.

from immich.

The logs might be a bit "dirty" as I was trying a lot of things today as I was trying to understand what is happening with #6196 (two of the photos missing there are found as duplicates in the logs).

immich_android_logs.zip

I tried uploading the two files on a different account directly to the web app and both were accepted. I can do other tests if needed.

from immich.

I'll need to check. An edge case bug is certainly possible. We can also just check if these files happen to produce a hash collision for sha1 (unlikely, but we should rule this out first)

from immich.

@bo0tzz I checked the code as I thought maybe it's a custom hash implementation but it's just SHA1, fat chance of random collisions there. The SHA1 hash of the old file is correct (GNrECa7lbm6smQHTdW5Rbf99NNU= in base64), but then the second one should be JpWUN7FvqDoak41elMrIc7wsH64= but it's somehow mixed with the first one in the app.

What I also did today (a lot) was to remove and re-add libraries into the app to try to find a pattern for #6196 - that might also be related somehow to indexing, as it seems that the app is not showing photos from one library when it is showing photos from another. I have 3 libraries on the phone with > 6000 photos each. Sometimes I added multiple libraries at the same time. But #6196 seems to be deterministic, the same photos were missing from the app every time I added the libraries. So it might be that the same bug that assigns the wrong hash to some of the photos is causing the missing photos too (maybe overwriting other data?).

from immich.

I can think of one scenario causing issues:
Did you restore your entire phone, your photos, or the immich app from a Google cloud backup etc.?

from immich.

You are right. I changed my phone a while back and used a migration app to move the data (via cable connection). It moved the photos and I think also the immich local storage. That was around 1st of March 2024. I remember that I had to do a login and resync at that point, but I don't think the local DB was cleaned.

I cleared the app data and I'm doing a new sync now. There were ~500 images in that Camera album that were not synced before and they were synced after the cleanup. Not all of them were reported as duplicates, but all of them were shot on the new phone.

from immich.