Consider S3 md5 without streaming bytes about flysystem-hash HOT 3 CLOSED

Thanks for your input, it's certainly worth considering, although I'm not quite sure if it's a good idea to do adapter-specific optimizations in such a simple plugin, since I'd like to keep it as generic as possible. But I'm open for it.

There are adapters where you can get certain checksums directly from the API (I'm using something similar to your code for Azure Blob Storage where the MD5 hash is returned as a response header), but it's not very consistent unfortunately. The ETag in S3 for example, isn't guaranteed to be an MD5 hash. Quoting from the S3 API docs:

The ETag may or may not be an MD5 digest of the object data. Whether or not it is depends on how the object was created and how it is encrypted [...]

from flysystem-hash.

I feel you. I'm not sure it belongs in a generic plugin either.

Additionally I was not aware the ETag wasn't consistent. Oops. Good to know!

For us streaming isn't a real problem as we store the hash after it has been detected locally in persistence anyway. So a file would only be streamed once if its not known yet.

Only noticed because we have some files on record without a checksum set (which was a bug) and need to repair these file records.

thanks! Bas

from flysystem-hash.

I'll close this issue then. But thanks for your input, much appreciated anyway.

from flysystem-hash.