Utilize SHA1 Attr on Large File Upload about blazer HOT 8 CLOSED

Are you sure? The X-Bz-Content-Sha1 header on the download should contain the SHA1 passed in during the upload: https://www.backblaze.com/b2/docs/b2_download_file_by_id.html and https://www.backblaze.com/b2/docs/b2_download_file_by_name.html

from blazer.

From what I can tell, that doesn't provide anything beyond allowing you to verify the checksum on download.

I think this probably fits into a larger discussion about validating files on download in general. Right now blazer does no verification at all, and I have some ideas about how to change that, but specifically for large files I definitely want to avoid reading the file twice.

from blazer.

It definitely would be useful during Download verification, and I already do this outside your library using a MultiWriter to compute checksums as the data is downloaded.

It would be useful to have the hash available on large files to begin with though.

from blazer.

I'm considering adding a method Verify(ctx) (error, bool) to the Reader, which would perform the round-trip and verify the file if it can. It would only work when (a) the SHA1 hash is available on the remote end, and (b) when the entire file has been downloaded (so no range readers).

But I'm still not sure how to allow users to set the sha1 hash for large files on upload. Doing it manually (by e.g. exposing a field on Writer) feels like punting. An alternative could be a field that simply says "performing multiple passes when uploading this is okay", and then I can collect all the hashes that I need in two passes in-place.

(And then the file type is starting to get complicated and I need to look at passing options to that... blep.)

from blazer.

I think it may make more sense to just do the check behind the scenes and trigger a retry or return an error on the download if there is a checksum mismatch, why make it an additional call when you'll probably compute the checksum as you download it and know the file to be bad already?

In my use case, I already have the checksum computed so I don't need this package to do it for me. you can do something like: If provided by user, utilize this value, else, if option is set to compute checksum, compute checksum before beginning upload (obviously, introduce an option as you mentioned)

from blazer.

In this case the additional call is to determine what value I should check against, since I don't get that information during the download.

from blazer.

Oh, look at that, that's super convenient. Yeah that'll make it a lot easier.

from blazer.

Confirmed to be working, thanks!

from blazer.