Store files to unique paths instead of filenames about cdp-backend HOT 4 CLOSED

Hi. I am interested in working on this issue.

from cdp-backend.

Update: I think this and #119 are the two primary reasons that the pipeline is currently failing for seattle-staging.

There are some files we are using that have filenames / URIs like https://some-domain.com/route?some-guid and when we save the files with our current pattern they get saved to our storage as gs://{infra}/route?some-guid... the ? or any special URI handling character causes fsspec.exists to fail it's check which fails the whole event upload.

Definitely need to save these files with their content hash as the filename.

from cdp-backend.

You can see this behavior:

fireo.fields.errors.FieldValidationFailed: "TextField" failed validation with value gs://cdp-jackson-dev-065.appspot.com/photo-1523665307251-7f4e0e7b23fc?ixid=MnwxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8

from cdp-backend.

Update: I think this and #119 are the two primary reasons that the pipeline is currently failing for seattle-staging.

There are some files we are using that have filenames / URIs like https://some-domain.com/route?some-guid and when we save the files with our current pattern they get saved to our storage as gs://{infra}/route?some-guid... the ? or any special URI handling character causes fsspec.exists to fail it's check which fails the whole event upload.

Definitely need to save these files with their content hash as the filename.

Really nice find, I didn't realize that the special characters would actually cause exists to fail. And I agree that the filename should be the content hash in this case.

from cdp-backend.