Glob on s3 bucket should be coherent with other implementations? about universal_pathlib HOT 1 CLOSED

There is a difference between fsspec glob syntax and pathlib glob syntax, as noted in the example notebook (link added):

The glob method is also available for most filesystems. Note the syntax here is as detailed in fsspec docs, rather than that of pathlib.

The fsspec implementation has some issues, see fsspec/filesystem_spec#1019, which seems like it may be the root cause.

Based on some limited testing, the pattern /**/* seems to be treated by fsspec as an absolute path within the bucket, so it may only be a circumstantial work around if you are operating at the root of the bucket, and pathlib doesn't even accept the pattern because it says only relative patterns are accepted. There is possibly a separate bug in fsspec, because the resulting paths are incorrectly assembled.

Open PR #96 includes an improved rglob, which might be a better work around for this issue once it gets merged.

I think that addressing the issues mentioned here should happen in the fsspec implementation. I am not sure if the fsspec behavior is intended to differ from pathlib. If not, then fixing that is what is needed, but if the difference is intended, then maybe some consideration ought to be given to implementing a more pathlib compliant approach here, and I could see reasons to go either way.

Hope this helps.

from universal_pathlib.