Feature request: Use adlfs to access public blobs about adlfs HOT 5 CLOSED

I've added anonymous authentication to AzureBlobFileSystem with #106 and #107 so the above goes example works as follows:

storage_options = {'account_name': 'goes'}
fs = AzureBlobFileSystem(**storage_options)
fs.ls("noaa-goes16")

An example Jupyter Notebook on Azure can be found here. The above doesn't work if you just try fs.ls("/").

I believe @raybellwaves is right -- the trick is making sure you get the account_name and container right.

BTW -- The GOES data is stored in xarray, so I didn't try opening the file.

Also successfully accessed the data here as follows:

storage_options={"account_name": "hlssa"}
fs = AzureBlobFileSystem(**storage_options)
fs.ls("hls")
ddf = dd.read_csv("abfs://hls/S2_TilingSystem2-1.txt", storage_options=storage_options)

Both .head() and .tail() work
tag: @lostmygithubaccount

from adlfs.

from adlfs.

I believe public data in Azure is stored in a slightly different way to AWS.

For example,GOES-16 on az is stored at https://goes.blob.core.windows.net. What's the nomenclature for this in az/adlfs connection_string? what you call the goes part? storage name? blob name?
I see an example here: BlobEndpoint=https://storagesample.blob.core.windows.net;. Once you have that you can access the data in the noaa-goes16 container. e.g. the path to a file would be noaa-goes16/ABI-L2-MCMIPF/2020/001/00/OR_ABI-L2-MCMIPF-M6_G16_s20200010000216_e20200010009536_c20200010010028.nc

For reference if you mount the blob the Storage Explorer you see addition info when copying the file
{"CloudHub.Azure.Storage.Blobs":{"connectionString":"BlobEndpoint=https://goes.blob.core.windows.net;SharedAccessSignature=","containerName":"noaa-goes16","subscription":null,"accountUri":"https://goes.blob.core.windows.net","sourceFolder":"ABI-L2-MCMIPF/2020/001/00/","items":[{"relativePath":"OR_ABI-L2-MCMIPF-M6_G16_s20200010000216_e20200010009536_c20200010010028.nc","snapshot":""}],"sasToken":"","service":"blob"}}

In AWS (and using s3fs) you can grab a file (note files do not overlap as have different frequency) as:

import s3fs

fs = s3fs.S3FileSystem(anon=True)
file = "OR_ABI-L2-MCMIPC-M6_G16_s20200010001184_e20200010003557_c20200010004112.nc"
fs.get("noaa-goes16/ABI-L2-MCMIPC/2020/001/00/{}".format(file), file)

Therefore would probably need extra arg
adlfs.AzureBlobFileSystem(credential=None, storage_name="goes")

That's because other datasets are hosted on different storage
e.g. https://azure.microsoft.com/en-us/services/open-datasets/catalog/hls/ is at https://hlssa.blob.core.windows.net

from adlfs.

are there any workarounds to read from public blobs currently? trying to read from Azure open datasets hosted on a public blob

from adlfs.

will this be published to pypi soon @hayesgb ?

from adlfs.