builder for zarr files in the same file structure about intake-esm-datastore HOT 8 CLOSED

My case is extremely simple since all zarr stores are stored in a very regular directory structure and I am using consolidated metadata. So I just list all of the .zmetadata files and parse the paths into a pandas dataframe. This is the script we used on the GC collection itself. If you use glob to create a list of all .zmetadata files, then you can just use the second half of the script.

import datetime
import io

from google.cloud import storage
import pandas as pd
def index_cmip6(request):
    
    SOURCE_BUCKET = 'pangeo-cmip6'
    TARGET_BUCKET = 'pangeo-cmip6'
    TARGET_FILENAME = 'pangeo-cmip6.csv'
    
    storage_client = storage.Client()
    zarr_blobs = [blob.name for blob in storage_client.list_blobs(SOURCE_BUCKET)
                                if '.zmetadata' in blob.name]

    df = pd.DataFrame(zarr_blobs, columns=['store'])

    files = df.store.values
    ddict = {}
    for item, tdir in enumerate(files):
        store = 'gs://cmip6/' + tdir.split('.zmetadata')[0]
        vlist = tdir.split('/')[-9:-1]
        vlist += [store]
        ddict[item] = vlist

    dz = pd.DataFrame.from_dict(ddict, orient='index')
    dz = dz.rename(columns={0: "activity_id", 1: "institution_id", 2:"source_id",
                            3:"experiment_id",4:"member_id",5:"table_id",6:"variable_id",
                            7:"grid_label",8:"zstore"})

    bucket = storage_client.get_bucket(TARGET_BUCKET)
    blob = bucket.blob(TARGET_FILENAME)
    with io.StringIO() as f:
        dz.to_csv(f, mode='w', index=False)
        f.seek(0)
        blob.upload_from_string(f.read(), content_type='text/csv')

from intake-esm-datastore.

That is very helpful. I think my setup is actually very similar! Ill give those a shot. I will also try your suggestion @andersy005 and give you some feedback on what is returned.

from intake-esm-datastore.

@andersy005 , quite right. I have fixed the problem in the data catalog and added a check in the scripts. What happens is that, if two separate requests are being filled at the same time, asking for the same dataset, the one zarr store is put inside the other (hence the 'gn/gn' in the pathname). It is kind of a freak occurance, but now my scripts should catch it. Thanks!

from intake-esm-datastore.

Quick reply. I was able to construct a catalog by modifying @naomi-henderson script. Haven’t been able to try to use the intake-ram-datastore builders yet. Ill try to get to it soon.

from intake-esm-datastore.

Closing this as it appears to have been addressed.

from intake-esm-datastore.

Can you post an example path of one of your zarr stores?

This seems very similar to what @naomi-henderson has done for the pangeo cloud. Are those scripts available publicly?

Ccing @charlesbluca as he may have pointers to the scripts used for the CMIP6 data.

from intake-esm-datastore.

Assuming you saved your zarr stores with .zarr extension, can try calling get_asset_list() function to see if you can get the list of all zarr stores at least?

I am not 100% confident that it will work out of the box...

Once you have a list of your zarr stores, constructing the dataframe should be straightforward.

from intake-esm-datastore.

@naomi-henderson

For some reason, there are two zarr stores in the Pangeo catalog whose attributes were not properly parsed. See screenshot below.

Notice how there's activity_id = 'AWI', I believe the activity_id for these stores should have been CMIP, institution_id='AWI', etc...

from intake-esm-datastore.