logon does not allow access to all ESGF nodes about esgf-pyclient HOT 13 CLOSED

Hey @valeriupredoi , thanks for coming back to this. Actually, this still doesn not work for me at least. However, i somehow suspect that this is maybe an issue with my openid. I just tried again with a fresh install and i still don't have access to esgf.ceda.ac.uk while esgf.dwd.de and cordexesg.dmi.dk work fine. Note, that i can download those files using the ESGF web portal, so that works fine. But consider, .e.g, this:

from netCDF4 import Dataset
from pyesgf.logon import LogonManager

urls = [
    "https://esgf.ceda.ac.uk/thredds/dodsC/esg_cordex/CORDEX/output/EUR-11/MOHC/ECMWF-ERAINT/evaluation/r1i1p1/MOHC-HadREM3-GA7-05/v1/mon/tas/v20200330/tas_EUR-11_ECMWF-ERAINT_evaluation_r1i1p1_MOHC-HadREM3-GA7-05_v1_mon_198212-199012.nc",
    "http://cordexesg.dmi.dk/thredds/dodsC/cordex_general/cordex/output/EUR-11/DMI/ECMWF-ERAINT/evaluation/r1i1p1/DMI-HIRHAM5/v1/mon/pr/v20140620/pr_EUR-11_ECMWF-ERAINT_evaluation_r1i1p1_DMI-HIRHAM5_v1_mon_198901-199012.nc",
    "https://esgf.dwd.de/thredds/dodsC/esgf2_1/cordex/output/EUR-11/CLMcom/MIROC-MIROC5/rcp26/r1i1p1/CLMcom-CCLM4-8-17/v1/mon/tas/v20180707/tas_EUR-11_MIROC-MIROC5_rcp26_r1i1p1_CLMcom-CCLM4-8-17_v1_mon_200601-201012.nc",
]

lm = LogonManager()

# logon
if not lm.is_logged_on():
    myproxy_host = "esgf-data.dkrz.de"
    lm.logon(hostname=myproxy_host, interactive=True, bootstrap=True)

print(lm.is_logged_on())

for url in urls:
    try:
        Dataset(url)
        print(f"This worked: {url}")
    except Exception as e:
        print(e)
        print(f"Failed: {url}")

which only works for the dwd data node:

[Errno -90] NetCDF: file not found: 'https://esgf.ceda.ac.uk/thredds/dodsC/esg_cordex/CORDEX/output/EUR-11/MOHC/ECMWF-ERAINT/evaluation/r1i1p1/MOHC-HadREM3-GA7-05/v1/mon/tas/v20200330/tas_EUR-11_ECMWF-ERAINT_evaluation_r1i1p1_MOHC-HadREM3-GA7-05_v1_mon_198212-199012.nc'
Failed: https://esgf.ceda.ac.uk/thredds/dodsC/esg_cordex/CORDEX/output/EUR-11/MOHC/ECMWF-ERAINT/evaluation/r1i1p1/MOHC-HadREM3-GA7-05/v1/mon/tas/v20200330/tas_EUR-11_ECMWF-ERAINT_evaluation_r1i1p1_MOHC-HadREM3-GA7-05_v1_mon_198212-199012.nc

syntax error, unexpected $end, expecting SCAN_ATTR or SCAN_DATASET or SCAN_ERROR
context: ^

This worked: http://cordexesg.dmi.dk/thredds/dodsC/cordex_general/cordex/output/EUR-11/DMI/ECMWF-ERAINT/evaluation/r1i1p1/DMI-HIRHAM5/v1/mon/pr/v20140620/pr_EUR-11_ECMWF-ERAINT_evaluation_r1i1p1_DMI-HIRHAM5_v1_mon_198901-199012.nc
This worked: https://esgf.dwd.de/thredds/dodsC/esgf2_1/cordex/output/EUR-11/CLMcom/MIROC-MIROC5/rcp26/r1i1p1/CLMcom-CCLM4-8-17/v1/mon/tas/v20180707/tas_EUR-11_MIROC-MIROC5_rcp26_r1i1p1_CLMcom-CCLM4-8-17_v1_mon_200601-201012.nc

I think it would be good to still keep this open since it's different from #84 and i'll ask some colleagues to check this too. For me, the main drawback is not having access to the CEDA ESGF data node. I wish this CORDEX logon could be removed after all sometime... 😒

from esgf-pyclient.

This looks very similar to #84. Is this a different issue?

from esgf-pyclient.

Today i tried accessing over the web portal after it didn't work in my script. i got something i have never seen before (i'm registered with CORDEX_Commercial/Research)

Just want to drop it off here since it might be related...

from esgf-pyclient.

We have currently some issues with cordexesg.dmi.dk after moving to new hardware. We are working on a solution. Sorry for that.

from esgf-pyclient.

@soay Thanks! I'll try again soon!

from esgf-pyclient.

Hi @larsbuntemeyer - did it eventually work? Can this be closed, please? Cheers 🍺

from esgf-pyclient.

Note, i updated my comment, since i had a typo in the dmi url.

from esgf-pyclient.

I've seen some really fishy stuff going on lately with the CEDA Thredds catalog, might have to ping them figure out what they're up to, BTW I don't see any dodsC in their main catalog at all https://esgf.ceda.ac.uk/thredds/catalog/catalog.html

from esgf-pyclient.

at any rate, I am the new maintainer of this package, not in the know much about the actual functional subtleties, for that prob best to ping @bouweandela and/or @alaniwi 👍

from esgf-pyclient.

However, i somehow suspect that this is maybe an issue with my openid.

Could you try if you can reproduce the issue using e.g. wget? Then we are sure that it is not related to esgf-pyclient and you could ask for help on the ESGF mailinglist.

from esgf-pyclient.

@bouweandela Yes, i can confirm that i also have this issue uswing wget/curl. However, i can access esgf-data1.ceda.ac.uk via https download, e.g.,

curl https://esgf-data1.ceda.ac.uk/thredds/fileServer/esg_cordex/CORDEX/output/EUR-11/MOHC/ECMWF-ERAINT/evaluation/r1i1p1/MOHC-HadREM3-GA7-05/v1/mon/tas/v20200330/tas_EUR-11_ECMWF-ERAINT_evaluation_r1i1p1_MOHC-HadREM3-GA7-05_v1_mon_198212-199012.nc -o tas_EUR-11_ECMWF-ERAINT_evaluation_r1i1p1_MOHC-HadREM3-GA7-05_v1_mon_198212-199012.nc

but i can not access the same file via opendap here:

http://esgf.ceda.ac.uk/thredds/dodsC/esg_cordex/CORDEX/output/EUR-11/MOHC/ECMWF-ERAINT/evaluation/r1i1p1/MOHC-HadREM3-GA7-05/v1/mon/tas/v20200330/tas_EUR-11_ECMWF-ERAINT_evaluation_r1i1p1_MOHC-HadREM3-GA7-05_v1_mon_198212-199012.nc

It was a little surprised that i can now access the data without login! That's great! I would confirm that this is not an issue with esgf-pyclient.

from esgf-pyclient.

I've heard 2 things that probably explain the above observations:

1. opendap on ESGF is deprecated and will not be available on systems that make use of the new deployment
2. logging in to access CORDEX data will no longer be needed 'soon'

but I'm not personally involved in running ESGF, so your best bet to get some answers is contacting the ESGF user mailing list.

from esgf-pyclient.

@bouweandela Thanks for the updates, i'll check out the mailing list. I think this could be closed then!

from esgf-pyclient.