Cutout.prepare() no reponse about atlite HOT 15 CLOSED

1. The documentation on https://atlite.readthedocs.io is for the upcoming version of atlite which we have yet to finish in the next weeks. You seem to be running on another version. Which version do you have installed?
2. Preparing the cutout does take rather long and uses all CPU ressources it can get its hands on. This will get better with the new version.
3. The code you posted seems incomplete. Is this the same code that produces the error when you run the script from your command line?

from atlite.

@euronion Thank you for the comment!

1. Thanks for the information. I am using version 0.0.2.
2. I have updated the original problem description with the complete code and error log. Yes, there is no error reported when I run it in Spyder, but the problem is that it keeps runs for at least 2hrs without any response/output.
3. I run the same code in the command line, and it firstly shows the same message ( "...Starting running them on all processors") and then keeps printing repeatedly the error I reported. Sorry for the formatting.

from atlite.

Hi @fang1103 ,

Thanks for the code update. Are you 100% sure it is the one you are running? Your output contains DEBUG output from the logging package, which is only configured for INFO in your code above.

I tried reproducing the problem on windows and on linux. On both plattforms the code above runs without problems and finishes within a minute or so on my laptop.

INFO:atlite.preparation:Starting preparation of cutout 'ukie-2011-01'
INFO:atlite.preparation:1 tasks have been collected. Starting running them on all processors.
INFO:atlite.preparation:Merging variables into monthly compound files
DEBUG:atlite.preparation:Completed file 201101.nc
INFO:atlite.preparation:Cutout 'ukie-2011-01' has been successfully prepared

Unfortunatley it is not clear to me, how your error above comes to be.

from atlite.

Hi @euronion

Sorry for the original misleading post, the output containing DEBUG output was when I used the following code

import warnings
import logging

warnings.simplefilter('default', DeprecationWarning)
logging.captureWarnings(True)
logging.basicConfig(level=logging.INFO)

I updated the message log after suppressing the DEBUG output in my original post.
But the problem is the same, my computer keeps running for hours without any output after
INFO:atlite.preparation:1 tasks have been collected. Starting running them on all processors. with Spyder, and it keeps reporting the error repeatedly with the command line.

It is not normal, right? I've tested the CDSAPI with the following code, and it works perfectly with a download rate 4.7M/s. I've also removed and reinstalled atlite, but didn't help...I will try to run it on Linux to see what happens.

import cdsapi
c = cdsapi.Client()
c.retrieve("reanalysis-era5-single-levels",
{
"variable": "runoff",
"product_type": "reanalysis",
"year": "2008",
"month": "01",
"day": "01",
"time": "12:00",
"format": "netcdf"
},
"runoff.nc")

from atlite.

@euronion Hi, I come back to you after I set nprocesses=1 in cutout.prepare(), which can help to see more clearly the error bug:
The following message keeps showing repeatedly when I am running the code in the command line:

C:\Users\Yiping\Miniconda3\lib\site-packages\pyproj\crs\crs.py:55: FutureWarning: '+init=<authority>:<code>' syntax is deprecated. '<authority>:<code>' is the preferred initialization method. When making the change, be mindful of axis order changes: https://pyproj4.github.io/pyproj/stable/gotchas.html#axis-order-changes-in-proj-6
  return _prepare_from_string(" ".join(pjargs))
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 114, in _main
    prepare(preparation_data)
  File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 225, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\Yiping\Miniconda3\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
    run_name="__mp_main__")
  File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 263, in run_path
    pkg_name=pkg_name, script_name=fname)
  File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "C:\Users\Yiping\Miniconda3\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\Yiping\Documents\Python Workspace\REUTE\climate\atlite_example.py", line 30, in <module>
    months=slice(1,1))
  File "C:\Users\Yiping\Miniconda3\lib\site-packages\atlite\cutout.py", line 60, in __init__
    assert False
AssertionError

from atlite.

Thanks for the updates.

The code modifications were not significant, but sometimes it can happen that one accidentally uses different code than the one that is posted - just making sure that's not the case :)

Setting nprocesses=1 really helped making things more clear.

You are running into these lines

Can you check whether the directory $cutout_dir/uk-2011-01 for storing the cutout files exists?
If it does, delete or rename it, including all contents, and try again.

from atlite.

@euronion Thanks for the quick message! I've tracked to this part, but still haven't figured out what's the problem.

The directory $cutout_dir/uk-2011-01 does not exist when I was running the code and having the above-reported message, then I have this directory created by the code after I interrupted its running with Ctrl+C (since it keeps reporting the AssertionError continuously)

If the directory $cutout_dir/uk-2011-01 exists, I got the following error and the code stops runing:

C:\Users\Yiping\Miniconda3\lib\site-packages\pyproj\crs\crs.py:55: FutureWarning: '+init=<authority>:<code>' syntax is deprecated. '<authority>:<code>' is the preferred initialization method. When making the change, be mindful of axis order changes: https://pyproj4.github.io/pyproj/stable/gotchas.html#axis-order-changes-in-proj-6
  return _prepare_from_string(" ".join(pjargs))
Traceback (most recent call last):
  File "C:\Users\Yiping\Documents\Python Workspace\REUTE\climate\atlite_example.py", line 29, in <module>
    months=slice(1,1))
  File "C:\Users\Yiping\Miniconda3\lib\site-packages\atlite\cutout.py", line 60, in __init__
    assert False
AssertionError

from atlite.

That's weird, it should stop at this point if the directories it wants to write only partially exist.

From which source are you installing? PyPI, conda-forge or GitHub?
Could you retry with a fresh conda environment based on the environment.yaml file + conda install -c conda-forge cartopy atlite cdsapi ?

from atlite.

@euronion I installed atlite from the conda-forge using conda install atlite with:

> channels:
>   - conda-forge
>   - defaults
> channel_priority: flexible

Following your suggestion, I created a fresh conda environment using the environment.yaml file by
conda env create -f environment.yaml
and installed all the packages
conda install -c conda-forge cartopy atlite cdsapi
and run the code there, unfortunately it is the same result, very weird...

from atlite.

Ok, so it doesn't seem to be a package / dependency-specific issue.
Let's wait and see what your results from testing it on linux are.

from atlite.

Hi @euronion, it works when I set up the conda environment and run the code in the VM Virtualbox-based Ubuntu system. I had the cutout successfully.
Then, I re-install the anaconda and re-setup all the things in my windows but still has the same problem, it just doesn't work...very weird
Which setting are you using in your windows?

I looked at the code again, and suspect it might be the problem of the multiprocessing, for which Windows and Linux have different mechanisms?

from atlite.

hi @fang1103, in a sense you are right that it is multiprocessing causing the problem here and it would be interesting to compare how each of you start their code.

How multiprocessing in windows works is that it starts a new python interpreter and makes it load the same module/script again. Up to now, I always thought, it would only use the module from which one sets up the multiprocessing pool but in your case the stacktrace shows clearly that multiprocessing starts your script again. So to fix your problem, move your logic into a main function. I would wagger that @euronion runs your code from within an ipykernel either through ipython or jupyter in which case multiprocessing will not re-run the script in the newly spawned python engines, so it goes through fine. On linux you don't have the problem since you can just fork instead of start a new python interpreter. Multiprocessing is weird. Let's get v0.2 out and get rid of that bit of code!

import geopandas as gpd
import cartopy.io.shapereader as shpreader
import atlite
import logging

def main():
    shpfilename = shpreader.natural_earth(resolution='10m',
                                        category='cultural',
                                        name='admin_0_countries')
    reader = shpreader.Reader(shpfilename)
    UkIr = gpd.GeoSeries({r.attributes['NAME_EN']: r.geometry
                        for r in reader.records()},
                        crs={'init': 'epsg:4326'}
                        ).reindex(['United Kingdom', 'Ireland'])

    # Define the cutout; this will not yet trigger any major operations
    cutout = atlite.Cutout(name="uk-2011-01",
                        cutout_dir="./",
                        module="era5",
                        bounds=UkIr.unary_union.bounds,
                        years=slice(2011,2011),
                        months=slice(1,1))

    # This is where all the work happens.
    cutout.prepare()

if __name__ == "__main__":
    logging.basicConfig(level=logging.INFO)
    main()

from atlite.

@coroa Thank you very much for the clear explanation!! Now it works, seems we need to always protect the multiprocessing code.

from atlite.

from atlite.

[...] I would wagger that @euronion runs your code from within an ipykernel either through ipython or jupyter [...]

Correct, I was testing it in jupyter lab.

Thanks for the help @coroa !

from atlite.