Original reported by: @sujata91

This can cause issues for users, as easymore has shown to be fussy about /'s in the path addresses. This can be easily avoided by using either pathlib or os.path modules.

I think the solution is fairly straightforward.

Would be nice to resolve as it adds up in the log file.

/globalhome/drc858/HPC/geopandas-env/lib/python3.7/site-packages/easymore/easymore.py:374: DeprecationWarning: np.bool is a deprecated alias for the builtin bool. To silence this warning, use bool by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use np.bool_ here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
lat_value = np.array(ncid.variables[self.var_lat])
/globalhome/drc858/HPC/geopandas-env/lib/python3.7/site-packages/easymore/easymore.py:375: DeprecationWarning: np.bool is a deprecated alias for the builtin bool. To silence this warning, use bool by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use np.bool_ here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
lon_value = np.array(ncid.variables[self.var_lon])

The package currently lacks unit tests, and there is no way to tell if each contribution still maintains the sanity of the numbers generated by the package.

This issue is to raise awareness that some unit tests are mandatory for the package. It is recommended to submit test scripts by pytest.

EASYMORE checks if lat and lon cells are spaced equally as follows:

In a case with more than 2 latitude or longitude cells, this works as expected:

nc_file_8lat = 'C:/Globus endpoint/CAMELS_spat/camels-spat-data/basin_data/CAN_01AD002/forcing/raw/ERA5_1950-01.nc'
with nc4.Dataset(nc_file_8lat) as nc:
    lat_temp  = np.array(nc.variables['latitude'][:])
    print(f'lat_temp = {lat_temp}')

    lat_temp_diff = np.diff(lat_temp)
    print(f'lat_temp_diff = {lat_temp_diff}')

    lat_temp_diff_2 = np.diff(lat_temp_diff)
    print(f'lat_temp_diff_2 = {lat_temp_diff_2}')

    max_lat_temp_diff_2 = max(abs(lat_temp_diff_2))
    print('max difference of lat values in source nc files are : ', max_lat_temp_diff_2)
----------
lat_temp = [47.75 47.5  47.25 47.   46.75 46.5  46.25 46.  ]
lat_temp_diff = [-0.25 -0.25 -0.25 -0.25 -0.25 -0.25 -0.25]
lat_temp_diff_2 = [0. 0. 0. 0. 0. 0.]
max difference of lat values in source nc files are :  0.0

In cases with exactly 2 lat or lon cells the second np.diff() returns an empty array, and the max_lat_temp_diff_2 line crashes as follows:

nc_file_2lat = 'C:/Globus endpoint/CAMELS_spat/camels-spat-data/basin_data/CAN_01AF007/forcing/raw/ERA5_1977-01.nc'
with nc4.Dataset(nc_file_2lat) as nc:
    lat_temp  = np.array(nc.variables['latitude'][:])
    print(f'lat_temp = {lat_temp}')

    lat_temp_diff = np.diff(lat_temp)
    print(f'lat_temp_diff = {lat_temp_diff}')

    lat_temp_diff_2 = np.diff(lat_temp_diff)
    print(f'lat_temp_diff_2 = {lat_temp_diff_2}')

    max_lat_temp_diff_2 = max(abs(lat_temp_diff_2))
    print('max difference of lat values in source nc files are : ', max_lat_temp_diff_2)

--------------
lat_temp = [47.5  47.25]
lat_temp_diff = [-0.25]
lat_temp_diff_2 = []

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[27], line 12
      9 lat_temp_diff_2 = np.diff(lat_temp_diff)
     10 print(f'lat_temp_diff_2 = {lat_temp_diff_2}')
---> 12 max_lat_temp_diff_2 = max(abs(lat_temp_diff_2))
     13 print('max difference of lat values in source nc files are : ', max_lat_temp_diff_2)

ValueError: max() arg is an empty sequence

Wrapping the code block above in checks if len(lat_temp) > 2 and if len(lon_temp) > 2, while setting max_lat_temp_diff2 and max_lon_temp_diff2 to 0 if not, might be enough to fix this.

There are multiple deprecation warnings printed to the users. An example is in the following:

/path/toEASYMORE/src/easymore/remapper.py:521: UserWarning: Column names longer than 10 characters will be truncated when saved to ESRI Shapefile.
  shp_int.to_file(self.temp_dir+self.case_name+'_intersected_shapefile.shp') # save the intersected files
/path/to/EASYMORE/src/easymore/remapper.py:558: PendingDeprecationWarning: dropping variables using `drop` will be deprecated; using drop_vars is encouraged.
  attr = attr.drop('index')
/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Compiler/gcc9/eccodes/2.25.0/lib/python3.10/site-packages/cfgrib-0.9.10.1-py3.10.egg/cfgrib/xarray_plugin.py:11: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
/path/to/virtual-envs/scienv2/lib/python3.10/site-packages/setuptools/_distutils/version.py:345: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  other = LooseVersion(other)

It would be great if these (or mainly, the deprecation ones) could be addressed.

Hi Shervan et al.,

I was trying to install the EASYMORE package on Graham (following the steps outlined here: https://github.com/ShervanGharari/EASYMORE/tree/main/env#linux) and I encountered an error while loading the package. The error was related to the mpi library import error ompi_mpi_info_null while loading the netCDF4 library.

I fixed the problem by adjusting the required modules by loading python 3.9.6 and the latest mpi4py. I used the following command module load python/3.9.6 scipy-stack/2022a mpi4py

I also made sure to install easymore using python -m pip install easymore to ensure that I use the current python version (3.9.6) to install the compatible libraries.

EASYMORE is working fine after those tweaks (based on the test case).

EASYMORE nc_remapper() may not work with non-standard calendars such as 365. This will be fixed in the next version.
Meanwhile, you can use local installation by following the steps below.

git clone https://github.com/ShervanGharari/EASYMORE.git
cd EASYMORE/easymore

in EASYMORE/easymore folder replace line 1328 of easymore.py with (make sure with the right indentation as the current code):

ds_temp = ds.isel(time=m)

Finally, install locally making sure the existing version is uninstalled

pip uninstall easymore
pip install path/to/where/EASYMORE/is/cloned/.

gdal-config is a Unix-only command: https://gdal.org/programs/gdal-config.html

Equivalent on Widnows would be:

> gdalinfo --version
GDAL 3.2.1, released 2020/12/29

A brief description of the remapped .csv file could be helpful. The starting point for rows and columns in the file is not described properly, and unsure whether indexing starts from 0 or 1.

Processing dataset r10i2p1r4 results in an error for the wind variable. All other variables are OK.
I've attached the error file as well as the easymore Python file and the output file.
easymore_wind_files.zip

All other variables are OK.
There does not seem to be any issue with the datatool as running it did not produce errors.

A

Currently, the remapped output nc files are named using the case name and an extension of starting date and time. This may result in nc files with a similar starting date and time to be overwritten during remapping. The following will be added to the code:

1. Make sure the input source nc files have a unique name even if they are located in different folders.
2. Use the original file name for the remapped file with remapped extension
3. include the source nc file name (and path) in the global attribute of remapped nc file.

One in the main folder, one in subfolder /candex. Can one be removed?

Use case:

User (for whatever reason) tries to remap a netcdf file that only has spatial dimensions of size 1 (i.e., a single latitude and longitude value; 1 grid cell).

Current behaviour:

EASYMORE crashes on calling nc_remapper():

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)

File C:\Globus endpoint\CAMELS_spat\camels-spat-env\lib\site-packages\easymore\easymore.py:152, in easymore.nc_remapper(self)
    150     print(self.temp_dir+self.case_name+'_target_shapefile.shp')
    151 # create source shapefile
--> 152 source_shp_gpd = self.create_source_shp()
    153 if self.save_temp_shp:
    154     source_shp_gpd.to_file(self.temp_dir+self.case_name+'_source_shapefile.shp')

File C:\Globus endpoint\CAMELS_spat\camels-spat-env\lib\site-packages\easymore\easymore.py:286, in easymore.create_source_shp(self)
    284 import geopandas as gpd
    285 # find the case, 1: regular, 2: rotated, 3: irregular Voronoi diagram creation if not provided
--> 286 self.NetCDF_SHP_lat_lon()
    287 # create the source shapefile for case 1 and 2 if shapefile is not provided
    288 if (self.case == 1 or self.case == 2):

File C:\Globus endpoint\CAMELS_spat\camels-spat-env\lib\site-packages\easymore\easymore.py:716, in easymore.NetCDF_SHP_lat_lon(self)
    714 lat_temp_diff = np.diff(lat_temp)
    715 lat_temp_diff_2 = np.diff(lat_temp_diff)
--> 716 max_lat_temp_diff_2 = max(abs(lat_temp_diff_2))
    717 print('max difference of lat values in source nc files are : ', max_lat_temp_diff_2)
    718 lon_temp = np.array(ncid.variables[self.var_lon][:])

ValueError: max() arg is an empty sequence

Desired behavior:

Options:

1. EASYMORE performs a quick check on self.source_nc to see if at least one of self.var_lat and self.var_lon is larger than 1. If not, gracefully exit and inform user that for this file remapping is not needed.
2. Perform remapping anyway (essentially just skip any remapping steps and simply create the target netCDF file with identical data values as source netCDF)

This error comes up with easymore installation using pip install easymore from PyPI. it will be fixed in the next release.
It is fixed in the code on the repo (both branches). So please follow the local install and do not forget to uninstall the exiting installed easymore before installing it locally (pip uninstall easymore).

UnboundLocalError                         Traceback (most recent call last)
<ipython-input-33-2ad4158b6f4a> in <module>
----> 1 esmr.nc_remapper()
D:\PostDoc\EASYMORE-main\EASYMORE-main\easymore\easymore.py in nc_remapper(self)
     75             self.check_source_nc()
     76             # find the case
---> 77             self.NetCDF_SHP_lat_lon()
     78             # create the source shapefile for case 1 and 2 if shapefile is not provided
     79             if (self.case == 1 or self.case == 2)  and (self.source_shp == ''):
D:\PostDoc\EASYMORE-main\EASYMORE-main\easymore\easymore.py in NetCDF_SHP_lat_lon(self)
    596             self.lat = lat
    597             self.lon = lon
--> 598             self.lat_expanded = lat_expanded
    599             self.lon_expanded = lon_expanded
    600         # case #2 rotated lat/lon
UnboundLocalError: local variable 'lat_expanded' referenced before assignment

easymore remapper takes significant time to check the similarity of lat/lon or name of variables across thousand of source nc files.
This should be written in a more efficient way.

While running easymore for our .nc file, there was an error during the longitude values correction from 0 and 360 to -180 to 180. We solved the 365 calendar day issue as described in 'Handeling calendar 365 [or non-standard] #49' but had to use the following command to fix the current error.

ncap2 -O -s 'where(lon>180) lon=lon-360' input.nc output.nc

Can_pr_crop_time.zip
FRB_SubBasins_polygon.zip
03_CC_extract_Lat_Lon.zip

The portion above seems to be missing from CLI features. Something to be added for users' comfort.

Reported-by: Zelalem Tesemma

Hello,

I am recently using EASYMORE to map some large dataset, for instance lat/lon dimension 3500/7000. Current version raises errors when writing shp file larger than ~4Gb. One workaround would be using geopandas to write geopackage instead of pyshp. It would be great to address it if others like me are interested in macro-scale hydrologic modeling.

I forked this repo and modified some of your codes:

1. writing a large dataset (>4Gb)
2. replace some for-loops with numba acceleration (for example method lat_lon_SHP contains double for-loops).
3. Implemented multiprocessing to fully utilize cpu in server (only tested for case 1 and 2).
4. use a configuration file to handle model inputs

Hope this is helpful :)
you can find my fork here:
https://github.com/chrimerss/EASYMORE/tree/main/easymore

There is an issue when SLURM job submission files are provided externally. The error is like the following:

Traceback (most recent call last):
  File "/home/kasra545/virtual-envs/scienv2/bin/easymore", line 33, in <module>
    sys.exit(load_entry_point('easymore', 'console_scripts', 'easymore')())
  File "/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Compiler/gcc9/eccodes/2.25.0/lib/python3.10/site-packages/click-8.1.2-py3.10.egg/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Compiler/gcc9/eccodes/2.25.0/lib/python3.10/site-packages/click-8.1.2-py3.10.egg/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Compiler/gcc9/eccodes/2.25.0/lib/python3.10/site-packages/click-8.1.2-py3.10.egg/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Compiler/gcc9/eccodes/2.25.0/lib/python3.10/site-packages/click-8.1.2-py3.10.egg/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Compiler/gcc9/eccodes/2.25.0/lib/python3.10/site-packages/click-8.1.2-py3.10.egg/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/kasra545/github-repos/EASYMORE/src/easymore/scripts/main.py", line 113, in from_cli
    submit_hpc_job(kwargs[cache],
  File "/home/kasra545/github-repos/EASYMORE/src/easymore/scripts/main.py", line 168, in submit_hpc_job
    job_str = pkgutil.get_data(__name__, job_conf_file).decode()
  File "/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Core/python/3.10.2/lib/python3.10/pkgutil.py", line 639, in get_data
    return loader.get_data(resource_name)
  File "<frozen importlib._bootstrap_external>", line 1073, in get_data
FileNotFoundError: [Errno 2] No such file or directory: '/home/kasra545/github-repos/EASYMORE/src/easymore/scripts/home/kasra545/scratch/easymore-job.slurm'

The suggested solution is to make sure if the file is externally provided, then the pkgutil in the following lines are not used, as files starting with `'/' are not allowed:

Problem statement:

I am trying to use easymore with a NetCDF file that it's NaNs values are defined as 32,768.

The issue is that easymore fails to consider a user-defined value for NaNs (or missing values). Therefore, the area averaged data is corrupted.

The shapefile used as the target and sample of such a NetCDF file is attached below for further discussion.

05BB001_shapefile.zip
MOD16A2.006_500m_aid0001_05BB001.nc.zip

System details:

• Operating System: Windows 10 - 64-bit

• Python: 3.7.10 (default, Feb 26 2021, 13:06:18) [MSC v.1916 64 bit (AMD64)] :: Anaconda, Inc. on win32

• EASYMORE: 0.0.0 (?)

It would be good to add the capability to use xarray.Dataset objects instead of only reading a list of netCDF files as input.

This gives flexibility, as xarray.Datasets could be loaded using Dask.

Problem statement:

I am trying to use easymore with a NetCDF file that uses a non-geographical coordinate system for its latitude and longitude gird cells.

The issue is that easymore always assumes the grid cells are defined using degrees.

The shapefile used as the target and sample of such a NetCDF file is attached below for further discussion. It should be noted that the NaNs are defined as 32,768 here.

MOD16A2.006_500m_aid0001_05BB001_meters.nc.zip
05BB001_shapefile.zip

System details:

• Operating System: Windows 10 - 64-bit

• Python: 3.7.10 (default, Feb 26 2021, 13:06:18) [MSC v.1916 64 bit (AMD64)] :: Anaconda, Inc. on win32

• EASYMORE: 0.0.0 (?)

The package seems to have issues when there is a "jump" in longitude values? It is better to convert all input longitude values from whatever range into a [0, 360] and do necessary calculations with this assumption on mind.

The following may provide further assistance for such a conversion: https://gis.stackexchange.com/a/201796

easymore configuration cannot be populated from the config file on Graham, more testing/coding is needed to make sure the key names are imported properly across platforms.

Often times errors like this are thrown at the user:

SystemExit: the provided output folder for EASYMORE should end with (/)

This could be avoided using os.path or pathlib libraries inside the package.

The logic in

The --submit-job flag can enable accepting more argument for scheduler options. The more users are using this platform, the more flexibility is needed.

If values of the target_shp_ID column within the target_shp is of type string, then the resulting remapped_var_ID values are NaNs.

MCVE

import pandas as pd
import geopandas as gpd
import xarray as xr

import requests
import re
import json

# ArcGIS REST API endpoint for the specific ATS layer
alta_ats = "https://geospatial-sit.alberta.ca/titan/rest/services/base/alberta_township_system/MapServer/7/query"
# list of townships needed
township_list = [
    'T001R08W4',
    'T001R12W4',
    'T001R13W4',
    'T001R18W4',
    'T001R19W4',
    'T001R20W4',
    'T001R22W4',
    'T001R23W4',
    'T002R11W4',
    'T002R12W4',
    'T002R13W4',
    'T002R14W4',
    'T002R15W4',
    'T002R16W4',
    'T002R17W4',
    'T002R18W4',
    'T002R21W4',
    'T002R23W4'
]
# list of township IDs based on the `PID` column of the source ATS data
ids_list = [''.join(re.split("[TRW]", ats)[::-1]) for ats in township_list]
# string of list items
list_str = ', '.join([f"'{item}'" for item in ids_list])
# creating a custrom SQL query statement to find `ids_list` values in the `PID` column
sql_query = f"FILE_NAME IN ({list_str})"
# Specifying query parameters
params_ag = {
    'where': sql_query,
    'outFields': '*',
    'returnGeometry': 'true',
    'f': 'geojson',  # Output format as GeoJSON
}

# Make the request to the ArcGIS REST API
response_ag = requests.get(alta_ats, params=params_ag)

# Check if the request was successful
if response_ag.status_code == 200:
    # Load GeoJSON data into a GeoDataFrame
    gdf_ag = gpd.read_file(response_ag.text)
else:
    print(f"Error: {response_ag.status_code} - {response_ag.text}")

The Shapefile looks like the following:

# printing the GeoDataFrame
print(gdf_ag)
       PID FILE_NAME     TRM  M  RGE  TWP            DESCRIPTOR  OBJECTID  \
0   419001    419001  001194  4   19    1  TWP-001 RGE-19 MER-4       158   
1   413002    413002  002134  4   13    2  TWP-002 RGE-13 MER-4       370   
2   411002    411002  002114  4   11    2  TWP-002 RGE-11 MER-4      1634   
3   416002    416002  002164  4   16    2  TWP-002 RGE-16 MER-4      1959   
4   418002    418002  002184  4   18    2  TWP-002 RGE-18 MER-4      2002   
5   418001    418001  001184  4   18    1  TWP-001 RGE-18 MER-4      2752   
6   415002    415002  002154  4   15    2  TWP-002 RGE-15 MER-4      2808   
7   414002    414002  002144  4   14    2  TWP-002 RGE-14 MER-4      3741   
8   412001    412001  001124  4   12    1  TWP-001 RGE-12 MER-4      3757   
9   420001    420001  001204  4   20    1  TWP-001 RGE-20 MER-4      3947   
10  421002    421002  002214  4   21    2  TWP-002 RGE-21 MER-4      4306   
11  423001    423001  001234  4   23    1  TWP-001 RGE-23 MER-4      4948   
12  417002    417002  002174  4   17    2  TWP-002 RGE-17 MER-4      5068   
13  412002    412002  002124  4   12    2  TWP-002 RGE-12 MER-4      5425   
14  423002    423002  002234  4   23    2  TWP-002 RGE-23 MER-4      6052   
15  413001    413001  001134  4   13    1  TWP-001 RGE-13 MER-4      6066   
16  408001    408001  001084  4    8    1  TWP-001 RGE-08 MER-4      6695   
17  422001    422001  001224  4   22    1  TWP-001 RGE-22 MER-4      6959   

    SHAPE.STArea()  SHAPE.STLength()  \
0         0.011785          0.443881   
1         0.011649          0.441466   
2         0.011654          0.441594   
3         0.011660          0.441703   
4         0.011630          0.441222   
5         0.011703          0.442220   
6         0.011661          0.441647   
7         0.011678          0.442181   
8         0.011959          0.445959   
9         0.011701          0.441603   
10        0.011663          0.441734   
11        0.011728          0.442683   
12        0.011669          0.441912   
13        0.011657          0.441644   
14        0.011641          0.441307   
15        0.011881          0.444818   
16        0.011854          0.444674   
17        0.011764          0.443244   

                                             geometry  
0   POLYGON ((-112.41021 48.99883, -112.41021 49.0...  
1   POLYGON ((-111.62536 49.17359, -111.62572 49.1...  
2   POLYGON ((-111.37926 49.17368, -111.38062 49.1...  
3   POLYGON ((-112.00954 49.17366, -112.01234 49.1...  
4   POLYGON ((-112.33750 49.17373, -112.34289 49.1...  
5   POLYGON ((-112.31545 49.08647, -112.32094 49.0...  
6   POLYGON ((-112.00954 49.08620, -112.00926 49.0...  
7   POLYGON ((-111.82152 49.17359, -111.82180 49.1...  
8   POLYGON ((-111.48086 49.08632, -111.48635 49.0...  
9   POLYGON ((-112.62693 49.08680, -112.63238 49.0...  
10  POLYGON ((-112.67709 49.08661, -112.67709 49.0...  
11  POLYGON ((-112.94430 48.99872, -112.94430 49.0...  
12  POLYGON ((-112.25877 49.17370, -112.25974 49.1...  
13  POLYGON ((-111.49763 49.17367, -111.50238 49.1...  
14  POLYGON ((-113.07739 49.08641, -113.07191 49.0...  
15  POLYGON ((-111.69200 49.08629, -111.69748 49.0...  
16  POLYGON ((-111.07417 48.99769, -111.06868 48.9...  
17  POLYGON ((-112.81625 48.99854, -112.81076 48.9...  

And the easymore's setup is as follows:

# using easymore to create aerial average values
from easymore import Easymore

easymore_conf = {
    'case_name': 'AB-townships-data',
    'target_shp': '/path/to/results/townships.shp', 
    'target_shp_ID': 'DESCRIPTOR',
    'temp_dir': '/path/to/temp_dir/', # only available locally
    'source_nc': '/path/to/met_vars.nc',
    'var_names': ['var1', 'var2'],
    'var_lon': 'lon',
    'var_lat': 'lat',
    'var_time': 'time',
    'remapped_var_id': 'DESCRIPTOR',
    'remapped_dim_id': 'DESCRIPTOR',
    'output_dir': '/path/to/township-ag/', # only available locally
}

remap_exp = Easymore.from_dict(easymore_conf)

Which runs successfully with:

remap_exp.nc_remapper()

The version of easymore is 2.0.0 likely the latest commit of the develop_2.0.0 branch.

I am trying to remap a bunch of station data to sub-basins and facing a long runtime issue. I have attached the NetCDF file I am trying to remap, and the sub-basins. Also, I have included the workflow below:

# specifying EASYMORE needed arguments
kwargs = {
    'case_name' : 'wolf-creek',
    'temp_dir' : './temp/',
    'source_nc' : './source_nc/wolf-creek-research-basin-obs.nc',
    'source_shp_ID': 'id',
    'var_names' : ['gflux'],
    'var_lon' : 'longitude',
    'var_lat' : 'latitude',
    'var_ID' : 'id',
    'var_station' : 'id',
    'var_time' : 'time',
    'target_shp' : './target_shp/wolf-creek-research-basin-catchments.shp',
    'target_shp_ID' : 'LINKNO',
    'output_dir' : './outputs/',
    'format_list' : ['f4'],
    'fill_value_list' : ['-9999.00'],
}

# initializing EASYMORE object
esmr = Easymore(**kwargs)

# run
esmr.nc_remapper()

Here is an MCVE:
mcve_irregular.zip

The parallel flag needs to be activated when the job is submitted to a job scheduler on a HPC.

Example 1 currently stops on:

cndx.run_candex()

An exception has occurred, use %tb to see the full traceback.
SystemExit: please project your shapefile to WGS84 (epsg:4326)

while the CRS is set correctly:

shp = gpd.read_file(cndx.sink_shp)
shp.crs

{'init': 'epsg:4326'}

This can be traced to the checks used in check_target_shp() (lines 195-198):

if (str(shp.crs).lower() != 'epsg:4326'):
            sys.exit('please project your shapefile to WGS84 (epsg:4326)')
        if (str(shp.crs).lower() == 'epsg:4326'): # check if the projection is WGS84 (or epsg:4326)
            print('candex detects that target shapefile is in WGS84 (epsg:4326)')

check_source_nc_shp() (lines 350-353):

if (str(shp.crs).lower() != 'epsg:4326'):
            sys.exit('please project your source shapefile and varibales in source nc files to WGS84 (epsg:4326)')
        if (str(shp.crs).lower() == 'epsg:4326'): # check if the projection is WGS84 (or epsg:4326)
            print('candex detects that source shapefile is in WGS84 (epsg:4326)')

and run_candex() (line 106):

if (str(shp_1.crs).lower() == str(shp_2.crs).lower()) and (str(shp_1.crs).lower() == 'epsg:4326'):

In a general case, an unknown option/argument is being supplied to the main constructor using CLI:

Traceback (most recent call last):
  File "/home/kasra545/virtual-envs/scienv/bin/easymore", line 8, in <module>
    sys.exit(main())
  File "/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Compiler/gcc9/eccodes/2.25.0/lib/python3.10/site-packages/click-8.1.2-py3.10.egg/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Compiler/gcc9/eccodes/2.25.0/lib/python3.10/site-packages/click-8.1.2-py3.10.egg/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Compiler/gcc9/eccodes/2.25.0/lib/python3.10/site-packages/click-8.1.2-py3.10.egg/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Compiler/gcc9/eccodes/2.25.0/lib/python3.10/site-packages/click-8.1.2-py3.10.egg/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Compiler/gcc9/eccodes/2.25.0/lib/python3.10/site-packages/click-8.1.2-py3.10.egg/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/home/kasra545/virtual-envs/scienv/lib/python3.10/site-packages/easymore/scripts/main.py", line 147, in from_conf
    json_exp = Easymore.from_json_file(json)
  File "/home/kasra545/virtual-envs/scienv/lib/python3.10/site-packages/easymore/remapper.py", line 352, in from_json_file
    return cls.from_dict(json_dict)
  File "/home/kasra545/virtual-envs/scienv/lib/python3.10/site-packages/easymore/remapper.py", line 324, in from_dict
    return cls(**init_dict)
TypeError: Easymore.__init__() got an unexpected keyword argument 'overwrite_remapped_nc'

  shp_int.to_file(self.temp_dir+self.case_name+'_intersected_shapefile.shp') # save the intersected files
File "/home/gut428/.local/lib/python3.8/site-packages/geopandas/geodataframe.py", line 1086, in to_file
  _to_file(self, filename, driver, schema, index, **kwargs)
File "/home/gut428/.local/lib/python3.8/site-packages/geopandas/io/file.py", line 330, in _to_file
  colxn.writerecords(df.iterfeatures())
File "/home/gut428/.local/lib/python3.8/site-packages/fiona/collection.py", line 342, in writerecords
  self.session.writerecs(records, self)
File "fiona/ogrext.pyx", line 1198, in fiona.ogrext.WritingSession.writerecs
RuntimeError: Failed to write record:

saving the intersected shapefile for North Asia failed.
In this issue there is a netcdf file with format 0 to 360 and a shapefile (merit hydro basin) with longitude more than 180 degrees. It should be tested were the issue of failure in saving is coming from. more test are needed.