dem_create hangs when using gdal 3.4.1 and pyroSAR 0.16 about pyrosar HOT 7 CLOSED

@johntruckenbrodt thanks a lot for fixing this! Using all available threads, with version 0.16.1 the execution time of dem_create has been reduced to just 2 seconds. :-)

from pyrosar.

Thanks @peterfriedl. I will investigate this next week. In the meantime you can make use of the new arguments pbar and threads to check the status and (if it doesn't hang but is just slow) increase the speed.

from pyrosar.

Thanks @johntruckenbrodt for the hint to the pbar and threads arguments. Indeed, dem_create does not hang, it's just super slow when using its default configuration (a few seconds with pyroSAR 0.15.1 and gdal 3.3 vs. > 11 minutes with pyroSAR 0.16 and gdal 3.4.1). I tried different numbers of 56 available threads and found the "sweet spot" of our system to be at only 2 threads. Adding more threads made things even worse by leading to execution times of up to > 30 minutes. Using threads=2 reduced the execution time to 7 minutes, but this is still far away from the previous execution times of just a few seconds, achieved with the default configurations of dem_create.

from pyrosar.

Thanks @peterfriedl, this is super valuable feedback. I did notice dem_create taking quite long, but thought it was just due to the large 10 m DEM mosaics I was creating at the time. The configuration of 0.15.1 was to use multithreading but leave the GDAL_NUM_THREADS variable untouched. I introduced the threads argument because it was so slow. I suspect this could have something to do with the GDAL version. Would you mind checking this? You could create a conda env like so:

conda create --name test pyroSAR gdal=3.3

from pyrosar.

@johntruckenbrodt I am not sure whether it has something to do with the gdal-version in general. I tried both: a) gdal 3.3.3 + pyroSAR 0.16 and b) gdal 3.4.1 + pyroSAR 0.16. in case of b) dem_create runs super slow and in case of a) dem_create hangs.

from pyrosar.

If it hangs then this might be due to function get_egm_lookup. However, there are definitely strange things going on. I have opened a new GDAL issue: OSGeo/gdal#5414.

from pyrosar.

Okay after the discussion in the GDAL issue above pyroSAR is now using different EGM lookup files, see 598ac39.
@peterfriedl, I will publish pyroSAR version 0.16.1 later. Perhaps this already fixes this issue.
Also interesting in this context: OSGeo/gdal#3630.

from pyrosar.