ifgram_inversion: need command that waits until all dask workers are completed. about mintpy HOT 10 CLOSED

Hi @Ovec8hkin, NumPy and scipy with open-blas binding could usually take advantage of multiple cores on local laptop, which is installed by default using conda or macports. In my case, the ifgram_inversion.py use ~%400 of CPU. Therefore, I don't see a significant gain from parallel locally and believe this should be a low priority for now.

from mintpy.

Hello @2gotgrossman @yunjunz ,

I have a question related with dask enabling on local systems.
I enabled dask parallelization in my template and ran into some problems in ifgram inversion.
First I needed to add cores and memory to line 1131 in ifgram_inversion.py

cluster = LSFCluster(walltime=inps.walltime, python=python_executable_location,cores=1,memory='5 GB')

but after that dask tried to create and use LSF jobs (which is expected with use of LSFCluster). I was wondering if there is a dask for local option which hasn't been implemented yet?

from mintpy.

Hi @ehavazli, I don't have an answer to your question. @2gotgrossman is more suitable to answer it but he won't be at work anytime soon. All I know is that PBS scheduler can be easily added, not sure about all the other options.

from mintpy.

from mintpy.

@falkamelung

I am running a test case on my laptop with the following options in my template:

## Parallel processing with Dask for HPC
mintpy.networkInversion.parallel        = yes #[yes / no], auto for no, parallel processing using dask
mintpy.networkInversion.numWorker       = 4   #[int > 0], auto for 40, number of works for dask cluster to use
mintpy.networkInversion.walltime        = 03:00 #[HH:MM], auto for 00:40, walltime for dask workers

when I run smallbaselineApp.py I receive the error below:

split 13372 lines into 4 patches for processing
    with each patch up to 4000 lines
reference pixel in y/x: (10651, 3739) from dataset: unwrapPhase
Traceback (most recent call last):
  File "/Users/havazli/MintPy/mintpy/smallbaselineApp.py", line 1067, in <module>
    main()
  File "/Users/havazli/MintPy/mintpy/smallbaselineApp.py", line 1057, in main
    app.run(steps=inps.runSteps, plot=inps.plot)
  File "/Users/havazli/MintPy/mintpy/smallbaselineApp.py", line 1001, in run
    self.run_network_inversion(sname)
  File "/Users/havazli/MintPy/mintpy/smallbaselineApp.py", line 508, in run_network_inversion
    mintpy.ifgram_inversion.main(scp_args.split())
  File "/Users/havazli/MintPy/mintpy/ifgram_inversion.py", line 1258, in main
    ifgram_inversion(inps.ifgramStackFile, inps)
  File "/Users/havazli/MintPy/mintpy/ifgram_inversion.py", line 1131, in ifgram_inversion
    cluster = LSFCluster(walltime=inps.walltime, python=python_executable_location)
  File "/Users/havazli/miniconda3/envs/ts/lib/python3.7/site-packages/dask_jobqueue/lsf.py", line 89, in __init__
    super(LSFCluster, self).__init__(config_name=config_name, **kwargs)
  File "/Users/havazli/miniconda3/envs/ts/lib/python3.7/site-packages/dask_jobqueue/core.py", line 231, in __init__
    "You must specify how many cores to use per job like ``cores=8``"
ValueError: You must specify how many cores to use per job like ``cores=8``

Did you run into this problem before or am I doing something wrong? I would like to activate dask on my local machine (MacOS) right now and don't have a PBS system.

from mintpy.

from mintpy.

@ehavazli This is old now, but I have been working on Dask for a bit and have been updating MintPy so as to be more flexible in how it performs its parallel computations. Are you still encountering this problem? I can take a look into how to make MintPy run in parallel locally, as that would be a useful default option for a lot of people that don't have access to an HPC cluster.

from mintpy.

from mintpy.

Hi @falkamelung, to followup on the main topic of this issue, is it still current?

from mintpy.

part of this is discussed in a new issue

from mintpy.