insarlab / miaplpy Goto Github PK

View Code? Open in Web Editor NEW

72.0 11.0 20.0 15.87 MB

MIAmi Phase Linking in PYthon

License: Other

Python 88.92% Cython 11.08%

timeseries deformation earthquake geodesy geophysics geoscience insar landslide phase-linking remotesensing

miaplpy's People

Contributors

Stargazers

Watchers

miaplpy's Issues

out of data range error although subset is within SLC area

I am getting an out of data range error, although I have selected a subset that is fully within the SLCs. It turns out that the LENGTH and WIDTH in the *.rsc files are too small. Some multilooking seems to have been applied and this is what causes the problem? I darkly remember that we had this before, but I don't remember details. Any idea?

miaplpyApp.py /work2/05861/tg851601/stampede2/test3_frontera/rsmas_insar/samples/unittestGalapagosSenDT128.template --dir miaplpy --start load_data
--------------------------------------------------

_________________________________________________      

  /##      /## /##                     / ## /#######
 | ###    /###|__/|                    | ##| ##__  ##
 | ####  /#### /##|/ ##### /##         | ##| ##  \ ## /##   /##
 | ## ##/## ##| ##| ##__  ### /####### | ##| #######/| ##  | ##
 | ##  ###| ##| ##| ##__  ###| ##__  ##| ##| ##      | ##  | ##
 | ##\  # | ##| ##| ##   \###| ##   \##| ##| ##      | ##  | ##
 | ## \/  | ##| ##|  ##### ##| #######/| ##| ##      |  #######
 |__/     |__/|__/| \____/|_/| ##____/ |__/|__/       \____  ##
                             | ##                     /##  | ## 
                             | ##                    |  ######/         
                             |__/                     \______/
 Miami Non-Linear Phase Linking software in Python   
          MiaplPy 0.2.0, 2021-09-14
_________________________________________________

--RUN-at-2022-10-14 21:04:15.426400--
Current directory: /scratch/05861/tg851601/unittestGalapagosSenDT128
Run routine processing with arg_parser.py on steps: ['load_data', 'phase_linking', 'concatenate_patches', 'generate_ifgram', 'unwrap_ifgram', 'load_ifgram', 'ifgram_correction', 'invert_network', 'timeseries_correction']
Remaining steps: ['phase_linking', 'concatenate_patches', 'generate_ifgram', 'unwrap_ifgram', 'load_ifgram', 'ifgram_correction', 'invert_network', 'timeseries_correction']
Project name: unittestGalapagosSenDT128
--------------------------------------------------
20221014:210415 * miaplpyApp.py /work2/05861/tg851601/stampede2/test3_frontera/rsmas_insar/samples/unittestGalapagosSenDT128.template --dir miaplpy --start load_data
Project name: unittestGalapagosSenDT128
Go to work directory: /scratch/05861/tg851601/unittestGalapagosSenDT128/miaplpy
copy default template file /work2/05861/tg851601/stampede2/test3_frontera/rsmas_insar/sources/MintPy/mintpy/defaults/smallbaselineApp.cfg to work directory
read custom template file: /work2/05861/tg851601/stampede2/test3_frontera/rsmas_insar/samples/unittestGalapagosSenDT128.template
update default template based on input custom template
    mintpy.compute.cluster: auto --> local
    mintpy.compute.numWorker: auto --> 6
    mintpy.load.processor: auto --> isce
    mintpy.load.autoPath: auto --> yes
    mintpy.subset.lalo: auto --> -0.86:-0.81,-91.19:-91.13
    mintpy.reference.lalo: auto --> -0.82,-91.14
    mintpy.troposphericDelay.method: auto --> no
    mintpy.save.kmz: auto --> yes
    mintpy.save.hdfEos5: auto --> yes
    mintpy.save.hdfEos5.update: auto --> yes
copy unittestGalapagosSenDT128.template to inputs   directory for backup.
copy smallbaselineApp.cfg               to inputs   directory for backup.
copy unittestGalapagosSenDT128.template to pic      directory for backup.
copy smallbaselineApp.cfg               to pic      directory for backup.
read default template file: /scratch/05861/tg851601/unittestGalapagosSenDT128/miaplpy/smallbaselineApp.cfg
copy default template file /work2/05861/tg851601/stampede2/test3_frontera/rsmas_insar/sources/MiaplPy/miaplpy/objects/../defaults/miaplpyApp.cfg to work directory
update default template based on input custom template
    miaplpy.load.processor: auto --> isce
    miaplpy.load.autoPath: auto --> yes
No new option value found, skip updating /scratch/05861/tg851601/unittestGalapagosSenDT128/miaplpy/custom_smallbaselineApp.cfg
copy miaplpyApp.cfg to inputs   directory for backup.
read default template file: /scratch/05861/tg851601/unittestGalapagosSenDT128/miaplpy/miaplpyApp.cfg
    miaplpy.load.compression: auto --> no
    miaplpy.load.autoPath: yes --> yes
    miaplpy.subset.lalo: auto --> -0.86:-0.81,-91.19:-91.13
SAR platform/sensor : Sen
processor: isce
check auto path setting for Univ of Miami users for processor: isce
--------------------------------------------------
prepare metadata files for isce products
prep_slc_isce.py -s /scratch/05861/tg851601/unittestGalapagosSenDT128/merged/SLC -f *.slc.full -m /scratch/05861/tg851601/unittestGalapagosSenDT128/reference/IW1.xml -b /scratch/05861/tg851601/unittestGalapagosSenDT128/baselines -g /scratch/05861/tg851601/unittestGalapagosSenDT128/merged/geom_reference
['/scratch/05861/tg851601/unittestGalapagosSenDT128/reference/data.rsc'] exists and is newer than ['/scratch/05861/tg851601/unittestGalapagosSenDT128/reference/IW1.xml'] --> skip.
prepare .rsc file for geometry files
read perp baseline time-series from /scratch/05861/tg851601/unittestGalapagosSenDT128/baselines
preparing RSC file for  *.slc.full
[==================================================] 20160605_20160828    0s /     0s
Done.
Traceback (most recent call last):
  File "/work2/05861/tg851601/stampede2/test3_frontera/rsmas_insar/sources/MiaplPy/miaplpy/miaplpyApp.py", line 801, in <module>
    main()
  File "/work2/05861/tg851601/stampede2/test3_frontera/rsmas_insar/sources/MiaplPy/miaplpy/miaplpyApp.py", line 82, in main
    app.open()
  File "/work2/05861/tg851601/stampede2/test3_frontera/rsmas_insar/sources/MiaplPy/miaplpy/miaplpyApp.py", line 176, in open
    self.date_list, self.num_pixels, self.metadata = read_initial_info(self.workDir, self.templateFile)
  File "/work2/05861/tg851601/stampede2/test3_frontera/rsmas_insar/sources/MiaplPy/miaplpy/objects/utils.py", line 1223, in read_initial_info
    metadata = read_subset_box(iDict)
  File "/work2/05861/tg851601/stampede2/test3_frontera/rsmas_insar/sources/MiaplPy/miaplpy/objects/utils.py", line 1156, in read_subset_box
    pix_box = coord.check_box_within_data_coverage(pix_box)
  File "/work2/05861/tg851601/stampede2/test3_frontera/rsmas_insar/sources/MintPy/mintpy/objects/coord.py", line 477, in check_box_within_data_coverage
    raise ValueError(msg)
ValueError: ERROR: input index is out of data range!
data   range in (x0,y0,x1,y1): (0, 0, 1364, 565)
subset range in (x0,y0,x1,y1): (7009, 1486, 8912, 2010)
data   range in (W, N, E, S): None
subset range in (W, N, E, S): None

unittestGalapagosSenDT128/merged/SLC/20160605[1121] cat *.full.rsc | grep -E 'LENGTH|length'
FILE_LENGTH               565
LENGTH                    565
length                    2826

For comparison, here for a different dataset which works fine:

MiamiSenAT48/merged/SLC/20150921[1029] cat *.full.rsc | grep -E 'LENGTH|length'
FILE_LENGTH               5531
LENGTH                    5531
length                    5531

Here the *template file for the problem case:

cat /work2/05861/tg851601/stampede2/test3_frontera/rsmas_insar/samples/unittestGalapagosSenDT128.template 
######################################################
cleanopt                          = 0   # [ 0 / 1 / 2 / 3 / 4]   0,1: none 2: keep merged,geom_master,SLC 3: keep MINTPY 4: everything
processor                         = isce
ssaraopt.platform                 = SENTINEL-1A,SENTINEL-1B
ssaraopt.relativeOrbit            = 128
ssaraopt.startDate                = 20160601
ssaraopt.endDate                  = 20160831
insarmaps_flag                    = False
image_products_flag               = False

######################################################

topsStack.boundingBox             = -1 -0.6 -91.9 -90.7      # -1 0.15 -91.9 -90.6
topsStack.boundingBox             = -0.81 -0.80 -90.9 -90.86    # -1 0.15 -91.9 -90.6
topsStack.subswath                = 1    # '1 2'
topsStack.numConnections          = 3    # comment
topsStack.azimuthLooks            = 5    # comment
topsStack.rangeLooks              = 15   # comment
topsStack.filtStrength            = 0.2  # comment
topsStack.unwMethod               = snaphu  # comment
topsStack.coregistration          = auto  # [NESD geometry], auto for NESD
topsStack.slcDir                  = $TESTDATA_ISCE/unittestGalapagosSenDT128/SLC

######################################################

mintpy.load.autoPath              = yes
mintpy.compute.cluster            = local #[local / slurm / pbs / lsf / none], auto for none, cluster type
mintpy.compute.numWorker          = 6 #[int > 1 / all], auto for 4 (local) or 40 (non-local), num of workers
mintpy.reference.lalo             = -0.82,-91.14     # S of SN
mintpy.networkInversion.parallel  = yes  #[yes / no], auto for no, parallel processing using dask
mintpy.troposphericDelay.method   = no    # pyaps  #[pyaps / height_correlation / base_trop_cor / no], auto for pyaps
mintpy.save.hdfEos5               = yes   #[yes / update / no], auto for no, save timeseries to UNAVCO InSAR Archive format
mintpy.save.hdfEos5.update        = yes   #[yes / no], auto for no, put XXXXXXXX as endDate in output filename
mintpy.save.hdfEos5.subset        = yes   #[yes / no], auto for no, put XXXXXXXX as endDate in output filename
mintpy.save.kmz                   = yes  #[yes / no], auto for yes, save geocoded velocity to Google Earth KMZ file

######################################################

miaplpy.load.processor      = isce  
miaplpy.load.autoPath       = yes
miaplpy.compute.num_workers = 46
miaplpy.subset.lalo        = -0.86:-0.81,-91.19:-91.13    #[31.5:32.5,130.5:131.0 / no], auto for no
miaplpy.interferograms.networkType = sequential     # [single_reference, sequential, combine, list] default: single_reference

Is it possible to use old version of ISCE output

Hi,

I was wondering if it is possible to use the old version of ISCE2 which for instance they used master instead of reference folder?
Which kind of modification might needed if we want to import data from them?

Best regards,
M

Mean velocity result

@mirzaees I've got some interesting results of landslide about the fading signal issue. So, I've been doing some experiments and trying out different data processing techniques ? the result seems unexpected . The results from each method are surprisingly similar. What would be the reason?

ImportError: cannot import name 'utils' from 'miaplpy.lib' (/mnt/d/SAR/MiaplPy/src/miaplpy/lib/init.py)

Hello,
I install the new release version of miaplpy follow by docs/installation.md, and run it by example data. and It show an error as follow at step 2. It seems python could not found the module in miaplpy.lib. How to fix it?

<><><><><><><><><> step 2 - phase_linking (MiaplPy) <><><><><><><><><><>
Generate /mnt/d/work/test/PichinchaSenDT142/miaplpy/network_single_reference/run_files/run_02_miaplpy_phase_linking
Total number of PATCHES: 27
Number of tasks for step phase inversion: 4
Traceback (most recent call last):
File "/home/hzq/miniforge3/envs/miaplpy/bin/phase_linking.py", line 5, in
from miaplpy.phase_linking import main
File "/mnt/d/SAR/MiaplPy/src/miaplpy/phase_linking.py", line 17, in
from miaplpy.lib import utils as iut
ImportError: cannot import name 'utils' from 'miaplpy.lib' (/mnt/d/SAR/MiaplPy/src/miaplpy/lib/init.py)
Traceback (most recent call last):
File "/home/hzq/miniforge3/envs/miaplpy/bin/phase_linking.py", line 5, in
from miaplpy.phase_linking import main
File "/mnt/d/SAR/MiaplPy/src/miaplpy/phase_linking.py", line 17, in
from miaplpy.lib import utils as iut
ImportError: cannot import name 'utils' from 'miaplpy.lib' (/mnt/d/SAR/MiaplPy/src/miaplpy/lib/init.py)
Traceback (most recent call last):
File "/home/hzq/miniforge3/envs/miaplpy/bin/phase_linking.py", line 5, in
from miaplpy.phase_linking import main
File "/mnt/d/SAR/MiaplPy/src/miaplpy/phase_linking.py", line 17, in
from miaplpy.lib import utils as iut
ImportError: cannot import name 'utils' from 'miaplpy.lib' (/mnt/d/SAR/MiaplPy/src/miaplpy/lib/init.py)
Traceback (most recent call last):
File "/home/hzq/miniforge3/envs/miaplpy/bin/phase_linking.py", line 5, in
from miaplpy.phase_linking import main
File "/mnt/d/SAR/MiaplPy/src/miaplpy/phase_linking.py", line 17, in
from miaplpy.lib import utils as iut
ImportError: cannot import name 'utils' from 'miaplpy.lib' (/mnt/d/SAR/MiaplPy/src/miaplpy/lib/init.py)
Go back to directory: /mnt/d/work/test/PichinchaSenDT142/miaplpy

###############################################################
Normal end of Non-Linear time series processing workflow!
##############################################################
Go back to directory: /mnt/d/work/test/PichinchaSenDT142/miaplpy

how to process ds-insar?

Can Miaplpy be processed ds-insar?How do i do this?

AttributeError: module 'generate_temporal_coherence' has no attribute 'main'

Hello,
When i run miaplpy by example data. When it run in step 8 it shows an error that is "AttributeError: module 'generate_temporal_coherence' has no attribute 'main'". Why it happen, I saw the main method has already defined in generate_temporal_coherence.py? and How to fix it?

MiaplPy version v0.2.1, date 2023-08-22
--RUN-at-2023-08-27 23:08:35.233485--
Current directory: /mnt/d/work/test/PichinchaSenDT142
Run routine processing with arg_parser.py on steps: ['invert_network']
Remaining steps: ['timeseries_correction']
Project name: PichinchaSenDT142

20230827:230835 * miaplpyApp.py PichinchaSenDT142.txt --dostep invert_network --dir ./miaplpy
Project name: PichinchaSenDT142
Go to work directory: /mnt/d/work/test/PichinchaSenDT142/miaplpy
copy default template file /home/hzq/miniforge3/envs/miaplpy/lib/python3.11/site-packages/mintpy/defaults/smallbaselineApp.cfg to work directory
read custom template file: /mnt/d/work/test/PichinchaSenDT142/PichinchaSenDT142.txt
update default template based on input custom template
mintpy.compute.cluster: auto --> local
mintpy.compute.numWorker: auto --> 8
mintpy.reference.lalo: auto --> -0.1786, -78.5933
mintpy.troposphericDelay.method: auto --> no
copy PichinchaSenDT142.txt to inputs directory for backup.
copy smallbaselineApp.cfg to inputs directory for backup.
copy PichinchaSenDT142.txt to pic directory for backup.
copy smallbaselineApp.cfg to pic directory for backup.
read default template file: /mnt/d/work/test/PichinchaSenDT142/miaplpy/smallbaselineApp.cfg
update default template based on input custom template
No new option value found, skip updating /mnt/d/work/test/PichinchaSenDT142/miaplpy/miaplpyApp.cfg
mintpy.load.processor: auto --> isce
copy PichinchaSenDT142.txt to inputs directory for backup.
copy miaplpyApp.cfg to inputs directory for backup.
read default template file: /mnt/d/work/test/PichinchaSenDT142/miaplpy/miaplpyApp.cfg
----- Number of interferograms in the selected network: 45 -----

<><><><><><><><><> step 8 - invert_network (MiaplPy) <><><><><><><><><><>
Generate /mnt/d/work/test/PichinchaSenDT142/miaplpy/network_single_reference/run_files/run_08_miaplpy_invert_network
20230827:230839 * network_inversion.py /mnt/d/work/test/PichinchaSenDT142/miaplpy/network_single_reference/inputs/ifgramStack.h5 --template /mnt/d/work/test/PichinchaSenDT142/PichinchaSenDT142.txt --temp_coh /mnt/d/work/test/PichinchaSenDT142/miaplpy/inverted/tempCoh_full --mask-thres 0.5 --norm L2 --smooth_factor 0.001 --weight-func var --work_dir /mnt/d/work/test/PichinchaSenDT142/miaplpy/network_single_reference

ifgram_inversion_L1L2.py /mnt/d/work/test/PichinchaSenDT142/miaplpy/network_single_reference/inputs/ifgramStack.h5 -t /mnt/d/work/test/PichinchaSenDT142/PichinchaSenDT142.txt --update --norm L2 --tcoh /mnt/d/work/test/PichinchaSenDT142/miaplpy/inverted/tempCoh_full --mask-threshold 0.5 --smooth_factor 0.001
use dataset "unwrapPhase" by default

update mode: ON

output files already exist: ['timeseries.h5', 'temporalCoherence.h5', 'numInvIfgram.h5'].
output dataset is newer than input dataset: unwrapPhase.
all key configuration parameters are the same: ['obsDatasetName', 'numIfgram', 'weightFunc', 'maskDataset', 'maskThreshold', 'minRedundancy', 'minNormVelocity'].
run or skip: skip.

generate_temporal_coherence.py -d /mnt/d/work/test/PichinchaSenDT142/miaplpy/network_single_reference
Traceback (most recent call last):
File "/home/hzq/miniforge3/envs/miaplpy/bin/network_inversion.py", line 33, in
sys.exit(load_entry_point('miaplpy==0.2.1', 'console_scripts', 'network_inversion.py')())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/hzq/miniforge3/envs/miaplpy/lib/python3.11/site-packages/miaplpy/network_inversion.py", line 67, in main
generate_temporal_coherence.main(iargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'generate_temporal_coherence' has no attribute 'main'
Go back to directory: /mnt/d/work/test/PichinchaSenDT142/miaplpy

################################################
Normal end of miaplpyApp processing!
################################################
Time used: 00 mins 4.7 secs

prep_slc_isce.py module load error: isceobj

Hi there,
I have recently installed MiaplPy on my local HPC system. Everything seems to install correctly but when I try and run miaplpyApp.py I run into the below error related to module importing during prep_slc_isce.py:

I have MiaPlPy working successfully on a local machine (installed last week) and did not run into this issue, I also have ISCE2 working correctly on the HPC system independently of the MiaPlPy installation. MintPy is also installed on the HPC system and works correctly with the prep_isce.py script that they utilize at the start of their workflow.

(miaplpy) [jaha2600@shas0136 miaplpy]$ miaplpyApp.py kigar.txt --dir miaplpy

--RUN-at-2022-09-27 16:34:05.402889--
Current directory: /gpfs/summit/scratch/jaha2600/GREENLAND/2022/stack_mintpy/kigarsima_slc/miaplpy
Run routine processing with arg_parser.py on steps: ['load_data', 'phase_linking', 'concatenate_patches', 'generate_ifgram', 'unwrap_ifgram', 'load_ifgram', 'ifgram_correction', 'invert_network', 'timeseries_correction']
Remaining steps: ['phase_linking', 'concatenate_patches', 'generate_ifgram', 'unwrap_ifgram', 'load_ifgram', 'ifgram_correction', 'invert_network', 'timeseries_correction']
Project name: kigar

20220927:163405 * miaplpyApp.py kigar.txt --dir miaplpy
Project name: kigar
Go to work directory: /gpfs/summit/scratch/jaha2600/GREENLAND/2022/stack_mintpy/kigarsima_slc/miaplpy/miaplpy
copy default template file /projects/jaha2600/software/miaplpy_programs/MintPy/mintpy/defaults/smallbaselineApp.cfg to work directory
read custom template file: /gpfs/summit/scratch/jaha2600/GREENLAND/2022/stack_mintpy/kigarsima_slc/miaplpy/kigar.txt
update default template based on input custom template
mintpy.compute.cluster: auto --> local
mintpy.compute.numWorker: auto --> 8
mintpy.troposphericDelay.method: auto --> no
copy smallbaselineApp.cfg to inputs directory for backup.
copy smallbaselineApp.cfg to pic directory for backup.
read default template file: /gpfs/summit/scratch/jaha2600/GREENLAND/2022/stack_mintpy/kigarsima_slc/miaplpy/miaplpy/smallbaselineApp.cfg
update default template based on input custom template
miaplpy.load.compression: no --> auto
copy miaplpyApp.cfg to inputs directory for backup.
read default template file: /gpfs/summit/scratch/jaha2600/GREENLAND/2022/stack_mintpy/kigarsima_slc/miaplpy/miaplpy/miaplpyApp.cfg
miaplpy.load.compression: auto --> no
miaplpy.load.autoPath: yes --> yes
There are 24 workers available, numWorker is changed to 24
SAR platform/sensor : None
processor: isce
check auto path setting for Univ of Miami users for processor: isce

prepare metadata files for isce products
prep_slc_isce.py -s /scratch/summit/jaha2600/GREENLAND/2022/stack_mintpy/kigarsima_slc/merged/SLC -f *.slc.full -m /scratch/summit/jaha2600/GREENLAND/2022/stack_mintpy/kigarsima_slc/reference/IW3.xml -b /scratch/summit/jaha2600/GREENLAND/2022/stack_mintpy/kigarsima_slc/baselines -g /scratch/summit/jaha2600/GREENLAND/2022/stack_mintpy/kigarsima_slc/merged/geom_reference
Traceback (most recent call last):
File "/projects/jaha2600/software/miaplpy_programs/MiaplPy/miaplpy/prep_slc_isce.py", line 26, in
from isceobj.Planet.Planet import Planet
ModuleNotFoundError: No module named 'isceobj'
box to read for datasets in y/x: None
Traceback (most recent call last):
File "/projects/jaha2600/software/miaplpy_programs/MiaplPy/miaplpy/miaplpyApp.py", line 798, in
main()
File "/projects/jaha2600/software/miaplpy_programs/MiaplPy/miaplpy/miaplpyApp.py", line 82, in main
app.open()
File "/projects/jaha2600/software/miaplpy_programs/MiaplPy/miaplpy/miaplpyApp.py", line 176, in open
self.date_list, self.num_pixels, self.metadata = read_initial_info(self.workDir, self.templateFile)
File "/projects/jaha2600/software/miaplpy_programs/MiaplPy/miaplpy/objects/utils.py", line 1225, in read_initial_info
num_pixels = (box[2] - box[0]) * (box[3] - box[1])
TypeError: 'NoneType' object is not subscriptable

Have you come across this issue before and have any ideas?
Any help would be fantastic.

snaphu unwrapping not working in tile mode

When I run snaphu in tile mode it never finishes. For a relatively large file is runs for 40 minutes and then restarts. To solve the problem I switch off tiling by setting in defaults/miaplpyApp_auto.cfg:

miaplpy.unwrap.snaphu.tileNumPixels       = 1000000000

That works but unwrapping is painfully slow.

Here a number of observed suspicious inconsistencies that might help to debug: miaplpyApp.py produces this command including --num_tiles 3:

unwrap_ifgram.py --ifg /scratch/05861/tg851601/EifelSenDT139/miaplpy_44_201410_202307/inverted/interferograms_single_reference/20190627_20141015/filt_fine.int --coherence /scratch/05861/tg851601/EifelSenDT139/miaplpy_44_201410_202307/inverted/tempCoh_average --unwrapped_ifg /scratch/05861/tg851601/EifelSenDT139/miaplpy_44_201410_202307/inverted/interferograms_single_reference/20190627_20141015/filt_fine.unw --max_discontinuity 1.2 --init_method MCF --length 3602 --width 10149 --height 703006.0668887326 --num_tiles 3 --earth_radius 6373617.816423893  --wavelength 0.05546576 -m /scratch/05861/tg851601/EifelSenDT139/miaplpy_44_201410_202307/inverted/mask_unwrap

However, the snaphu command produced has --tile 2 1 and --nproc 3:

snaphu -f /scratch/05861/tg851601/EifelSenDT139/miaplpy_44_201410_202307/inverted/interferograms_single_reference/20190627_20141015/config_all -d /scratch/05861/tg851601/EifelSenDT139/miaplpy_44_201410_202307/inverted/interferograms_single_reference/20190627_20141015/filt_fine.int 10149 -o /scratch/05861/tg851601/EifelSenDT139/miaplpy_44_201410_202307/inverted/interferograms_single_reference/20190627_20141015/filt_fine.unw --tile 2 1 500 500 --nproc 3

However, the top command shows four snaphu processes.

   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                  
231390 tg851601  20   0 7668336   7.3g    632 R 100.0  3.9  10:32.37 snaphu                   
231393 tg851601  20   0 7668336   7.3g    632 R 100.0  3.9  10:29.44 snaphu
231415 tg851601  20   0 7668336   7.3g    632 R 100.0  3.9  10:13.76 snaphu                   
231419 tg851601  20   0 7668336   7.3g    632 R 100.0  3.9  10:10.45 snaphu                   
229930 tg851601  20   0  130760   3396   1608 R   1.0  0.0   0:05.84 top

miaplpyApp.py continues after fatal error

Hi @mirzaees
When miaplpyApp.py encounters an error it does not always exist. For example, I got the error below but it continued, producing garbage. I have seen this also for other run_steps. It would be much preferred to alway exit so that you know where the problem is.

<><><><><><><><><> step 4 - generate_ifgram (MiaplPy) <><><><><><><><><><>
Generate /scratch2/05861/tg851601/MiamitestTsxSMD14/miaplpy/network_single_reference/run_files/run_04_miaplpy_generate_ifgram
20221207:185450 * generate_unwrap_mask.py --geometry /scratch2/05861/tg851601/MiamitestTsxSMD14/miaplpy/inputs/geometryRadar.h5
There is no shadow mask in geometryRadar.h5, all values set to 1
Traceback (most recent call last):
  File "/work2/05861/tg851601/stampede2/code/rsmas_insar/sources/MiaplPy/miaplpy/generate_unwrap_mask.py", line 110, in <module>
    main()
  File "/work2/05861/tg851601/stampede2/code/rsmas_insar/sources/MiaplPy/miaplpy/generate_unwrap_mask.py", line 47, in main
    mintpy.generate_mask.main(args_shm.split())
AttributeError: module 'mintpy' has no attribute 'generate_mask'
20221207:185450 * generate_ifgram.py --reference 20200313 --secondary 20200324 --output_dir /scratch2/05861/tg851601/MiamitestTsxSMD14/miaplpy/inverted/interferograms_single_reference/20200313_20200324 --azimuth_looks 1 --range_looks 1 --filter_strength 0 -

unwrap_ifgram.py generates a different filename

unwrap_ifgram.py generates a different filename when unwrapping is tiled. It generates the filename snaphu_tileinit_270422_filt_fine.unw instead of just filt_fine.unw. I checked the code and I suspect that the naming issue is caused by snaphu. Please see the screenshot below of my directory for your reference.

hdfeos5 file is not created

When you say mintpy.save.hdfEos5 = yes then I get in the run)9 step this error:

******************** step - hdfeos5 ********************
--------------------------------------------

save_hdfeos5.py /scratch/05861/tg851601/qunittestGalapagosSenDT128/miaplpy/network_single_reference/geo/geo_timeseries_demErr.h5 --tc /scratch/05861/tg851601/qunittestGalapagosSenDT128/miaplpy/network_single_reference/geo/geo_temporalCoherence.h5 --asc /scratch/05861/tg851601/qunittestGalapagosSenDT128/miaplpy/network_single_reference/geo/geo_avgSpatialCoh.h5 -m /scratch/05861/tg851601/qunittestGalapagosSenDT128/miaplpy/network_single_reference/geo/geo_maskTempCoh.h5 -g /scratch/05861/tg851601/qunittestGalapagosSenDT128/miaplpy/network_single_reference/geo/geo_geometryRadar.h5 -t /scratch/05861/tg851601/qunittestGalapagosSenDT128/miaplpy/network_single_reference/smallbaselineApp.cfg
read options from template file: smallbaselineApp.cfg
Traceback (most recent call last):
  File "/work2/05861/tg851601/stampede2/code/rsmas_insar/sources/MintPy/mintpy/cli/smallbaselineApp.py", line 213, in <module>
    main(sys.argv[1:])
  File "/work2/05861/tg851601/stampede2/code/rsmas_insar/sources/MintPy/mintpy/cli/smallbaselineApp.py", line 208, in main
    run_smallbaselineApp(inps)
  File "/work2/05861/tg851601/stampede2/code/rsmas_insar/sources/MintPy/mintpy/smallbaselineApp.py", line 1117, in run_smallbaselineApp
    app.run(steps=inps.runSteps)
  File "/work2/05861/tg851601/stampede2/code/rsmas_insar/sources/MintPy/mintpy/smallbaselineApp.py", line 922, in run
    self.run_save2hdfeos5(sname)
  File "/work2/05861/tg851601/stampede2/code/rsmas_insar/sources/MintPy/mintpy/smallbaselineApp.py", line 862, in run_save2hdfeos5
    mintpy.cli.save_hdfeos5.main(iargs)
  File "/work2/05861/tg851601/stampede2/code/rsmas_insar/sources/MintPy/mintpy/cli/save_hdfeos5.py", line 107, in main
    save_hdfeos5(inps)
  File "/work2/05861/tg851601/stampede2/code/rsmas_insar/sources/MintPy/mintpy/save_hdfeos5.py", line 437, in save_hdfeos5
    meta = prep_metadata(
  File "/work2/05861/tg851601/stampede2/code/rsmas_insar/sources/MintPy/mintpy/save_hdfeos5.py", line 64, in prep_metadata
    unavco_meta = metadata_mintpy2unavco(meta, ts_obj.dateList, geom_file)
  File "/work2/05861/tg851601/stampede2/code/rsmas_insar/sources/MintPy/mintpy/save_hdfeos5.py", line 118, in metadata_mintpy2unavco
    lons = [meta['LON_REF1'],
KeyError: 'LON_REF1'
Go back to directory: /scratch/05861/tg851601/qunittestGalapagosSenDT128/miaplpy

MintPy does not throw this error.
It turns out that the data.rsc lacks the fields LAT_REF1, etc, whereas the data.rsc generated by mintpy has them:

qunittestGalapagosSenDT128/reference[1089] sdiff data.rsc ../reference_mintpy/data.rsc 
ALOOKS                1					      |	ALOOKS                    5
ANTENNA_SIDE          -1				      |	ANTENNA_SIDE              -1
AZIMUTH_PIXEL_SIZE    15.616065431619585		      |	AZIMUTH_PIXEL_SIZE        78.08032715809793
CENTER_LINE_UTC       42609.0				      |	CENTER_INCIDENCE_ANGLE    33.853077
EARTH_RADIUS          6337323.067122917			      |	CENTER_LINE_UTC           42609.0
HEADING               -167.96908041323653		      |	EARTH_RADIUS              6337323.067122917
HEIGHT                698984.5823960749			      |	FILE_LENGTH               565
NCORRLOOKS            0.5988245993484209		      |	HEADING                   -167.96908041323653
ORBIT_DIRECTION       DESCENDING			      |	HEIGHT                    698984.5823960749
PLATFORM              sen				      |	LAT_REF1                  -0.6879028124910035
POLARIZATION          VV				      |	LAT_REF2                  -0.5250735971614573
PRF                   1717.128973878037			      |	LAT_REF3                  -1.0360017147852283
PROCESSOR             isce				      |	LAT_REF4                  -0.8728346561708716
RANGE_PIXEL_SIZE      2.329562114715323			      |	LENGTH                    565
RLOOKS                1					      |	LON_REF1                  -90.79578040798087
STARTING_RANGE        799547.2189154713			      |	LON_REF2                  -91.55101356159743
WAVELENGTH            0.05546576			      |	LON_REF3                  -90.86902369155685
altitude              698984.5823960749			      |	LON_REF4                  -91.62443101326208
azimuthPixelSize      15.616065431619585		      |	NCORRLOOKS                44.91184495113157
azimuthResolution     22.5				      |	ORBIT_DIRECTION           DESCENDING

The problem is at follows:

LON_REF1, etc is read from the .xml file here. However, the extract_geometry_metadata function is never reached, because it thinks that there are no geom_files. It checks for the existence of e.g. lat.rdr.full here. But the file does not exist. Only lat.rdr.full.vrt exists.

I hope there is an easy fix??

KeyError: 'width' for load_ifgram (MiaplPy)

when i run "miaplpy.interferograms.networkType = delaunay" , the following error occurs. I don't know why will appear this mistake, because there is no error operation "miaplpy.interferograms.net workType = single_reference"

<><><><><><><><><> step 6 - load_ifgram (MiaplPy) <><><><><><><><><><>
Generate /home/gmtsar-user/test/LuD/Process/miaplpy/network_delaunay_4/run_files/run_06_miaplpy_load_ifgram
20240102:231249 * load_ifgram.py --template /home/gmtsar-user/test/LuD/Process/miaplpy/miaplpyApp.cfg /home/gmtsar-user/test/LuD/Process/LUD --project LUD --output /home/gmtsar-user/test/LuD/Process/miaplpy/network_delaunay_4/inputs/ifgramStack.h5 /home/gmtsar-user/test/LuD/Process/miaplpy/network_delaunay_4/inputs/geometryRadar.h5 /home/gmtsar-user/test/LuD/Process/miaplpy/network_delaunay_4/inputs/geometryGeo.h5
SAR platform/sensor : unknown from project name "LUD"
check auto path setting for Univ of Miami users for processor: isce

prepare metadata files for isce products
['../reference/data.rsc'] exists and is newer than ['../reference/IW1.xml'] --> skip.
preparing RSC file for geometry files
write file: /home/gmtsar-user/test/LuD/Process/merged/geom_reference/shadowMask.rdr.rsc
read perp baseline time-series from ../baselines
preparing RSC file for: ./inverted/interferograms_delaunay/*/fine.unw
Traceback (most recent call last):
File "/home/gmtsar-user/Softwares/MiaplPy/src/miaplpy/load_ifgram.py", line 474, in
main()
File "/home/gmtsar-user/Softwares/MiaplPy/src/miaplpy/load_ifgram.py", line 359, in main
inpsDict = read_subset_box(inpsDict)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/gmtsar-user/Softwares/MiaplPy/src/miaplpy/load_ifgram.py", line 250, in read_subset_box
pix_box = (0, 0, int(atr['width']), int(atr['length'])) # coord.bbox_geo2radar(geo_box)
~~~^^^^^^^^^
KeyError: 'width'

What is the difference between MiaplPy and Fringe ？

--mask-ps option for save_hdfeos5.py

It would be nice to save the maskPS in the S1* file as from our correspondence with Yunjun below. It also would be nice to have an option for automatic geolocation correction. This could be either an extra step of MiaplPy or with MintPy. If you do MintPy we have to add a blurb to cite the MiaplPy paper when this feature is used.

Hi Falk,

In the S1* file it would be good to have two masks.The regular mask based on the temporal coherence and the PS mask. Do you think this is a good idea?

Yes, this sounds good to me. Nitpicking for style consistency, I would use --mask-ps maskPS.h5, and for the h5 dataset name “maskPS”.

We are trying to get the PS data into insarmaps. Is there anything that comes to your mind that we should consider?

I remember the geolocation correction that Sara has is very useful. It would be helpful to add an option to feed this correction, to update the corresponding geometry datasets (lat/lon/hgt).

Yunjun

Problem with ISCE stackStripMap.py to MiaplPy

I'm confused about how to set stackStripMap.py parameters in ISCE to make it compatible with Miaplpy.

Also, what should be the correct way to set input paths for Miaplpy with stackStripMap.py output?

Maybe someone could share some experiences on this.

Though I mainly process ERS and Envisat data, any experience with stripmap data should be quite helpful (e.g., TerraSAR).

Assistance Required with MiaplPy InSAR Data Processing Issue

Dear Developer,

I am a novice in processing InSAR data using "MiaplPy." After successfully installing the software and running the test command "miaplpyApp.py -h," I proceeded to import data for processing. However, I encountered an error during the second step, which is as follows:

<><><><><><><><> step 2 - phase_linking (MiaplPy) <><><><><><><><><><>
Generate /media/yzr/BHTD/1212test_slc/miaplpy/network_single_reference/run_files/run_02_miaplpy_phase_linking
Total number of PATCHES: 3024
Number of tasks for step phase inversion: 756
Traceback (most recent call last):
File "/home/yzr/tools/MiaplPy/miaplpy/phase_linking.py", line 17, in
from miaplpy.lib import utils as iut
ImportError: cannot import name 'utils' from 'miaplpy.lib' (unknown location)
...
I have attempted to add the path "MiaplPy/src/miaplpy/lib/" to the system environment, but the same error persists. Could you please advise on how to resolve this issue?

Thank you for your assistance.

Kind regards,

yzr

Installation with conda hangs

Following the installation instructions, it hangs for me in the conda installing isce2 step (linux). The work-around is to install isce2 with mamba but it still takes a few hours. I would recommend to remove as much as possible from the requirements.txt (e.g. the minsar dependencies).

That conda install isce2 did not work (or was too slow) is of course weird. Did I do something stupid? It happened on consecutive days. I I did not investigate as mamba worked fine.

load_ifgram.py (step 6) fails when running multiple unwrapping networks

When I am trying to run network_single_reference. While this is running I started network_sequential_3 to compare results from two unwrap networks. However, network_single_reference throws an error because it wants to use data to be generated by network_sequential_3.

The network_single_reference run throws this error:

prep_isce.py -m /scratch/05861/tg851601/MiamiSenAT48/reference/IW3.xml -g /scratch/05861/tg851601/MiamiSenAT48/merged/geom_reference -b /scratch/05861/tg851601/MiamiSenAT48/baselines  -f /scratch/05861/tg851601/MiamiSenAT48/miaplpy/inverted/interferograms_sequential_3/*/fi*.unw 
['/scratch/05861/tg851601/MiamiSenAT48/reference/data.rsc'] exists and is newer than ['/scratch/05861/tg851601/MiamiSenAT48/reference/IW3.xml'] --> skip.
preparing RSC file for geometry files
read perp baseline time-series from /scratch/05861/tg851601/MiamiSenAT48/baselines
preparing RSC file for: /scratch/05861/tg851601/MiamiSenAT48/miaplpy/inverted/interferograms_sequential_3/*/fi*.unw
Traceback (most recent call last):
  File "/work2/05861/tg851601/stampede2/test3_frontera/rsmas_insar/sources/MintPy/mintpy/cli/prep_isce.py", line 116, in <module>
    main(sys.argv[1:])
  File "/work2/05861/tg851601/stampede2/test3_frontera/rsmas_insar/sources/MintPy/mintpy/cli/prep_isce.py", line 111, in main
    prep_isce(inps)
  File "/work2/05861/tg851601/stampede2/test3_frontera/rsmas_insar/sources/MintPy/mintpy/prep_isce.py", line 214, in prep_isce
    prepare_stack(
  File "/work2/05861/tg851601/stampede2/test3_frontera/rsmas_insar/sources/MintPy/mintpy/prep_isce.py", line 137, in prepare_stack
    raise FileNotFoundError(f'NO file found with path pattern: {obs_file}')
FileNotFoundError: NO file found with path pattern: /scratch/05861/tg851601/MiamiSenAT48/miaplpy/inverted/interferograms_sequential_3/*/fi*.unw
WARNING: mintpy.subset.lalo is not supported if 1) no lookup file AND    2) radar/unkonwn coded dataset
	ignore it and continue.

As you see, it looks for miaplpy/inverted/interferograms_sequential_3/*/fi*.unw but it has no business to look in an sequential_3 folder. This is because of the * in

MiaplPy/miaplpy/load_ifgram.py

Line 307 in 26da97f

cmd += ' -f {f} '.format(f=obs_dir + '/*/' + obs_file)

Maybe the interferograms_sequential_3 directory should be in network_sequential_3/interferograms?

SNAP compatibility

I use SNAP to preprocess data for MintPy and I wonder if MiaplPy has the same compatibility.

And a quick question if it is compatible. I was looking at the config and it looks similar to MintPy however there is this option which I'm not familiar with:

##---------Coregistered SLC images:        
miaplpy.load.slcFile        = ../merged/SLC/*/*.slc.full  #[path2slc_file]

Is that pointing to the wrapped phase imagery?

attributes missing when creating S1* file in radar coordinates

This is a MiaplyPy issue, because the creation of a S1* file in radar coordinates is only relevant for MiaplyPy data.

If I say mintpy.save.hdfEos5.subset = yes I get this error:

save_hdfeos5.py timeseries_demErr.h5 --tc temporalCoherence.h5 --asc avgSpatialCoh.h5 -m  ../maskPS.h5 -g inputs/geometryRadar.h5 -t smallbaselineApp.cfg --suffix PS $update
read options from template file: smallbaselineApp.cfg
Input file is not geocoded, no data_footprint without X/Y_FIRST/STEP info.
## UNAVCO Metadata:
-----------------------------------------
  atmos_correct_method      None
  beam_mode                 IW
  beam_swath                3
  first_date                2015-09-21
  first_frame               81
  flight_direction          A
  history                   2022-08-02
  last_date                 2021-11-12
  last_frame                82
  look_direction            R
  mission                   S1
  polarization              VV
  post_processing_method    MintPy
  prf                       1685.817302492702
  processing_dem            Unknown
  processing_software       isce
  processing_type           LOS_TIMESERIES
  relative_orbit            48
  scene_footprint           POLYGON((-80.23931223791634 25.40699917238294,-80.37648662818698 26.089217886949804,-79.58717570767169 26.215613743859954,-79.45471838400576 25.53393687281258,-80.23931223791634 25.40699917238294))
  unwrap_method             Unknown
  wavelength                0.05546576

-----------------------------------------
Subset mode is enabled, put subset range info in output filename.
Traceback (most recent call last):
  File "/work2/05861/tg851601/stampede2/codet/rsmas_insar/sources/MintPy/mintpy/save_hdfeos5.py", line 487, in <module>
    main(sys.argv[1:])
  File "/work2/05861/tg851601/stampede2/codet/rsmas_insar/sources/MintPy/mintpy/save_hdfeos5.py", line 469, in main
    out_file = get_output_filename(metadata=meta,
  File "/work2/05861/tg851601/stampede2/codet/rsmas_insar/sources/MintPy/mintpy/save_hdfeos5.py", line 274, in get_output_filename
    lat1 = float(metadata['Y_FIRST'])
KeyError: 'Y_FIRST'

It should use the relevant attributes from geometryRadar.h5. Alfredo has sort-of fixed similar issues in the insarmaps scripts when an S1* files is in radar coordinate. I can put a example dataset on Jetstream if you need.

How to get full resolution result?

I used the config text file "PichinchaSenDT142.txt" as provided in the example and did not modify any options before running it.

I followed the steps outlined in juyter notebook exactly as described.
I also executed the "3. Correct for geolocation step" at the bottom of the notebook.

I know that the resolution of Sentinel-1 is 5 x 20 m, but the result of above steps is like:

Above image is from QGIS Measure function

Are there any options to get the full resolution(5x20m)?
Why is it not a square but a rectangle in shape?
Can it be converted into a square?

TSX: shadowMask not loading into geometryRadar.h5

Hi @mirzaees ,
In the run_04 step I am getting the error below running generate_unwrap_mask.py because there is no shadowMask in the geometryRadar.h5 file. It turns out that it is looking for ./merged/geom_reference/shadowMask.rdr.full which exist for Sentinel-1 but not for TSX (there is only `/merged/geom_reference/shadowMask.rdr'). Saying

miaplpy.load.shadowMaskFile = ./merged/geom_reference/shadowMask.rdr   #[path2shadow_file], optional

seems to resolve it (should it?) but I wonder whether the autopath is working properly/.

cd MiamiTsxSMD14

ls -1 merged/*/*shadowMask*
merged/geom_reference/shadowMask.rdr
merged/geom_reference/shadowMask.rdr.rsc
merged/geom_reference/shadowMask.rdr.vrt
merged/geom_reference/shadowMask.rdr.xml

generate_unwrap_mask.py --geometry /scratch/05861/tg851601/MiamiTsxSMD14/miaplpy/inputs/geometryRadar.h5 
20221118:163300 * generate_unwrap_mask.py --geometry /scratch/05861/tg851601/MiamiTsxSMD14/miaplpy/inputs/geometryRadar.h5
There is no shadow mask in geometryRadar.h5, all values set to 1
Traceback (most recent call last):
  File "/work2/05861/tg851601/stampede2/code/rsmas_insar/sources/MiaplPy/miaplpy/generate_unwrap_mask.py", line 110, in <module>
    main()
  File "/work2/05861/tg851601/stampede2/code/rsmas_insar/sources/MiaplPy/miaplpy/generate_unwrap_mask.py", line 47, in main
    mintpy.generate_mask.main(args_shm.split())
AttributeError: module 'mintpy' has no attribute 'generate_mask'

install problem

Thanks for your great work. and I want to test it.
When i finished the install of miaplpy, an type "miaplpyApp.py -h" in terminal it shows:

from miaplpy.objects.arg_parser import MiaplPyParser
ModuleNotFoundError: No module named 'miaplpy'

Am i mssing some step?

Is it phase unwrapping without a mask？

Hello，Is it phase unwrapping without a mask？What is the difference between this and the unwinding method of stamps?

MintPy import error in ifgram_inversion_L1L2.py

With the test data I am getting this error. This was not an issue in MinoPy. I am using MintPy v1.4.1


<><><><><><><><><> step 8 - invert_network (MiaplPy) <><><><><><><><><><>
Generate /scratch2/05861/tg851601/PichinchaSenDT142/miaplpy/network_single_reference/run_files/run_08_miaplpy_invert_network
Traceback (most recent call last):
  File "/work2/05861/tg851601/stampede2/test3_frontera/rsmas_insar/sources/MiaplPy/miaplpy/network_inversion.py", line 11, in <module>
    from miaplpy.dev import ifgram_inversion_L1L2
  File "/work2/05861/tg851601/stampede2/test3_frontera/rsmas_insar/sources/MiaplPy/miaplpy/dev/ifgram_inversion_L1L2.py", line 24, in <module>
    from mintpy.utils import readfile, writefile, ptime, utils as ut, arg_group
ImportError: cannot import name 'arg_group' from 'mintpy.utils' (/work2/05861/tg851601/stampede2/test3_frontera/rsmas_insar/sources/MintPy/mintpy/utils/__init__.py)

unwrap_ifgram step errro with snaphu (NaN or infinity found in input float data)

I tried to run unwrap_ifgram step with miaplpyApp factory.txt --dir factory2 --dostep unwrap_ifgram
The result like:

20231003:183621 * unwrap_ifgram.py --ifg /home/github/isce2/project/synspective_factory_area/ascending/miaplpy/factory2/inverted/interferograms_single_reference/20200702_20200209/filt_fine.int --coherence /home/github/isce2/project/synspective_factory_area/ascending/miaplpy/factory2/inverted/tempCoh_average --unwrapped_ifg /home/github/isce2/project/synspective_factory_area/ascending/miaplpy/factory2/inverted/interferograms_single_reference/20200702_20200209/filt_fine.unw --max_discontinuity 1.2 --init_method MCF --length 2834 --width 21579 --height 697946.3676910399 --num_tiles 6 --earth_radius 6339244.253900407 --wavelength 0.05546576 -m /home/github/isce2/project/synspective_factory_area/ascending/miaplpy/factory2/inverted/mask_unwrap
snaphu -f /home/github/isce2/project/synspective_factory_area/ascending/miaplpy/factory2/inverted/interferograms_single_reference/20200702_20200221/config_all -d /home/github/isce2/project/synspective_factory_area/ascending/miaplpy/factory2/inverted/interferograms_single_reference/20200702_20200221/filt_fine.int 21579 -o /home/github/isce2/project/synspective_factory_area/ascending/miaplpy/factory2/inverted/interferograms_single_reference/20200702_20200221/filt_fine.unw --tile 2 3 500 500 --nproc 6
snaphu -f /home/github/isce2/project/synspective_factory_area/ascending/miaplpy/factory2/inverted/interferograms_single_reference/20200702_20200328/config_all -d /home/github/isce2/project/synspective_factory_area/ascending/miaplpy/factory2/inverted/interferograms_single_reference/20200702_20200328/filt_fine.int 21579 -o /home/github/isce2/project/synspective_factory_area/ascending/miaplpy/factory2/inverted/interferograms_single_reference/20200702_20200328/filt_fine.unw --tile 2 3 500 500 --nproc 6
20231003:183621 * unwrap_ifgram.py --ifg /home/github/isce2/project/synspective_factory_area/ascending/miaplpy/factory2/inverted/interferograms_single_reference/20200702_20200421/filt_fine.int --coherence /home/github/isce2/project/synspective_factory_area/ascending/miaplpy/factory2/inverted/tempCoh_average --unwrapped_ifg /home/github/isce2/project/synspective_factory_area/ascending/miaplpy/factory2/inverted/interferograms_single_reference/20200702_20200421/filt_fine.unw --max_discontinuity 1.2 --init_method MCF --length 2834 --width 21579 --height 697946.3676910399 --num_tiles 6 --earth_radius 6339244.253900407 --wavelength 0.05546576 -m /home/github/isce2/project/synspective_factory_area/ascending/miaplpy/factory2/inverted/mask_unwrap
snaphu -f /home/github/isce2/project/synspective_factory_area/ascending/miaplpy/factory2/inverted/interferograms_single_reference/20200702_20200209/config_all -d /home/github/isce2/project/synspective_factory_area/ascending/miaplpy/factory2/inverted/interferograms_single_reference/20200702_20200209/filt_fine.int 21579 -o /home/github/isce2/project/synspective_factory_area/ascending/miaplpy/factory2/inverted/interferograms_single_reference/20200702_20200209/filt_fine.unw --tile 2 3 500 500 --nproc 6
20231003:183621 * unwrap_ifgram.py --ifg /home/github/isce2/project/synspective_factory_area/ascending/miaplpy/factory2/inverted/interferograms_single_reference/20200702_20200304/filt_fine.int --coherence /home/github/isce2/project/synspective_factory_area/ascending/miaplpy/factory2/inverted/tempCoh_average --unwrapped_ifg /home/github/isce2/project/synspective_factory_area/ascending/miaplpy/factory2/inverted/interferograms_single_reference/20200702_20200304/filt_fine.unw --max_discontinuity 1.2 --init_method MCF --length 2834 --width 21579 --height 697946.3676910399 --num_tiles 6 --earth_radius 6339244.253900407 --wavelength 0.05546576 -m /home/github/isce2/project/synspective_factory_area/ascending/miaplpy/factory2/inverted/mask_unwrap
snaphu -f /home/github/isce2/project/synspective_factory_area/ascending/miaplpy/factory2/inverted/interferograms_single_reference/20200702_20200421/config_all -d /home/github/isce2/project/synspective_factory_area/ascending/miaplpy/factory2/inverted/interferograms_single_reference/20200702_20200421/filt_fine.int 21579 -o /home/github/isce2/project/synspective_factory_area/ascending/miaplpy/factory2/inverted/interferograms_single_reference/20200702_20200421/filt_fine.unw --tile 2 3 500 500 --nproc 6
snaphu -f /home/github/isce2/project/synspective_factory_area/ascending/miaplpy/factory2/inverted/interferograms_single_reference/20200702_20200304/config_all -d /home/github/isce2/project/synspective_factory_area/ascending/miaplpy/factory2/inverted/interferograms_single_reference/20200702_20200304/filt_fine.int 21579 -o /home/github/isce2/project/synspective_factory_area/ascending/miaplpy/factory2/inverted/interferograms_single_reference/20200702_20200304/filt_fine.unw --tile 2 3 500 500 --nproc 6
Terminated

the log from inverted/interferograms_single_reference/20200702_20200104/snaphu_tiles_334061 is like

snaphu (pid 334077): unwrapping tile at row 0, column 0

Current working directory is /home/github/isce2/project/synspective_factory_area/ascending/miaplpy/factory2
Reading wrapped phase from file /home/github/isce2/project/synspective_factory_area/ascending/miaplpy/factory2/inverted/interferograms_single_reference/20200702_20200104/filt_fine.int
NaN or infinity found in input float data
Abort

how to fix `NaN or infinity found in input float data` error?

Thanks

the command of ISCE2 was:

stackSentinel.py -s SLC/ -d DEM/*.dem.wgs84 -a AuxDir/ -o Orbits/ -b 11.0295 11.1651 106.5815 106.7192 -c 2

the miaplpy .txt file is like:

miaplpy.load.processor      = isce  #[isce,snap,gamma,roipac], auto for isceTops
miaplpy.load.updateMode     = auto  #[yes / no], auto for yes, skip re-loading if HDF5 files are complete
miaplpy.load.compression    = auto  #[gzip / lzf / no], auto for no.
miaplpy.load.autoPath       = no # [yes, no] auto for no
##---------Coregistered SLC images:        
miaplpy.load.slcFile        = /home/github/isce2/project/synspective_factory_area/ascending/merged/SLC/*/*.slc.full  #[path2slc_file]
##---------for ISCE only:
# miaplpy.load.metaFile       = /home/github/isce2/project/synspective_factory_area/ascending/reference/IW*.xml
miaplpy.load.metaFile       = /home/github/isce2/project/synspective_factory_area/ascending/reference/IW*.xml
# miaplpy.load.baselineDir    = /home/github/isce2/project/synspective_factory_area/ascending/baselines
miaplpy.load.baselineDir    = /home/github/isce2/project/synspective_factory_area/ascending/baselines
##---------geometry datasets:
miaplpy.load.demFile          = /home/github/isce2/project/synspective_factory_area/ascending/merged/geom_reference/hgt.rdr.full
miaplpy.load.lookupYFile      = /home/github/isce2/project/synspective_factory_area/ascending/merged/geom_reference/lat.rdr.full
miaplpy.load.lookupXFile      = /home/github/isce2/project/synspective_factory_area/ascending/merged/geom_reference/lon.rdr.full
miaplpy.load.incAngleFile     = /home/github/isce2/project/synspective_factory_area/ascending/merged/geom_reference/los.rdr.full
miaplpy.load.azAngleFile      = /home/github/isce2/project/synspective_factory_area/ascending/merged/geom_reference/los.rdr.full
miaplpy.load.shadowMaskFile   = /home/github/isce2/project/synspective_factory_area/ascending/merged/geom_reference/shadowMask.rdr.full
miaplpy.load.waterMaskFile    = None
##---------interferogram datasets:
miaplpy.load.unwFile        = ./inverted/interferograms_single_reference/*/*fine*.unw
miaplpy.load.corFile        = ./inverted/interferograms_single_reference/*/*fine*.cor
miaplpy.load.connCompFile   = ./inverted/interferograms_single_reference/*/*.unw.conncomp

##---------subset (optional):
## if both yx and lalo are specified, use lalo option unless a) no lookup file AND b) dataset is in radar coord
# miaplpy.subset.lalo         = -0.199:-0.142,-78.612:-78.578

# MiaplPy options
miaplpy.compute.numCores   = 8
miaplpy.interferograms.type = single_reference

## Mintpy options
mintpy.compute.cluster     = local  # if dask is not available, set this option to no
mintpy.compute.numWorker   = 8
mintpy.geocode.fillValue   = 0

snaphu unwrapping problem

Hi, I use snaphu to unwrapping interferograms under maiplpy framwork. the size of interferogram is 7259 * 27044 generated by part of full frame sentinel-1A SAR IW data. But the snaphu tile mode doesn't work, it shows Exceeded maximum number of secondary nodes Decrease TILECOSTTHRESH and/or increase MINREGIONSIZE. I try to adjust TILECOSTTHRESH and MINREGIONSIZE, and unwrapping still auto exit in some places. I have to increase the snaphu.tileNumPixels to force a single piece process, it works. but the one piece process consumes a lot of time and memory, I cann't image how to process full frame data.
Does anyone to Unwrapping full frame interferogram by snaphu? cause I saw the Chen and Zebker's paper it shows a 23240 * 4800 pixel interferogram unwrapping result in 2002. It shouldn't be a problem by now. how can i solve this problem？

another problem is I set the mask file path in config file, but the snaphu still unwrapping the full region even some area outside the mask, why?

smallbaselineApp.cfg not updated for --dostep load_ifgram

For reference point update, I am aware that I need to run
miaplpyApp.py $TE/MiamiSenAT48.template --dir miaplpy_Surfside_2016_202211 --start ifgram_correction,
but there seems to be a bug which confuses things. When I run after modifying the *.template file

miaplpyApp.py $TE/MiamiSenAT48.template --dir miaplpy_Surfside_2016_202211 --dostep load_ifgram

the reference.lalo field in smallbaselineApp.cfg does not get updated which I think is a bug. It gets updated only in the ifgram_correction step.

ifgram_inversion_L1L2.py not on path?

Hi @mirzaees ,
Quick question:I don't see that miaplpy/dev is added to the path. Should it be? Else should ifgram_inversion_L1L2.py be in a different directory?

Stack ISCE, SLC with os.scandir(src) as itr: FileNotFoundError: [Errno 2] No such file or directory: ''

Hi

Thanks for your answer, but in my case I run the stack with ISCE2 with stackSentinel.py and according to the example template
minopy.load.slcFile = ../merged/SLC//.slc.full #[path2slc_file]
I only have the *.slc.full.vrt and *slc.full.xml for each date.

The message is:

File "/home//tools/MiaplPy/miaplpy/miaplpyApp.py", line 801, in
main()
File "/home/tools/MiaplPy/miaplpy/miaplpyApp.py", line 82, in main
app.open()
File "/home/stools/MiaplPy/miaplpy/miaplpyApp.py", line 176, in open
self.date_list, self.num_pixels, self.metadata = read_initial_info(self.workDir, self.templateFile)
File "/home/tools/MiaplPy/miaplpy/objects/utils.py", line 1221, in read_initial_info
iDict = read_inps2dict(inps_loadSlc)
File "/home/tools/MiaplPy/miaplpy/objects/utils.py", line 1293, in read_inps2dict
shutil.copytree(reference_dir, out_reference)

with os.scandir(src) as itr:
FileNotFoundError: [Errno 2] No such file or directory: ''

The same of @gongzhiqiang97

Thanks

Alejandro

Originally posted by @alejobeap in #45 (comment)

Compatible version of ISCE

Hi,

I tried to run the package with one of the sample datasets but I got the following error at the beginning step:

Traceback (most recent call last):
File "/home/insar/tools/MiaplPy/miaplpy/phase_linking.py", line 18, in
from miaplpy.lib import invert as iv
File "invert.pyx", line 13, in init invert
ModuleNotFoundError: No module named 'isceobj'
Traceback (most recent call last):
File "/home/insar/tools/MiaplPy/miaplpy/phase_linking.py", line 18, in
from miaplpy.lib import invert as iv
File "invert.pyx", line 13, in init invert
ModuleNotFoundError: No module named 'isceobj'
Traceback (most recent call last):
File "/home/insar/tools/MiaplPy/miaplpy/phase_linking.py", line 18, in
from miaplpy.lib import invert as iv
File "invert.pyx", line 13, in init invert
ModuleNotFoundError: No module named 'isceobj'

I believe that I set the correct environment path to all ISCE, MintPy, and MiapIPy packages (as you can see in the attachment) since I can run the stackSentinel.py correctly. Could you please kindly let me know which version of ISCE is compatible with your package?

bashrc.txt

Sincerely,
Sayyed

Module 'mintpy.load_data' has no attribute 'print_write_setting'

After updating to the latest version of MintPy, a new error came up. After looking into it, the error was cause by the removal of the function print_write_setting in mintpy.load_data due to the PR https://github.com/insarlab/MintPy/pull/796/files#diff-99744ebb08e7cd1faee6bef21777ee1f3ce58975550918bd3213d804e6f6a502. I am stuck on how to properly addressed this new issue.

I attached below the error I encountered.

searching geometry files info
input data files:
height          : /home/data/INSAR_processing/bjm_ts/kalimantan/a1_stripmapstack/merged/geom_reference/hgt.rdr
latitude        : /home/data/INSAR_processing/bjm_ts/kalimantan/a1_stripmapstack/merged/geom_reference/lat.rdr
longitude       : /home/data/INSAR_processing/bjm_ts/kalimantan/a1_stripmapstack/merged/geom_reference/lon.rdr
incidenceAngle  : /home/data/INSAR_processing/bjm_ts/kalimantan/a1_stripmapstack/merged/geom_reference/los.rdr
azimuthAngle    : /home/data/INSAR_processing/bjm_ts/kalimantan/a1_stripmapstack/merged/geom_reference/los.rdr
shadowMask      : /home/data/INSAR_processing/bjm_ts/kalimantan/a1_stripmapstack/merged/geom_reference/shadowMask.rdr
Traceback (most recent call last):
  File "/home/bryanjim001/tools/MiaplPy/miaplpy/load_slc_geometry.py", line 222, in <module>
    main()
  File "/home/bryanjim001/tools/MiaplPy/miaplpy/load_slc_geometry.py", line 85, in main
    updateMode, comp, box, boxGeo = mld.print_write_setting(iDict)
AttributeError: module 'mintpy.load_data' has no attribute 'print_write_setting'

A confusing typo in generate_ifgram.py

I suppose in this line (as well as in the function call) you mean run_interferogram or run_ifgram,

MiaplPy/src/miaplpy/generate_ifgram.py

Line 68 in 7dbf0c1

def run_inreferogram(inps, ifg_file):

right?

phase_linking.py does not exit for fatal error `PATCH_0360 is not inverted`

Hi @mirzaee
I mentioned previously that miaplpy does not always stops when an error occurs (#37). Here an example: run_03_miaplpy_concatenate_patches_0 ran for many hours and created a 137GB run_03_miaplpy_concatenate_patches_0__1.o file containing:

Error: PATCH_0360 is not inverted, run previous step (phase_linking) to complete
Error: PATCH_0360 is not inverted, run previous step (phase_linking) to complete

If we replace in

MiaplPy/miaplpy/phase_linking.py

Line 143 in 30fa971

 print('Error: PATCH_{:04.0f} is not inverted, run previous step (phase_linking) to complete'.format(index)) 

print by raise RuntimeError, so that the line reads

raise RuntimeError('Error: PATCH_{:04.0f} is not inverted, run previous step (phase_linking) to complete'.format(index))

it will exit with an exception. Could you do this? My PRs don't seem to get through.

How to set ISCE stackSentinel.py parameters in order to run MiaplPy?

Hello,

I used to analyze Sentinel1 images with MintPy and I found that MiaplPy seems to need different input. However, I'm confused about how to set stackSentinel.py parameters so that I can get coregistered SLC data in full resolution. Is there any document/discussion that could be referenced? (I can hardly find one so far)

Thank you

redundant code in miaplpyApp.py

There is some lines twice in miaplpyApp.py:

MiaplPy/miaplpy/miaplpyApp.py

Lines 139 to 149 in 30fa971

 name_ifg_network = self.template['miaplpy.interferograms.networkType'] 

 if self.template['miaplpy.interferograms.networkType'] == 'delaunay': 

 name_ifg_network += '_{}'.format(self.template['miaplpy.interferograms.delaunayBaselineRatio']) 

 elif self.template['miaplpy.interferograms.networkType'] == 'sequential': 

 name_ifg_network += '_{}'.format(self.template['miaplpy.interferograms.connNum']) 

 name_ifg_network = self.template['miaplpy.interferograms.networkType'] 

 if self.template['miaplpy.interferograms.networkType'] == 'delaunay': 

 name_ifg_network += '_{}'.format(self.template['miaplpy.interferograms.delaunayBaselineRatio']) 

 elif self.template['miaplpy.interferograms.networkType'] == 'sequential': 

 name_ifg_network += '_{}'.format(self.template['miaplpy.interferograms.connNum'])

The platform metadata is missing from load_ifgram.py

The platform metadata is missing from the load_ifgram.py which causes my timeseries_correction processing step to fail during correct_LOD.
A temporary workaround for now is I defined the platform on the miaplpyApp.cfg.

`NaN or infinity found in input float data` error in unwrapping step

I am occasionally getting this error in the unwrapping step. Any idea why this occurs? I have processed this dataset before, possibly with slightly different options (e.g. different starting date). Any idea why this occurs?

cat run_05_miaplpy_unwrap_ifgram_0__10.o
20221029:213429 * unwrap_ifgram.py --ifg /scratch/05861/tg851601/MiamiSenAT48/miaplpy/inverted/interferograms_single_reference/20210104_20190912/filt_fine.int --coherence /scratch/05861/tg851601/MiamiSenAT48/miaplpy/inverted/tempCoh_average --unwrapped_ifg /scratch/05861/tg851601/MiamiSenAT48/miaplpy/inverted/interferograms_single_reference/20210104_20190912/filt_fine.unw --max_discontinuity 1.2 --init_method MCF --length 629 --width 1296 --height 698454.1600760925 --num_tiles 1 --earth_radius 6348066.309525905 --wavelength 0.05546576 -m /scratch/05861/tg851601/MiamiSenAT48/miaplpy/inverted/mask_unwrap --tmp
snaphu -f /scratch/05861/tg851601/MiamiSenAT48/miaplpy/inverted/interferograms_single_reference/20210104_20190912/config_all -d /scratch/05861/tg851601/MiamiSenAT48/miaplpy/inverted/interferograms_single_reference/20210104_20190912/filt_fine.int 1296 -o /scratch/05861/tg851601/MiamiSenAT48/miaplpy/inverted/interferograms_single_reference/20210104_20190912/filt_fine.unw
b'NaN or infinity found in input float data\nAbort\n'
Time spent: 0.00882656176884969 m

In any case, this error should show up in the *.e file. I will make a note in the Suggested improvements issue.

Improvement suggestions

Here two suggestions
- run_files directory :
It would be good to have two directories:
miaplpy/run_files for run_01_* to run_03_* and
miaplpy/network_single_reference/run_files for run_04* to run_09_*

- interferogram.networkType option
Given that a directory network_sequential_3 or network_sequential_5 is generated, it would be helpful to have the ability to optionally specify the network as

interferogram.networkType=sequential_3
interferogram.networkType=sequential_5

with interferogram.numConn being read from the sequential_3 string. Similar for the Delaunay network options (which I have not tried). That would make it easier and cleaner to compare results for different networks

Multilook SLCs

Hi Sara,

I am working with full SLCs for InSAR time series. I think it could be computationally expensive for non linear phase inversion. Can I use multilooked SLCs instead?

Best regards,
Mohammad

No slcStack.h5 with gamma processor

There are always errors when i run as follows:

miaplpyApp.py YuzhouSen.template --dostep load_data ./

........
.......
processor: gamma
Traceback (most recent call last):
File "/home/sar02/tools/MiaplPy/miaplpy/miaplpyApp.py", line 801, in
main()
File "/home/sar02/tools/MiaplPy/miaplpy/miaplpyApp.py", line 82, in main
app.open()
File "/home/sar02/tools/MiaplPy/miaplpy/miaplpyApp.py", line 176, in open
self.date_list, self.num_pixels, self.metadata = read_initial_info(self.workDir, self.templateFile)
File "/home/sar02/tools/MiaplPy/miaplpy/objects/utils.py", line 1221, in read_initial_info
iDict = read_inps2dict(inps_loadSlc)
File "/home/sar02/tools/MiaplPy/miaplpy/objects/utils.py", line 1293, in read_inps2dict
shutil.copytree(reference_dir, out_reference)
File "/home/sar02/anaconda3/envs/insar/lib/python3.8/shutil.py", line 555, in copytree
with os.scandir(src) as itr:
FileNotFoundError: [Errno 2] No such file or directory: ''

here is my template:
miaplpy.load.processor = gamma
miaplpy.load.unwFile = ./PROCESS/DONE/IFG*/diff* rlks.unw
miaplpy.load.corFile = ./PROCESS/DONE/IFG* /* filt*rlks.cor
miaplpy.load.connCompFile = None
miaplpy.load.intFile = None

miaplpy.load.slcFile = ./merged/SLC/* /*.slc.full #[path2slc_file]

miaplpy.load.demFile = ./PROCESS/SIM/sim_${m_date12}/sim*.hgt_sim
miaplpy.load.lookupYFile = ./PROCESS/SIM/sim_${m_date12}/sim*.UTM_TO_RDC
miaplpy.load.lookupXFile = ./PROCESS/SIM/sim_${m_date12}/sim*.UTM_TO_RDC
miaplpy.load.incAngleFile = None
miaplpy.load.azAngleFile = None
miaplpy.load.shadowMaskFile = None
miaplpy.load.bperpFile = ./merged/baselines/* /* .base_perp

miaplpy.multiprocessing.numProcessor = 4
miaplpy.interferograms.type = single_reference

mintpy.compute.cluster = local # if dask is not available, set this option to no
mintpy.compute.numWorker = 8

how can i create the file of slcStack.h5?
i have prepared all files of this needed and checked all path, those are ok. so how can i solve this problem? hope your advice! thanks!

How to prepare gamma SLC data for MiapIpy ?

I processed some s1 interferograms by gamma and I want us Miapipy to do time series analysis. I found There is no prep_slc_gamma.py scripts. What I need to do in the prep_slc_gamma.py scripts?

wrong option in Pichincha example file

It should be: miaplpy.interferograms.networkType but it currently is miaplpy.interferograms.type.

When you update please consider to add these options:

mintpy.troposphericDelay.method   = no    # pyaps  #[pyaps / height_correlation / base_trop_cor / no], auto for pyaps
mintpy.save.hdfEos5                         = yes   #[yes / update / no], auto for no, save timeseries to UNAVCO InSAR Archive format

I also would add the other options similar to the defaults: # [mini_stacks, single_reference, sequential, delaunay] default: single_reference

How to crop data prepared for miaplpy

Hello, I see that the size of the sample data given by miaplpy is very small, may I ask how to crop the data in isce processing, looking forward to your reply!

ValueError: Cannot return member number (operation not supported for type class)

Hi,
Does anyone know what this error is in the load_ifgram step?
thx

create HDF5 file /home/insar/1/Fast/Demo/proc/merged/miaplpy/network_delaunay_4/inputs/ifgramStack.h5 with w mode
create dataset /unwrapPhase of <class 'numpy.float32'> in size of (51, 2140, 6140) with compression = lzf
[==================================================] 20210810_20210822 19s / 0s
create dataset /coherence of <class 'numpy.float32'> in size of (51, 2140, 6140) with compression = lzf
[==================================================] 20210810_20210822 20s / 0s
create dataset /connectComponent of <class 'numpy.int16'> in size of (51, 2140, 6140) with compression = lzf
[==================================================] 20210810_20210822 3s / 0s
create dataset /wrapPhase of <class 'numpy.float32'> in size of (51, 2140, 6140) with compression = lzf
[> 2% ] 20170608_20170714 0s / 0sValueError: Cannot return member number (operation not supported for type class)
Exception ignored in: 'h5py._proxy.make_reduced_type'
Traceback (most recent call last):
File "/insar/miniconda3/lib/python3.11/site-packages/h5py/_hl/dataset.py", line 1009, in setitem
self.id.write(mspace, fspace, val, mtype, dxpl=self._dxpl)
ValueError: Cannot return member number (operation not supported for type class)
Traceback (most recent call last):
File "/insar/MiaplPy/miaplpy/load_ifgram.py", line 474, in
main()
File "/insar/MiaplPy/miaplpy/load_ifgram.py", line 390, in main
stackObj.write2hdf5(outputFile=inps.outfile[0],
File "/insar/miniconda3/lib/python3.11/site-packages/mintpy/objects/stackDict.py", line 215, in write2hdf5
ds[i, :, :] = data
~~^^^^^^^^^
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "/insar/miniconda3/lib/python3.11/site-packages/h5py/_hl/dataset.py", line 1009, in setitem
self.id.write(mspace, fspace, val, mtype, dxpl=self._dxpl)
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "h5py/h5d.pyx", line 280, in h5py.h5d.DatasetID.write
File "h5py/_proxy.pyx", line 110, in h5py._proxy.dset_rw
File "h5py/_proxy.pyx", line 324, in h5py._proxy.needs_proxy
ValueError: Not a datatype (not a datatype)

Occasional in step 9 in save_hdfeos5.py: key error: LON_REF1

Occasionally I am getting this LON_REF1 attribute error in the last step. For the Pichincha test data I don't get it. A couple of days ago it worked fine with this data but slight different parameters.
Any idea?

******************** step - hdfeos5 ********************
--------------------------------------------

save_hdfeos5.py /scratch2/05861/tg851601/MiamiSenAT48/miaplpy/network_single_reference/geo/geo_timeseries_demErr.h5 --tc /scratch2/05861/tg851601/MiamiSenAT48/miaplpy/network_single_reference/geo/geo_temporalCoheren
ce.h5 --asc /scratch2/05861/tg851601/MiamiSenAT48/miaplpy/network_single_reference/geo/geo_avgSpatialCoh.h5 -m /scratch2/05861/tg851601/MiamiSenAT48/miaplpy/network_single_reference/geo/geo_maskTempCoh.h5 -g /scratc
h2/05861/tg851601/MiamiSenAT48/miaplpy/network_single_reference/geo/geo_geometryRadar.h5 -t /scratch2/05861/tg851601/MiamiSenAT48/miaplpy/network_single_reference/smallbaselineApp.cfg
read options from template file: smallbaselineApp.cfg
Traceback (most recent call last):
  File "/work2/05861/tg851601/stampede2/test3_frontera/rsmas_insar/sources/MintPy/mintpy/cli/smallbaselineApp.py", line 213, in <module>
    main(sys.argv[1:])
  File "/work2/05861/tg851601/stampede2/test3_frontera/rsmas_insar/sources/MintPy/mintpy/cli/smallbaselineApp.py", line 208, in main
    run_smallbaselineApp(inps)
  File "/work2/05861/tg851601/stampede2/test3_frontera/rsmas_insar/sources/MintPy/mintpy/smallbaselineApp.py", line 1115, in run_smallbaselineApp
    app.run(steps=inps.runSteps)
  File "/work2/05861/tg851601/stampede2/test3_frontera/rsmas_insar/sources/MintPy/mintpy/smallbaselineApp.py", line 921, in run
    self.run_save2hdfeos5(sname)
  File "/work2/05861/tg851601/stampede2/test3_frontera/rsmas_insar/sources/MintPy/mintpy/smallbaselineApp.py", line 861, in run_save2hdfeos5
    mintpy.cli.save_hdfeos5.main(iargs)
  File "/work2/05861/tg851601/stampede2/test3_frontera/rsmas_insar/sources/MintPy/mintpy/cli/save_hdfeos5.py", line 107, in main
    save_hdfeos5(inps)
  File "/work2/05861/tg851601/stampede2/test3_frontera/rsmas_insar/sources/MintPy/mintpy/save_hdfeos5.py", line 437, in save_hdfeos5
    meta = prep_metadata(
  File "/work2/05861/tg851601/stampede2/test3_frontera/rsmas_insar/sources/MintPy/mintpy/save_hdfeos5.py", line 64, in prep_metadata
    unavco_meta = metadata_mintpy2unavco(meta, ts_obj.dateList, geom_file)
  File "/work2/05861/tg851601/stampede2/test3_frontera/rsmas_insar/sources/MintPy/mintpy/save_hdfeos5.py", line 118, in metadata_mintpy2unavco
    lons = [meta['LON_REF1'],
KeyError: 'LON_REF1'
Go back to directory: /scratch2/05861/tg851601/MiamiSenAT48/miaplpy

###############################################################

reference date not available as attribute in processing products

Hi @mirzaees , here a suggestion for improvement: For single_reference network (and mini stacks), it would be useful to include the reference date

FileNotFoundError: [Errno 2] No such file or directory: '' (*.slc.full from isce2 is missing)

I used ISCE2 to create an interferogram stack for use in Miaplpy.
When i run miaplpyApp.py factory.txt

miaplpy.load.processor      = isce  #[isce,snap,gamma,roipac], auto for isceTops
miaplpy.load.updateMode     = auto  #[yes / no], auto for yes, skip re-loading if HDF5 files are complete
miaplpy.load.compression    = auto  #[gzip / lzf / no], auto for no.
miaplpy.load.autoPath       = yes    # [yes, no] auto for no
##---------Coregistered SLC images:        
miaplpy.load.slcFile        = ../merged/SLC/*/*.slc.full  #[path2slc_file]
##---------for ISCE only:
miaplpy.load.metaFile       = ../reference/IW*.xml
miaplpy.load.baselineDir    = ../baselines
##---------geometry datasets:
miaplpy.load.demFile          = ../merged/geom_reference/hgt.rdr.full
miaplpy.load.lookupYFile      = ../merged/geom_reference/lat.rdr.full
miaplpy.load.lookupXFile      = ../merged/geom_reference/lon.rdr.full
miaplpy.load.incAngleFile     = ../merged/geom_reference/los.rdr.full
miaplpy.load.azAngleFile      = ../merged/geom_reference/los.rdr.full
miaplpy.load.shadowMaskFile   = ../merged/geom_reference/shadowMask.rdr.full
miaplpy.load.waterMaskFile    = None
##---------interferogram datasets:
miaplpy.load.unwFile        = ./inverted/interferograms_single_reference/*/*fine*.unw
miaplpy.load.corFile        = ./inverted/interferograms_single_reference/*/*fine*.cor
miaplpy.load.connCompFile   = ./inverted/interferograms_single_reference/*/*.unw.conncomp

##---------subset (optional):
## if both yx and lalo are specified, use lalo option unless a) no lookup file AND b) dataset is in radar coord
# miaplpy.subset.lalo         = -0.199:-0.142,-78.612:-78.578

# MiaplPy options
miaplpy.compute.numCores   = 8
miaplpy.interferograms.type = single_reference

## Mintpy options
mintpy.compute.cluster     = local  # if dask is not available, set this option to no
mintpy.compute.numWorker   = 8

And the errors are below

    miaplpy.load.compression: no --> no
    miaplpy.load.autoPath: no --> no
SAR platform/sensor : None
processor: isce
Traceback (most recent call last):
  File "/root/miniconda3/envs/miaplpy/bin/miaplpyApp", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/root/miniconda3/envs/miaplpy/lib/python3.11/site-packages/miaplpy/miaplpyApp.py", line 82, in main
    app.open()
  File "/root/miniconda3/envs/miaplpy/lib/python3.11/site-packages/miaplpy/miaplpyApp.py", line 170, in open
    self.date_list, self.num_pixels, self.metadata = read_initial_info(self.workDir, self.templateFile)
                                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/miaplpy/lib/python3.11/site-packages/miaplpy/objects/utils.py", line 1223, in read_initial_info
    iDict = read_inps2dict(inps_loadSlc)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/miniconda3/envs/miaplpy/lib/python3.11/site-packages/miaplpy/objects/utils.py", line 1295, in read_inps2dict
    shutil.copytree(reference_dir, out_reference)
  File "/root/miniconda3/envs/miaplpy/lib/python3.11/shutil.py", line 559, in copytree
    with os.scandir(src) as itr:
         ^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: ''

I've tried setting the path to an absolute path, but I'm still getting the same error.

When I tried the Pichincha volcano example provided by Miaplpy, it ran without any issues.
However, when I tried it with the results generated by isce2, I encountered this error.

In my opinion, the error is occurring because the file *.slc.full does not exist in the results I generated with isce2.

The list of files used in Miaplpy appears as follows:
(This is the example data of PichinchaSenDT142 in Miaplpy)

However, the files I generated using ISCE2 do not include *.slc.full files.

The command I used in isce2 is as follows:

stackSentinel.py -s SLC/ -d DEM/*.dem.wgs84 -a AuxDir/ -o Orbits/ -b "{bbox}" -c 2

How can I generate the *.slc.full files?

The new installation is not adding the relevant files at */defaults

Hi,

I have installed Miaplpy with pip and when running it I got errors such as
FileNotFoundError: [Errno 2] No such file or directory: '/home/.../site-packages/miaplpy/defaults/conf.full'
(I added it manually), and then:
FileNotFoundError: [Errno 2] No such file or directory: '/home/.../site-packages/miaplpy/objects/../defaults/miaplpyApp.cfg'

`/reference/data.rsc` confusion between MiaplPy and MintPy

Hi @mirzaees
There are a number of issues with /reference/data.rsc which is created by both prep_isce.py of MintPy and prep_slc_isce.py of MiaplPy. Maybe we should rename to /reference/data_slc.rsc ? This might be better then always creating a new data.rsc as I suggested in this issue.

In the load_ifgram step of MiaplPy it uses the existing /reference/data.rsc . This causes a problem in save_HDF5EOS step of MintPy as LON_REF1 etc attributes are missing, which is described in this issue

Edited:
An *rsc file in the /merged directory is also giving trouble. I need to run rm merged/*/*rsc merged/*/*/*rsc reference/data.rsc so that the first issue above does not appear (I suspect in merged/geom_reference but I did not check)

	name_ifg_network = self.template['miaplpy.interferograms.networkType']
	if self.template['miaplpy.interferograms.networkType'] == 'delaunay':
	name_ifg_network += '_{}'.format(self.template['miaplpy.interferograms.delaunayBaselineRatio'])
	elif self.template['miaplpy.interferograms.networkType'] == 'sequential':
	name_ifg_network += '_{}'.format(self.template['miaplpy.interferograms.connNum'])

	name_ifg_network = self.template['miaplpy.interferograms.networkType']
	if self.template['miaplpy.interferograms.networkType'] == 'delaunay':
	name_ifg_network += '_{}'.format(self.template['miaplpy.interferograms.delaunayBaselineRatio'])
	elif self.template['miaplpy.interferograms.networkType'] == 'sequential':
	name_ifg_network += '_{}'.format(self.template['miaplpy.interferograms.connNum'])

insarlab / miaplpy Goto Github PK

miaplpy's People

Contributors

Stargazers

Watchers

Forkers

miaplpy's Issues

MiaplPy version v0.2.1, date 2023-08-22 --RUN-at-2023-08-27 23:08:35.233485-- Current directory: /mnt/d/work/test/PichinchaSenDT142 Run routine processing with arg_parser.py on steps: ['invert_network'] Remaining steps: ['timeseries_correction'] Project name: PichinchaSenDT142

how to fix NaN or infinity found in input float data error?

miaplpyApp.py YuzhouSen.template --dostep load_data ./

Recommend Projects

Recommend Topics

Recommend Org

MiaplPy version v0.2.1, date 2023-08-22
--RUN-at-2023-08-27 23:08:35.233485--
Current directory: /mnt/d/work/test/PichinchaSenDT142
Run routine processing with arg_parser.py on steps: ['invert_network']
Remaining steps: ['timeseries_correction']
Project name: PichinchaSenDT142

how to fix `NaN or infinity found in input float data` error?