Giter VIP home page Giter VIP logo

hyp3-gamma's Introduction

HyP3 documentation

DOI

HyP3 documentation is built using MkDocs and the ASF Theme.

How to

Setting up a development environment

In order to automatically document some of our APIs, we use a conda environment with our APIs installed. You can get Miniconda (recommended) here:

https://docs.conda.io/en/latest/miniconda.html

Once conda is installed, from the repository root, you can create and activate a conda environment with all the necessary dependencies

conda env create -f environment.yml
conda activate hyp3-docs

Later, you can update the environment's dependencies with

conda env update -f environment.yml

Build and view the documentation site

With the hyp3-docs conda environment activated, run

mkdocs serve

to generate the documentation. This will allow you to view it at http://127.0.0.1:8000/. MkDocs will automatically watch for new/changed files in this directory and rebuild the website so you can see your changes live (just refresh the webpage!).

Note: mkdocs serve captures your terminal; use crtl+c to exit. It is recommended you use a second/dedicated terminal so you can keep this command running.

Deploy

This documentation site is deployed as a GitHub Organization website with a CNAME so that it's viewable at https://hyp3-docs.asf.alaska.edu/. The website is served out of the special https://github.com/ASFHyP3/ASFHyP3.github.io repository. Deployment is handled automatically with the .github/workflows/deploy_to_github_io.yml GitHub Action for any merge to main.

There is also a test site deployed to https://hyp3-docs.asf.alaska.edu/hyp3-docs, which tracks the develop branch of this repo and is served out of the gh-pages branch of this repo.

Markdown formatting

The way MkDocs and GitHub parse the markdown documents are slightly different. Some compatibility tips:

  • Raw links should be wrapped in angle brackets: <https://example.com>

  • MkDocs is pickier about whitespace between types (e.g., headers, paragraphs, lists) and seems to expect indents to be 4 spaces. So to get a representation like:


    • A list item

      A sub list heading
      • A sub-list item

    in MkDocs, you'll want to write it like:

    Good

    - A list item
    
        ##### A sub list heading
        - A sub-list item
    

    Bad

    - A list item
      ##### A sub list heading
      - A sub-list item
    
    - A list item
        ##### A sub list heading
        - A sub-list item
    
    - A list item
    
      ##### A sub list heading
      - A sub-list item
    

hyp3-gamma's People

Contributors

andrewplayer3 avatar asjohnston-asf avatar cirrusasf avatar dependabot[bot] avatar emlundell avatar forrestfwilliams avatar hjkristenson avatar jacquelynsmale avatar jhkennedy avatar jlrine2 avatar jtherrmann avatar khogenso avatar ninneman avatar rudigens avatar talogan avatar tools-bot avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

hyp3-gamma's Issues

Fix inconsistent logging output

Jira: https://asfdaac.atlassian.net/browse/TOOL-2392

Note: The above link is accessible only to members of ASF.


Logging output appears inconsistent, e.g. for this job the log lines at the end of the file do not include the <timestamp> - <level> - prefix (I replaced some lines with [...] to make this example shorter):

12/01/2023 05:59:23 PM - INFO - Proc: system time (s):       0.000
12/01/2023 05:59:23 PM - INFO - Proc: elapsed time (s):      0.000
12/01/2023 05:59:23 PM - INFO - Finished: offset_add 20200308_20200401.off.it 20200308_20200401.off_3 20200308_20200401.off_3.temp
12/01/2023 05:59:23 PM - ERROR - ERROR: Found azimuth offset of               0.03181
!
Running command: rasSLC 20200308_003.slc 23781 1 0 50 10
subprocess return value was 0
Proc: *** DISP Program rasSLC ***
Proc: *** Copyright 2022, Gamma Remote Sensing, v1.6 22-Jun-2022 clw/cm ***
Proc: *** Calculate a multilook intensity raster image from complex data (FCOMPLEX, SCOMPLEX) using power-law scaling ***
Proc: OPENMP: number of physical processors:                     4
Proc: OPENMP: default max. number of threads (program specific): 6
Proc: OPENMP: current max. number of available threads:          4

[...]

Proc: system time (s):       0.420
Proc: elapsed time (s):      2.200
Finished: rasSLC 20200401_003.slc 23779 1 0 50 10

Also, it's annoying that the ERROR line is not the last line of the log, so I want to confirm that the log lines are getting written in the correct order, or if there's something we can do to make the container fail earlier if that ERROR is the reason this job failed.

`util.unzip_granule` fails with `StopIteration` for Sentinel-1 scenes acquired since 2023-09-26

09/29/2023 04:18:06 PM - INFO - Downloading https://sentinel1.asf.alaska.edu/GRD_HD/SA/S1A_IW_GRDH_1SDV_20230928T113814_20230928T113839_050527_0615EC_864E.zip
09/29/2023 04:18:53 PM - INFO - Unzipping S1A_IW_GRDH_1SDV_20230928T113814_20230928T113839_050527_0615EC_864E.zip
Traceback (most recent call last):
  File "/usr/local/bin/hyp3_gamma", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/dist-packages/hyp3_gamma/__main__.py", line 37, in main
    process_entry_point.load()()
  File "/usr/local/lib/python3.10/dist-packages/hyp3_gamma/__main__.py", line 87, in rtc
    safe_dir = util.get_granule(args.granule)
  File "/usr/local/lib/python3.10/dist-packages/hyp3_gamma/util.py", line 38, in get_granule
    safe_dir = unzip_granule(zip_file, remove=True)
  File "/usr/local/lib/python3.10/dist-packages/hyp3_gamma/util.py", line 46, in unzip_granule
    safe_dir = next(item.filename for item in z.infolist() if item.is_dir() and item.filename.endswith('.SAFE/'))
StopIteration

Older Sentinel-1 zip files include entries for directories in their zip manifest in addition to individual files:

$ unzip -l S1B_IW_GRDH_1SDV_20191030T075444_20191030T075520_018702_0233FC_6DCD.zip
Archive:  S1B_IW_GRDH_1SDV_20191030T075444_20191030T075520_018702_0233FC_6DCD.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  2019-10-30 02:47   S1B_IW_GRDH_1SDV_20191030T075444_20191030T075520_018702_0233FC_6DCD.SAFE/
        0  2019-10-30 02:47   S1B_IW_GRDH_1SDV_20191030T075444_20191030T075520_018702_0233FC_6DCD.SAFE/support/
     6427  2019-10-30 02:47   S1B_IW_GRDH_1SDV_20191030T075444_20191030T075520_018702_0233FC_6DCD.SAFE/support/s1-level-1-calibration.xsd
   147374  2019-10-30 02:47   S1B_IW_GRDH_1SDV_20191030T075444_20191030T075520_018702_0233FC_6DCD.SAFE/support/s1-level-1-product.xsd
    60608  2019-10-30 02:47   S1B_IW_GRDH_1SDV_20191030T075444_20191030T075520_018702_0233FC_6DCD.SAFE/support/s1-object-types.xsd
     7290  2019-10-30 02:47   S1B_IW_GRDH_1SDV_20191030T075444_20191030T075520_018702_0233FC_6DCD.SAFE/support/s1-level-1-noise.xsd
      450  2019-10-30 02:47   S1B_IW_GRDH_1SDV_20191030T075444_20191030T075520_018702_0233FC_6DCD.SAFE/support/s1-map-overlay.xsd
      469  2019-10-30 02:47   S1B_IW_GRDH_1SDV_20191030T075444_20191030T075520_018702_0233FC_6DCD.SAFE/support/s1-level-1-quicklook.xsd
      440  2019-10-30 02:47   S1B_IW_GRDH_1SDV_20191030T075444_20191030T075520_018702_0233FC_6DCD.SAFE/support/s1-product-preview.xsd
      471  2019-10-30 02:47   S1B_IW_GRDH_1SDV_20191030T075444_20191030T075520_018702_0233FC_6DCD.SAFE/support/s1-level-1-measurement.xsd
        0  2019-10-30 02:47   S1B_IW_GRDH_1SDV_20191030T075444_20191030T075520_018702_0233FC_6DCD.SAFE/measurement/
1189675334  2019-10-30 02:50   S1B_IW_GRDH_1SDV_20191030T075444_20191030T075520_018702_0233FC_6DCD.SAFE/measurement/s1b-iw-grd-vh-20191030t075444-20191030t075520-018702-0233fc-002.tiff
1189675334  2019-10-30 02:49   S1B_IW_GRDH_1SDV_20191030T075444_20191030T075520_018702_0233FC_6DCD.SAFE/measurement/s1b-iw-grd-vv-20191030t075444-20191030t075520-018702-0233fc-001.tiff
        0  2019-10-30 02:46   S1B_IW_GRDH_1SDV_20191030T075444_20191030T075520_018702_0233FC_6DCD.SAFE/annotation/
  2220187  2019-10-30 02:47   S1B_IW_GRDH_1SDV_20191030T075444_20191030T075520_018702_0233FC_6DCD.SAFE/annotation/s1b-iw-grd-vh-20191030t075444-20191030t075520-018702-0233fc-002.xml
  2231079  2019-10-30 02:47   S1B_IW_GRDH_1SDV_20191030T075444_20191030T075520_018702_0233FC_6DCD.SAFE/annotation/s1b-iw-grd-vv-20191030t075444-20191030t075520-018702-0233fc-001.xml
        0  2019-10-30 02:46   S1B_IW_GRDH_1SDV_20191030T075444_20191030T075520_018702_0233FC_6DCD.SAFE/annotation/calibration/
   596883  2019-10-30 02:46   S1B_IW_GRDH_1SDV_20191030T075444_20191030T075520_018702_0233FC_6DCD.SAFE/annotation/calibration/noise-s1b-iw-grd-vh-20191030t075444-20191030t075520-018702-0233fc-002.xml
   596883  2019-10-30 02:47   S1B_IW_GRDH_1SDV_20191030T075444_20191030T075520_018702_0233FC_6DCD.SAFE/annotation/calibration/noise-s1b-iw-grd-vv-20191030t075444-20191030t075520-018702-0233fc-001.xml
  1394957  2019-10-30 02:46   S1B_IW_GRDH_1SDV_20191030T075444_20191030T075520_018702_0233FC_6DCD.SAFE/annotation/calibration/calibration-s1b-iw-grd-vh-20191030t075444-20191030t075520-018702-0233fc-002.xml
  1394957  2019-10-30 02:46   S1B_IW_GRDH_1SDV_20191030T075444_20191030T075520_018702_0233FC_6DCD.SAFE/annotation/calibration/calibration-s1b-iw-grd-vv-20191030t075444-20191030t075520-018702-0233fc-001.xml
    20122  2019-10-30 02:47   S1B_IW_GRDH_1SDV_20191030T075444_20191030T075520_018702_0233FC_6DCD.SAFE/S1B_IW_GRDH_1SDV_20191030T075444_20191030T075520_018702_0233FC_6DCD.SAFE-report-20191030T103353.pdf
    21740  2019-10-30 02:47   S1B_IW_GRDH_1SDV_20191030T075444_20191030T075520_018702_0233FC_6DCD.SAFE/manifest.safe
        0  2019-10-30 02:47   S1B_IW_GRDH_1SDV_20191030T075444_20191030T075520_018702_0233FC_6DCD.SAFE/preview/
   246914  2019-10-30 02:47   S1B_IW_GRDH_1SDV_20191030T075444_20191030T075520_018702_0233FC_6DCD.SAFE/preview/quick-look.png
     1022  2019-10-30 02:47   S1B_IW_GRDH_1SDV_20191030T075444_20191030T075520_018702_0233FC_6DCD.SAFE/preview/map-overlay.kml
     3196  2019-10-30 02:47   S1B_IW_GRDH_1SDV_20191030T075444_20191030T075520_018702_0233FC_6DCD.SAFE/preview/product-preview.html
        0  2019-10-30 02:47   S1B_IW_GRDH_1SDV_20191030T075444_20191030T075520_018702_0233FC_6DCD.SAFE/preview/icons/
    95280  2019-10-30 02:47   S1B_IW_GRDH_1SDV_20191030T075444_20191030T075520_018702_0233FC_6DCD.SAFE/preview/icons/logo.png
---------                     -------
2388397417                     29 files

Very recent Sentinel-1 granules do not include entries for directories:

$ unzip -l S1A_IW_GRDH_1SDV_20230928T113814_20230928T113839_050527_0615EC_864E.zip
Archive:  S1A_IW_GRDH_1SDV_20230928T113814_20230928T113839_050527_0615EC_864E.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
    15677  2023-09-28 13:46   S1A_IW_GRDH_1SDV_20230928T113814_20230928T113839_050527_0615EC_864E.SAFE/S1A_IW_GRDH_1SDV_20230928T113814_20230928T113839_050527_0615EC_864E.SAFE-report-20230928T133505.pdf
   988213  2023-09-28 13:46   S1A_IW_GRDH_1SDV_20230928T113814_20230928T113839_050527_0615EC_864E.SAFE/annotation/calibration/calibration-s1a-iw-grd-vh-20230928t113814-20230928t113839-050527-0615ec-002.xml
   988213  2023-09-28 13:46   S1A_IW_GRDH_1SDV_20230928T113814_20230928T113839_050527_0615EC_864E.SAFE/annotation/calibration/calibration-s1a-iw-grd-vv-20230928t113814-20230928t113839-050527-0615ec-001.xml
   417318  2023-09-28 13:46   S1A_IW_GRDH_1SDV_20230928T113814_20230928T113839_050527_0615EC_864E.SAFE/annotation/calibration/noise-s1a-iw-grd-vh-20230928t113814-20230928t113839-050527-0615ec-002.xml
   417318  2023-09-28 13:46   S1A_IW_GRDH_1SDV_20230928T113814_20230928T113839_050527_0615EC_864E.SAFE/annotation/calibration/noise-s1a-iw-grd-vv-20230928t113814-20230928t113839-050527-0615ec-001.xml
    47860  2023-09-28 13:46   S1A_IW_GRDH_1SDV_20230928T113814_20230928T113839_050527_0615EC_864E.SAFE/annotation/rfi/rfi-s1a-iw-grd-vh-20230928t113814-20230928t113839-050527-0615ec-002.xml
    47860  2023-09-28 13:46   S1A_IW_GRDH_1SDV_20230928T113814_20230928T113839_050527_0615EC_864E.SAFE/annotation/rfi/rfi-s1a-iw-grd-vv-20230928t113814-20230928t113839-050527-0615ec-001.xml
  1584612  2023-09-28 13:46   S1A_IW_GRDH_1SDV_20230928T113814_20230928T113839_050527_0615EC_864E.SAFE/annotation/s1a-iw-grd-vh-20230928t113814-20230928t113839-050527-0615ec-002.xml
  1584622  2023-09-28 13:46   S1A_IW_GRDH_1SDV_20230928T113814_20230928T113839_050527_0615EC_864E.SAFE/annotation/s1a-iw-grd-vv-20230928t113814-20230928t113839-050527-0615ec-001.xml
    24534  2023-09-28 13:46   S1A_IW_GRDH_1SDV_20230928T113814_20230928T113839_050527_0615EC_864E.SAFE/manifest.safe
843826770  2023-09-28 13:46   S1A_IW_GRDH_1SDV_20230928T113814_20230928T113839_050527_0615EC_864E.SAFE/measurement/s1a-iw-grd-vh-20230928t113814-20230928t113839-050527-0615ec-002.tiff
843826770  2023-09-28 13:46   S1A_IW_GRDH_1SDV_20230928T113814_20230928T113839_050527_0615EC_864E.SAFE/measurement/s1a-iw-grd-vv-20230928t113814-20230928t113839-050527-0615ec-001.tiff
    95280  2023-09-28 13:46   S1A_IW_GRDH_1SDV_20230928T113814_20230928T113839_050527_0615EC_864E.SAFE/preview/icons/logo.png
     1022  2023-09-28 13:46   S1A_IW_GRDH_1SDV_20230928T113814_20230928T113839_050527_0615EC_864E.SAFE/preview/map-overlay.kml
     3673  2023-09-28 13:46   S1A_IW_GRDH_1SDV_20230928T113814_20230928T113839_050527_0615EC_864E.SAFE/preview/product-preview.html
   340521  2023-09-28 13:46   S1A_IW_GRDH_1SDV_20230928T113814_20230928T113839_050527_0615EC_864E.SAFE/preview/quick-look.png
   116672  2023-09-28 13:46   S1A_IW_GRDH_1SDV_20230928T113814_20230928T113839_050527_0615EC_864E.SAFE/preview/thumbnail.png
     6427  2023-09-28 13:46   S1A_IW_GRDH_1SDV_20230928T113814_20230928T113839_050527_0615EC_864E.SAFE/support/s1-level-1-calibration.xsd
      471  2023-09-28 13:46   S1A_IW_GRDH_1SDV_20230928T113814_20230928T113839_050527_0615EC_864E.SAFE/support/s1-level-1-measurement.xsd
     7290  2023-09-28 13:46   S1A_IW_GRDH_1SDV_20230928T113814_20230928T113839_050527_0615EC_864E.SAFE/support/s1-level-1-noise.xsd
   149999  2023-09-28 13:46   S1A_IW_GRDH_1SDV_20230928T113814_20230928T113839_050527_0615EC_864E.SAFE/support/s1-level-1-product.xsd
      469  2023-09-28 13:46   S1A_IW_GRDH_1SDV_20230928T113814_20230928T113839_050527_0615EC_864E.SAFE/support/s1-level-1-quicklook.xsd
    16595  2023-09-28 13:46   S1A_IW_GRDH_1SDV_20230928T113814_20230928T113839_050527_0615EC_864E.SAFE/support/s1-level-1-rfi.xsd
      450  2023-09-28 13:46   S1A_IW_GRDH_1SDV_20230928T113814_20230928T113839_050527_0615EC_864E.SAFE/support/s1-map-overlay.xsd
    61552  2023-09-28 13:46   S1A_IW_GRDH_1SDV_20230928T113814_20230928T113839_050527_0615EC_864E.SAFE/support/s1-object-types.xsd
      440  2023-09-28 13:46   S1A_IW_GRDH_1SDV_20230928T113814_20230928T113839_050527_0615EC_864E.SAFE/support/s1-product-preview.xsd
---------                     -------
1694570628                     26 files

This causes util.unzip_granule to fail when attempting to determine the name of the .SAFE directory at https://github.com/ASFHyP3/hyp3-gamma/blob/develop/hyp3_gamma/util.py#L46

Output DEM included with RTC/InSAR products potentially has invalid data in areas where DEM data was not needed for processing

The DEM included via the --include-dem option for InSAR and RTC jobs has the same spatial extent as the other output products. However, the DEM geotiff is built from only DEM tiles that overlap areas of data, and therefore can have invalid data in the corners where DEM data was not needed for underlying processing.

This example is for S1A_IW_SLC__1SDH_20200307T133427_20200307T133455_031570_03A324_C47F . The output DEM (pictured with the backscatter geotiff overlayed) has an area of invalid data in the upper left of the image.

Screenshot from 2021-05-13 10-34-46

This issue does not impact RTC/InSAR processing, but has the potential to impact workflows that use the DEM included with the output products.

The lv_phi xml file in the INSAR productlv_phi gives a smaller data range

The bug

In the INSAR product, it includes the optional _lv_phi.tif.xml file. In this file, the data range of the lv_phi is described as -PI/2 to PI/2.

###Correction
The correct lv_phi data range should be -PI to PI. Following is the explanation.
The look vector is defined as the vector from pixel to satellite. The lv_phi is the angle between the East direction and the horizontal projection of the vector. The angle increases counterclockwise from the East and decreases clockwise. If the horizontal projection of the vector points below the East direction, then it value is negative. The range of the lv_phi is -PI to PI. For Sentinel-1 satellite, if the satellite is ascending, the lv_phi is -PI to -PI/2, where PI is 3.14. If the satellite is descending, lv_phi is in the range of 0 to PI/2.

Heading attribute of InSAR products expressed clockwise instead of counter-clockwise for descending scenes

Originally reported by @yunjunz at insarlab/MintPy#540 (comment)

Sentinel-1 headings are expressed in degrees counter-clockwise from north in the annotation xml files provided with SLC products (around -12 degrees for ascending scenes and -168 for descending scenes).

For descending pairs, the heading attribute for InSAR products is expressed in degrees clockwise from north in the .txt metadata file (around 193 degrees).
descending.txt

GAMMA's SLC_copy_S1_TOPS program appears to be the culprit, an input .par file with a -168 heading generates an output .par file with a positive heading.
https://github.com/ASFHyP3/hyp3-gamma/blob/develop/hyp3_gamma/insar/ifm_sentinel.py#L335
https://github.com/ASFHyP3/hyp3-lib/blob/8594f68c4496c9c0ad1fcdc704c3c8a59f90cc70/hyp3lib/SLC_copy_S1_fullSW.py#L31

This does not appear to impact InSAR products generated from ascending pairs; the heading is still expressed as ~-12 degrees.
ascending.txt

Error in execute.py

hyp3lib.exceptions.ExecuteError: dem_import: ERROR 1: PROJ: proj_as_wkt: SQLite error on SELECT name, ellipsoid_auth_name, ellipsoid_code, prime_meridian_auth_name, prime_meridian_code, area_of_use_auth_name, area_of_use_code, publication_date, deprecated FROM geodetic_datum WHERE auth_name = ? AND code = ?: no such column: area_of_use_auth_name

Screenshot from 2022-03-30 21-29-00
/

rtc_sentinel.py fails when the --nocrosspol option is enabled

hyp3_metadata.create_metadata_file_set assumes both .tif files always exist for dual-pol scenes when generating product xml files (https://github.com/ASFHyP3/hyp3-metadata-templates/blob/ec6ca816d9c01dab9077785811b95d06c72435c4/hyp3_metadata/create.py#L177). This results in an exception when rtc_sentinel was run with the --nocrosspol option.

Traceback (most recent call last):
  File "/home/talogan/miniconda3/envs/hyp3-rtc-gamma/bin/rtc_sentinel.py", line 33, in <module>
    sys.exit(load_entry_point('hyp3-rtc-gamma', 'console_scripts', 'rtc_sentinel.py')())
  File "/home/talogan/hyp3-rtc-gamma/hyp3_rtc_gamma/rtc_sentinel.py", line 694, in main
    include_area_map=args.include_area_map)
  File "/home/talogan/hyp3-rtc-gamma/hyp3_rtc_gamma/rtc_sentinel.py", line 627, in rtc_sentinel_gamma
    processor_version=gamma_version(),
  File "/home/talogan/miniconda3/envs/hyp3-rtc-gamma/lib/python3.7/site-packages/hyp3_metadata/create.py", line 50, in create_metadata_file_set
    files.extend(create_product_xmls(payload))
  File "/home/talogan/miniconda3/envs/hyp3-rtc-gamma/lib/python3.7/site-packages/hyp3_metadata/create.py", line 186, in create_product_xmls
    create_metadata_file(payload, 'product.xml.j2', reference_file)
  File "/home/talogan/miniconda3/envs/hyp3-rtc-gamma/lib/python3.7/site-packages/hyp3_metadata/create.py", line 233, in create_metadata_file
    info = gdal.Info(str(reference_file), format='json')
  File "/home/talogan/miniconda3/envs/hyp3-rtc-gamma/lib/python3.7/site-packages/osgeo/gdal.py", line 275, in Info
    ds = Open(ds)
  File "/home/talogan/miniconda3/envs/hyp3-rtc-gamma/lib/python3.7/site-packages/osgeo/gdal.py", line 4192, in Open
    return _gdal.Open(*args)
RuntimeError: S1A_IW_20200716T161245_DVP_RTC30_G_spuned_6515/S1A_IW_20200716T161245_DVP_RTC30_G_spuned_6515_VH.tif: No such file or directory

Incorrect map coordinates reported for phase unwrapping reference point

The InSAR workflow reports the reference point used for phase unwrapping in the metadata .txt file included with the final InSAR product. For example:

Azimuth line of the reference point in SAR space: 372
Range pixel of the reference point in SAR space: 89
Y coordinate of the reference point in the map projection: 1579231.323
X coordinate of the reference point in the map projection: 620226.5753
Latitude of the reference point (WGS84): 14.28225615
Longitude of the reference point (WGS84): -91.88538414

The azimuth/range coordinates are reported correctly in SAR space. The UTM and Lat/Lon coordinates are reported incorrectly. This is a result of not specifying the height of the chosen pixel when using sarpix_coord to convert from radar coordinates to map coordinates at https://github.com/ASFHyP3/hyp3-gamma/blob/develop/hyp3_gamma/insar/unwrapping_geocoding.py#L41

Requesting formal support for use of the IFSAR DEM

While the CO-OP team is currently using a workaround to allow for use of the IFSAR DEM, a more streamlined solution to allowing this and other DEMs to be easily integrated with the RTC GAMMA processor would be beneficial for both our team and ASF in general

The pull requests to merge the code changes needed to support this have been created in the following

ASFHyP3/hyp3-metadata-templates#37
ASFHyP3/hyp3-lib#223

Also, the directory I am using to build our version of the processor is on the coopSB mount at /mnt/coopSB/washreve_testing/coop-processor-v2-3-4/ and a text copy of the dockerfile is attached to this issue
Dockerfile.txt

10m RTC fails with segmentation fault (core dumped)

These three granules all failed RTC processing at 10m (linked to the log file):

with a segmentation fault (core dumped) during mk_geo_radcal2. Here's the relevant snippet from the logs:

09/08/2021 12:01:44 AM - INFO - Proc: ras_linear ./corrected.ls_map_rdc 32819 1 - 1 1 0 20 0 - ./corrected_0.ls_map_rdc.bmp
09/08/2021 12:01:44 AM - INFO - Proc: WARNING: BMP format image width out of valid range (1->32767): 32819
09/08/2021 12:01:44 AM - INFO - Proc: Segmentation fault (core dumped)
09/08/2021 12:01:44 AM - INFO - Proc: ERROR /usr/local/GAMMA_SOFTWARE-20191203/DIFF/scripts/mk_geo_radcal2: non-zero exit status: ras_linear ./corrected.ls_map_rdc 32819 1 - 1 1 0 20 0 - ./corrected_0.ls_map_rdc.bmp
09/08/2021 12:01:44 AM - INFO - Finished: mk_geo_radcal2 vv.mli vv.mli.par dem.image dem.par dem_seg dem_seg.par . corrected 10.0 0 -q
09/08/2021 12:01:44 AM - ERROR - Nonzero return value!
Traceback (most recent call last):
  File "/usr/local/bin/hyp3_gamma", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/dist-packages/hyp3_gamma/__main__.py", line 32, in main
    load_entry_point('hyp3_gamma', 'console_scripts', args.process)()
  File "/usr/local/lib/python3.6/dist-packages/hyp3_gamma/__main__.py", line 73, in rtc
    dem_name=args.dem_name,
  File "/usr/local/lib/python3.6/dist-packages/hyp3_gamma/rtc/rtc_sentinel.py", line 350, in rtc_sentinel_gamma
    run(f'mk_geo_radcal2 {mli_image} {mli_par} {dem_image} {dem_par} dem_seg dem_seg.par . corrected '
  File "/usr/local/lib/python3.6/dist-packages/hyp3_gamma/rtc/rtc_sentinel.py", line 147, in run
    execute(cmd, uselogging=True)
  File "/usr/local/lib/python3.6/dist-packages/hyp3lib/execute.py", line 77, in execute
    raise ExecuteError(tool + ': ' + line)
hyp3lib.exceptions.ExecuteError: mk_geo_radcal2: ERROR /usr/local/GAMMA_SOFTWARE-20191203/DIFF/scripts/mk_geo_radcal2: non-zero exit status: ras_linear ./corrected.ls_map_rdc 32819 1 - 1 1 0 20 0 - ./corrected_0.ls_map_rdc.bmp

It's possible that this is just hitting the instance memory limit, and upping the machine size may solve it, but I have not investigated this yet.


Note: All jobs were requested with these options:

job_definition = {
    'job_type': 'RTC_GAMMA',
    'job_parameters': {
        'resolution': 10,
        'speckle_filter': True,
        'radiometry': 'gamma0',
        'scale': 'power',
        'dem_matching': True,
    }
}

rtc_sentinel.py fails with `Unable to open datasource` when provided absolute path to SAFE directory

The leading / is incorrectly stripped from the SAFE directory parameter, potentially causing a pathing error when attempting to open files in the SAFE directory:

$ rtc_sentinel.py /Downloads/S1B_IW_SLC__1SDV_20171109T160515_20171109T160542_008207_00E816_E76D.SAFE
12/07/2021 08:55:29 PM - INFO - ===================================================================
12/07/2021 08:55:29 PM - INFO -                 Sentinel RTC Program - Starting
12/07/2021 08:55:29 PM - INFO - ===================================================================
12/07/2021 08:55:29 PM - INFO - Downloading https://scihub.copernicus.eu/gnss/odata/v1/Products('e3d5fada-3a44-45d0-b2d3-98fe25b9b5f9')/$value
12/07/2021 08:55:36 PM - INFO - Parameters for this run:
12/07/2021 08:55:36 PM - INFO -     SAFE directory            : Downloads/S1B_IW_SLC__1SDV_20171109T160515_20171109T160542_008207_00E816_E76D.SAFE
12/07/2021 08:55:36 PM - INFO -     Output resolution         : 30.0
<...clip...>
12/07/2021 08:55:36 PM - INFO - Preparing DEM
FAILURE:
Unable to open datasource `Downloads/S1B_IW_SLC__1SDV_20171109T160515_20171109T160542_008207_00E816_E76D.SAFE/preview/map-overlay.kml' with the following drivers.

InSAR fails with "no points available for determining average intensity" error

Impacted ~1 in 1,000 pairs out of ~20,000 insar jobs we ran for SARVIEWS.

Example pairs:

S1A_IW_SLC__1SDV_20200126T112153_20200126T112221_030971_038E72_7909,S1B_IW_SLC__1SDV_20200201T112118_20200201T112141_020075_025FEA_7523
S1A_IW_SLC__1SDV_20191115T112156_20191115T112224_029921_036A01_6C79,S1B_IW_SLC__1SDV_20191121T112120_20191121T112144_019025_023E71_1AD0

This is related to code in https://github.com/ASFHyP3/hyp3-gamma/blob/master/hyp3_gamma/insar/ifm_sentinel.py#L48 , related to the burst times for each scene not being within 0.2 seconds of each other.

03/05/2021 01:39:59 AM - INFO - total_bursts1, time1 10 [6201.7603385546, 6204.5188951092, 6207.2774516638, 6210.0339526621, 6212.792509216701, 6215.549010215001, 6218.305511213301, 6221.0661233242, 6223.8246798788, 6226.5811808771]
03/05/2021 01:39:59 AM - INFO - total_bursts2, time2 7 [285.4837895546, 288.2444016655, 291.0029582201, 293.7594592184, 296.5200713293, 299.2786

Sort out DEM resampling for InSAR

Possibly out of scope for this PR, but I notice the DEM geotiff comes out with a pixel size of 80m/40m for 20x4/10x2 looks, even though we originally downsample the input DEM to 160m/80m. I looks like that's because we set oversampling to 2 when calling GC_map_mod and gc_map, which resamples the DEM back to 80m/40m:
https://github.com/ASFHyP3/hyp3-lib/blob/develop/hyp3lib/SLC_copy_S1_fullSW.py#L63

That seems unproductive? Should we resample the input DEM to 80m/40m at the start and avoid possibly throwing away resolution?

Originally posted by @asjohnston-asf in #247 (comment)

ifm_sentinal.py always picks the older file as reference, and newer file as secondary.

Describe the bug
A clear and concise description of what the bug is.
I also found that even if we choose the file with later date time as the reference in the pair of files, the ifm_sentinal.py always picks the file with earlier date time as reference one, I suppose ifm-sentinal.py should produce the ifm with reference file is later than the secondary file.

For example,
Case1 (forward):
ref=S1A_IW_SLC__1SSV_20160112T205113_20160112T205140_009466_00DBA8_C430
sec=S1A_IW_SLC__1SSV_20160628T205108_20160628T205136_011916_01259C_EE11
output:S1AA_20160112T205113_20160628T205108_VVP168_INT80_G_ueF_5027.zip
Slant range near: 800208.3153
Slant range center: 879343.5365
Slant range far: 958478.7576
Case2 (backward):
Ref=S1A_IW_SLC__1SSV_20160628T205108_20160628T205136_011916_01259C_EE11
sec=S1A_IW_SLC__1SSV_20160112T205113_20160112T205140_009466_00DBA8_C430
Output: S1AA_20160112T205113_20160628T205108_VVP168_INT80_G_ueF_7E78.zip
Slant range near: 800208.3153
Slant range center: 879343.5365
Slant range far: 958478.7576

Actually, the metadata.txt in two cases are the same. Furthermore, I found in both cases, no matter if you reverse the ref and sec, ifm_snetinal.py always produces the same result which always uses a file with earlier time as a ref file.

To Reproduce
Steps to reproduce the behavior:

  1. Go to '...'
  2. Click on '....'
  3. Scroll down to '....'
  4. See error

Expected behavior
A clear and concise description of what you expected to happen.

Additional context
Add any other context about the problem here.

INSAR_GAMMA jobs near the south pole fail due to missing water mask tiles

INSAR_GAMMA jobs in the far south fail when attempting to build a water mask due to missing S80 tiles.

For an example, see https://hyp3-api.asf.alaska.edu/jobs/dfd67697-0f44-46b7-b1d4-16d40ee77409 :

ERROR 11: HTTP response code: 404
Warning 1: Can't open /vsicurl/https://asf-dem-west.s3.amazonaws.com/WATER_MASK/TILES/s80w165.tif. Skipping it
ERROR 11: HTTP response code: 404
Warning 1: Can't open /vsicurl/https://asf-dem-west.s3.amazonaws.com/WATER_MASK/TILES/s80w155.tif. Skipping it
0...10...20...30...40...50...60...70...80...90...100 - done.
Traceback (most recent call last):
  File "/usr/local/bin/hyp3_gamma", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.10/dist-packages/hyp3_gamma/__main__.py", line 38, in main
    process_entry_point.load()()
  File "/usr/local/lib/python3.10/dist-packages/hyp3_gamma/__main__.py", line 185, in insar
    product_name = insar_sentinel_gamma(
  File "/usr/local/lib/python3.10/dist-packages/hyp3_gamma/insar/ifm_sentinel.py", line 427, in insar_sentinel_gamma
    coords, ref_point_info = unwrapping_geocoding(reference, secondary, step="man", rlooks=rlooks, alooks=alooks,
  File "/usr/local/lib/python3.10/dist-packages/hyp3_gamma/insar/unwrapping_geocoding.py", line 253, in unwrapping_geocoding
    get_water_mask(f"{ifgname}.adf.cc", width, lt, demw, demn, dempar)
  File "/usr/local/lib/python3.10/dist-packages/hyp3_gamma/insar/unwrapping_geocoding.py", line 150, in get_water_mask
    create_water_mask(f'{temp_dir}/tmpgtiff_mask_geo.tif', 'water_mask.tif')
  File "/usr/local/lib/python3.10/dist-packages/hyp3_gamma/water_mask.py", line 136, in create_water_mask
    subprocess.run(build_vrt_command, check=True)
  File "/usr/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['gdalbuildvrt', 'merged.vrt', '/vsicurl/https://asf-dem-west.s3.amazonaws.com/WATER_MASK/TILES/s80w165.tif', '/vsicurl/https://asf-dem-west.s3.amazonaws.com/WATER_MASK/TILES/s80w155.tif']' returned non-zero exit status 1.

I've confirmed https://asf-dem-west.s3.amazonaws.com/WATER_MASK/TILES/s80w165.tif and https://asf-dem-west.s3.amazonaws.com/WATER_MASK/TILES/s80w155.tif both return "The specified key does not exist." as indicated in the error message.

job.download_files should provide feedback when no files are available to download

Is your feature request related to a problem? Please describe.
Currently in this situation download_files() prints a progress bar without data and exists successfully. It's not immediately intuitive from the output that there were no files available to download.

> file_list = jobs.download_files()
0/0 [00:00<?, ?it/s]

Describe the solution you'd like
Logging a warning message could provide more specific feedback in this situation.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

`shapely.errors.TopologicalError` during InSAR water masking

The bug

The insar workflow and INSAR_GAMMA jobs in HyP3 may fail with this stack trace when generating a water mask prior to phase unwrapping:

  File "/usr/local/lib/python3.10/dist-packages/hyp3_gamma/__main__.py", line 38, in main
    process_entry_point.load()()
  File "/usr/local/lib/python3.10/dist-packages/hyp3_gamma/__main__.py", line 185, in insar
    product_name = insar_sentinel_gamma(
  File "/usr/local/lib/python3.10/dist-packages/hyp3_gamma/insar/ifm_sentinel.py", line 427, in insar_sentinel_gamma
    coords, ref_point_info = unwrapping_geocoding(reference, secondary, step="man", rlooks=rlooks, alooks=alooks,
  File "/usr/local/lib/python3.10/dist-packages/hyp3_gamma/insar/unwrapping_geocoding.py", line 253, in unwrapping_geocoding
    get_water_mask(f"{ifgname}.adf.cc", width, lt, demw, demn, dempar)
  File "/usr/local/lib/python3.10/dist-packages/hyp3_gamma/insar/unwrapping_geocoding.py", line 150, in get_water_mask
    create_water_mask(f'{temp_dir}/tmpgtiff_mask_geo.tif', 'water_mask.tif')
  File "/usr/local/lib/python3.10/dist-packages/hyp3_gamma/water_mask.py", line 71, in create_water_mask
    mask = gpd.clip(mask, envelope)
  File "/usr/local/lib/python3.10/dist-packages/geopandas/tools/clip.py", line 192, in clip
    clipped = _clip_gdf_with_mask(gdf, combined_mask)
  File "/usr/local/lib/python3.10/dist-packages/geopandas/tools/clip.py", line 69, in _clip_gdf_with_mask
    ] = gdf_sub.geometry.values[non_point_mask].intersection(mask)
  File "/usr/local/lib/python3.10/dist-packages/geopandas/array.py", line 671, in intersection
    self._binary_method("intersection", self, other), crs=self.crs
  File "/usr/local/lib/python3.10/dist-packages/geopandas/array.py", line 611, in _binary_method
    return getattr(vectorized, op)(left._data, right, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/geopandas/_vectorized.py", line 942, in intersection
    return _binary_geo("intersection", data, other)
  File "/usr/local/lib/python3.10/dist-packages/geopandas/_vectorized.py", line 306, in _binary_geo
    data[:] = [
  File "/usr/local/lib/python3.10/dist-packages/geopandas/_vectorized.py", line 307, in <listcomp>
    getattr(s, op)(right) if s is not None and right is not None else None
  File "/usr/lib/python3/dist-packages/shapely/geometry/base.py", line 689, in intersection
    return geom_factory(self.impl['intersection'](self, other))
  File "/usr/lib/python3/dist-packages/shapely/topology.py", line 72, in __call__
    self._check_topology(err, this, other)
  File "/usr/lib/python3/dist-packages/shapely/topology.py", line 37, in _check_topology
    raise TopologicalError(
shapely.errors.TopologicalError: The operation 'GEOSIntersection_r' could not be performed. Likely cause is invalidity of the geometry <shapely.geometry.multipolygon.MultiPolygon object at 0x7f8d9ef30100>

To date, we've seen this error for these InSAR pairs run with looks=10x2 and apply_water_mask=true:

S1A_IW_SLC__1SDV_20170728T051208_20170728T051234_017667_01D93F_8329,S1A_IW_SLC__1SDV_20180711T051213_20180711T051240_022742_0276FD_617A
S1A_IW_SLC__1SDV_20180711T051213_20180711T051240_022742_0276FD_617A,S1B_IW_SLC__1SDV_20190712T051136_20190712T051204_017096_0202A5_C8F2
S1A_IW_SLC__1SDV_20210707T051231_20210707T051258_038667_04901E_08FA,S1A_IW_SLC__1SDV_20220702T051237_20220702T051304_043917_053E27_9215
S1A_IW_SLC__1SDV_20220702T051237_20220702T051304_043917_053E27_9215,S1A_IW_SLC__1SDV_20230615T051241_20230615T051308_048992_05E43B_4288
S1B_IW_SLC__1SDV_20190712T051136_20190712T051204_017096_0202A5_C8F2,S1B_IW_SLC__1SDV_20200730T051144_20200730T051212_022696_02B139_7027
S1B_IW_SLC__1SDV_20200718T051143_20200718T051211_022521_02ABE5_9C44,S1A_IW_SLC__1SDV_20210719T051232_20210719T051259_038842_04955D_7838
S1B_IW_SLC__1SDV_20200718T051143_20200718T051211_022521_02ABE5_9C44,S1A_IW_SLC__1SDV_20210731T051233_20210731T051259_039017_049A8F_A8F7
S1B_IW_SLC__1SDV_20200718T051143_20200718T051211_022521_02ABE5_9C44,S1B_IW_SLC__1SDV_20210713T051149_20210713T051217_027771_03506A_A8C6
S1B_IW_SLC__1SDV_20200718T051143_20200718T051211_022521_02ABE5_9C44,S1B_IW_SLC__1SDV_20210725T051149_20210725T051217_027946_03559A_B4F3
S1B_IW_SLC__1SDV_20200730T051144_20200730T051212_022696_02B139_7027,S1A_IW_SLC__1SDV_20210707T051231_20210707T051258_038667_04901E_08FA

This issue was first observed on 2023-11-16 and was likely introduced with the water masking changes in hyp3-gamma v6.4.1.

Per @cirrusasf :
Version does not work for some pairs, for example, The error that hyp3-gamma throw outs is related to water_mask code. THe error: File "/usr/local/lib/python3.10/dist-packages/hyp3_gamma/water_mask.py", line 71, in create_water_mask
mask = gpd.clip(mask, envelope)
It turns out the that geopandas.clip() will fail if both mask and envelope are in projection coordinates and the mask is very large. If we change them into wgs84 coordinates, it works.

INSAR_GAMMA job with 10x2 looks fails with OutOfMemoryError after GAMMA 20210701 upgrade

Running an INSAR_GAMMA job for this SLC pair with 10x2 looks results in OutOfMemoryError: Container killed due to memory usage during the execution of the mcf program to do phase unwrapping

S1B_IW_SLC__1SSV_20170124T232750_20170124T232827_003996_006E65_58A0
S1B_IW_SLC__1SSV_20170217T232750_20170217T232826_004346_0078D8_4CB7

10x2 InSAR jobs for this pair succeeded prior to the v4.8.0 release (as recently as Sep 13 20:14), and fail following the upgrade (as early as Sep 13 23:27).

To date, I've not identified any additional pairs that cause this memory error, out of ~3,800 10x2 InSAR jobs run since the release.

the impact of window size on the choice of the reference point in the InSAR calculation

The bug

To Reproduce

Additional context

In the InSAR calculation, choice of reference point is critical. In our algorithm of determining the reference point, the window size plays a import role. In general,we want to pick the reference point from the big area with high coherence value. so we better choose window with enough size. Here I share the reference point are calculated with 5x5 window and 21x21 window

Incorrect Conversion from Shapefile to Water Mask Raster

The bug

When 'Apply Water Mask' is selected, in some regions, the water mask incorrectly masks out most of the land and leads to incorrect unwrapping.
Here is an example, from Switzerland, where you can see the image should be mostly land with a couple lakes. However, the water mask is all nodata.
Screenshot 2023-10-30 at 1 09 01 PM
Screenshot 2023-10-30 at 1 09 22 PM

To Reproduce

Here are the parameters for a job that fails:

"job_parameters": {
        "apply_water_mask": true,
        "granules": [
          "S1A_IW_SLC__1SDV_20230602T172346_20230602T172413_048810_05DEAA_9971",
          "S1A_IW_SLC__1SDV_20230614T172347_20230614T172414_048985_05E406_AD4A"
        ],
        "include_dem": false,
        "include_displacement_maps": true,
        "include_inc_map": false,
        "include_look_vectors": false,
        "include_wrapped_phase": true,
        "looks": "10x2"
      }

RTC - only one mk_geo_radcal2 mode 3 log captured

The log output of mk_geo_radcal2 modes 0-3 is appended to the end of the log file distributed with the output product. Mode 0 runs exactly once, modes 1-2 are run once if dem matching is selected, and mode 3 runs once for each polarization. For dual pol granules, only one of the mode 3 logs is appended to the final log file.

Logs are appended at https://github.com/ASFHyP3/hyp3-gamma/blob/develop/hyp3_gamma/rtc/rtc_sentinel.py#L359 , at which point the cross-pol run has already clobbered the mk_geo_radcal_3.log file from the co-pol run.

NoData values for output products potentially mask valid data values

GeoTIFFs included in RTC and InSAR output products generally have NoData=0. However, zero lies inside the data range for a number of these GeoTIFFs, potentially masking valid pixels. An alternate NoData value might be more appropriate for at least these files:

_wrapped_phase.tif (InSAR)
_unw_phase.tif (InSAR)
_los_disp.tif (InSAR)
_vert_disp.tif (InSAR)
_corr.tif (InSAR)
_dem.tif (RTC, InSAR)
_lv_phi.tif (InSAR)
_lv_theta.tif (InSAR)

Invalid interferograms over Armenia and Azerbaijan

INSAR_GAMMA jobs rely on the Copernicus Global 30 meter public DEM (GLO-30 Public). This DEM is not available over Armenia and Azerbaijan. To lean more, visit Digital Elevation Models in our HyP3 documentation, or https://spacedata.copernicus.eu/collections/copernicus-digital-elevation-model

cop-missing-100

Because no DEM is available in this region, INSAR_GAMMA jobs over Armenia and Azerbaijan produce interferograms where topographic phase has not been removed over this area:

khowy

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.