Hello professor, I have some questions about the parameter file in "--parameter-file". For example, does the "0120m" in "autorift_solidearth_0120m.shp" refer to the resolution of 120 meters?

Secondly, is it suitable for application in coal mining subsidence areas?

Looking forward to your reply! Thank you.

ITS_LIVE website has updated "how to cite" guidelines:

The recommended citation for the Regional Glacier and Ice Sheet Surface Velocities is:
"Velocity data generated using auto-RIFT (Gardner et al., 2018) and provided by the NASA MEaSUREs ITS_LIVE project (Gardner et al., 2019)."

Gardner, A. S., M. A. Fahnestock, and T. A. Scambos, 2019 [update to time of data download]: ITS_LIVE Regional Glacier and Ice Sheet Surface Velocities. Data archived at National Snow and Ice Data Center; doi:10.5067/6II6VW8LLWJ7.

Gardner, A. S., G. Moholdt, T. Scambos, M. Fahnstock, S. Ligtenberg, M. van den Broeke, and J. Nilsson, 2018: Increased West Antarctic and unchanged East Antarctic ice discharge over the last 7 years, Cryosphere, 12(2): 521–547, doi:10.5194/tc-12-521-2018.

Production products should have the Gardner et al. (2019) reference (NSIDC DOI) in the references attribute.

All products should have the "NASA MEaSUREs ITS_LIVE project" soumewhere in the source attribute.

Currently, we hardcode the set of needed files for processing:

https://github.com/ASFHyP3/hyp3-autorift/blob/develop/hyp3_autorift/io.py#L47-L68

These however, at detailed in the new parameter shapefile and could be pulled from there instead to be more flexible to future change (see #47 )

Similar to ASFHyP3/hyp3#1022

hyp3_autorfit.process.get_datetime
https://github.com/ASFHyP3/hyp3-autorift/blob/develop/hyp3_autorift/process.py#L145-L146

will assume any Sentinel-2 scene with a name longer than 24 characters is a Sentinel-2 ESA name, not a COG name. But, there are valid 25 character COG names.

This may not be the only place this assumption exists (but I think it is).

AutoRIFT is a pixel-tracking algorithm and so requires at least some difference between the reference and secondary scene.

Unfortunately, autoRIFT will process a pair that has the same scene for reference and secondary all the way through and then finally fail due to NaN issues in the netCDF packaging step at the very end, like in this job:
https://hyp3-api.asf.alaska.edu/jobs/879efe81-adf2-48cb-89db-f4bd88c577e4

We should check up front that the reference and secondary scene provided aren't exactly the same and fail immediately.

With the downgrade to S-2 L1C products, URLs to the products now look like:

s3://sentinel-s2-l1c/tiles/22/W/EB/2020/6/12/0/B08.jp2

and our downloaded files look like:

S2B_MSIL1C_20200612T150759_N0209_R025_T22WEB_20200612T184700_B08.jp2

autoRIFT is expecting URLs to look like:

https://sentinel-cogs.s3.us-west-2.amazonaws.com/sentinel-s2-l2a-cogs/22/W/EB/2020/9/S2B_22WEB_20200903_0_L2A/B08.tif

and downloaded files to look like ???

Blocked: this needs to be resolved upstream

bug
I'm pointing autoRIFT to the new shapefile that in-turn points to the 120m gridded parameter files but they are producing 240m grids... tracing the issue a bit it looks like it must happen on the gdal call to retrieve the parameters.
For example the parameter file: http://its-live-data.jpl.nasa.gov.s3.amazonaws.com/autorift_parameters/v001/SPS_0120m_yMinChipSize.tif
has a 120 m grid spacing but the file "SPS_0120m_yMinChipSize.tif" created by the autoRIFT workflow has 240m grid spacing

this line: https://github.com/ASFHyP3/hyp3-autorift/blob/develop/hyp3_autorift/io.py#L86

hyp3_autorift/io.py:86
out_path, tif, outputBounds=output_bounds, xRes=240, yRes=240, targetAlignedPixels=True, multithread=True,

needs to get xRes and yRes from the input file

** ALSO MAKE SURE THAT PIXELS CENTERS ARE NOT BEING MOVED FROM ORIGINAL PARAMETER FILE **

$ python -m hyp3_autorift ++process hyp3_autorift S1A_IW_SLC__1SSH_20170118T091036_20170118T091104_014884_01846D_01C5 S1B_IW_SLC__1SSH_20170112T090955_20170112T091023_003813_0068DC_C750
Traceback (most recent call last):
  File "/home/asjohnston/mambaforge/envs/hyp3-autorift/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/asjohnston/mambaforge/envs/hyp3-autorift/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/asjohnston/src/hyp3-autorift/src/hyp3_autorift/__main__.py", line 47, in <module>
    main()
  File "/home/asjohnston/src/hyp3-autorift/src/hyp3_autorift/__main__.py", line 38, in main
    (process_entry_point,) = {process for process in eps if process.name == args.process}
ValueError: too many values to unpack (expected 1)

Reviewing importlib.metadata.entry_points, there are multiple entry points registered with the same hyp3_autorift name:

>>> from importlib.metadata import entry_points
>>> for ep in entry_points()['console_scripts']:
...     if ep.name == 'hyp3_autorift':
...         print(ep)
... 
EntryPoint(name='hyp3_autorift', value='hyp3_autorift.__main__:main', group='console_scripts')
EntryPoint(name='hyp3_autorift', value='hyp3_autorift.process:main', group='console_scripts')
EntryPoint(name='hyp3_autorift', value='hyp3_autorift.process:main', group='console_scripts')

This was possibly introduced when migrating from setup.py to pyproject.toml

Currently, testautoRIFT_ISCE.py sub-processes out a call to topsinsar_filename.py to just load back in a dictionary with product metadata:

This is bad and entirely not needed -- we could just patch the vendored script to call the function directly, or push changes upstream to actually package everything it needs appropriately.

If user provides username, or password there should be usefull feedback since .netrc won't be written

Since we only look at the reference scene to determine the platform, we may have some issues with the secondary scene:

• L7 reference scene w/ a L8 secondary will attempt to apply the Wallis filter and bonk:
  https://github.com/ASFHyP3/hyp3-autorift/blob/develop/hyp3_autorift/process.py#L269-L282

• L8 reference scene w/ a L7 secondary will not have the Wallis filter applied to it:
  https://github.com/ASFHyP3/hyp3-autorift/blob/develop/hyp3_autorift/process.py#L368-L370

Fortunately, in our current campaign pair list, 4/5 pairs are only crossed with 4/5 pairs, and 9 pairs are only crossed with 8, so we should only see issues around 7+8 pairs.

>>> import geopandas as gpd
>>> df = gpd.read_parquet('l45789.parquet')
>>> df.groupby(['ref_mission', 'sec_mission']).reference.count()
ref_mission  sec_mission
L4           L4                1058
             L5                3203
L5           L4                2988
             L5              767024
L7           L7             2003704
             L8              474339
L8           L7              416539
             L8             4103107
             L9              367178
L9           L8              112403
             L9              109936

Fix: AutoRIFT pre-processing defaults to the strictest filtering, so we should allow L8 scenes to be wallis filled if either pair is L7. And we should check both ref, sec platform when deciding to run the pre-processing filters or not in process.py

This is likely only needed for the Landsat mission.

Projection info should be in the metadata.

• confirm parameter files are based on the reference scene
• #187
• #188

The current autoRIFT parameter file:

aws s3 ls s3://its-live-data.jpl.nasa.gov/autorift_parameters/v001/autorift_parameters.shp

Assumes the area of interested in glaciers/land-ice and links to files with optimized search parameters for the research application.

Ideally, hyp3_autorift would take a parameter where you could specify the research application and select the right shape file (or right features out of the shape file) based on that.

Note: this will require action by JPL to generate appropriate input files and specify how to query based on research application

Cropping some of the Golden Test netCDF files takes over 10 minutes to perform, which seems ridiculously long.

Profiling the script on

LC08_L1TP_009011_20200703_20200913_02_T1_X_LC08_L1TP_009011_20200820_20200905_02_T1_G0120V02_P078_IL_ASF_OD.nc

shows 99.8% of the wallclock time is spent in the where method here:

Hello, professor. When I used Hyp3-autorift to track the pixel offset of the Turkish earthquake, some strange errors occurred. As shown in the figure, I used Sentinel-2 data and the command was: hyp3_autorift
"S2A_MSIL1C_20230120T082241_N0509_R121_T37SCB_20230120T091424"
"S2A_MSIL1C_20230209T082111_N0509_R121_T37SCB_20230209T091429"
The error reported is as follows:

ITS_LIVE requested that the parameter file version be recorded in the netCDF file.

However, parameter file can now be passed in on the CLI, and there are likely going to be different files for different research applications.

So, we should record the full path to the file used.

Unfortunately, we subprocess out to vendored script, so the vendored scripts should be updated upstream to (preferably) allow import and direct calling of script workflow or (alternatively) take parameter file as an input parameter.

The Malaspina datacube test is rife with these failures. You can get a list of pairs that failed due to miss-matched projection from the GeoDataFrame record of the test, located here:

aws s3 ls --profile hyp3 s3://enterprise-campaigns/its-live/tests/malasina.parquet

To get the list of pairs you can run:
import geopandas as gpd

df = gpd.read_parquet('malaspina.parquet')

projection_failures = df.loc[
    (df.status == 'FAILED')
    df.mission.str.startswith('L') 
    & df.look.str.startswith('Exception: The current version of geo_autoRIFT assumes the two images are in the same projection')
    ]

You can create an environment for these commands with the attached env yaml.

This is likely due to how netcdf-python handles string writing

• netCDF4 conventionsl
• Unidata/netcdf4-python #529

I think the issues is Matlab can't render NC_STRING type attributes, and needs NC_CHAR, so we just need to make sure it's encodable to ASCII

S2 names are currently given using the ESA identifier (e.g. S2B_MSIL2A_20200903T151809_N0214_R068_T22WEB_20200903T194353), we should change to the COG identifier (e.g. S2B_22WEB_20200903_0_L2A) because that's what Mark's pair list uses.

Also, consider validation of granule names and pairs here:

• both are valid names
• both are from the same mission

I've attached a list of ~1500 HTTP links to Sentinel-2 jobs which failed due to Google throwing HTTP 429s when trying to read the data.

From the logs it looks like GDAL is trying to open a pile of side-car files that may contain metadata, which we have no interest in and that may be causing the rate-limit issues. We are setting export GDAL_DISABLE_READDIR_ON_OPEN='EMPTY_DIR' to prevent GDAL From trying to list the entire cloud object store, but it's possible there's another GDAL environment variable we can set to prevent side-car files from being loaded.

To reproduce locally, you can use either the autorift environment (linux only) or docker container, which looks like:

hyp3_autorft <S2_REF> <S2_SEC>

Essentially, what's happening in the code is just a gdal.Open on the data, so we likely can reproduce by simply writing a python script that opens and reads two scenes. Psudo code:

ds = gdal.Open('/vsicurl/http...')
ds.readAsArray()

making sure the GDAL environment variables are set accordingly.

important parts of the hyp3-autorift:

• the processing workflow starts here: https://github.com/ASFHyP3/hyp3-autorift/blob/develop/hyp3_autorift/process.py#L317
  (it's a mess, sorry)
• We first make HTTP requests to google when we loaded the SAFE manifest file here:
  • https://github.com/ASFHyP3/hyp3-autorift/blob/develop/hyp3_autorift/process.py#L363-L364
    which calls
  • https://github.com/ASFHyP3/hyp3-autorift/blob/develop/hyp3_autorift/process.py#L94-L99
• from the metadata, we determine the path to the actual data file (*.jp2) and then pass it to Geogrid
  https://github.com/ASFHyP3/hyp3-autorift/blob/develop/hyp3_autorift/process.py#L448-L454
  which just ends up opening the files with GDAL
  https://github.com/ASFHyP3/hyp3-autorift/blob/develop/hyp3_autorift/vend/testGeogridOptical.py#L95

Hello, professors! I encountered some problems when using Hyp3-autorift to process Sentinel-2 data. First, I can quickly get the OT results of each small block through Hyp3-autorift, but when I splice all the OT results with gdal, the situation shown in the figure below appears. I don’t understand why there are so many gaps around.
I tried to crop the OT results of each small block around, and then when I performed a three-dimensional solution, the result of the second picture appeared.
I am very confused, please help me, professors. Thank you.