nasa / harmony Goto Github PK

Environment
Harmony-in-a-box, Harmony UAT, Harmony OPS

Steps to Reproduce
In UAT

1. http://{{harmony_host}}/C1238621141-POCLOUD/ogc-api-coverages/1.0.0/collections/analysed_sst/coverage/rangeset?width=36000&height=17999&format=image/png&granuleId=G1240564967-POCLOUD&forceAsync=true&turbo=true
2. Wait for job to complete
3. http://{{harmony_host}}/jobs/{{job_id}}

Expected Result

Job result should contain at least two links: one for the png file and one for the wld file.

Actual Result

Job result only contains link to png file.

Details

The asfgdal service has been updated to stage a wld file and return it as an additional asset in the item entry for a granule:
https://github.com/asfadmin/asf-harmony-gdal/blob/ea28edd81ad64f0e1a3edabed693120ab9a79113/gdal_subsetter/transform.py#L216-L228

There is a unit test that verifies the catalog returned from the service contains two items:
https://github.com/asfadmin/asf-harmony-gdal/blob/ea28edd81ad64f0e1a3edabed693120ab9a79113/tests/test_transform_no_download.py#L68-L70

When running harmony locally, I was able to extract the item.json produced as output from the harmony adapter itself, which also shows the two assets:

{
    "type": "Feature",
    "stac_version": "1.0.0-beta.2",
    "id": "d9874dac-59e5-424e-b346-9a6a62d2627c",
    "properties": {
        "start_datetime": "2021-04-29T21:00:00.000Z",
        "end_datetime": "2021-04-30T21:00:00.000Z",
        "datetime": null
    },
    "geometry": {"type":"Polygon","coordinates":[[[-179.9950055,-89.9949985],[-179.9950055,89.9949979],[-179.9949983,89.9949979],[-179.9949983,-89.9949985],[-179.9950055,-89.9949985]]]    },
    "links": [
        {
            "rel": "root",
            "href": "../catalog.json",
            "type": "application/json"
        },
        {
            "rel": "parent",
            "href": "../catalog.json",
            "type": "application/json"
        }
    ],
    "assets": {
        "data": {
            "href": "s3://local-staging-bucket/public/asfdataservices/gdal-subsetter/e07cb268-1cef-4bd1-90db-25bcab327506/20210430090000-JPL-L4_GHRSST-SSTfnd-MUR-GLOB-v02.0-fv04.1_analysed_sst_regridded.png",
            "type": "image/png",
            "title": "20210430090000-JPL-L4_GHRSST-SSTfnd-MUR-GLOB-v02.0-fv04.1_analysed_sst_regridded.png",
            "roles": ["data"]
        },
        "metadata": {
            "href": "s3://local-staging-bucket/public/asfdataservices/gdal-subsetter/e07cb268-1cef-4bd1-90db-25bcab327506/20210430090000-JPL-L4_GHRSST-SSTfnd-MUR-GLOB-v02.0-fv04.1_analysed_sst_regridded.wld",
            "type": "text/plain",
            "title": "20210430090000-JPL-L4_GHRSST-SSTfnd-MUR-GLOB-v02.0-fv04.1_analysed_sst_regridded.wld",
            "roles": ["metadata"]
        }
    },
    "bbox": [-179.9950055,-89.9949985,-179.9949983,89.9949979],
    "stac_extensions": []
}

DESCRIPTION:

A required function parameter isn't provided while calling the function. This is an error.

BAD PRACTICE:

def add_student(student, section):
students[section].append(student)

add_student("Aaron") # Missing parameter section

RECOMMENDED:

def add_student(student, section):
students[section].append(student)

add_student("Aaron", "10A")

No value for argument 'start_time' in method call can be found here: harmony/blob/main/workload/locustfile-prod.py#L113-L113

No value for argument 'name' in method call can be found here: harmony/blob/main/workload/locustfile-prod.py#L113-L113
/harmony/blob/main/workload/locustfile-prod.py#L91-L91

And so on.

DESCRIPTION

Usage of assert statement in application logic is discouraged. assert is removed with compiling to optimized byte code. Consider raising an exception instead. Ideally, assert statement should be used only in tests.

Python has an option to compile the optimized bytecode and create the respective .pyo files by using the options -O and -OO. When used, these basic optimizations are done:

All the assert statements are removed
All docstrings are removed (when -OO is selected)
Value of the debug built-in variable is set to False
It is recommended not to use assert in non-test files. A better way for internal self-checks is to check explicitly and raise respective error using an if statement.

Consider this code snippet:

def read_secret(self):
assert self.is_admin, "You are unauthorized to read this"
return self._secret
If python is run with the -O flag, the check for self.is_admin is completely ignored, which can cause secrets to be leaked.

This is how you can ensure the code always works:

def read_secret(self):
if not self.is_admin:
raise AssertionError("You are unauthorized to read this")

return self._secret

Here's a more detailed example. Consider the following script foo.py:

import sys

def run():
assert len(sys.argv) == 5 # Insecure, statement will be removed when compiled to optimized byte code
print("Argument variables are: ", sys.argv)

run()

When optimization is disabled:

$ python foo.py 1 2 3 4 5
Traceback (most recent call last):
File "foo.py", line 7, in
run()
File "foo.py", line 4, in run
assert len(sys.argv) == 5 # Insecure, statement will be removed when compiled to optimized byte code
AssertionError
When optimization is enabled:

$ python -O foo.pyo 1 2 3 4 5 6
Argument variables are: ['foo.pyo', '1', '2', '3', '4', '5', '6']

Here, all the internal self-checks using the assert statements are removed, as we can see. Therefore, there's a chance for an application to behave strangely in this case. It is better do raise the Exception explicitly:

import sys

def run():
if not len(sys.argv) == 5:
raise ValueError
print("Argument variables are: ", sys.argv)

run()

Use of assert detected. The enclosed code will be removed when compiling to optimised byte code. here:
harmony/blob/main/docs/notebook_helpers/init.py#L194-L194
L105-L105
L225-L225

and so on.

Discussion blog : https://discuss.deepsource.com/t/using-assert-outside-tests/79/2

DESCRIPTION:

Do not use a mutable like list or dictionary as a default value to an argument. Python’s default arguments are evaluated once when the function is defined. Using a mutable default argument and mutating it will mutate that object for all future calls to the function as well.

BAD PRACTICE:

def my_function(elem, l=[]):
l.append(elem)
return l

print(my_function(2)) # [2]
print(my_function(5)) # [2, 5]

RECOMMENDED:
def my_function(elem, l=None):
if l is None:
l = []
l.append(elem)
return l

print(my_function(2)) # [2]
print(my_function(5)) # [5]

Dangerous default value [] as argument found here:

harmony/blob/main/docs/notebook_helpers/init.py#L105-L105
harmony/blob/main/docs/notebook_helpers/init.py#L225-L225
/harmony/blob/main/docs/notebook_helpers/init.py#L281-L281