Port UFS_UTILS to Orion about ufs_utils HOT 24 CLOSED

Note to self: The 'emcsfc' builds use v2.5 of the NCEP g2 library. Under /apps/contrib/NCEPLIBS/orion/modulefiles, there is only v3.2.0. But I could not get emcsfc to compile without this error: undefined reference to `png_set_longjmp_fn

According to George V.: "This is an incompatibility between our version of png in NCEPLIBS and the version on the system. Some of the g2 build is picking up our version and some, the system's."

So I switched to George V's version of g2 v2.5 here: /apps/contrib/NCEPLIBS/lib/modulefiles to get emcsfc to compile.

Also, if I use George V's g2 v3 and his version of png, it compiles without error. Will contact the NCEPLIBS group.

from ufs_utils.

All programs within the repository were successfully compiled on Orion (ffab5c4).

Next, run any unit and regression tests. Note: there is no test for the 'nst_tf_chg' program, which is out-of-date (#90)

from ufs_utils.

The nemsio utilities - nemsio_get, nemsio_chgdate, nemsio_read, mkgdsnemsioctl - were tested using the script and data in: /work/noaa/da/ggayno/save/ufs_utils.git/orion_port/nemsio_utils

The specific tests are:

• Call nemsio_get to print header information from a nemsio file.
• Call nemsio_chgdate to change the date stamp in a nemsio file.
• Call nemsio_read to dump the contents of a nemsio file.
• Call mkgfsnemsioctl to create a Grads control file for viewing a nemsio file.

from ufs_utils.

The nemsio test was run using ffab5c4 of the feature/orion branch. Output from the first three utilities was stored to this log file: /work/noaa/da/ggayno/save/ufs_utils.git/orion_port/nemsio_utils/log.unit.test. All output looked correct. The Grads control file from the last utility was stored in the same directory: gdas.t00z.sfcanl.nemsio.ctl. The file was visualized using that control file and all fields displayed correctly.

from ufs_utils.

Next, the grid generation was tested. This includes the fre-nctools, orog and sfc_climo_gen programs. Using ffab5c4, C96 uniform, C96 stretched, C96 nested and C96 regional grids were created using the driver_grid.orion.sh script in ./UFS_UTILS/driver_scripts. The default grid specs in the script were not modified.

All grids were successfully created. The 'grid', 'orog' and 'sfc_climo' files were visualized using ncview and looked correct.

from ufs_utils.

Next, the snow2mdl program was tested using the script and data in /work/noaa/da/ggayno/save/ufs_utils.git/orion_port/snow2mdl. The test (using ffab5c4) blends IMS and AFWA data to create a T1534 global analysis of snow depth and cover (grib1 format).

The test ran successfully and the output file was visualized in Grads and looked correct. The same test was run on Hera using 2d7f796 of 'develop'. The output files were bit identical.

from ufs_utils.

Next, the global_cycle program was tested (ffab5c4) using a canned case. The run script and data are here: /work/noaa/da/ggayno/save/ufs_utils.git/orion_port/cycle. The test updates the surface fields for a C768 uniform global grid. The same test was run on Hera, using 'develop' at 2d7f796. The output tiled netCDF files from both tests were compared using the nccmp utility and were identical.

from ufs_utils.

Next, the serial version of chgres was tested (the branch at ffab5c4) using a canned case: /work/noaa/da/ggayno/save/ufs_utils.git/orion_port/chgres_serial. The test converts GFS gaussian nemsio files to C96 global uniform. The same test was run on Hera, using 'develop' at 2d7f796. chgres outputs atmospheric and surface tiled netCDF files. The files from both tests were identical according to the nccmp utility.

from ufs_utils.

Next, the emcsfc_ice_blend program was tested (used the branch at ffab5c4). The test blends IMS ice cover and EMC ice concentration data. The output blended file is 1/12-degree, grib 1 format. The script and data for the canned case are here: /work/noaa/da/ggayno/save/ufs_utils.git/orion_port/ice_blend. The same test was run on Hera using 'develop' at 2d7f796. The files from both tests were bit-identical.

Note: the ice blend process uses copygb to replace the bitmap with a flag value of 1.57 in the final file. This version of copygb gave a wrong answer: /apps/contrib/NCEPLIBS/orion/utils/grib_util.v1.2.0/exec/copygb. This version gave the correct answer: /apps/contrib/NCEPLIBS/lib/NCEPLIBS-grib_util/v1.1.1/exec/copygb. Will notify the developer of this. In summary, the ice_blend program is working on Orion.

from ufs_utils.

Finally, the chgres_cube regression test suite was run (using ffab5c4). The baseline comparison data was copied from Hera. All tests passed:

<<< C192 FV3 HISTORY TEST PASSED. >>>
<<< C96 FV3 GAUSSIAN NEMSIO TEST PASSED. >>>
<<< C96 GFS SIGIO TEST PASSED. >>>
<<< C96 GFS GAUSSIAN NEMSIO TEST PASSED. >>>
<<< C96 REGIONAL TEST PASSED. >>>
<<< C192 GFS GRIB2 TEST PASSED. >>>```

from ufs_utils.

All programs are working on Orion. One final task is to the test the ./utils/gdas_init utility. However, it requires access to HPSS. And Orion can't talk to HPSS. George V. is developing a work-around. Standing by.

from ufs_utils.

There are issues building g2 with png support. PNG 1.2.44 (and maybe others) has a header with different structure from earlier and also much more recent PNG releases. If different releases are used when the png library is built and when g2 is linked, there will be an unresolved png-longjmp external. This was a nasty problem for me for some weeks in 2017 getting NCEPLIBS to build on newer theia releases before I found it.

The solution is to do one of two things.

1. Use png/v1.2.59 along with g2.

2. When building the g2 library make sure that the png includes are NOT from the linux distribution ones in /usr/include or /usr/local/include or /opt/include. SPECIFY them by
   loading the png module used in post and other app builds.

When I build my G2, these are set

export PNG_INC=$NCEPLIBS/EXTERNAL/LOCAL_EXTERN/png/v1.2.44/include/

export JASPER_INC=$NCEPLIBS/EXTERNAL/LOCAL_EXTERN/jasper_v1.900.1/include
export Z_INC=$NCEPLIBS/EXTERNAL/LOCAL_EXTERN/z/v1.2.6/include

(I probably need to change this last one to z1.2.7 which is generally used elsewhere)

from ufs_utils.

Before merging, check with Kate about which versions of netcdf and esmf to use.

from ufs_utils.

@KateFriedman-NOAA What more needs to be done for this ticket?

from ufs_utils.

Nothing I can think of. I checked and see this is in the Orion build modulefile:

module use -a /apps/contrib/NCEPLIBS/lib/modulefiles
module load netcdfp/4.7.4.release
module load esmflocal/8_0_0.release

...which was the only bit I needed to check was in. I've been running a longer test on Orion since the maintenance last week ended and have no issues so we're good to finish this and merge back.

We will need to update those netcdf and esmf modules again when the updated versions are officially installed and ready. Can do a small PR for that when ready. Thanks!

Will we get a new UFS_UTILS tag afterwards?

from ufs_utils.

Nothing I can think of. I checked and see this is in the Orion build modulefile:

module use -a /apps/contrib/NCEPLIBS/lib/modulefiles
module load netcdfp/4.7.4.release
module load esmflocal/8_0_0.release

...which was the only bit I needed to check was in. I've been running a longer test on Orion since the maintenance last week ended and have no issues so we're good to finish this and merge back.

We will need to update those netcdf and esmf modules again when the updated versions are officially installed and ready. Can do a small PR for that when ready. Thanks!

Will we get a new UFS_UTILS tag afterwards?

The GDAS initialization scripts can be added later if they ever allow Orion to access HPSS.

Once I merge, I can create a tag or you can just point to the proper snapshot of 'develop'. Or are you referring to the 'release/ops-gfsv16' branch? That release branch is several commits behind 'develop'. I did not update the release branch for Orion (but I can).

from ufs_utils.

The GDAS initialization scripts can be added later if they ever allow Orion to access HPSS.

Yep...most archive/HPSS stuff will have to wait too. Sounds good, thanks!

Once I merge, I can create a tag or you can just point to the proper snapshot of 'develop'. Or are you referring to the 'release/ops-gfsv16' branch? That release branch is several commits behind 'develop'. I did not update the release branch for Orion (but I can).

We need to get Orion support into the release/ops-gfsv16 branch so yes, please update the release branch. Thanks!

from ufs_utils.

The GDAS initialization scripts can be added later if they ever allow Orion to access HPSS.

Yep...most archive/HPSS stuff will have to wait too. Sounds good, thanks!

Once I merge, I can create a tag or you can just point to the proper snapshot of 'develop'. Or are you referring to the 'release/ops-gfsv16' branch? That release branch is several commits behind 'develop'. I did not update the release branch for Orion (but I can).

We need to get Orion support into the release/ops-gfsv16 branch so yes, please update the release branch. Thanks!

Kate, sorry for being asleep at the wheel. I was totally focused on the head of develop. Will try to get it done by sometime next week.

from ufs_utils.

No worries and thanks! :)

from ufs_utils.

I will work the Orion port of the GFS v16 release branch under this issue #26.

from ufs_utils.

The branch at e55d18e was successfully compiled on Cray, Dell, Hera, Orion and Jet. All regression tests passed on those machines. Will merge.

from ufs_utils.

@KateFriedman-NOAA This was merged today. I will keep the 'feature/orion' branch available for a few more days. Then delete it.

from ufs_utils.

@GeorgeGayno-NOAA Great, thanks!

from ufs_utils.

Deleting branch and closing issue.

from ufs_utils.