jcsda / crtmv3 Goto Github PK

The test_forward_SOI_v.abi_g18 test fails when in the new tagged jedi bundle, but passes if crtm is built separately. Both builds are using ecbuild with CMAKE_BUILD_TYPE=Debug. This is with gfortran 11.3.1. Compilation occurs on an Intel Xeon Platinum 9242 CPU. The host OS is Rocky Linux 9.0.

The test outputs differ subtly, and there are only trivial differences in how ecbuild set flags between the two builds. I haven't been able to spot any OpenMP issues with the Intel Inspector tool, which is expected since I'm not seeing any OpenMP blocks around the SOI modules. I suspect a compiler bug somewhere.

ecbuild outputs:

• crtm only: crtm_ecbuild.txt
• crtm as part of the jedi bundle: jedi_ecbuild.txt

It may look odd, but f95 is a direct symlink to gfortran on this system. No idea why ecbuild prefers one over the other. Ecbuild gives essentially identical outputs for both builds.

Differences:

crtm:

-- [crtm] (3.0.0) [74cdc94]
-- Feature TESTS enabled

......

-- Fortran -- GNU 11.3.1
--     compiler   : /usr/bin/f95
--     flags      :  -D_REAL8_ -ffree-line-length-none -O0 -g -fcheck=bounds -ffpe-trap=invalid,zero,overflow -fbacktrace
--     link flags : -fopenmp

jedi-bundle:

-- ---------------------------------------------------------
-- Adding bundle project crtm
-- ---------------------------------------------------------
-- [crtm] (3.0.0) [74cdc94]
-- Feature TESTS enabled
-- Found OpenMP: TRUE (found version "4.5") found components: Fortran

......

-- Fortran -- GNU 11.3.1
--     compiler   : /usr/bin/gfortran
--     flags      :  -O0 -g -fcheck=bounds -fbacktrace -finit-real=snan
--     link flags :

Although the link flags differ, when I look further down at the build logs, they are consistent. This may be a display bug in ecbuild.

Build flags:

Both builds use identical flags when compiling test_forward_test_SOI. They use the same backend libraries. Both use -fopenmp. Both set -D_REAL8_.

crtm only build:

[ 86%] Linking Fortran executable test_forward_test_SOI
cd /mnt/ram/builds/skylab-5.0.0/crtm/build/gcc/test && /stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/cmake-3.23.1-jc2ddxt/bin/cmake -E remove /mnt/ram/builds/skylab-5.0.0/crtm/build/gcc/test/test_forward_test_SOI
cd /mnt/ram/builds/skylab-5.0.0/crtm/build/gcc/test && /stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/cmake-3.23.1-jc2ddxt/bin/cmake -E cmake_link_script CMakeFiles/test_forward_test_SOI.dir/link.txt --verbose=1
/usr/bin/f95 -fopenmp     -Wl,--disable-new-dtags -Wl,-rpath -Wl,/stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/openmpi-4.1.5-xnhbrj2/lib -Wl,-rpath -Wl,/stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/hwloc-2.9.0-ajeuevx/lib -Wl,-rpath -Wl,/stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/libevent-2.1.12-3pclbsf/lib -Wl,-rpath -Wl,/stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/pmix-4.2.3-s7kxvpi/lib -Wl,-rpath -Wl,/usr/lib64 -L/stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/hwloc-2.9.0-ajeuevx/lib -L/stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/libevent-2.1.12-3pclbsf/lib -L/stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/pmix-4.2.3-s7kxvpi/lib -L/usr/lib64  -D_REAL8_ -ffree-line-length-none -O0 -g -fcheck=bounds -ffpe-trap=invalid,zero,overflow -fbacktrace CMakeFiles/test_forward_test_SOI.dir/mains/regression/forward/test_SOI/test_SOI.f90.o -o test_forward_test_SOI  -Wl,-rpath,/mnt/ram/builds/skylab-5.0.0/crtm/build/gcc/lib:/stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/netcdf-fortran-4.6.0-kfoc7vc/lib:/stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/netcdf-c-4.9.2-3b7tvzn/lib:/stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/openmpi-4.1.5-xnhbrj2/lib ../lib/libcrtm.so /usr/lib/gcc/x86_64-redhat-linux/11/libgomp.so /usr/lib64/libpthread.a /stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/netcdf-fortran-4.6.0-kfoc7vc/lib/libnetcdff.so -L/stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/netcdf-fortran-4.6.0-kfoc7vc/lib -lnetcdff -lnetcdf -lnetcdf -lm /stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/netcdf-c-4.9.2-3b7tvzn/lib/libnetcdf.so -L/stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/netcdf-c-4.9.2-3b7tvzn/lib -lnetcdf /stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/openmpi-4.1.5-xnhbrj2/lib/libmpi_usempif08.so /stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/openmpi-4.1.5-xnhbrj2/lib/libmpi_usempi_ignore_tkr.so /stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/openmpi-4.1.5-xnhbrj2/lib/libmpi_mpifh.so /stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/openmpi-4.1.5-xnhbrj2/lib/libmpi.so 
make[2]: Leaving directory '/mnt/ram/builds/skylab-5.0.0/crtm/build/gcc'

jedi bundle build:

[ 80%] Linking Fortran executable test_forward_test_SOI
cd /mnt/ram/builds/skylab-5.0.0/build/gcc/crtm/test && /stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/cmake-3.23.1-jc2ddxt/bin/cmake -E remove /mnt/ram/builds/skylab-5.0.0/build/gcc/crtm/test/test_forward_test_SOI
cd /mnt/ram/builds/skylab-5.0.0/build/gcc/crtm/test && /stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/cmake-3.23.1-jc2ddxt/bin/cmake -E cmake_link_script CMakeFiles/test_forward_test_SOI.dir/link.txt --verbose=1
/usr/bin/gfortran -fopenmp     -Wl,--disable-new-dtags -Wl,-rpath -Wl,/stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/openmpi-4.1.5-xnhbrj2/lib -Wl,-rpath -Wl,/stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/hwloc-2.9.0-ajeuevx/lib -Wl,-rpath -Wl,/stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/libevent-2.1.12-3pclbsf/lib -Wl,-rpath -Wl,/stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/pmix-4.2.3-s7kxvpi/lib -Wl,-rpath -Wl,/usr/lib64 -L/stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/hwloc-2.9.0-ajeuevx/lib -L/stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/libevent-2.1.12-3pclbsf/lib -L/stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/pmix-4.2.3-s7kxvpi/lib -L/usr/lib64  -D_REAL8_ -ffree-line-length-none -O0 -g -fcheck=bounds -ffpe-trap=invalid,zero,overflow -fbacktrace CMakeFiles/test_forward_test_SOI.dir/mains/regression/forward/test_SOI/test_SOI.f90.o -o test_forward_test_SOI  -Wl,-rpath,/mnt/ram/builds/skylab-5.0.0/build/gcc/lib:/stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/netcdf-fortran-4.6.0-kfoc7vc/lib:/stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/netcdf-c-4.9.2-3b7tvzn/lib:/stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/openmpi-4.1.5-xnhbrj2/lib ../../lib/libcrtm.so /usr/lib/gcc/x86_64-redhat-linux/11/libgomp.so /usr/lib64/libpthread.a /stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/netcdf-fortran-4.6.0-kfoc7vc/lib/libnetcdff.so -L/stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/netcdf-fortran-4.6.0-kfoc7vc/lib -lnetcdff -lnetcdf -lnetcdf -lm /stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/netcdf-c-4.9.2-3b7tvzn/lib/libnetcdf.so -L/stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/netcdf-c-4.9.2-3b7tvzn/lib -lnetcdf /stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/openmpi-4.1.5-xnhbrj2/lib/libmpi_usempif08.so /stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/openmpi-4.1.5-xnhbrj2/lib/libmpi_usempi_ignore_tkr.so /stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/openmpi-4.1.5-xnhbrj2/lib/libmpi_mpifh.so /stacks/climacell/1.4.0/envs/nwp-1.4.0-atnorth-rocky9/install/gcc/11.3.1/openmpi-4.1.5-xnhbrj2/lib/libmpi.so 
make[2]: Leaving directory '/mnt/ram/builds/skylab-5.0.0/build/gcc'
[ 80%] Built target test_forward_test_SOI

Test logs:

• crtm only: log_nojedi_crtm.txt
• crtm as part of the jedi bundle: log_jedi_crtm_test.txt

CRTM release v3.1.0, new features, code enhancements, coefficients updates, and etc

Add active sensor module for radar and lidar applications to CRTMv3 as well.

draft issue

automatically generate link when file uploaded to FTP

Description

We need a better solution to store and disseminate CRTM coefficients. GDEX is free, but inconvenient as it requires the help from UCAR IT people to maintain the data sets. We should explore other options:

• AWS S3 (provided and paid by AWS)
• AWS S3 (provided and paid by us)
• NOAA data lake
• AWS S3-backed git-lfs (provided and paid by either AWS or us)
• Xenodo, Pangea, etc. - providers that store ESM/NWP data (and possibly also issue DOIs for it)

This issue covers exploring the different options, not implementing it across all JEDI repos.

I attempted to build the JEDI fv3-bundle which points to the develop branch of this repo. Compilation of this repo failed at this section of the code:

Updated and uploaded the fix folder for #51

The Community Surface Emissivity Model (CSEM) evolved from the existing surface modules of the Community Radiative Transfer Model (CRTM), but with completely redesigned model structure to facilitate the implementation of new surface RT models. Multiple model options may be easily implemented and accommodated in the CSEM framework. In practical applications, CSEM may be used as a standalone surface RT research tool, or used as a sub-system to provide the surface radiative conditions for CRTM, significantly leveraging the development and improvement efforts of CRTM.
CSEM V1.0.0 includes all the existing CRTM surface functionality, and is ready for the integration with CRTM3.0. A new branch will be created to develop the integrated CRTM version.

This issue is to keep track of the shasums for recent sensors and those that have been modified.
The goal is to incorporate the correct coefficient files in v3.1.0 release.

This issue will be edited heavily as I add new sensors, shasums, etc.

A checked off box means that the coefficient (a) exists, (b) is the preferred version of the coefficient, (c) will become part of the v3.1.0 tarball under development. LE = little endian format, BE = big endian format, NC = netcdf format, XX = missing, ZZ = somehow broken.

The work will be done in this google sheets, and then transferred here:

https://docs.google.com/spreadsheets/d/1CCTKRCoB-OSAJHL04NEKf7m1xWKAh8wacEQ17YMIy7E

This issue documents the test folder merge from v2.4.1 to v3.

Some of the source codes in test folders of v2.4.1 and v3 are different, per issue #21, and could be the source of some ctest failures per issue #19

Test folder updates will be gradually pushed to feature branch: https://github.com/JCSDA/CRTMv3/tree/feature/v241_merge_test
Please feel free to edit this branch as you see fit.

Merge CRTM 2.4.1 surface updates into CRTM v3, including support for

• netCDF surface emissivity LUTs
• New IR water emissivity LUT (Nalli2)
• New IR snow emissivity LUT (Nalli)

To merge the v2.4.1 updates in to this fresh v3 repository, we cannot use git to track code differences since the commits history are new in this repo.

Assuming the CRTM V3 develop branch was generated mostly based on Mark Liu's initial v3 branch, I started a draft PR (https://github.com/JCSDA-internal/crtm/pull/412) to show the code differences between Mark Liu's branch (with fix folder removed, feature/cd_qliuv3_merge_withoutfix) and the latest v2.4.1 updates before Issac's updates (feature/cd_snow_EM). The code differences are showing in roughly ~170 commits before/after after Mark Liu's branch was created: https://github.com/JCSDA-internal/crtm/pull/412/commits

Besides the aerosol and surface emissivity updates, some updates in these commits have already been included in V3 branch, but some have not.

For example:
v2.4.1: https://github.com/JCSDA-internal/crtm/blob/develop/src/CRTM_Tangent_Linear_Module.f90
v3: https://github.com/JCSDA/CRTMv3/blob/develop/src/CRTM_Tangent_Linear_Module.f90
If you search function CRTM_RTSolution_Zero, there are some differences, and this seems to be part of this commit: https://github.com/JCSDA-internal/crtm/pull/412/commits/cfc9f7c89fe56895cf40bb5048c17374d6816b5d

I'm not sure if such code differences are causing the failures in ctests, and wonder if there is a better way to track the code updates with this fresh repo.

In support of issue #19, this action specifically is attempting to debug the failing ctest 11 - test_forward_Simple_atms_n21 (SEGFAULT) to obtain a clearer idea of why this and the remaining CRTMv3 tests are currently failing.

Remove the unused travis CI yaml file from the CRTMv3 repository.

With the progressive merge of v2.4.1 into v3, and continuous generation of coefficients for new sensors, this issue tracks the update history of fix folder for testing/developing

JEDI (spack-stack 1.4.1) ctests 520 failed:

520 - Unit_test_emis_coeff_io_nc (Failed)

because Nalli2.IRsnow.EmisCoeff.nc4 has not been uploaded to/pub/s4/CRTM/file/crtm_coefficients_3.0.0_skylab_6.0.tar.gz

This is an optional emissivity coefficient file, that it won't impact any skylab tests.

Adding stokes and solar irradiance in RT NetCDF output module

Presently there's no check against the tarball that gets downloaded from the FTP site. This issue is to posit potential options for checking the validity of the file prior to uncompressing it.

Key considerations: needs to be easy to maintain.

Compile a list of source files that differ between crtm v2.4.1 and v3.0 (develop branch).

Adding a new IR snow emissivity table provided by Nick Nalli: Nalli_TFIRsnow_EmisCoeff-v11.nc

This table contains updated IR snow emissivities, in comparison to the current optional table: Nalli.IRsnow.EmisCoeff.nc4

CRTM will keep both LUTs per user request for testing.

With standalone ecbuild and branch feature/v241_merge_test, the test_k_matrix_AOD tests fail sometime, for example:

 ctest -R test_k_matrix_AOD
Test project /Users/dangch/Documents/CRTM/CRTM_dev/crtm_code_review/CRTMv3_test/build
    Start 75: test_k_matrix_AOD_cris-fsr_n21
1/6 Test #75: test_k_matrix_AOD_cris-fsr_n21 ...***Failed   13.29 sec
    Start 76: test_k_matrix_AOD_v.abi_g18
2/6 Test #76: test_k_matrix_AOD_v.abi_g18 ......   Passed    0.08 sec
    Start 77: test_k_matrix_AOD_cris399_npp
3/6 Test #77: test_k_matrix_AOD_cris399_npp ....   Passed    2.48 sec
    Start 78: test_k_matrix_AOD_v.abi_gr
4/6 Test #78: test_k_matrix_AOD_v.abi_gr .......   Passed    0.08 sec
    Start 79: test_k_matrix_AOD_abi_g18
5/6 Test #79: test_k_matrix_AOD_abi_g18 ........   Passed    0.13 sec
    Start 80: test_k_matrix_AOD_airs_aqua
6/6 Test #80: test_k_matrix_AOD_airs_aqua ......***Failed   14.80 sec

67% tests passed, 2 tests failed out of 6

Label Time Summary:
CRTM_Tests    =  30.85 sec*proc (6 tests)
crtm_tests    =  30.85 sec*proc (6 tests)
openmp        =  30.85 sec*proc (6 tests)
script        =  30.85 sec*proc (6 tests)

Total Test time (real) =  33.86 sec

The following tests FAILED:
	 75 - test_k_matrix_AOD_cris-fsr_n21 (Failed)
	 80 - test_k_matrix_AOD_airs_aqua (Failed)
Errors while running CTest

I went through all the code updates on aerosols, it is still unclear to me
(1) why some of these tests fail occasionally when the others pass with v3 code.
(2) why the same updates work well with v2.4.1 codes.
I'll spend more time to find out the cause with v3 codes.

Create NetCDF module for writing standalone CRTM outputs

In v2.4.0, PR on quiet print has been included per request by the JEDI team:
https://github.com/JCSDA-internal/crtm/pull/405

In V3, it seems most of the tests we have do not include Quiet, which caused ctest failures, see discussion:
#19 (comment)
The attempt to merge v2.4.1 Quiet PR (#20) caused the seg fault.

In the current V3 code, there is a temporary variable iQuiet introduced as a temporary fix:
https://github.com/JCSDA-internal/crtm/blob/23ebbf159ac39051fc29e36cd70e5ea994275a03/src/CRTM_LifeCycle.f90#L635

As a temporary fix, I'll use iQuiet in CRTM_Lifecycle.f90 instead of Quiet.

The get_crtm_coeffs test is failing with develop with the following message:

    Start 470: get_crtm_coeffs

470: Test command: /usr/local/Cellar/cmake/3.26.3/bin/cmake "-P" "/Users/fabiodiniz/jedi/jedi-bundle_crtmv3/build/crtm/test/get_data_get_crtm_coeffs.cmake"
470: Working Directory: /Users/fabiodiniz/jedi/jedi-bundle_crtmv3/build/crtm/test
470: Test timeout computed to be: 1500
470: (curl) downloading https://ftp.ssec.wisc.edu/pub/s4/CRTM/file/crtm_coefficients_3.0.0_skylab_4.0.tar.gz
470: Generating crtm_coefficients_3.0.0_skylab_4.0.tar.gz.localmd5
470: Generating crtm_coefficients_3.0.0_skylab_4.0.tar.gz.ok
470: make[1]: *** [crtm/test/crtm_coefficients_3.0.0_skylab_4.0.tar.gz.ok] Error 1
470: make: *** [__get_data_get_crtm_coeffs_crtm_coefficients_3_0_0_skylab_4_0_tar_gz/fast] Error 2
470: CMake Error at get_data_get_crtm_coeffs.cmake:5 (message):
470:   Error running
470: Call Stack (most recent call first):
470:   get_data_get_crtm_coeffs.cmake:9 (exec_check)
470: 
470: 
1/1 Test #470: get_crtm_coeffs ..................***Failed    0.90 sec

CRTM is attempting to continue when n_Profiles becomes zero (for whatever reason), this issue should add checks to ensure that CRTM quits gracefully.

CRTM_K_Matrix is where it was noticed in some UFO/JEDI runs.

This repository is a "fresh" start for v3 development, removing prior history. v2.x codes will be maintained in jcsda-internal/crtm (along with the history).

Please develop freely within this repository. Pull requests are necessary to merge codes, and the PRs must be reviewed by at least 2 reviewers.

Evaluate CRTM aerosol LUTs for WRF-Chem aerosols

Noticed that check_crtm.F90 is crashing in v3, when it reaches the point in CRTM_Forward checking if the option Apply_NLTE_Correction is true. Somewhere in the code, the Opt% structure is being corrupted.

Also, the profile_solution function is not returning properly, when the function is called, all code after the function are not executed. Probably related to the above issue.

Summary: As part of our gradual transition away from binary formats, we plan to implement the capability to read both binary format files and netCDF4 coefficient files in CRTM. The preference will be given to netCDF4 files when both types are available. This feature will ensure smooth transition and compatibility until all legacy coefficients are converted to netCDF4.

Details:

1. Dual Support: CRTM will be enhanced to read both binary and netCDF4 coefficient files. This will allow for greater flexibility and easier transition for users.
2. Preference: When both file types are available, CRTM will prioritize netCDF4 files over binary files.
3. Transition Phase: This mixed file type capability will be a transitional phase until all binary files are converted to netCDF4. We will provide a clear timeline and support for this conversion process.

Plan:

• Design and develop the capability to read both binary and netCDF4 coefficient files (essentially done, but needs verification that we can fully transition)
• Implement a mechanism to prioritize netCDF4 files over binary format when both are available (this should be on a per request basis, check for nc4 first, then down to binary).
• Thorough testing to ensure compatibility and correct prioritization.
• Update documentation to guide users about the new feature and the transitioning process.
• Release this feature in CRTM v3.1.0 (see #53)
• Work with NESDIS/STAR to start the conversion process for legacy coefficients.

Expected Release Date: TBD

We encourage everyone's input on this process. Please feel free to comment below with any feedback, suggestions, or concerns you may have regarding this planned feature. Thanks!

Merge CRTM 2.4.1 aerosol updates into CRTM v3, including support for

• GOCART LUT
• NAAPS LUT
• RTTOV LUT

Summary

Introduce a 4th field in the versioning scheme for CRTM binary data tarballs to improve consistency and traceability.

Background

CRTM is currently versioned using a 3-field versioning system (e.g., v3.1.0). However, the binary data tarballs lack a versioning scheme that can be directly tied to the code version. This makes it challenging to ensure consistency and traceability between different versions of CRTM and the associated binary datasets.

Proposed Changes

1. Adopt a 4-field versioning scheme for the binary data tarballs, where the first three fields match the CRTM code version and the 4th field indicates the tarball version. For example, v3.1.0.5 would indicate version 5 of the v3.1.0 tarball.
2. Update the build and packaging scripts to automatically generate the 4-field version tag for the tarball.
3. Update documentation to reflect the new versioning scheme and communicate with partners on the proposed changes and deprecation planning.
4. Deprecation: all tarballs will be deprecated with the deprecation of a certain version of CRTM. e.g. 2.3.0 would deprecate 2.3.0.x tarballs. For situations with rapid updates to tarballs (where versioning goes beyond 9, for exampe), we can deprecate oldest tarballs on a rolling window. So tarball v3.1.0.10 would automatically deprecate the v3.1.0.0 tarball. In this way at most 10 binary tarballs would need to be maintained for any given CRTM release.

Acceptance Criteria

• Tarballs should be versioned using the new 4-field scheme
• md5sum (or whatever hashing algorithm) would be immutable for a given tarball version number.
• The versioning should be automated as part of the build and packaging process (initially this would have to be manual)
• Documentation should be updated to describe the new versioning scheme and deprecation plans.

Additional Context

This change is crucial for ensuring that users can easily trace the binary datasets back to the specific version of CRTM they are using, thereby improving reproducibility and traceability.
This does have a risk associated with carrying around a history of binary tarballs, so a

Related Issues

None

Tasks

• Implement 4-field versioning in build and packaging scripts
• Update documentation to describe the new versioning scheme

Merge relative humidity calculation

[Release Planning] CRTM Version 3.1.0

Summary: We are preparing for the release of CRTM version 3.1.0. This update will incorporate all features from version 3.0.x and will introduce space-based radar simulation capabilities. We aim to ensure stability, backward compatibility, and the introduction of new and improved features.

New Features:

1. Space-Based Radar Simulation: The main highlight of this release will be the space-based radar simulation capabilities, and will contain the necessary code for space-based lidar support (#39).

2. Comprehensive netcdf4 support: Legacy binary formats will continue to be supported, but sensors that have netCDF4 support will only have the netcdf files.

Inherited Features:

This version inherits all features from CRTM 3.0.x.

Bug fixes vs. 3.0.x

1. [placeholder]

New sensors vs. 3.0.x

1. [placeholder]

Plan:

• Finalize the design and performance requirements for the new space-based radar simulation capabilities.
• Ensure it integrates well with existing platforms, such as UFO.
• Thorough testing to ensure stability and performance (ctests, partner evaluation)
• Update documentation to reflect new feature addition (https://github.com/JCSDA-internal/crtm-documentation)
• Transition to netCDF4 formats for all LUTs/coefficient files (#54)
• Release beta version to a limited number of users for early feedback (GMAO/EMC/STAR)
• Incorporate feedback and make necessary modifications.
• Officially release CRTM 3.1.0

Expected Release Date: TBD

We are in the early stages of planning this release, and we encourage everyone's input on this process. Please feel free to comment below with any feedback, suggestions, or concerns you may have regarding CRTM 3.1.0.

Per issue #31, adding unit test on Nalli2.IRsnow emissivity coefficient.

With the CRTM v3 test branch (feature/v241_merge_test), the other group of tests constantly failed are test_adjoint_Simple, see issue #19 :

The following tests FAILED:
	121 - test_adjoint_Simple_atms_n21 (Failed)
	122 - test_adjoint_Simple_cris-fsr_n21 (Failed)
	123 - test_adjoint_Simple_v.abi_g18 (Failed)
	124 - test_adjoint_Simple_atms_npp (Failed)
	125 - test_adjoint_Simple_cris399_npp (Failed)
	126 - test_adjoint_Simple_v.abi_gr (Failed)
	127 - test_adjoint_Simple_abi_g18 (Failed)
	128 - test_adjoint_Simple_modis_aqua (Failed)

Soil_Moisture_Content is specified by input as g/cm^3 (according to the code and CRTM user guide), but gets restricted to [0,1] via the max/min operation below. I suspect that this was a confusion between a typical fractional expression of soil moisture content (e.g., cm^3/cm^3) and the content (g/cm^3).

i.e., in NESDIS_LandEM_Module.f90:

 ! By default use the
    ! Assign local variable
    mv               = Soil_Moisture_Content  

...

    ! Check soil moisture content range
    mv = MAX(MIN(mv,ONE),ZERO)

Recommended solution for v3: use a fractional value for inputs, and update the documentation / calling routines accordingly. Will need to check GSI / UFO as well. I suspect the impact has been limited since soil moisture contents are frequently less than 1 g/cm^3.

I remain, however, confused about finding soil moisture content values in the literature that are as high as 1.6 g/cm^3, considering that the density of pure water is 1 g/cm^3... seems like it would be impossible to have a value greater than 1.

Minor release of CRTM v3.0, reflecting changes implemented during skylab-v6 release.

This will also update the coefficient tarball versions.

current branch feature/btj_REL-3.0.1 for release preparation.

To do:

• 1. develop robust checking toolset for verifying / inspecting BE/LE/netCDF SpcCoeff, TauCoeff, AerosolCoeff, and CloudCoeff files
• 2. ensure consistent BE / LE / netCDF coefficient for all existing files it looks like LE severely lacking compared to BE for N21, for example.
• 3. Test in JEDI / UFO (structurally, numerically)

Currently working fix_directories/tmp/crtm/3.0.1_skylab_6.0/ and fix_directories/fix_REL-3.0.1_20230923 are where the work is occurring. Intention is to merge the jedi-type tarballs (former) and CRTM type tarballs (latter), and only use one tarball type for all CRTM releases.

Feature Request: Add Visible Reflectance Output for GOES-ABI Simulation

Summary

Extend CRTM to include visible reflectance output capabilities for simulating GOES-ABI visible reflectances.

Background

Currently, CRTM lacks the ability to output visible reflectance, which is crucial for simulating GOES-ABI visible channels. This will enable direct assimilation of ABI visible channels (at a minimum).

Proposed Changes

1. Modify the solver and output pathways to include visible reflectance calculations.
2. Update the output data structure to include a field for visible reflectance.
3. Add an option in CRTM Options to toggle visible reflectance output.

Acceptance Criteria

• CRTM should be able to output visible reflectance values when the option is enabled.
• The output should be validated against known benchmarks or real GOES-ABI data.
• Documentation should be updated to describe the new feature.

Additional Context

This feature will be particularly useful for projects that require high-fidelity simulations of visible channels from GOES-ABI and similar sensors. Adjustments to the definition of TOA reflectance might need to be made for different sensors. If this turns out to be too cumbersome for CRTM to manage, we'll move this capability into UFO as a derived quantity.

Related Issues

None

Tasks

• Update radiative transfer equation solver if necessary
• Modify output data structure
• Update CRTM Options to set an output flag
• Validate against observations and other visible RT models
• Update documentation

Add DDA hydrometeor LUT for cloud scattering properties to CRTMv3 as well (this was already available in the scalar release v2.4.1).

This epic covers the conversion of TauCoeff and SpcCoeff CRTM instrument coefficient files from binary to netCDF format for the case where there is no netCDF file already available.

The following file types need to be converted:

• TauCoeff
• SpcCoeff

Conversion scripts for individual files are already available (SpcCoeff and TauCoeff, le -> nc4).
The remaining tasks are:

• Write a batch script to run the individual Fortran applications over all required instruments ( O(100) ).
• Run batch script over all cases.
• Error handling for special cases where the applications fail.

Create CRTM_RTSolution_ReadFile module for netCDF format

This was encountered using tempest-D_cubesat coefficients, but may be unrelated to that coefficient.

[Orion-22-72:381990:0:381990] Caught signal 8 (Floating point exception: integer divide by zero)

/work2/noaa/da/fdiniz/jedi/jedi-bundle/crtm/src/CRTM_K_Matrix_Module.f90: [ __crtm_k_matrix_module_MOD_crtm_k_matrix() ]
      ...

/work2/noaa/da/fdiniz/jedi/jedi-bundle/crtm/src/CRTM_K_Matrix_Module.f90: [ __crtm_k_matrix_module_MOD_crtm_k_matrix() ]
      ...

/work2/noaa/da/fdiniz/jedi/jedi-bundle/crtm/src/CRTM_K_Matrix_Module.f90: [ __crtm_k_matrix_module_MOD_crtm_k_matrix() ]
      ...
      425       CALL OMP_SET_MAX_ACTIVE_LEVELS(1)
      426     ELSE
      427       n_profile_threads = n_Profiles
==>   428       n_channel_threads = MIN(n_Channels, n_omp_threads / n_Profiles)
      429 !      n_channel_threads = MIN(n_Channels, n_omp_threads )
      430       if(n_channel_threads > 1) THEN
      425       CALL OMP_SET_MAX_ACTIVE_LEVELS(1)
      426     ELSE
      427       n_profile_threads = n_Profiles
==>   428       n_channel_threads = MIN(n_Channels, n_omp_threads / n_Profiles)
      429 !      n_channel_threads = MIN(n_Channels, n_omp_threads )
      430       if(n_channel_threads > 1) THEN
      431         CALL OMP_SET_MAX_ACTIVE_LEVELS(2)
      431         CALL OMP_SET_MAX_ACTIVE_LEVELS(2)

Per email request from MIT/LL - Vince Leslie, I verified that the Tropics coefficients (Little Endian) that were originally provided by Patrick were internally consistent in terms of polarization basis. His concern was the file dates were different, which suggested that the files hadn't been updated (still may be the case).

File: /data/users/bjohnson/CRTM/fix_directories/patrick/SpcCoeff/le/tropics_sv2.SpcCoeff.bin, WMO_Sensor_ID: 1, Release.Version: 8.3
Channel 1: Mixed polarization with constant mixing angle
Channel 2: Mixed polarization with constant mixing angle
Channel 3: Mixed polarization with constant mixing angle
Channel 4: Mixed polarization with constant mixing angle
Channel 5: Mixed polarization with constant mixing angle
Channel 6: Mixed polarization with constant mixing angle
Channel 7: Mixed polarization with constant mixing angle
Channel 8: Mixed polarization with constant mixing angle
Channel 9: Mixed polarization with constant mixing angle
Channel 10: Mixed polarization with constant mixing angle
Channel 11: Mixed polarization with constant mixing angle
Channel 12: Mixed polarization with constant mixing angle
Fixed Polarization Angle:
Channel 1:
1.000000000000000E-08
Channel 2:
1.000000000000000E-08
Channel 3:
1.000000000000000E-08
Channel 4:
1.000000000000000E-08
Channel 5:
1.000000000000000E-08
Channel 6:
1.000000000000000E-08
Channel 7:
1.000000000000000E-08
Channel 8:
1.000000000000000E-08
Channel 9:
9.000000000000000E+01
Channel 10:
9.000000000000000E+01
Channel 11:
9.000000000000000E+01
Channel 12:
9.000000000000000E+01

File: /data/users/bjohnson/CRTM/fix_directories/patrick/SpcCoeff/le/tropics_sv3.SpcCoeff.bin, WMO_Sensor_ID: 1, Release.Version: 8.3
Channel 1: Mixed polarization with constant mixing angle
Channel 2: Mixed polarization with constant mixing angle
Channel 3: Mixed polarization with constant mixing angle
Channel 4: Mixed polarization with constant mixing angle
Channel 5: Mixed polarization with constant mixing angle
Channel 6: Mixed polarization with constant mixing angle
Channel 7: Mixed polarization with constant mixing angle
Channel 8: Mixed polarization with constant mixing angle
Channel 9: Mixed polarization with constant mixing angle
Channel 10: Mixed polarization with constant mixing angle
Channel 11: Mixed polarization with constant mixing angle
Channel 12: Mixed polarization with constant mixing angle
Fixed Polarization Angle:
Channel 1:
1.000000000000000E-08
Channel 2:
1.000000000000000E-08
Channel 3:
1.000000000000000E-08
Channel 4:
1.000000000000000E-08
Channel 5:
1.000000000000000E-08
Channel 6:
1.000000000000000E-08
Channel 7:
1.000000000000000E-08

Channel 8:
1.000000000000000E-08
Channel 9:
9.000000000000000E+01
Channel 10:
9.000000000000000E+01
Channel 11:
9.000000000000000E+01
Channel 12:
9.000000000000000E+01

File: /data/users/bjohnson/CRTM/fix_directories/patrick/SpcCoeff/le/tropics_sv4.SpcCoeff.bin, WMO_Sensor_ID: 1, Release.Version: 8.3
Channel 1: Mixed polarization with constant mixing angle
Channel 2: Mixed polarization with constant mixing angle
Channel 3: Mixed polarization with constant mixing angle
Channel 4: Mixed polarization with constant mixing angle
Channel 5: Mixed polarization with constant mixing angle
Channel 6: Mixed polarization with constant mixing angle
Channel 7: Mixed polarization with constant mixing angle
Channel 8: Mixed polarization with constant mixing angle
Channel 9: Mixed polarization with constant mixing angle
Channel 10: Mixed polarization with constant mixing angle
Channel 11: Mixed polarization with constant mixing angle
Channel 12: Mixed polarization with constant mixing angle
Fixed Polarization Angle:
Channel 1:
1.000000000000000E-08
Channel 2:
1.000000000000000E-08
Channel 3:
1.000000000000000E-08
Channel 4:
1.000000000000000E-08
Channel 5:
1.000000000000000E-08
Channel 6:
1.000000000000000E-08
Channel 7:
1.000000000000000E-08
Channel 8:
1.000000000000000E-08
Channel 9:
9.000000000000000E+01
Channel 10:
9.000000000000000E+01
Channel 11:
9.000000000000000E+01
Channel 12:
9.000000000000000E+01

File: /data/users/bjohnson/CRTM/fix_directories/patrick/SpcCoeff/le/tropics_sv5.SpcCoeff.bin, WMO_Sensor_ID: 1, Release.Version: 8.3
Channel 1: Mixed polarization with constant mixing angle
Channel 2: Mixed polarization with constant mixing angle
Channel 3: Mixed polarization with constant mixing angle
Channel 4: Mixed polarization with constant mixing angle
Channel 5: Mixed polarization with constant mixing angle
Channel 6: Mixed polarization with constant mixing angle
Channel 7: Mixed polarization with constant mixing angle
Channel 8: Mixed polarization with constant mixing angle
Channel 9: Mixed polarization with constant mixing angle
Channel 10: Mixed polarization with constant mixing angle
Channel 11: Mixed polarization with constant mixing angle
Channel 12: Mixed polarization with constant mixing angle
Fixed Polarization Angle:
Channel 1:
1.000000000000000E-08
Channel 2:
1.000000000000000E-08
Channel 3:
1.000000000000000E-08
Channel 4:
1.000000000000000E-08
Channel 5:
1.000000000000000E-08
Channel 6:
1.000000000000000E-08
Channel 7:
1.000000000000000E-08
Channel 8:
1.000000000000000E-08
Channel 9:
9.000000000000000E+01
Channel 10:
9.000000000000000E+01
Channel 11:
9.000000000000000E+01
Channel 12:
9.000000000000000E+01

File: /data/users/bjohnson/CRTM/fix_directories/patrick/SpcCoeff/le/tropics_sv6.SpcCoeff.bin, WMO_Sensor_ID: 1, Release.Version: 8.3
Channel 1: Mixed polarization with constant mixing angle
Channel 2: Mixed polarization with constant mixing angle
Channel 3: Mixed polarization with constant mixing angle
Channel 4: Mixed polarization with constant mixing angle
Channel 5: Mixed polarization with constant mixing angle
Channel 6: Mixed polarization with constant mixing angle
Channel 7: Mixed polarization with constant mixing angle
Channel 8: Mixed polarization with constant mixing angle
Channel 9: Mixed polarization with constant mixing angle
Channel 10: Mixed polarization with constant mixing angle
Channel 11: Mixed polarization with constant mixing angle
Channel 12: Mixed polarization with constant mixing angle
Fixed Polarization Angle:
Channel 1:
1.000000000000000E-08
Channel 2:
1.000000000000000E-08
Channel 3:
1.000000000000000E-08
Channel 4:
1.000000000000000E-08
Channel 5:
1.000000000000000E-08
Channel 6:
1.000000000000000E-08
Channel 7:
1.000000000000000E-08
Channel 8:
1.000000000000000E-08
Channel 9:
9.000000000000000E+01
Channel 10:
9.000000000000000E+01
Channel 11:
9.000000000000000E+01
Channel 12:
9.000000000000000E+01

File: /data/users/bjohnson/CRTM/fix_directories/patrick/SpcCoeff/le/tropics_sv7.SpcCoeff.bin, WMO_Sensor_ID: 1, Release.Version: 8.3
Channel 1: Mixed polarization with constant mixing angle
Channel 2: Mixed polarization with constant mixing angle
Channel 3: Mixed polarization with constant mixing angle
Channel 4: Mixed polarization with constant mixing angle
Channel 5: Mixed polarization with constant mixing angle
Channel 6: Mixed polarization with constant mixing angle
Channel 7: Mixed polarization with constant mixing angle
Channel 8: Mixed polarization with constant mixing angle
Channel 9: Mixed polarization with constant mixing angle
Channel 10: Mixed polarization with constant mixing angle
Channel 11: Mixed polarization with constant mixing angle
Channel 12: Mixed polarization with constant mixing angle
Fixed Polarization Angle:
Channel 1:
1.000000000000000E-08
Channel 2:
1.000000000000000E-08
Channel 3:
1.000000000000000E-08
Channel 4:
1.000000000000000E-08
Channel 5:
1.000000000000000E-08
Channel 6:
1.000000000000000E-08
Channel 7:
1.000000000000000E-08
Channel 8:
1.000000000000000E-08
Channel 9:
9.000000000000000E+01
Channel 10:
9.000000000000000E+01
Channel 11:
9.000000000000000E+01
Channel 12:
9.000000000000000E+01

develop branch, tested with gfortran 12.0 on macos.

The following tests FAILED:
	 36 - test_k_matrix_Simple_atms_n21 (SEGFAULT)
	 37 - test_k_matrix_Simple_cris-fsr_n21 (SEGFAULT)
	 38 - test_k_matrix_Simple_v.abi_g18 (SEGFAULT)
	 39 - test_k_matrix_Simple_modis_aqua (SEGFAULT)
	 67 - test_adjoint_Simple_atms_n21 (Failed)
	 68 - test_adjoint_Simple_cris-fsr_n21 (Failed)
	 69 - test_adjoint_Simple_v.abi_g18 (Failed)
	 70 - test_adjoint_Simple_modis_aqua (Failed)

36: Program received signal SIGSEGV: Segmentation fault - invalid memory reference.
36: 
<...>
1/1 Test #36: test_k_matrix_Simple_atms_n21 ....***Exception: SegFault  0.28 sec

67:      Comparing calculated results with saved ones...
67:  test_Simple(INFORMATION) : Atmosphere_AD save file does not exist. Creating...
67:  test_Simple(INFORMATION) : Surface_AD save file does not exist. Creating...
67:  test_Simple(FAILURE) : Atmosphere_AD Adjoints are different!
67: STOP 1
1/1 Test #67: test_adjoint_Simple_atms_n21 .....***Failed    0.17 sec

0% tests passed, 1 tests failed out of 1

This one is more odd, beause it creates the result, and compares it immediately, and fails that comparison. so there's potentially an issue in the comparison structures or in the structures being passed. The forward model and TL model seems to work fine.
My first suspicion is that the aerosol structure is working properly.

Looking a bit deeper, the forward operator for modis_aqua is outputting NaNs in the aerosol portion of the atmosphere structure. I @chengdang I suspect that the something might be mismatched in the aerosol code update -- do the ctests need to be updated?

Title: Migrate build system from ecbuild to CMake

Is your feature request related to a problem? Please describe.

As the developer team, we're finding it increasingly difficult to maintain our build system using ecbuild, due to its less familiar syntax and fewer resources compared to CMake. This hinders our productivity and slows down the progress of our project.

Describe the solution you'd like

We would like to transition from ecbuild to CMake. CMake has a more standard syntax and more community resources available, making it easier to use and troubleshoot. It would simplify the build process for developers, leading to a more efficient and effective development process.

Describe alternatives you've considered

As an alternative, we've considered continuing with ecbuild, while investing in more in-depth training and resources for our team. However, considering the overall benefits of CMake, migrating seems like the more beneficial choice in the long run.

Additional context

This will be a substantial project, so it's recommended to divide it into manageable stages:

1. Research and planning - Understand the current ecbuild setup and plan out how we will recreate this in CMake.
2. Initial conversion - Start creating the CMake setup based on the plan.
3. Testing and debugging - Thoroughly test the new CMake build system and fix any bugs that arise.
4. Documentation - Update all relevant documentation to reflect the change from ecbuild to CMake.
5. Deployment - Roll out the CMake build system to all developers and provide any necessary training.
6. Post-implementation review - After a few weeks or months, gather feedback to understand how well the new system is working and make any necessary adjustments.

It would be helpful if team members could comment with any specific challenges they've had with ecbuild that we should be sure to address in the CMake implementation.

Add the missing CRTM coefficient download script Get_CRTM_Binary_Files.sh from JCSDA-internal/crtm (v2.4.1) to the CRTMv3 repository.

Transitioning Radar operator code (separated from the hydrometeor update). This will be done prior to the code freeze.

The hydrometeor update will require a bit of additional work prior to merging.

This will extend "QUIET" to cover all output messages from CRTM, since CRTM can be called millions of times in operational runs, this options should reduce logfile sizes substantially.

Remove the legacy autoconf build system from the CRTMv3 repository as support is dropped.

ctest 1 (get_crtm_coeffs) is currently failing with the following error:

test 1
    Start 1: get_crtm_coeffs

1: Test command: /opt/homebrew/Cellar/cmake/3.25.3/bin/cmake "-P" "/Users/stegmann/workspace/Fortran_dev/CRTM/REL-3.0/CRTMv3/build/test/get_data_get_crtm_coeffs.cmake"
1: Working Directory: /Users/stegmann/workspace/Fortran_dev/CRTM/REL-3.0/CRTMv3/build/test
1: Test timeout computed to be: 1500
1: (curl) downloading https://ftp.ssec.wisc.edu/pub/s4/CRTM/file/crtm_coefficients_3.0.0_skylab_4.0.tar.gz
1: Generating crtm_coefficients_3.0.0_skylab_4.0.tar.gz.localmd5
1: Generating crtm_coefficients_3.0.0_skylab_4.0.tar.gz.ok
1: gmake[1]: *** [test/CMakeFiles/__get_data_get_crtm_coeffs_crtm_coefficients_3_0_0_skylab_4_0_tar_gz.dir/build.make:86: test/crtm_coefficients_3.0.0_skylab_4.0.tar.gz.ok] Error 1
1: gmake: *** [Makefile:581: __get_data_get_crtm_coeffs_crtm_coefficients_3_0_0_skylab_4_0_tar_gz/fast] Error 2
1: CMake Error at get_data_get_crtm_coeffs.cmake:5 (message):
1:   Error running
1: Call Stack (most recent call first):
1:   get_data_get_crtm_coeffs.cmake:9 (exec_check)
1: 
1: 
1/1 Test #1: get_crtm_coeffs ..................***Failed  177.68 sec

This will cause issues with subsequent dependent tests.