Temporal and spatial accuracy when overlay aircraft data in a tile plot - testing VERDI 2.1.4 20221109 builds about verdi HOT 20 OPEN

Daiwen,

Can you confirm if the timeStamps in the aircraft data is also in UTC ? I did set the "time.vars" to "UTC" before runing read.ICARTT() function.
Thanks,
Yadong @dkang2

from verdi.

@yadongxuEPA

The time stamps in the aircraft data are the seconds after midnight "UTC". I have R code to work with it.

from verdi.

There is another issue we need to pay attention to when overlay aircraft data as observations in a tile plot for model data, that is to figure out which vertical layer a specific instantaneous aircraft observation should locate. Aircraft observations were taken at lower altitude (see below the Y-axis).

P3B_Discover-AQ_15s_CMAQ_Overlay_2011-07-28_O3.pdf
They should not appear in the upper layers of the model data in a tile plot.

In this case, we need to deal with layer vs altitude matching. However, the altitude values for each layer are not straightforward from the WRF file itself. According to Daiwen's shared R script, the layer data need to be extracted from /work/MOD3EVAL/dkj/LTNG/Anal_Vertical/data/mcip/METCRO3D_20110711

from verdi.

Tested VERDI 2.1.4 20221206 builds on Atmos. Found that some improvements have been made but the issue has not resolved, see the details as below:
Test 1: Overlay aircraft data: “discoveraq-UMDAircraft_UMD-AIRCRAFT_20110728_R4_L1.ict” with model data: “2011ek_cb6cmaq5.2_wrf3.8_zero_lnox.CONC.20110729” from /work/MOD3EVAL/dkj/LTNG/Anal_Vertical/data/Base
Now VERDI can correctly match the timestamps and display an error message to remind user that "OBS data does not contain readings ..." (There is a typo in the error message "Observactional" should be "Observational" )

Test 2: Overlay aircraft data: “discoveraq-UMDAircraft_UMD-AIRCRAFT_20110728_R4_L1.ict” with model data: “2011ek_cb6cmaq5.2_wrf3.8_zero_lnox.CONC.20110728” from /work/MOD3EVAL/dkj/LTNG/Anal_Vertical/data/Base
When came to the timestamps (hour 13, 14, 15 UTC) when the aircraft Obs data was available, nothing displayed in the title plot but an error message popped-up as below:

from verdi.

Tested VERDI 2.1.4 20230103 builds on Atmos. Found that some improvements have been made but the following issues need to be worked on:

1. Model data and aircraft data need to be matched hour by hour. For example, when overlay aircraft data: “discoveraq-UMDAircraft_UMD-AIRCRAFT_20110728_R4_L1.ict” with model data: “2011ek_cb6cmaq5.2_wrf3.8_zero_lnox.CONC.20110728” from /work/MOD3EVAL/dkj/LTNG/Anal_Vertical/data/Base
   the aircraft Obs data was only available at hour 13, 14, 15 UTC, we need to pick out those observations for each hour to match with the model data at that hour.
2. We need to figure out which vertical layer each aircraft observation is located and only display them in the corresponding layer. In general, those overlaid circles should not appear in lower (for example, layer 1) layers, nor upper layers (for example, layer 34).
   

from verdi.

Tested VERDI 2.1.4 20230201 builds on Atmos. Found that some improvements have been made in terms of timesteps matching between model data and aircraft data. Use “discoveraq-UMDAircraft_UMD-AIRCRAFT_20110728_R4_L1.ict” as an example, now only the aircraft measurements for a specific hour (hour 13, 14, 15 UTC) showed up in the overlaid tile plot.

We still need to work on vertical layer matching for each aircraft observation. For example, the aircraft data points should not show up in the tile plot when we switch to layer 35 shown as below.

from verdi.

I was able to visualize the O3 data following Yadong's instructions above. However, if I tried to visualize both O3 and NO2, then VERDI seemed to either freeze or give a memory error.

./verdi.sh
Exception in thread "AWT-EventQueue-0" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "AWT-EventQueue-0" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "TimerQueue" java.lang.OutOfMemoryError: GC overhead

Output from top shows high cpu and memory usage

PID    COMMAND      %CPU  TIME     #TH    #WQ  #PORT MEM    PURG   CMPRS  PGRP  PPID  STATE    BOOSTS           %CPU_ME %CPU_OTHRS UID  FAULTS    COW    MSGSENT    MSGRECV    SYSBSD
79918  java         98.1  05:00.05 31/1   3    409   3797M+ 0B     195M   79914 79914 running  *46[88]          0.02332 0.00000    502  1480477+  1861   556303+    115840+    612153+

I had to use Force Quit on the mac to get VERDI to stop.

The model data file covered the CONUS and contained 6 variables so it was a 2.7G file. I used m3xtract and m3wndw to create a file with only O3 and NO2, and a reduced spatial dimension that covers only columns 352-362 and rows 147-157, which is within the area that the observational data is located.
The file name and size is now:

ls -lht 2011ek_cb6cmaq5.2_wrf3.8_zero_lnox.CONC.O3.NO2.wndw.20110728
-rw-rw-r-- 1 lizadams rc_cep-emc_psx 839K Feb 15 13:27 2011ek_cb6cmaq5.2_wrf3.8_zero_lnox.CONC.O3.NO2.wndw.20110728

Also added the observational dataset to the VERDI so it will be available in future builds under VERDI_2.1.4/data/obs/discoveraq-UMDAircraft_UMD-AIRCRAFT_20110728_R4_L1.ict

from verdi.

working on extracting the same windowed domain for the 3.7G METCRO3D_20110711 file. I am getting an error.

Program version: 
     $Id:: m3xtract.f 117 2019-06-15 14:56:29Z coats               $
      
     
     Value for PROMPTFLAG not defined;returning default:   TRUE
     Enter logical name for  INPUT FILE [INFILE] >>
     Error opening file at path-name:
     netCDF error number -128  processing file "INFILE"

Found this error described in the CMAS User Forum: https://forum.cmascenter.org/t/error-abort-in-subroutine-m3wndw/2199/3

Need to use ncatted to replace the following attribute. (or some value that will allow the m3wndw program to run successfully.

 :IOAPI_VERSION = "ioapi-3.2: $Id: init3.F90 1 2017-06-10 18:05:20Z coats $   

with

 :IOAPI_VERSION = "ioapi-3.2: $Id: init3.F90 200 2021-05-10 14:06:20Z coats $ 

The METCRO3D_20110728 file didn't have the same issue.
It looks like perhaps the source of the problem was this attribute: _NCProperties = "version=1|netcdflibversion=4.4.1|hdf5libversion=1.8.18" ;
comparing the difference between
diff METCRO3D_20110711.header METCRO3D_20110728.header

< 		:_NCProperties = "version=1|netcdflibversion=4.4.1|hdf5libversion=1.8.18" ;
< 		:IOAPI_VERSION = "ioapi-3.2: $Id: init3.F90 1 2017-06-10 18:05:20Z coats $                        " ;
---
> 		:IOAPI_VERSION = "$Id:: init3.F 1 2014-03-14 20:22:54Z coats                  $        

I didn't have the same issue using m3wndw on the METCRO3D_20110728 file.

I've added the METCRO3D_20110728 file windowed to the region of interest to the VERDI repo. The next build should contain all of the files required to do vertical layer matching for the aircraft observation testing.

TODO:

• look for another *.ict file that contains a different variable name such as ALTP for the altitude rather than GPSAlt that is in the discoveraq-UMDAircraft_UMD-AIRCRAFT_20110728_R4_L1.ict that we already have.

It looks like the ict files that contain merge in the name: merge are for flights covering a different location, different lat, lon values. I tried extracting only the July 28 data from the 0701-0729 file, but I can't get VERDI to plot the data, as the obs values are located outside of the CONUS.
I am also not sure that any of these files contained data for O3 or NO2, as some of the values are -99999.

• upload another *.ict file that contains different units for the altitude km versus m

I will need help identifying files that contain these different variations of variables and units that covers the same time and region.

from verdi.

@yadongxuEPA
Following the list of todo items in the comment above, I took an additional look at the files that contained ALTP for the altitude rather than GPSAlt.

Here is a slice of the data from discoveraq-mrg01-p3b_merge_20110728_R3_thru20110728.ict
I chose one row of the data for the JDAY 209.

The latitude, longitude values are:
37.936733, 284.536545

VERDI didn't plot this data correctly. When I reached timestep 14, the following pop-up window appeared.

I used awk to change any -999999 value to 60 for the O3_ppbv column, and was able to get past this error message.
So that indicates that VERDI needs to handle NaN when doing the Obs Overlay.

For the merge file, the Latitude and Longitude values are in all CAPPS, and I don't think that VERDI is currently recognizing that, and the Logitude value may need to be adjusted by subtracting 360 degrees: 284.536545 - 360 = -75.463455

This location on Google Maps is near Wallops Field in Virginia.
https://www.google.com/maps/place/37%C2%B056'12.2%22N+75%C2%B027'48.4%22W/@37.9367372,-75.4984739,13z/data=!4m4!3m3!8m2!3d37.936733!4d-75.463455

 Fractional_Day,  UTC, JDAY, INDEX, FLIGHT, LOCAL_SUN_TIME, LATITUDE, LONGITUDE, ALTP, PRESSURE, TEMPERATURE, THETA, O3COLUMN, SZA, WNS, WND, FMS_TAS, FMS_SAT, FMS_GRD_SPD, Heading, FMS_TRK, IRS_PITCH, IRS_ROLL, IRS_VERT_ACC, ADC_IAS, GPS_ALT, A_DewPoint, A_CabinPressure, A_SurfTemp, A_TotalTemp, A_JNO2_Nadir, A_JNO2_Zenith, A_RadarAlt, C_DiffPressure, C_PotTempDegK, C_MachNumber, C_CabAltitude, C_VaporPresWater, C_SatVaporPresWater, C_SatVaporPresIce, C_MixingRatio, C_RelHumidity, H2O(DLH), NO, NOy, NO2_NCAR, O3_ppbv, Carbon_Monoxide_mixing_ratio, Methane_mixing_ratio, Carbon_dioxide_mixing_ratio, NO2_LIF, PNs_TD-LIF, ANs_TD-LIF, HNO3_TD-LIF, CH2O_DFGAS, Abs470tot, Abs532tot, Abs660tot, nAPS_total, sAPS_total, vAPS_total, nAPS_fine(Daero_lt_1.0um), sAPS_fine(Daero_lt_1.0um), vAPS_fine(Daero_lt_1.0um), nAPS_coarse(Daero_ge_1.0um), sAPS_coarse(Daero_ge_1.0um), vAPS_coarse(Daero_ge_1.0um), BlackCarbonMassConcentration, CN>3nm, CN>10nm, nonvolCN>10nm, Scat450tot, Scat550tot, Scat700tot, Scat450sub, Scat550sub, Scat700sub, nUHSAS(60-1000_nm), sUHSAS(60-1000_nm), vUHSAS(60-1000_nm), nLAS(0.9-7.7_um), sLAS(0.9-7.7_um), vLAS(0.9-7.7_um), Angstrom_Exponent_of_Scattering_at_450and700nm, Angstrom_Exponent_of_Scattering_at_450and550nm, Angstrom_Exponent_of_Absorption_at_450and700nm, Angstrom_Exponent_of_Absorption_at_450and550nm, SingleScatteringAlbedo_at_450nm, SingleScatteringAlbedo_at_550nm, SingleScatteringAlbedo_at_700nm, SingleScatteringAlbedo_at_550nm(ambient), nSMPS, sSMPS, vSMPS, Acetaldehyde, Acetonitrile, Acetone, Methanol, Toluene, Isoprene, MVK_MAC, Xylenes_ETB, Acetic_acid, Monoterpenes, RHamb, RHdry, RHwet, SCdry, SCwet, gamma, fRH_80_20, SCamb, EXTamb532, EXTdry532, SCamb532, SCdry532, ABSdry532, Chloride, Nitrite, Nitrate, Sulfate, Sodium, Ammonium, Potassium, Magnesium, Calcium, Water-Soluble_Organic_Carbon_Mass, discoveraq-SiteFlag-1sec, ProfileSequenceNum
       209.60079,51908,   209,          130002.00,         13.000000,  9.388130778,   37.936733,   284.536545,   -0.03989832049,   1019.03,   302.0799939,   300.4573479,   0,   38.83436106,    -999999,   -999999,  68.27133241,   301.6499939,   70.78179961,   212.01,   212.7,   0.47,   0.44,   0.029,   68.77033921,   -0.01392936017,   295.1499939,   1019.528127,   319.0499939,   304.3899939,   0.000598,   0.007919,   0.000161239202,   27.516,   300.467,   0.195,   -0.0519494116,   26.5538,   40.0809,   52.7245,   16.64112,   66.25,   24926.9,    -999999,   -999999,   -999999,   -999999,   100.0,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,  18.79,   0.25,   0,   0,   0,   0,    -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,  43.26,   4.78,   0.19,    -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,  66.25,    -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,   -999999,                0,                0

from verdi.

@lizadams The Latitude and Logitude are 37.936733 and 284.536545, respectively.

from verdi.

from verdi.

Another thought, it is probably good to use the pressure levels (or at least as an option). In general, the pressure levels should be in both the observations and model outputs.

from verdi.

Attaching a gzipped version of the file where I extracted JDAY 209 or 2011209. In case it helps troubleshoot this issue.

discoveraq-mrg01-p3b_merge_20110728_R3_thru20110728.ict.gz

from verdi.

@dkang2 Is there a formula to convert pressure levels to layer height? I did find this document for a method to compare satellite data to CMAQ output. https://www.ldeo.columbia.edu/~amfiore/haqast_TT_files/HAQAST_SIP_TT_GT_Technical_Guidance_Comparison_of_CMAQ_to_OMI-V1.0.pdf

from verdi.

@lizadams There are some algorithms to convert pressure levels to layer height. It can be interpolated using the bottom and top pressure values and the sigma values, p=sigma*(psfc-ptop)+ptop to find the bottom and top pressures at each layer. It becomes more complicated when using the hybrid vertical structure. However, I think that any interpolation should be good enough to make the comparisons, especially the aircraft observations are mainly in the lower to middle troposphere. In the LTNG_DEFN.F in CMAQ code, the lightning NO is matched to the model layer in this way (find the BOTTOM and TOP variables). The most simple and accurate method is still using the ZF variable.

from verdi.

Reference

Agree with Daiwen's comments.
Yadong

from verdi.

Tested VERDI 2.1.4 20230224 builds on Atmos. Found that we still have timesteps matching issue between model data and aircraft data for some aircraft data files. When used “discoveraq-UMDAircraft_UMD-AIRCRAFT_20110728_R4_L1.ict” to overlay on model data “2011ek_cb6cmaq5.2_wrf3.8_zero_lnox.CONC.20110728”, the timesteps matching worked ok. But when switched to use another aircraft file "discoveraq-mrg01-p3b_merge_20110701_R3_thru20110729.ict", I got the following error message:

from verdi.

Retested the timesteps matching issue with VERDI 2.1.4 20230911 builds on Atmos. Found that we don't have error messages showed up when load the merged file such as "discoveraq-mrg01-p3b_merge_20110701_R3_thru20110729.ict", but no data points showed up in the map after we add them as observations.

Furthermore, there was an error message popped-up when used the aircraft data for one specific day such as “discoveraq-UMDAircraft_UMD-AIRCRAFT_20110728_R4_L1.ict” to overlay on model data “2011ek_cb6cmaq5.2_wrf3.8_zero_lnox.CONC.20110728”.

from verdi.

Retested the timesteps matching issue with VERDI 2.1.4 20230924 builds on Atmos. Found that this issue has improved and but the timesteps matching is not consistent with the ones we saw in our previous tests.
When use “discoveraq-UMDAircraft_UMD-AIRCRAFT_20110728_R4_L1.ict” to overlay on model data “2011ek_cb6cmaq5.2_wrf3.8_zero_lnox.CONC.20110728”. We found that only two hours (T14 and T15 UTC have O3 observations)

But our previous testing results showed that we have three hours (T13, T14, T15 UTC) with O3 observations, I also checked the csv files extracted from the *.ict file and confirmed that we do have several data points within T13.
DaiWen,
Can you double-check if “discoveraq-UMDAircraft_UMD-AIRCRAFT_20110728_R4_L1.ict” contains O3 observations at T13 UTC?
Thanks,
Yadong

from verdi.

Continued testing the timesteps matching issue with VERDI 2.1.4 20230924 builds on Atmos.
When use the merged aircraft data “discoveraq-mrgLARGE-pilsIC-p3b_merge_20110701_R3_thru20110729.ict” to overlay on model data “2011ek_cb6cmaq5.2_wrf3.8_zero_lnox.CONC.20110728”. Found that the timesteps when observations showed up in the map are not consistent with what we see from the extracted data in CSV file.
In VERDI, we can see observational data points appeared at 2011-07-28 T00 UTC through 2011-07-28 T13 UTC.

from verdi.