Comments (20)
Daiwen,
Can you confirm if the timeStamps in the aircraft data is also in UTC ? I did set the "time.vars" to "UTC" before runing read.ICARTT() function.
Thanks,
Yadong @dkang2
from verdi.
The time stamps in the aircraft data are the seconds after midnight "UTC". I have R code to work with it.
from verdi.
There is another issue we need to pay attention to when overlay aircraft data as observations in a tile plot for model data, that is to figure out which vertical layer a specific instantaneous aircraft observation should locate. Aircraft observations were taken at lower altitude (see below the Y-axis).
P3B_Discover-AQ_15s_CMAQ_Overlay_2011-07-28_O3.pdf
They should not appear in the upper layers of the model data in a tile plot.
In this case, we need to deal with layer vs altitude matching. However, the altitude values for each layer are not straightforward from the WRF file itself. According to Daiwen's shared R script, the layer data need to be extracted from /work/MOD3EVAL/dkj/LTNG/Anal_Vertical/data/mcip/METCRO3D_20110711
from verdi.
Tested VERDI 2.1.4 20221206 builds on Atmos. Found that some improvements have been made but the issue has not resolved, see the details as below:
Test 1: Overlay aircraft data: “discoveraq-UMDAircraft_UMD-AIRCRAFT_20110728_R4_L1.ict” with model data: “2011ek_cb6cmaq5.2_wrf3.8_zero_lnox.CONC.20110729” from /work/MOD3EVAL/dkj/LTNG/Anal_Vertical/data/Base
Now VERDI can correctly match the timestamps and display an error message to remind user that "OBS data does not contain readings ..." (There is a typo in the error message "Observactional" should be "Observational" )
Test 2: Overlay aircraft data: “discoveraq-UMDAircraft_UMD-AIRCRAFT_20110728_R4_L1.ict” with model data: “2011ek_cb6cmaq5.2_wrf3.8_zero_lnox.CONC.20110728” from /work/MOD3EVAL/dkj/LTNG/Anal_Vertical/data/Base
When came to the timestamps (hour 13, 14, 15 UTC) when the aircraft Obs data was available, nothing displayed in the title plot but an error message popped-up as below:
from verdi.
Tested VERDI 2.1.4 20230103 builds on Atmos. Found that some improvements have been made but the following issues need to be worked on:
- Model data and aircraft data need to be matched hour by hour. For example, when overlay aircraft data: “discoveraq-UMDAircraft_UMD-AIRCRAFT_20110728_R4_L1.ict” with model data: “2011ek_cb6cmaq5.2_wrf3.8_zero_lnox.CONC.20110728” from /work/MOD3EVAL/dkj/LTNG/Anal_Vertical/data/Base
the aircraft Obs data was only available at hour 13, 14, 15 UTC, we need to pick out those observations for each hour to match with the model data at that hour. - We need to figure out which vertical layer each aircraft observation is located and only display them in the corresponding layer. In general, those overlaid circles should not appear in lower (for example, layer 1) layers, nor upper layers (for example, layer 34).
from verdi.
Tested VERDI 2.1.4 20230201 builds on Atmos. Found that some improvements have been made in terms of timesteps matching between model data and aircraft data. Use “discoveraq-UMDAircraft_UMD-AIRCRAFT_20110728_R4_L1.ict” as an example, now only the aircraft measurements for a specific hour (hour 13, 14, 15 UTC) showed up in the overlaid tile plot.
We still need to work on vertical layer matching for each aircraft observation. For example, the aircraft data points should not show up in the tile plot when we switch to layer 35 shown as below.
from verdi.
I was able to visualize the O3 data following Yadong's instructions above. However, if I tried to visualize both O3 and NO2, then VERDI seemed to either freeze or give a memory error.
./verdi.sh
Exception in thread "AWT-EventQueue-0" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "AWT-EventQueue-0" java.lang.OutOfMemoryError: GC overhead limit exceeded
Exception in thread "TimerQueue" java.lang.OutOfMemoryError: GC overhead
Output from top shows high cpu and memory usage
PID COMMAND %CPU TIME #TH #WQ #PORT MEM PURG CMPRS PGRP PPID STATE BOOSTS %CPU_ME %CPU_OTHRS UID FAULTS COW MSGSENT MSGRECV SYSBSD
79918 java 98.1 05:00.05 31/1 3 409 3797M+ 0B 195M 79914 79914 running *46[88] 0.02332 0.00000 502 1480477+ 1861 556303+ 115840+ 612153+
I had to use Force Quit on the mac to get VERDI to stop.
The model data file covered the CONUS and contained 6 variables so it was a 2.7G file. I used m3xtract and m3wndw to create a file with only O3 and NO2, and a reduced spatial dimension that covers only columns 352-362 and rows 147-157, which is within the area that the observational data is located.
The file name and size is now:
ls -lht 2011ek_cb6cmaq5.2_wrf3.8_zero_lnox.CONC.O3.NO2.wndw.20110728
-rw-rw-r-- 1 lizadams rc_cep-emc_psx 839K Feb 15 13:27 2011ek_cb6cmaq5.2_wrf3.8_zero_lnox.CONC.O3.NO2.wndw.20110728
Also added the observational dataset to the VERDI so it will be available in future builds under VERDI_2.1.4/data/obs/discoveraq-UMDAircraft_UMD-AIRCRAFT_20110728_R4_L1.ict
from verdi.
working on extracting the same windowed domain for the 3.7G METCRO3D_20110711 file. I am getting an error.
Program version:
$Id:: m3xtract.f 117 2019-06-15 14:56:29Z coats $
Value for PROMPTFLAG not defined;returning default: TRUE
Enter logical name for INPUT FILE [INFILE] >>
Error opening file at path-name:
netCDF error number -128 processing file "INFILE"
Found this error described in the CMAS User Forum: https://forum.cmascenter.org/t/error-abort-in-subroutine-m3wndw/2199/3
Need to use ncatted to replace the following attribute. (or some value that will allow the m3wndw program to run successfully.
:IOAPI_VERSION = "ioapi-3.2: $Id: init3.F90 1 2017-06-10 18:05:20Z coats $
with
:IOAPI_VERSION = "ioapi-3.2: $Id: init3.F90 200 2021-05-10 14:06:20Z coats $
The METCRO3D_20110728 file didn't have the same issue.
It looks like perhaps the source of the problem was this attribute: _NCProperties = "version=1|netcdflibversion=4.4.1|hdf5libversion=1.8.18" ;
comparing the difference between
diff METCRO3D_20110711.header METCRO3D_20110728.header
< :_NCProperties = "version=1|netcdflibversion=4.4.1|hdf5libversion=1.8.18" ;
< :IOAPI_VERSION = "ioapi-3.2: $Id: init3.F90 1 2017-06-10 18:05:20Z coats $ " ;
---
> :IOAPI_VERSION = "$Id:: init3.F 1 2014-03-14 20:22:54Z coats $
I didn't have the same issue using m3wndw on the METCRO3D_20110728 file.
I've added the METCRO3D_20110728 file windowed to the region of interest to the VERDI repo. The next build should contain all of the files required to do vertical layer matching for the aircraft observation testing.
TODO:
- look for another *.ict file that contains a different variable name such as ALTP for the altitude rather than GPSAlt that is in the discoveraq-UMDAircraft_UMD-AIRCRAFT_20110728_R4_L1.ict that we already have.
It looks like the ict files that contain merge in the name: merge are for flights covering a different location, different lat, lon values. I tried extracting only the July 28 data from the 0701-0729 file, but I can't get VERDI to plot the data, as the obs values are located outside of the CONUS.
I am also not sure that any of these files contained data for O3 or NO2, as some of the values are -99999.
- upload another *.ict file that contains different units for the altitude km versus m
I will need help identifying files that contain these different variations of variables and units that covers the same time and region.
from verdi.
@yadongxuEPA
Following the list of todo items in the comment above, I took an additional look at the files that contained ALTP for the altitude rather than GPSAlt.
Here is a slice of the data from discoveraq-mrg01-p3b_merge_20110728_R3_thru20110728.ict
I chose one row of the data for the JDAY 209.
The latitude, longitude values are:
37.936733, 284.536545
VERDI didn't plot this data correctly. When I reached timestep 14, the following pop-up window appeared.
I used awk to change any -999999 value to 60 for the O3_ppbv column, and was able to get past this error message.
So that indicates that VERDI needs to handle NaN when doing the Obs Overlay.
For the merge file, the Latitude and Longitude values are in all CAPPS, and I don't think that VERDI is currently recognizing that, and the Logitude value may need to be adjusted by subtracting 360 degrees: 284.536545 - 360 = -75.463455
This location on Google Maps is near Wallops Field in Virginia.
https://www.google.com/maps/place/37%C2%B056'12.2%22N+75%C2%B027'48.4%22W/@37.9367372,-75.4984739,13z/data=!4m4!3m3!8m2!3d37.936733!4d-75.463455
Fractional_Day, UTC, JDAY, INDEX, FLIGHT, LOCAL_SUN_TIME, LATITUDE, LONGITUDE, ALTP, PRESSURE, TEMPERATURE, THETA, O3COLUMN, SZA, WNS, WND, FMS_TAS, FMS_SAT, FMS_GRD_SPD, Heading, FMS_TRK, IRS_PITCH, IRS_ROLL, IRS_VERT_ACC, ADC_IAS, GPS_ALT, A_DewPoint, A_CabinPressure, A_SurfTemp, A_TotalTemp, A_JNO2_Nadir, A_JNO2_Zenith, A_RadarAlt, C_DiffPressure, C_PotTempDegK, C_MachNumber, C_CabAltitude, C_VaporPresWater, C_SatVaporPresWater, C_SatVaporPresIce, C_MixingRatio, C_RelHumidity, H2O(DLH), NO, NOy, NO2_NCAR, O3_ppbv, Carbon_Monoxide_mixing_ratio, Methane_mixing_ratio, Carbon_dioxide_mixing_ratio, NO2_LIF, PNs_TD-LIF, ANs_TD-LIF, HNO3_TD-LIF, CH2O_DFGAS, Abs470tot, Abs532tot, Abs660tot, nAPS_total, sAPS_total, vAPS_total, nAPS_fine(Daero_lt_1.0um), sAPS_fine(Daero_lt_1.0um), vAPS_fine(Daero_lt_1.0um), nAPS_coarse(Daero_ge_1.0um), sAPS_coarse(Daero_ge_1.0um), vAPS_coarse(Daero_ge_1.0um), BlackCarbonMassConcentration, CN>3nm, CN>10nm, nonvolCN>10nm, Scat450tot, Scat550tot, Scat700tot, Scat450sub, Scat550sub, Scat700sub, nUHSAS(60-1000_nm), sUHSAS(60-1000_nm), vUHSAS(60-1000_nm), nLAS(0.9-7.7_um), sLAS(0.9-7.7_um), vLAS(0.9-7.7_um), Angstrom_Exponent_of_Scattering_at_450and700nm, Angstrom_Exponent_of_Scattering_at_450and550nm, Angstrom_Exponent_of_Absorption_at_450and700nm, Angstrom_Exponent_of_Absorption_at_450and550nm, SingleScatteringAlbedo_at_450nm, SingleScatteringAlbedo_at_550nm, SingleScatteringAlbedo_at_700nm, SingleScatteringAlbedo_at_550nm(ambient), nSMPS, sSMPS, vSMPS, Acetaldehyde, Acetonitrile, Acetone, Methanol, Toluene, Isoprene, MVK_MAC, Xylenes_ETB, Acetic_acid, Monoterpenes, RHamb, RHdry, RHwet, SCdry, SCwet, gamma, fRH_80_20, SCamb, EXTamb532, EXTdry532, SCamb532, SCdry532, ABSdry532, Chloride, Nitrite, Nitrate, Sulfate, Sodium, Ammonium, Potassium, Magnesium, Calcium, Water-Soluble_Organic_Carbon_Mass, discoveraq-SiteFlag-1sec, ProfileSequenceNum
209.60079,51908, 209, 130002.00, 13.000000, 9.388130778, 37.936733, 284.536545, -0.03989832049, 1019.03, 302.0799939, 300.4573479, 0, 38.83436106, -999999, -999999, 68.27133241, 301.6499939, 70.78179961, 212.01, 212.7, 0.47, 0.44, 0.029, 68.77033921, -0.01392936017, 295.1499939, 1019.528127, 319.0499939, 304.3899939, 0.000598, 0.007919, 0.000161239202, 27.516, 300.467, 0.195, -0.0519494116, 26.5538, 40.0809, 52.7245, 16.64112, 66.25, 24926.9, -999999, -999999, -999999, -999999, 100.0, -999999, -999999, -999999, -999999, -999999, -999999, -999999, -999999, -999999, -999999, -999999, -999999, -999999, -999999, -999999, -999999, -999999, -999999, -999999, -999999, -999999, -999999, -999999, -999999, -999999, -999999, -999999, -999999, -999999, 18.79, 0.25, 0, 0, 0, 0, -999999, -999999, -999999, -999999, -999999, -999999, -999999, -999999, 43.26, 4.78, 0.19, -999999, -999999, -999999, -999999, -999999, -999999, -999999, -999999, -999999, -999999, 66.25, -999999, -999999, -999999, -999999, -999999, -999999, -999999, -999999, -999999, -999999, -999999, -999999, -999999, -999999, -999999, -999999, -999999, -999999, -999999, -999999, -999999, -999999, 0, 0
from verdi.
@lizadams The Latitude and Logitude are 37.936733 and 284.536545, respectively.
from verdi.
from verdi.
Another thought, it is probably good to use the pressure levels (or at least as an option). In general, the pressure levels should be in both the observations and model outputs.
from verdi.
Attaching a gzipped version of the file where I extracted JDAY 209 or 2011209. In case it helps troubleshoot this issue.
discoveraq-mrg01-p3b_merge_20110728_R3_thru20110728.ict.gz
from verdi.
@dkang2 Is there a formula to convert pressure levels to layer height? I did find this document for a method to compare satellite data to CMAQ output. https://www.ldeo.columbia.edu/~amfiore/haqast_TT_files/HAQAST_SIP_TT_GT_Technical_Guidance_Comparison_of_CMAQ_to_OMI-V1.0.pdf
from verdi.
@lizadams There are some algorithms to convert pressure levels to layer height. It can be interpolated using the bottom and top pressure values and the sigma values, p=sigma*(psfc-ptop)+ptop to find the bottom and top pressures at each layer. It becomes more complicated when using the hybrid vertical structure. However, I think that any interpolation should be good enough to make the comparisons, especially the aircraft observations are mainly in the lower to middle troposphere. In the LTNG_DEFN.F in CMAQ code, the lightning NO is matched to the model layer in this way (find the BOTTOM and TOP variables). The most simple and accurate method is still using the ZF variable.
from verdi.
Reference
Agree with Daiwen's comments.
Yadong
from verdi.
Tested VERDI 2.1.4 20230224 builds on Atmos. Found that we still have timesteps matching issue between model data and aircraft data for some aircraft data files. When used “discoveraq-UMDAircraft_UMD-AIRCRAFT_20110728_R4_L1.ict” to overlay on model data “2011ek_cb6cmaq5.2_wrf3.8_zero_lnox.CONC.20110728”, the timesteps matching worked ok. But when switched to use another aircraft file "discoveraq-mrg01-p3b_merge_20110701_R3_thru20110729.ict", I got the following error message:
from verdi.
Retested the timesteps matching issue with VERDI 2.1.4 20230911 builds on Atmos. Found that we don't have error messages showed up when load the merged file such as "discoveraq-mrg01-p3b_merge_20110701_R3_thru20110729.ict", but no data points showed up in the map after we add them as observations.
Furthermore, there was an error message popped-up when used the aircraft data for one specific day such as “discoveraq-UMDAircraft_UMD-AIRCRAFT_20110728_R4_L1.ict” to overlay on model data “2011ek_cb6cmaq5.2_wrf3.8_zero_lnox.CONC.20110728”.
from verdi.
Retested the timesteps matching issue with VERDI 2.1.4 20230924 builds on Atmos. Found that this issue has improved and but the timesteps matching is not consistent with the ones we saw in our previous tests.
When use “discoveraq-UMDAircraft_UMD-AIRCRAFT_20110728_R4_L1.ict” to overlay on model data “2011ek_cb6cmaq5.2_wrf3.8_zero_lnox.CONC.20110728”. We found that only two hours (T14 and T15 UTC have O3 observations)
But our previous testing results showed that we have three hours (T13, T14, T15 UTC) with O3 observations, I also checked the csv files extracted from the *.ict file and confirmed that we do have several data points within T13.
DaiWen,
Can you double-check if “discoveraq-UMDAircraft_UMD-AIRCRAFT_20110728_R4_L1.ict” contains O3 observations at T13 UTC?
Thanks,
Yadong
from verdi.
Continued testing the timesteps matching issue with VERDI 2.1.4 20230924 builds on Atmos.
When use the merged aircraft data “discoveraq-mrgLARGE-pilsIC-p3b_merge_20110701_R3_thru20110729.ict” to overlay on model data “2011ek_cb6cmaq5.2_wrf3.8_zero_lnox.CONC.20110728”. Found that the timesteps when observations showed up in the map are not consistent with what we see from the extracted data in CSV file.
In VERDI, we can see observational data points appeared at 2011-07-28 T00 UTC through 2011-07-28 T13 UTC.
from verdi.
Related Issues (20)
- Obs overlays not working in VERDI 2.1.4 20221109 version HOT 19
- Saving/opening VERDI project HOT 14
- The purpose of the "Domain" window in the Datasets panel HOT 8
- Failed to load GOES-16/-17 GLM datasets (in NetCDF format) in VERDI 20230425 builds HOT 7
- save tileplot as shapefile from gui HOT 8
- Opening MPAS mesh file with an error HOT 12
- Incorrect values for the probed grid cells after using "maximum_8hour_mean" stats in a tile plot HOT 6
- Support sub-domain on a Vertical Cross Section Plot HOT 9
- Incorrect Map Projection for Regional MPAS Mesh HOT 3
- Add ability to read MPAS "cdf5"-formatted output HOT 3
- High Priority: Update Java in VERDI to Java SE 17.0.8 or greater HOT 6
- create ascii file from tileplot variable from the command line and obtain projection information for GIS from shapefile with same name
- whitespace after configFile=filename in batch script causes verdi.sh -batch to ignore the config file HOT 4
- No loaders are available for the selected file HOT 6
- Visualize differences between time-independent model files HOT 1
- Reading data from one grid and adding the value of a cell to another grid
- Reading netCDF files with unconventional dimension names HOT 5
- Tile plot for 3D grid data with layer mean or sum quantities shown in 2D space HOT 11
- batch script does not turn off column/row labels and legend as specified in the configuration file HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from verdi.