usepa / amet Goto Github PK
View Code? Open in Web Editor NEWCode base for the U.S. EPA’s Atmospheric Model Evaluation Tool (AMET)
Home Page: https://www.epa.gov/CMAQ/atmospheric-model-evaluation-tool
Code base for the U.S. EPA’s Atmospheric Model Evaluation Tool (AMET)
Home Page: https://www.epa.gov/CMAQ/atmospheric-model-evaluation-tool
Add $AMETBASE/obs/site_metadata_files to the distribution rep.
Minor change to AQ_Soccerplot.R to work with the new credentials:
Change root_login and root_pass to amet_login and amet_pass in the following two lines:
mysql <- list(login=root_login, passwd=root_pass, server=mysql_server, dbase=dbase, maxrec=1000000)
con <- dbConnect(MySQL(),user=root_login,password=root_pass,dbname=dbase,host=mysql_server)
The setting of network_names to NULL in the script Network.input (line 41) breaks the batch scripts. The Network.input script sets the network names based on if (inc_improve=="y")
type conditionals. These types of conditionals don't get set in the batch mode of the scripts and so initializing these vectors to NULL breaks the batch scripts because they never get set back to actual network names after the Network.input script is sourced. The offending "source" command that unsets these variables happens at line 1174 of R_analysis_code/AQ_Misc_Functions.R.
Suggested fix is to wrap these lines in Network.input in a conditional to ignore them if the scripts are in batch mode:
if ( batch != 'y' ){
network_names <- NULL
network_label <- NULL
}
Hi,
Is there an option to run AMET without having a MySQL connection, maybe by somehow using SQLite when compiling?
If not, any plans in the future to implement this?
Thanks,
Lisa
The documentation for v1.3 (Section 6.2) refers to a combine subdirectory under $AMETBASE/scripts_db/aqExample. Is this directory missing in the installation or should I omit this reference in the documentation?
Add a directory of all example plots to the user's guide. Reference these examples in the text of the guide.
According to the AMET_Install_Guide_v14b under the section Install AMET Source Code and Tier 3 Software there should be a src directory containing the programs bldoverlay, sitecmp, and sitecmp_dailyo3. The src directory appears to be missing in the 14b release. There is a directory called tools_src which seems to have the source for these programs and the config.amet file. However, they are missing any make files to build bldoverlay, sitecmp, and sitecmp_dailyo3.
Currently there is an issue with the AQ_Stacked_Barplot_AE6.R script that results in incorrect bars being plotted. This is likely due to an error in the computation of PM_other in the script and the aggregation function being applied. Users should not use this script until the issue is resolved.
Users are advised to use the other stacked bar plotting scripts available, namely the AQ_Stacked_Barplot_AE6_ggplot.R and AQ_Stacked_Barplot_AE6_plotly.R scripts. These scripts appear to be functioning correctly.
This issue will be updated when the problem with the script is fixed.
There is an error (since fixed) in the setup of the AQ site_metadata table that will prevent the site metadata for each network from loading properly. There is a missing column definition, namely "co_network", that when missing causes the MySQL queries to fail, and hence no metadata being loaded in the site_metadata table. This will then prevent database queries from functioning correctly. This error does not affect results when not using the database for AQ analysis.
This issue was fixed by making the simple change below (in bold) to the AQ_setup_dbase.R script.
aq_new_3 <- "create table site_metadata (stat_id varchar(25) UNIQUE KEY, num_stat_id int(10), stat_name varchar(100), network varchar(15), co_network varchar(200), state varchar(4), city varchar(50), start_date varchar(20), end_date varchar(20), lat double, lon double, elevation int(10), landuse varchar(50), loc_setting varchar(50), county varchar(100), country varchar(100), near_road varchar(10), timezone varchar(20), GMT_Offset double, NLCD2011_Imperv_Surf_Frac double,NLCD2011_Imperv_Surf_Loc_Setting varchar(100),NLCD2006_Imperv_Surf_Frac double, NLCD2006_Imperv_Surf_Loc_Setting varchar(100), NLCD2001_Imperv_Surf_Frac double, NLCD2001_Imperv_Surf_Loc_Setting varchar(100))"
I've fixed the code with the above change. If you downloaded the AMET code prior to 11/6/2019, you should either download the full AMET code again, download and replace your existing AQ_setup_dbase.R script in the R_db_code/ directory of your AMET installation, or edit your existing AQ_setup_dbase.R script with the change noted above.
Note that if you've already run the AMET setup, you'll need to either manually delete the "site_metadata" table from your AMET database so that it can be recreated without the column missing, or delete your entire AMET setup and start fresh using the script provided with the AMET code. I would recommend that if you've already done analysis and loaded data into the database that you attempt to manually delete "site_metadata" table and have it recreated automatically when you run the aqExample_post_only.csh script.
A third option would be to simply edit your existing "site_metadata" table in the MySQL database to add the missing column manually. This can be accomplished by logging into MySQL with your username and password, switching to your AMET database, and then editing the existing table. An example is given below.
mysql -h mysql_host_name -u user_name -p
enter password
use AMET_DATABASE;
alter table site_metadata add column co_network varchar(200) after network;
exit
You should then re-run your aqExample_post_only.csh script with the reload_metadata flag set to T. That should then properly load your site metadata.
Wyat
The update mode for the AQ database scripts doesn't appear to work as expected. Once a project has been created, the units table does not get updated correctly. It actually looks like it's being clobbered (networks deleted) when you try to update the table.
When I grep on the log from the aqProject.csh script, run in update mode, I see this message (check to see which networks are being loaded into the db:
Finshed populating database for network IMPROVE
Finshed populating database for network CSN
Finshed populating database for network CASTNET
Finshed populating database for network CASTNET_Hourly
Finshed populating database for network CASTNET_Daily
Finshed populating database for network CASTNET_Drydep
Finshed populating database for network NADP
Finshed populating database for network MDN
Finshed populating database for network AMON
When I query the database, I'm only seeing that the last network processed exists in the units table:
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| amet_example |
| amet_iwdw |
| mysql |
| performance_schema |
+--------------------+
5 rows in set (0.00 sec)
mysql> use amet_iwdw ;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
mysql> show tables;
+----------------------+
| Tables_in_amet_iwdw |
+----------------------+
| WAQS_Base11b |
| WAQS_Base11b_CAMx12k |
| WAQS_Base11b_CAMx4k |
| aq_project_log |
| project_units |
| site_metadata |
+----------------------+
6 rows in set (0.00 sec)
mysql> select network from project_units;
+---------+
| network |
+---------+
| AMON |
+---------+
1 row in set (0.00 sec)
This is a problem because the units are used in the plotting scripts for various conditionals. If the units are NULL in the database, the plotting scripts don't work.
There are settings missing from AMET_batch.input that result in failure of the scatter density plots. Add these fields to the release script:
density_zlim<-NULL
num_dens_bins<-NULL
bin_by_mod<-""
The new run_AMET_batch.csh script is looking for:
setenv AMETRINPUT $AMETBASE/scripts_analysis/aqExample/input_files/AMET_batch.R
I'm only seeing AMET_batch.input in the input_files directory. It seems to work to use the *.input file. Is this just a type in the repo distribution?
Need to add Obs data for 2015/16 to the CMAS data clearinghouse. Currently we have data for 2001-2014: https://drive.google.com/drive/folders/0B2kjxCwKICxUZ3ZyNEVld2F6bHc
New unified met/aq scripts use a single database script to create projects (if needed) and load data. There is a way to include your login and password as variables in the script, otherwise, if these variables aren't set the database load script asks for your login info upon execution.
should we include login and password env vars as examples/optional in the script (e.x., aqProject.csh) to bypass the interactive mode?
how do these variables connect to the amet-config.R scripts? Is there a way to reference that centralized configuration script from these database loading scripts?
The documentation states that the numerator and denominator should be summed separately to calculate fractional bias and error.
However, the code for DomainStats in AQ_Misc_Functions.R appears to calculate these metrics as the sum after the fraction is calculated for each observation/model pair. The code also multiplies the result by a 100 to report as a percentage rather than a fraction.
https://github.com/USEPA/AMET/blob/1.5/R_analysis_code/AQ_Misc_Functions.R#L652
fb <- signif(c(fb,(sum((data.df$mod_val-data.df$ob_val)/((data.df$mod_val+data.df$ob_val)/2)))/length(data.df$stat_id))*100,3)
fe <- signif(c(fe,(sum(abs(data.df$mod_val-data.df$ob_val)/((data.df$mod_val+data.df$ob_val/2)))/length(data.df$stat_id))*100,3)
Am I correct that this is a discrepancy, and can this be updated to be consistent with the correct approach?
AMET result in a nebulous error when you try to plot a species from a network that doesn't include said species. The error may look something like this:
Error in `$<-.data.frame`(`*tmp*`, "good_ob", value = 0) :
replacement has 1 row, data has 0
Calls: Average -> $<- -> $<-.data.frame
Execution halted
The system would be much more user friendly if (1) there was a sanity check on the species (AMET_AQSPECIES) and network (e.g. AMET_CSN) pairs; and (2) a table (maybe a new Appendix in the UG) that shows what species are available for each network. When the check is written for (1), there would be an informative error that says something like "AMET_AQSPECIES O3_8hrmax is not available in the network AMET_CSN; please refer to Table XXX in the AMET user's guide to see the list of species available for each network in the AMET database."
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.