jgcri / ceds Goto Github PK

Hello,

The wiki says that the "IEA energy statistics database needs to be purchased from the IEA". I am attempting to figure out how expensive it is, however I'm not sure which database this refers to. I have found this one: https://webstore.iea.org/world-energy-statistics-2018, however it says that "this product is no longer available". Any help would be appreciated! Thanks!

The most recent GCP 2021 shows 2019 CO2 emissions at 10017 MtC (fossil fuels only, no land use included!). Looking at the most recent version of CEDS for 2019 yields 36040300.7 (value from summing over all rows) / 3.664 (CO2 => C) / 1000 (units) = 9836 MtC. This is certainly ballpark but I am curious if the devs have additional insight here. GCP is broken down into: Coal | Oil | Gas | Cement.emission | Flaring | Other. This is of course different from CEDS but I was expecting the final values to be much closer, the goals ought to be closer. And my assumption is that CEDS does not include land use emissions.

The filenames and titles of the data releases on Zenodo (https://zenodo.org/doi/10.5281/zenodo.3606752) are a bit confusing and I believe some might just be due to copy-and-paste errors.

The titles of the last two versions seem not to be correct, they should be 2021-04-21 I think:

The notes of these two releases (https://zenodo.org/records/4737769 and https://zenodo.org/records/4741285) refer to May 5 and May 6 versions.

This seems otherwise only be referenced in the JSON metadata in the created field.

Below excerpts from the metadata for https://zenodo.org/records/4741285:

"created": "2021-05-06T22:01:09.285650+00:00",
  
"publication_date": "2021-04-06",

"title": "CEDS v_2021_04_21 Release Emission Data",
"version": "v_2021_02_05"

"updated": "2021-05-07T01:48:09.552962+00:00",

I think the version field is a copy-and-paste error, for the others I'm not sure.

Maybe you can fix (some of) these, I believe metadata can be edited on Zenodo even after publication (https://help.zenodo.org/docs/deposit/manage-records/).

In any case writing this down here might help others who are also confused about the versions and releases.

USA NMVOC emissions from 1B2_Fugitive-petr-and-gas and other fugitive sectors have a discontinuity between 1970 and 1975. This has been fixed for future releases.

USA off road emissions are underestimated in of v_2020_9_11 (particularly for CO) due to a mismatch between sector definitions between the energy and emissions data. We are working to correct this.

Combustion emissions becomes zero in earlier years for several countries that have inconsistent temporal coverage in the IEA energy data. These include: Sint Maarten, Suriname, Cambodia, Mongolia, Palau, Botswana, Namibia, and Niger. Some of these instances, where alternative data sources are available, will be corrected in the next release.

The current CEDS gridded data exhibits seasonality for industrial emissions that is likely too high. The source of this seasonality information was the ECLIPSE V5a inventory. Future CEDS versions will likely have lower assumed seasonality for industrial emissions.

Hello CEDS team,

the Readme in CEDS v_2021_02_05 Release Emission Data (https://doi.org/10.5281/zenodo.4509372) states:

Note that country totals in these summary files do not include international shipping or aircraft emissions. International shipping and aircraft emissions are reported under the "global" iso.

However, it seems that domestic aviation (1A3aii_Domestic-aviation) is also included under the "global" code.

select em, country, sector, X2015, X2019 
from 'CEDS_v2021-02-05_emissions/CO2_CEDS_emissions_by_sector_country_2021_02_05.csv' 
where country = 'global' and X2015 > 0;

┌─────────┬─────────┬──────────────────────────────┬──────────────────┬──────────────────┐
│   em    │ country │            sector            │      X2015       │      X2019       │
│ varchar │ varchar │           varchar            │      double      │      double      │
├─────────┼─────────┼──────────────────────────────┼──────────────────┼──────────────────┤
│ CO2     │ global  │ 1A3ai_International-aviation │ 521158.412509509 │ 579694.569865003 │
│ CO2     │ global  │ 1A3aii_Domestic-aviation     │ 333777.180523926 │  387323.06918934 │
│ CO2     │ global  │ 1A3di_International-shipping │ 802392.812086916 │ 857915.691810566 │
└─────────┴─────────┴──────────────────────────────┴──────────────────┴──────────────────┘

There is no data for 1A3aii_Domestic-aviation under the other isos:

select * from 'CEDS_v2021-02-05_emissions/CO2_CEDS_emissions_by_sector_country_2021_02_05.csv' 
where country != 'global' and sector = '1A3aii_Domestic-aviation';

┌─────────┬─────────┬─────────┬─────────┬────────┬────────┬────────┬────────┬────────┬───┬────────┬────────┬────────┬────────┬────────┬────────┬────────┬────────┐
│   em    │ country │ sector  │  units  │ X1750  │ X1751  │ X1752  │ X1753  │ X1754  │ … │ X2012  │ X2013  │ X2014  │ X2015  │ X2016  │ X2017  │ X2018  │ X2019  │
│ varchar │ varchar │ varchar │ varchar │ double │ double │ double │ double │ double │   │ double │ double │ double │ double │ double │ double │ double │ double │
├────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│                                                                             0 rows                                                                             │
└────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

For domestic navigation data is reported for the individual countries.

select * from 'CEDS_v2021-02-05_emissions/CO2_CEDS_emissions_by_sector_country_2021_02_05.csv' 
where country != 'global' and sector like '%navigation%';

┌─────────┬─────────┬──────────────────────┬─────────┬────────┬───┬────────────────────┬────────────────────┬───────────────────┬───────────────────┐
│   em    │ country │        sector        │  units  │ X1750  │ … │       X2016        │       X2017        │       X2018       │       X2019       │
│ varchar │ varchar │       varchar        │ varchar │ double │   │       double       │       double       │      double       │      double       │
├─────────┼─────────┼──────────────────────┼─────────┼────────┼───┼────────────────────┼────────────────────┼───────────────────┼───────────────────┤
│ CO2     │ abw     │ 1A3dii_Domestic-na…  │ ktCO2   │    0.0 │ … │   1.38446052728173 │   1.38446052728173 │  1.42765714815472 │  1.38929773657336 │
│ CO2     │ afg     │ 1A3dii_Domestic-na…  │ ktCO2   │    0.0 │ … │   263.568424417394 │   274.548492398693 │  286.682211432831 │  305.237378100436 │
│ CO2     │ ago     │ 1A3dii_Domestic-na…  │ ktCO2   │    0.0 │ … │         55.2611808 │         39.9108528 │  41.9928360831718 │  43.6637264790247 │
│ CO2     │ alb     │ 1A3dii_Domestic-na…  │ ktCO2   │    0.0 │ … │   101.369890918464 │   107.524453648079 │  107.340930162444 │  108.797904541088 │
│ CO2     │ are     │ 1A3dii_Domestic-na…  │ ktCO2   │    0.0 │ … │                0.0 │                0.0 │               0.0 │               0.0 │
│ CO2     │ arg     │ 1A3dii_Domestic-na…  │ ktCO2   │    0.0 │ … │   1113.32306612885 │   925.514704820981 │  882.383940094613 │  868.028704426801 │
│ CO2     │ arm     │ 1A3dii_Domestic-na…  │ ktCO2   │    0.0 │ … │                0.0 │                0.0 │               0.0 │               0.0 │
│ CO2     │ asm     │ 1A3dii_Domestic-na…  │ ktCO2   │    0.0 │ … │  0.209548166077035 │  0.209548166077035 │ 0.216086288693238 │ 0.210280312870691 │
│ CO2     │ atg     │ 1A3dii_Domestic-na…  │ ktCO2   │    0.0 │ … │  0.291465374164575 │  0.291465374164575 │ 0.300559399611521 │ 0.292483734015446 │
│ CO2     │ aus     │ 1A3dii_Domestic-na…  │ ktCO2   │    0.0 │ … │    1706.0042246903 │   1715.17525419925 │  1761.89110594095 │  1748.73585329427 │
...

Is this a bug or simplification due to constraints of source data? In any case it might be helpful to clarify this in the Readme.

I have also checked this with the 2021-04-21 release (where the Readme file was missing, https://doi.org/10.5281/zenodo.4741285).

Hello.

I have successfully been working with files from the PNNL DataHub of individual VOCs. However, downloading a particular file:
VOC17-other-arom-em-speciated-VOC-anthro_input4MIPs_emissions_CMIP_CEDS-2021-04-21-supplemental-data_gn_185001-189912.nc

(i.e. https://g-83fdd0.1beed.03c0.data.globus.org/CEDS/CEDS_gridded_data_2021-04-21/data/VOC-speciated/VOC17-other/individual_files/VOC17-other-arom-em-speciated-VOC-anthro_input4MIPs_emissions_CMIP_CEDS-2021-04-21-supplemental-data_gn_185001-189912.nc )

... seems to be significantly smaller size than e.g. the same species for 1750-1799 (not just on my machine but the size noted on PNNL DataHub). Then ncdump gives errors like below. Any chance someone can check whether this file is corrupted?

Best, Greg

globus_downloads/2021-04-21> ncdump -h VOC17-other-arom-em-speciated-VOC-anthro_input4MIPs_emissions_CMIP_CEDS-2021-04-21-supplemental-data_gn_185001-189912.nc | more
HDF5-DIAG: Error detected in HDF5 (1.10.4) thread 46912530532224:
  #000: H5F.c line 509 in H5Fopen(): unable to open file
    major: File accessibilty
    minor: Unable to open file
  #001: H5Fint.c line 1400 in H5F__open(): unable to open file
    major: File accessibilty
    minor: Unable to open file
  #002: H5Fint.c line 1700 in H5F_open(): unable to read superblock
    major: File accessibilty
    minor: Read failed
  #003: H5Fsuper.c line 623 in H5F__super_read(): truncated file: eof = 162138234, sblock->base_addr = 0, stored_eof = 559945761
    major: File accessibilty
    minor: File has been truncated
ncdump: VOC17-other-arom-em-speciated-VOC-anthro_input4MIPs_emissions_CMIP_CEDS-2021-04-21-supplemental-data_gn_185001-189912.nc: NetCDF: HDF error

I have cloned the repo and now have access to all folders in CEDS, and am running Module G before I edit to extract data points (instead of gridded values).

The fourth line of code: initialize("G1.1.grid_bulk_emissions.R", log_msg, headers)) gives the following error:

initialize( "G1.1.grid_bulk_emissions.R", log_msg, headers )initialize( "G1.1.grid_bulk_emissions.R", log_msg, headers )Error in file(file, if (append) "a" else "w") : cannot open the connection In addition: Warning message: In file(file, if (append) "a" else "w") : cannot open file '../code/parameters//../../logs/G1.1.grid_bulk_emissions.R.log': No such file or directory Called from: file(file, if (append) "a" else "w")

@ssmithClimate has been my main point of contact and very helpful!

It seems the new aircraft data files (e.g., SO2-em-AIR-anthro_input4MIPs_emissions_CMIP_CEDS-2017-08-30_gn_185101-189912.nc; md5sum: c8d79ff6e213cbbfb620c6e551385003) contain non-zero values for aircraft emissions before 1920. So far as I have checked, the aircraft emissions of other species (BC, OC, etc) are zero until 1920.

The default CO2 emission coefficient used for heavy oil is too high (4.321 kg/kg) instead of ~ 3.092 kg/kg. Further, some country specific emission coefficients for heavy_oil are also incorrect. These will be corrected in the next release.

Hello, thanks for providing CEDS!

Could it be that the unit in the CSV files for CO2, available in the corrected supplementary at https://www.geosci-model-dev.net/11/369/2018/gmd-11-369-2018-assets.html is wrong?

It looks off by the usual 3.667, so seems to be ktCO2 instead of ktC.

Scaled with that it matches up the Global Carbon Budget's GtC nicely.

The link to the CEDS homepage in the sidebar on the right is broken: http://www.globalchange.umd.edu/ceds/

It redirects to https://www.pnnl.gov/projects/jgcri/ceds/

This link works: https://www.pnnl.gov/projects/ceds

Thor et al. have identified an inconsistency in latitudinal distribution of gridded aviation emissions present in current CEDS gridded data.

This is documented here:

https://gmd.copernicus.org/preprints/gmd-2022-250/

I think the CH4 year range in the docx file Corrected Supplement Explanation.docx in the supplementary at https://www.geosci-model-dev.net/11/369/2018/gmd-11-369-2018-assets.html

should be changed from
(annual estimates 1750 – 2014)

to

(annual estimates 1970 – 2014)

Hi CEDS team,

Excellent work updating emissions to 2022. Very long anticipated - thanks.

In the Zenodo dataset at https://zenodo.org/records/10904361 there may be a typo. The dataset title is v_2023_04_01, but the data files are labelled v_2024_04_01.

The historical CO2 emission cdf files had an associated csv file to sanity check to global values. I cannot find similar files for the scenario files. Have I missed them or were they not made for the future emission files?

Hi,

Can anyone help with the following error, please?

When I try to create any of the emissions, the execution halts with the following message:

Error in disaggregate_country(original_data = A.IEA_others_bunkers, trend_data = cdiac_trend_bunkers,  : 
  in disaggregate_country(): "combined_iso" is not in "original_data" 
Execution halted

This happens after the 8th call to disaggregate_country(), which I think refers to line 740 of A1.3.IEA_downscale_ctry.R.

I'm not sure what the solution is here and appreciate any help.

Setup

• CEDS v2021_2_05
• R 3.6.3
• r_env 0.13.2
• IEA energy statistics 2020 edition

Many thanks,
Luke

The Australia_scaling_mapping.xlsx file currently groups Australia NPI emissions for 'Coal Mining' with CEDS fugitive emissions (1B1, 1B2, 1B2d), 'Metal Ore Mining' with CEDS metal production (2C, 1A2b, 1A2a) and 'Other Non-Metallic Mineral Mining and Quarrying' with CEDS non-metallic minerals (2A1, 2A2, 2A6, 1A2f, and 1A2g (Ind-Comb-mining-quarrying)). Based on the magnitude of the Australia NPI mining emissions (Coal + metal + non-metal), it appears that these NPI mining sectors should instead be mapped to the CEDS industry scaling sector, along with CEDS sector 1A2g (ind-comb-mining). Due to the magnitude of NPI coal mining emissions, the current mapping is resulting in a CEDS scaling factor of 100 for fugitive emissions in recent years. I believe that this is resulting in aggregate CEDS ENE emissions that are slightly too large and CEDS IND emissions that are too low (since mining emissions are currently included in the aggregate ENE sector, but not the IND sector).

In addition, the Australia NPI agricultural emissions do not appear to include soil emissions from fertilizer application. CEDS soil emissions are currently scaled to all other reported NPI AGR emissions (livestock and grain farming), which will cause the final CEDS soil emissions over Australia to be too low.

SO2 emissions in the western US are too high compared to US EPA state level emissions data. This is a consequence of our use of EDGAR emissions grids which use global proxy databases that evidently do not incorporate sub-regional information. This spatial allocation error is much larger than the uncertainty in emissions over the western US. (Over the Eastern US, the difference, is likely of comparable magnitude to emissions uncertainty.)

Hi, I am trying to run my first instance of CEDS,

will i get anywhere without IEA data? i am not in a position to purchase proprietary data, will CEDS offer any results at all?

The fail appears to happen on the second file read with no metadata, for example see issue 24

For now the fix is to add:
, meta = FALSE )

to the end of each readData commands to turn off its attempt to read metadata.

As discussed in the Hoesly et al journal paper, BC (and likely OC) emissions are overestimated in the current dataset. This difference appears to be larger than emissions uncertainty. Further assessment is warranted.

In the recent release, there is a discontinuity in Switzerland's total NH3 emissions in 1990, dropping from 75 to 65 kt NH3 from 1989 to 1990. This discontinuity is also present in the EMEP inventory (agriculture and other agriculture sectors).

Thanks to Stephan Buhofer for bringing to our attention.

BC emissions from the industrial sector in the Netherlands show a large increase in 2007, particularly for BC. This is due to a increase in combustion of natural gas liquids in the IEA energy data. natural gas liquids was mapped to the aggregate diesel fuel category in the current sion of CEDS, resulting in an increase in BC emissions, which is not realistic. This mapping will be changed in future versions of the dataset, which will remove this increase.

The CH4 extension file in https://www.geosci-model-dev.net/11/369/2018/gmd-11-369-2018-assets.html doesn't have a unit column. It's the same as the CH4 file but might be worth including if you do an update anyways (#9).

Related question, are the decadal estimates to be understood as estimates of emissions in the respective years?

In NH3_CEDS_emissions_by_sector_country_v2016_07_26.csv there seems to be an unnecessary (tiny) value:

75:"NH3","global","5D_Wastewater-handling","ktNH3",0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,1.38613861386139e-05,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0

Hi,

i was trying to understand the sector definitions and compared the definitions in

https://github.com/JGCRI/CEDS/blob/master/input/mappings/Master_Sector_Level_map.csv

with the sectors i find from the emissions by sector file from the 2021-04-21 release.

I found that there are some working_sector_v1 values which don't appear in the CSV files and vice versa.

In Master_Sector_Level_map but not found in CSVs:

'1A1bc_Other-feedstocks',
'1B2_Fugitive-petr-and-gas',
'2A6_Other-minerals',
'2D3_Chemical-products-manufacture-processing',
'2D3_Other-product-use',
'3F_Agricultural-residue-burning-on-fields',
'5C_Waste-incineration'

In CSVs:

'2Ax_Other-minerals',
'2D_Chemical-products-manufacture-processing',
'2D_Other-product-use',
'5C_Waste-combustion'

Some of these are pretty close obviously, but i can't find in the code base where these renamings happen, e.g.
https://github.com/JGCRI/CEDS/search?q=5C_Waste-combustion

Is working_sectors_v1 the right place for the sector names used in the CSVs?

Several modelers have requested CH4 emissions prior to 1970. CH4 emissions were released from 1970 forward because we have not processed historical data necessary to reliably extend CH4 emissions for key CH4 sectors (e.g., fossil fuel production, rice production, livestock statistics) before that point.

The supplement of the paper states in the PDF

© Author(s) 2018. This work is distributed under
the Creative Commons Attribution 3.0 License.

and below

The copyright of individual parts of the supplement might differ from the CC BY 3.0 License.

The input datasets at https://doi.org/10.22033/ESGF/input4MIPs.1241 are under

Creative Commons Attribution 4.0 International License (CC BY-SA 4.0)

What is license for the CSV files in the supplement? They are derived from the gridded CMIP data, arent'y they?

Due to a data error, SO2 emissions in the US are overestimated in all current data releases from 1961 to 1969. The overestimate averages 22% over this period. The previous and corrected time series are shown in the Known Issues section of the wiki. This difference, avenged over the decade, is likely somewhat larger than the uncertainty in emissions over that time scale.

It appears that NOx emissions from agriculture (fertilizer and animal manure) were not included in the USA and Canada emission inventories used to produced the CEDS emissions data. As a result, agricultural NOx emissions in the CEDS data to date are too low. (Thanks to Maria Val Martin for pointing this out.)

Hi - the PNNL DataHub does not appear to make data available for download (https://release.datahub.pnnl.gov/released_data/1677) , I cannot get beyond the in-progress status for "File Retrieval" (Chrome, edge, Firefox all the same - 3 spinning dots constantly, have left it for many hours). thanks

This should not be included in CEDS and can result in some double-counting. This will be fixed in the next release.