Motivation

The GDAS data referred to in #29 uses Template 5.3/7.3, but it sometimes cannot be decoded with current decoder:

% ./target/debug/gribber decode nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gdas.20230111/12/atmos/gdas.t12z.pgrb2.0p25.f000 0.0
error: ComplexPackingDecodeError(
    NotSupported,
)

Detailed Information on the Data

• URL is documented in #29

Additional Context

Octet 49 (numberOfOctetsExtraDescriptors in ecCodes) in some submessages of this data is 3, although this crate currently supports only 2. It is the reason of this error. That octet in other submessages is 2.

Simple Description on the Bug

Lat/lon computation of grid points fails for testdata/CMC_glb_TMP_ISBL_1_latlon.24x.24_2021051800_P000.grib2.

It is because longitude range [180000000, 179760000] has a final value smaller than the initial value, although it is expected to increase from the scanning mode 0b01000000.

Steps to Reproduce

Test code using a grid definition extracted from that data is like this:

    #[test]
    fn cmc_gdps_lat_lon_grid() {
        let grid = LatLonGridDefinition::new(
            1500,
            751,
            -90000000,
            180000000,
            90000000,
            179760000,
            ScanningMode(0b01000000),
        );
        assert!(grid.latlons().is_ok())
    }

Expected Behavior

Test passes.

Actual Behavior

Test fails.

Additional Context

No response

Simple Description on the Bug

An assertion fails when calling the .dispatch() function on a Grib2SubmessageDecoder that is attempting to read a GRIB2 message of the 5.3/7.3 format. It fails with the error

... 
panicked at 'assertion failed: `(left == right)`
    left: `[-45, 45]`,
    right: [6942, 6942]`', .../src/decoders/complex.rs:94:5
...

Steps to Reproduce

1. Download data from the GFS forecast site (e.g. https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gdas.20230111/12/atmos/gdas.t12z.pgrb2.0p25.f000, Note that this exact URL will become unavailable as the forecast data becomes historical data and will have to be modified accordingly.). I am using the global 0.25 degree forecast data.
2. Attempt to read and decode the values in the submessages.

// open file
let f = std::fs::File::open("test_grib_file").unwrap();
let rd = std::io::BufReader::new(f);
let gr = grib::from_reader(rd).unwrap();
// set up decoder
let (_, first_submessage) = gr.iter().next().unwrap();
let dc = Grib2SubmessageDecoder::from(first_submessage).unwrap();
let dv = dc.dispatch().unwrap(); // SHOULD PANIC HERE

Expected Behavior

Read the grid values from the file.

Actual Behavior

Assertion panic.

Additional Context

No response

Simple Description on the Bug

When attempting to read the next message based on starting reading a GRIB file from an offset, the code panics instead of parsing the values even though it can correctly read the section information.

Steps to Reproduce

Here is the test I ran with the test file found here. Attempting to use other offsets that are found in the GRIB index file (second value, delimited by :) will produce similar results, sometimes on the first message instead of the second. Starting from an offset of 0 allows the reader to read the file normally with no errors encountered.

#[test]
fn test_manual_offset_read_grib() {
    // offset to the second submessage
    let offset = 983_303;
    let mut f = File::open(GRIB_TEST_FILE).unwrap();
    f.seek(SeekFrom::Start(offset)).unwrap(); // TOGGLE COMMENTING THIS LINE TO GET ERROR
    let buf = BufReader::new(f);
    let gr = grib::from_reader(buf).unwrap();

    for (i, (_, lyr)) in gr.submessages().enumerate() {
        println!("{}", lyr.describe());

        let d = Grib2SubmessageDecoder::from(lyr).unwrap();
        let v = d.dispatch().unwrap(); // SHOULD RETURN AN ERROR ON THE SECOND SUBMESSAGE

        for i in v.take(5) {
            println!("{:?}", i);
        }

        println!("index: {}", i);
    }
}

Expected Behavior

The message offset-to and each following message are properly parsed.

Actual Behavior

When testing on the test file that I mentioned above, the decoder can properly handle all the submessages in the GRIB file. However, when attempting to start from an offset in the file greater than 0, the decoder returns a ComplexDecodingError for the second message.

Additional Context

No response

Simple Description on the Bug

When parsing GRIB data in the 5.3/7.3 format, sometimes there is an out-of-bounds error when calling .dispatch(). This may be related to #30, but I am unsure.

Steps to Reproduce

Using the same data as #29 (NOAA GFS data), I iterated through the submessages and attempted to parse and print a few values of each, along with the definition information about the messages.

// read grib file
let fname = "./test_files/test_grib_file";
let f = std::fs::File::open(fname).unwrap();
let rd = std::io::BufReader::new(f);
let gr = grib::from_reader(rd).unwrap();

// set up decoder
for lyr in gr.iter() {
    let (_, fm) = lyr;
    println!(
        "{}\n{:?}\n{:?}\n{:?}\n{:?}",
        fm.describe(),
        fm.indicator(),
        fm.grid_def(),
        fm.prod_def(),
        fm.grid_def()
    );
    let dc = Grib2SubmessageDecoder::from(fm).unwrap();
    match dc.dispatch() { // WILL PANIC ON SOME MESSAGES
        Ok(dv) => {
            println!("size hint: {:?}", dv.size_hint());

            dv.take(10).for_each(|v| {
                println!("{:?}", v);
            });
        }
        Err(e) => {
            eprintln!("\n\n{:?}\n\n", e);
        }
    };
}

Expected Behavior

Parse GRIB file normally.

Actual Behavior

Here is the backtrace along with the console output about the GRIB layer definitions. The source of the error is at src/decoders/complex.rs:36:14.

Grid:                                   Latitude/longitude
  Number of points:                     1038240
Product:                                Analysis or forecast at a horizontal level or in a horizontal layer at a point in time
  Parameter Category:                   Moisture
  Parameter:                            Relative humidity
  Generating Proceess:                  Forecast
  Forecast Time:                        12
  Forecast Time Unit:                   Hour
  1st Fixed Surface Type:               Isobaric surface
  1st Scale Factor:                     0
  1st Scaled Value:                     10
  2nd Fixed Surface Type:               code '255' is not implemented
  2nd Scale Factor:                     0
  2nd Scaled Value:                     0
Data Representation:                    Grid point data - complex packing and spatial differencing
  Number of represented values:         1038240

Indicator { discipline: 0, total_length: 983303 }
GridDefinition { payload: [0, 0, 15, 215, 160, 0, 0, 0, 0, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 160, 0, 0, 2, 209, 0, 0, 0, 0, 255, 255, 255, 255, 5, 93, 74, 128, 0, 0, 0, 0, 48, 133, 93, 74, 128, 21, 113, 89, 112, 0, 3, 208, 144, 0, 3, 208, 144, 0] }
ProdDefinition { payload: [0, 0, 0, 0, 1, 1, 2, 0, 96, 0, 0, 0, 1, 0, 0, 0, 12, 100, 0, 0, 0, 0, 10, 255, 0, 0, 0, 0, 0] }
GridDefinition { payload: [0, 0, 15, 215, 160, 0, 0, 0, 0, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 160, 0, 0, 2, 209, 0, 0, 0, 0, 255, 255, 255, 255, 5, 93, 74, 128, 0, 0, 0, 0, 48, 133, 93, 74, 128, 21, 113, 89, 112, 0, 3, 208, 144, 0, 3, 208, 144, 0] }
thread 'read_grib::read_grib_tests::test_read_grib_file' panicked at 'range end index 4 out of range for slice of length 3', /home/lafewessel/.cargo/git/checkouts/grib-rs-1a2632789d91bd49/44a1c1f/src/decoders/complex.rs:36:14
stack backtrace:
   0: rust_begin_unwind
             at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/std/src/panicking.rs:575:5
   1: core::panicking::panic_fmt
             at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/panicking.rs:65:14
   2: core::slice::index::slice_end_index_len_fail_rt
             at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/slice/index.rs:76:5
   3: core::slice::index::slice_end_index_len_fail
             at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/slice/index.rs:69:9
   4: <core::ops::range::Range<usize> as core::slice::index::SliceIndex<[T]>>::index
             at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/slice/index.rs:408:13
   5: core::slice::index::<impl core::ops::index::Index<I> for [T]>::index
             at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/slice/index.rs:18:9
   6: grib::decoders::complex::decode
             at /home/lafewessel/.cargo/git/checkouts/grib-rs-1a2632789d91bd49/44a1c1f/src/decoders/complex.rs:36:14
   7: grib::decoders::common::Grib2SubmessageDecoder::dispatch
             at /home/lafewessel/.cargo/git/checkouts/grib-rs-1a2632789d91bd49/44a1c1f/src/decoders/common.rs:121:67
   8: crate::read_grib::read_grib_tests::test_read_grib_file::test_impl
             at ./src/read_grib.rs:300:19
   9: crate::read_grib::read_grib_tests::test_read_grib_file
             at ./src/read_grib.rs:272:5
  10: crate::read_grib::read_grib_tests::test_read_grib_file::{{closure}}
             at ./src/read_grib.rs:272:5
  11: core::ops::function::FnOnce::call_once
             at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/ops/function.rs:251:5
  12: core::ops::function::FnOnce::call_once
             at /rustc/69f9c33d71c871fc16ac445211281c6e7a340943/library/core/src/ops/function.rs:251:5

Additional Context

Some of the other layers were unable to be parsed and returned a DecodeError(ComplexPackingDecodeError(NotSupported)), but I'm guessing that those errors are related to #30.

Here is a link to my Google Drive folder containing the GRIB file that I am currently testing with. link

Simple Description on the Bug

When iterating through submessages, the size indicated by submessage.indicator() is constant whereas iterating through the sections shows different sizes for each submessage.

Steps to Reproduce

fn test_read_grib_file() {
    let f = std::fs::File::open(GRIB_TEST_FILE).unwrap();
    let rd = std::io::BufReader::new(f);
    let gr = grib::from_reader(rd).unwrap();

    // NOTE the submessage sizes indicated by the Section0 sections
    for s in gr.sections() {
        println!("{:?}", s);
    }

    // set up decoder
    for lyr in gr.iter() {
        let ((i, j), fm) = lyr;
        println!(
            "{}, {}\n{}\n{:?}\n{:?}\n{:?}\n{:?}\n{:?}",
            i,
            j,
            fm.describe(),
            // NOTE the submessage sizes indicated by indicator()
            fm.indicator(),
            fm.grid_def(),
            fm.prod_def(),
            fm.grid_def(),
            fm.repr_def()
        );
    }
}

Expected Behavior

submessage.indicator() shows the proper size of the message.

Actual Behavior

A constant size is shown for all submessages.

Additional Context

I used the GRIB file found here.

Motivation

Currently, the crate (particularly list_surfaces.rs example) shows only MSLP surface in NOAA GFS file, while the file actually contains much more data (geopotential, temperature, wind etc. on both isobaric and height levels).

I believe GFS data is probably one of the most commonly used in the world. Thus, it may make this crate useful for more users. Also, it contains many fields that are present also in data from other NWP models. So support for GFS data will bring a support for multiple other datasets.

Detailed Information on the Data

Data is available at this URL:
https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.<date>/<run>/atmos/gfs.t<run>z.pgrb2.0p25.f<forecast_time>

There is also additional data in pgrb2b files, but they contain mostly data at additional levels, so supporting data from pgrb2 files will bring most of the full support of GFS data.

Additional Context

I am more than happy to help with achieveing the goal set by this issue, but I am not not yet familiar enough with this crate to know where to start. I wanted to create this issue to keep track of functionality I need.

I believe, it will be easier to resolve this issue by splitting it into several more specific issues (directly relating to the parts of code that need work). If you would like to create such issues, then I also will be interested in solving them.

Description

Thanks for the motivation to replace wgrib2.
I tried to install the repo with cargo. But unfortunately a build error occurs. It seems that some data is missing.

System Configuration:

• Fedora 33 - 5.11.7-200.fc33.x86_64
• rustc 1.50.0 (cb75ad5db 2021-02-10)
• cargo 1.50.0 (f04e7fab7 2021-02-04)

Steps to reproduce

cargo build
``´
The error also occurs with `cargo test`  and `cargo run` .

## Error message

(base) [daniel@localhost:~/dev/grib-rs]$ cargo build
Compiling grib-build v0.1.0 (/home/daniel/dev/grib-rs/gen)
Compiling grib v0.2.0 (/home/daniel/dev/grib-rs)
error: failed to run custom build command for grib v0.2.0 (/home/daniel/dev/grib-rs)

Caused by:
process didn't exit successfully: /home/daniel/dev/grib-rs/target/debug/build/grib-1c47537ab65b4df2/build-script-build (exit code: 101)
--- stderr
thread 'main' panicked at 'called Result::unwrap() on an Err value: Os { code: 2, kind: NotFound, message: "No such file or directory" }', gen/src/cct11.rs:10:31
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace

## Additional information

The report is related to #1 .

Thank you very much,

Daniel

Simple Description on the Bug

Iterator::size_hint()s implemented in this library return the same length after consuming items.

As per description in the API reference, the implementation of size_hint() should return the bounds on the remaining length of the iterator.

Steps to Reproduce

1. Check the length returned by size_hint()
2. Call next()
3. Check again the length

Expected Behavior

When consuming items, the returned length of size_hint() decreases.

Actual Behavior

Even after consuming items, the returned length of size_hint() does not change.

Additional Context

No response

Simple Description on the Bug

A for loop:

let grib2 = Grib2::<SeekableGrib2Reader<BufReader<File>>>::read_with_seekable(f)?;

for submessage in grib2.iter() {}

returns only the first message from the .grib file, instead of all messages that this file contains, even though all messages can be read with the crate.

Steps to Reproduce

The folder contains files from the ERA5 dataset (converted with ecCodes library), where _p# files are a submessages from the era5.grib copied into separate files:

user@user-Computer:~/grib-test$ ls -l
total 26388
-rw-rw-r-- 1 user user 13498296 lis  9 17:01 era5.grib
-rw-rw-r-- 1 user user  2076676 lis  9 17:09 era5_p1.grib
-rw-rw-r-- 1 user user  2076676 lis  9 17:09 era5_p2.grib
-rw-rw-r-- 1 user user  2076676 lis  9 17:09 era5_p3.grib
-rw-rw-r-- 1 user user  2076676 lis  9 17:09 era5_p4.grib
-rw-rw-r-- 1 user user  3114916 lis  9 17:10 era5_p5.grib
-rw-rw-r-- 1 user user  2076676 lis  9 17:10 era5_p6.grib
user@user-Computer:~/grib-test$ grib_ls -p "edition,shortName,gridType,packingType,level,typeOfLevel" era5.grib
era5.grib
edition      shortName    gridType     packingType  level        typeOfLevel  
2            10u          regular_ll   grid_simple  10           heightAboveGround 
2            10v          regular_ll   grid_simple  10           heightAboveGround 
2            2d           regular_ll   grid_simple  2            heightAboveGround 
2            2t           regular_ll   grid_simple  2            heightAboveGround 
2            z            regular_ll   grid_simple  0            surface     
2            sp           regular_ll   grid_simple  0            surface     
6 of 6 messages in era5.grib

6 of 6 total messages in 1 files
user@user-Computer:~/grib-test$ grib_ls -p "edition,shortName,gridType,packingType,level,typeOfLevel" era5_p1.grib
era5_p1.grib
edition      shortName    gridType     packingType  level        typeOfLevel  
2            10u          regular_ll   grid_simple  10           heightAboveGround 
1 of 1 messages in era5_p1.grib

1 of 1 total messages in 1 files
user@user-Computer:~/grib-test$ grib_ls -p "edition,shortName,gridType,packingType,level,typeOfLevel" era5_p2.grib
era5_p2.grib
edition      shortName    gridType     packingType  level        typeOfLevel  
2            10v          regular_ll   grid_simple  10           heightAboveGround 
1 of 1 messages in era5_p2.grib

1 of 1 total messages in 1 files
user@user-Computer:~/grib-test$ grib_ls -p "edition,shortName,gridType,packingType,level,typeOfLevel" era5_p3.grib
era5_p3.grib
edition      shortName    gridType     packingType  level        typeOfLevel  
2            2d           regular_ll   grid_simple  2            heightAboveGround 
1 of 1 messages in era5_p3.grib

1 of 1 total messages in 1 files
user@user-Computer:~/grib-test$ grib_ls -p "edition,shortName,gridType,packingType,level,typeOfLevel" era5_p4.grib
era5_p4.grib
edition      shortName    gridType     packingType  level        typeOfLevel  
2            2t           regular_ll   grid_simple  2            heightAboveGround 
1 of 1 messages in era5_p4.grib

1 of 1 total messages in 1 files
user@user-Computer:~/grib-test$ grib_ls -p "edition,shortName,gridType,packingType,level,typeOfLevel" era5_p5.grib
era5_p5.grib
edition      shortName    gridType     packingType  level        typeOfLevel  
2            z            regular_ll   grid_simple  0            surface     
1 of 1 messages in era5_p5.grib

1 of 1 total messages in 1 files
user@user-Computer:~/grib-test$ grib_ls -p "edition,shortName,gridType,packingType,level,typeOfLevel" era5_p6.grib
era5_p6.grib
edition      shortName    gridType     packingType  level        typeOfLevel  
2            sp           regular_ll   grid_simple  0            surface     
1 of 1 messages in era5_p6.grib

1 of 1 total messages in 1 files

Running this code (modified list_surfaces.rs example) in that folder produces below output:

Code:

use grib::codetables::{CodeTable4_2, Lookup};
use grib::context::Grib2;
use grib::reader::SeekableGrib2Reader;
use std::error::Error;
use std::fs::File;
use std::io::BufReader;
use std::path::Path;

fn main() -> Result<(), Box<dyn Error>> {
        println!("One file:");
        let path = Path::new("era5.grib");
        list_surfaces(path)?;

        println!("\nSplit files:");
        list_surfaces(Path::new("era5_p1.grib"))?;
        list_surfaces(Path::new("era5_p2.grib"))?;
        list_surfaces(Path::new("era5_p3.grib"))?;
        list_surfaces(Path::new("era5_p4.grib"))?;
        list_surfaces(Path::new("era5_p5.grib"))?;
        list_surfaces(Path::new("era5_p6.grib"))?;

        Ok(())
}

fn list_surfaces(path: &Path) -> Result<(), Box<dyn Error>> {
    // Open the input file in a normal way.
    let f = File::open(&path)?;
    let f = BufReader::new(f);
    let grib2 = Grib2::<SeekableGrib2Reader<BufReader<File>>>::read_with_seekable(f)?;

    for submessage in grib2.iter() {

        let discipline = submessage.indicator().discipline;
        let category = submessage.prod_def().parameter_category().unwrap();
        let parameter = submessage.prod_def().parameter_number().unwrap();
        let parameter = CodeTable4_2::new(discipline, category).lookup(usize::from(parameter));
        let forecast_time = submessage.prod_def().forecast_time().unwrap();
        let (first, _second) = submessage.prod_def().fixed_surfaces().unwrap();
        let elevation_level = first.value();

        println!(
            "{:<31} {:>14} {:>17}",
            parameter.to_string(),
            forecast_time.to_string(),
            elevation_level
        );
    }

    Ok(())
}

Output:

One file:
u-component of wind                      0 [h]                10

Split files:
u-component of wind                      0 [h]                10
v-component of wind                      0 [h]                10
Dewpoint temperature                     0 [h]                 2
Temperature                              0 [h]                 2
Geopotential                             0 [h]               NaN
Pressure                                 0 [h]               NaN

Clearly, the iterator does not return all messages in the file.

Expected Behavior

The SubMessageIterator should iterate over all messages present in the .grib file instead of returning only the first one.

Actual Behavior

SubMessageIterator .next() method returns only first message from the file and then returns None.

Additional Context

No response

Simple Description on the Bug

If the length is zero inside the next() method in FixedValueIterator we get a subtract with overflow.

Steps to Reproduce

This happened to me when processing grib data from the GFS

Expected Behavior

Not crashing on valid grib data.

Actual Behavior

Crashing on valid grib data.

Additional Context

The bug can be fixed by changing the next method in decoder/stream.rs to not subtract the length unless there is some value.

Going from this:

impl<T> Iterator for FixedValueIterator<T>
where
    T: Copy,
{
    type Item = T;

    fn next(&mut self) -> Option<Self::Item> {
        let val = if self.length > 0 { 
            Some(self.val)
        } else {
            None
        };
        self.length -= 1; // <--- Crash here
        val
    }
    
    // -- snip -- //
}

To this:

impl<T> Iterator for FixedValueIterator<T>
where
    T: Copy,
{
    type Item = T;

    fn next(&mut self) -> Option<Self::Item> {
        let val = if self.length > 0 {
            self.length -= 1;
            Some(self.val)
        } else {
            None
        };
        val
    }
    
    // -- snip -- //
}

I'll be sending in a PR for this shortly.

Simple Description on the Bug

I'm not sure if this is a broken data file or a bug.

The DWD ICON "Total precipitation accumulated since model start" first output file crashes with a LengthMismatch error, but I think it contains an all-zero data series.

Steps to Reproduce

Get the grib file from reprod.zip (GitHub doesn't support grib files directly, hence it's packed into a ZIP). Then run:

$ cargo run -- decode icon_global_icosahedral_single-level_2021112018_000_TOT_PREC.grib2 0

You'll need #14 to prevent a panic.

Expected Behavior

The data is supposed to be the "Total precipitation accumulated since model start". This is the first output file (at model start) from the DWD ICON model, so I THINK that it's probably an all-zero data series.

Actual Behavior

It crashes with DecodeError(SimplePackingDecodeError(LengthMismatch)).

Additional Context

Data taken from https://opendata.dwd.de/weather/nwp/icon/grib/18/tot_prec/. See https://www.dwd.de/SharedDocs/downloads/DE/modelldokumentationen/nwv/icon/icon_dbbeschr_aktuell.pdf?view=nasPublication&nn=495490 for a model description.

Simple Description on the Bug

There is a segmentation fault when reading JPEG2000 streams due to unsafe code. This is due to the image struct being dropped at the end of the function, so the pointer given to from_raw_parts is no longer valid.

I found this when trying to read data from the ECMWF provided files (namely the U/V components of wind)

Steps to Reproduce

1. Download some of the U/V component files from ECMWF

2. Run gribber decode <file.grib2.bin> 0

Expected Behavior

I get a nice array of values

Actual Behavior

Segfault on Linux

Additional Context

A trivial way would be to create a vec so that the values are owned:

        let vec = unsafe {
            std::slice::from_raw_parts(comp_gray.data, (width * height) as usize)
                .iter()
                .map(|x| *x as i32)
                .collect::<Vec<_>>()
        };
        Ok(vec.into_iter())

Or the SimplePackingDecodeIterator could be inlined into the function, ensuring the slice memory is alive until the return

Simple Description on the Bug

When I try to decode some submessages, the program panics with a message 'attempt to multiply with overflow'.

Steps to Reproduce

1. Use the program described in report #32, but change dv.take(10) to dv.take(10000) on line 23
2. Use the crate after merging #33
3. Use the same data as #29

Expected Behavior

Parse GRIB file normally.

Actual Behavior

The encoded values are incrementally increasing, and the program panicks with the following error:

0
,Grid:                                   Latitude/longitude
  Number of points:                     1038240
Product:                                Analysis or forecast at a horizontal level or in a horizontal layer at a point in time
  Parameter Category:                   Mass
  Parameter:                            Pressure reduced to MSL
  Generating Proceess:                  Forecast
  Forecast Time:                        0
  Forecast Time Unit:                   Hour
  1st Fixed Surface Type:               Mean sea level
  1st Scale Factor:                     0
  1st Scaled Value:                     0
  2nd Fixed Surface Type:               code '255' is not implemented
  2nd Scale Factor:                     0
  2nd Scaled Value:                     0
Data Representation:                    Grid point data - complex packing and spatial differencing
  Number of represented values:         1038240

Indicator { discipline: 0, total_length: 990935 }
GridDefinition { payload: [0, 0, 15, 215, 160, 0, 0, 0, 0, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 160, 0, 0, 2, 209, 0, 0, 0, 0, 255, 255, 255, 255, 5, 93, 74, 128, 0, 0, 0, 0, 48, 133, 93, 74, 128, 21, 113, 89, 112, 0, 3, 208, 144, 0, 3, 208, 144, 0] }
ProdDefinition { payload: [0, 0, 0, 0, 3, 1, 2, 0, 81, 0, 0, 0, 1, 0, 0, 0, 0, 101, 0, 0, 0, 0, 0, 255, 0, 0, 0, 0, 0] }
GridDefinition { payload: [0, 0, 15, 215, 160, 0, 0, 0, 0, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 160, 0, 0, 2, 209, 0, 0, 0, 0, 255, 255, 255, 255, 5, 93, 74, 128, 0, 0, 0, 0, 48, 133, 93, 74, 128, 21, 113, 89, 112, 0, 3, 208, 144, 0, 3, 208, 144, 0] }
size hint: (1038240, Some(1038240))
101752.59
101752.59
106462.2
:
212740210.0
214157800.0
215580060.0
thread 'main' panicked at 'attempt to multiply with overflow', /Users/noritada/gitwc/private/grib-rs/src/decoders/complex.rs:207:29
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Additional Context

No response

Motivation

Problem

I encountered an error while trying to parse a GRIB2 file using the grib crate in Rust. The code fails with the error NotSupported("template 40").

Steps to Reproduce

1. Open a GRIB2 file named gfs.t06z.sfluxgrbf000.grib2.
2. Attempt to read the file and find a specific submessage.
3. The code crashes with the error NotSupported("template 40").

Code

use std::fs::File;
use std::io::BufReader;
use grib::Grib;

fn main() {
    let fname = "gfs.t06z.sfluxgrbf000.grib2";
    // Open the input file in a normal way.
    let f = File::open(fname).unwrap();
    let f = BufReader::new(f);

    // Read with the reader.
    let grib2 = grib::from_reader(f).unwrap();
    
    // Find the target submessage.
    let (_index, submessage) = grib2
        .iter()
        .find(|(index, _)| *index == (1, 0))
        .ok_or("no such index").unwrap();
        
    let latlons = submessage.latlons().unwrap();
}

Error

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: NotSupported("template 40")', src/main.rs:XX:YY

Environment

• Operating System: MacOS
• Rust Version: 1.79.0
• grib Crate Version: 0.9.2

Additional Information

The error suggests that the specific GRIB2 template (template 40) is not supported by the grib crate. It would be helpful if support for this template could be added or if there is a workaround to handle this template.

Expected Behavior

The code should parse the GRIB2 file without encountering the NotSupported("template 40") error and should successfully retrieve the latitude and longitude information.

Proposed Solution

Additional Context

• source file https://nomads.ncep.noaa.gov/pub/data/nccf/com/gfs/prod/gfs.20240617/06/atmos/gfs.t06z.pgrb2.0p25.f051

Proposed Feature

I think it would be useful to add the example script presenting how to use this library to read a GRIB file in own project.

Motivation

As the library code is rather complex, it is not straightforward how to use the library in own project. Moreover, even small libraries include short presentation how to use them. I believe that adding the example script would lower a level of technical knowledge necessary for using this library. Therefore the library could also be used by atmospheric scientists, who might not have as extensive technical knowledge as programmers.

Additional Info

The example script could be included as .rs file in examples folder or as a code snippet in README.

Simple Description on the Bug

Copying and pasting list_surfaces.rs example into main.rs of a new binary crate and adding grib = "0.3.0" as a dependency into Cargo.toml results in build errors.

Steps to Reproduce

1. Create a new bin crate with cargo new
2. Copy paste list_surfaces.rs from examples into main.rs
3. Add grib = "0.3.0" as a dependency into Cargo.toml
4. Build errors occur

Expected Behavior

No build errors. Example should just successfully compile and run.

Actual Behavior

Build errors:

error[E0277]: the trait bound `GribError: std::error::Error` is not satisfied
   --> src/main.rs:31:85
    |
31  |     let grib2 = Grib2::<SeekableGrib2Reader<BufReader<File>>>::read_with_seekable(f)?;
    |                                                                                     ^ the trait `std::error::Error` is not implemented for `GribError`
    |
    = note: required because of the requirements on the impl of `From<GribError>` for `Box<dyn std::error::Error>`
    = note: required because of the requirements on the impl of `FromResidual<Result<Infallible, GribError>>` for `Result<(), Box<dyn std::error::Error>>`
note: required by `from_residual`
   --> /home/quba/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/library/core/src/ops/try_trait.rs:339:5
    |
339 |     fn from_residual(residual: R) -> Self;
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

error[E0599]: no method named `iter` found for struct `Grib2` in the current scope
  --> src/main.rs:34:29
   |
34 |     for submessage in grib2.iter() {
   |                             ^^^^ method not found in `Grib2<SeekableGrib2Reader<BufReader<File>>>`

Some errors have detailed explanations: E0277, E0599.
For more information about an error, try `rustc --explain E0277`.

Additional Context

The examples build correctly when the latest crate version from the repo is linked (`git clone --recurse-submodules -j8 https://github.com/noritada/grib-rs.git and link the crate locally). So the example is working just okay, and the issue is with the outdated version of the crate published on crates.io

Thus, the solution is simply updating the crate on crates.io (see issue #9).

Motivation

When parse grib2 file with 36x5011x7501 shape, the data encode with JPEG 2000, the performance is very slow. with one submessage 5011x7501 spent 10s, for parse all 36 submessage is almost need 6 min on my labtop.

Proposed Solution

Is there some optimization? Change openjpeg-sys to other jpeg 2000 library?

Additional Context

No response

Motivation

It would be nice to add support for GRIB2 Template 5.41, Portable Network Graphics (PNG) format.

Detailed Information on the Data

One of data using Template 5.41 is available in MRMS product page.

Additional Context

I have already downloaded a data file and started to work on implementation.

Motivation

Crate version published on crates.io is well behind the latest version of the crate in Github repo. Because of that, many latest features are missing (e.g. partial support of GFS files reading).

Proposed Solution

Release a new version of the crate (should be probably 0.4.x due to upgraded Rust edition). This will also be a great occasion to update outdated dependencies.

Additional Context

The impact of this release will probably be minimal as the crate has (unfortunately) not too much users. But can be advantegous by adding new features to the crate, showing on crates.io site that the crate is actively maintained and in result bringing new users.

Simple Description on the Bug

The build has been failing on all of ubuntu-latest, macOS-latest, and windows-latest on GitHub Actions since run on 2021-12-17 21:00 UTC.

Steps to Reproduce

1. Run cargo build

Expected Behavior

Successful build is expected.

Actual Behavior

Build fails with error messages like this (on ubuntu-latest):

error: linking with `cc` failed: exit status: 1
  |
  = note: "cc" "-m64" "/home/runner/work/grib-rs/grib-rs/target/debug/deps/gribber-80f0190eb821eb8f.1222uwsbdpw5s8e2.rcgu.o" (snip) "-Wl,--end-group" "/home/runner/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-2a0b2a4f96acb821.rlib" "-Wl,-Bdynamic" "-lc" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-Wl,--eh-frame-hdr" "-Wl,-znoexecstack" "-L" "/home/runner/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-o" "/home/runner/work/grib-rs/grib-rs/target/debug/deps/gribber-80f0190eb821eb8f" "-Wl,--gc-sections" "-pie" "-Wl,-zrelro" "-Wl,-znow" "-nodefaultlibs"
  = note: /usr/bin/ld: /home/runner/work/grib-rs/grib-rs/target/debug/deps/libopenjpeg_sys-70c5f6d04fad6c88.rlib(t1.o): in function `opj_t1_clbl_decode_processor':
          /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/openjpeg-sys-1.0.4/vendor/src/lib/openjp2/t1.c:1690: undefined reference to `opj_t1_ht_decode_cblk'
          collect2: error: ld returned 1 exit status

  = help: some `extern` functions couldn't be found; some native libraries may need to be installed or have their path specified
  = note: use the `-l` flag to specify native libraries to link
  = note: use the `cargo:rustc-link-lib` directive to specify the native libraries to link with Cargo (see https://doc.rust-lang.org/cargo/reference/build-scripts.html#cargorustc-link-libkindname)

Additional Context

Successful builds before 2021-12-16 21:00 UTC used openjpeg-sys 1.0.3, while builds after 2021-12-17 21:00 UTC uses 1.0.4, and this change may have
triggered the issue.

Simple Description on the Bug

When parsing data at a constant Isobaric level that is encoded with Complex Packing, there are some messages that have a "Missing" 2nd Scale Factor that cannot be parsed resulting in incorrect values being produced.

Steps to Reproduce

Attempt to parse a GRIB file with a supposedly missing 2nd Scale Factor. I have posted an example GRIB file that displays this here: https://drive.google.com/drive/folders/1Zxs21uamCSF9LVUWHYO-dE-A9rzb91S_?usp=sharing. The error_grib_file contains the troublesome data and the correct_test_file contains GRIB data that can be properly parsed.

Note that I am only parsing messages that are at Isobaric layers (Code Table 4-5 100), not those that are altitude layers. The .idx index file shows which messages are of which type. The specific message that I am testing on is number 295 in the index file.

The following code explains how I was comparing the data from the test file to a different file that should have fairly similar data. Note that this was copy-pasted then adapted, so it will likely require some changes to function properly.

    #[test]
    fn test_grib_format() {
        // read through every message in a grib file and parse the first few values from each
        // message
        let p = "error_grib_file";
        let f = std::fs::File::open(&p).unwrap();
        let rd = std::io::BufReader::new(f);
        let f1 = grib::from_reader(rd).unwrap();

        let p = "correct_test_file";
        let f = std::fs::File::open(&p).unwrap();
        let rd = std::io::BufReader::new(f);
        let f2 = grib::from_reader(rd).unwrap();

        // set up decoder
        let f1 = f1
            .iter()
            .map(|lyr| {
                let ((i, j), fm) = lyr;
                let strn = format!(
                    "\n\nIncorrect\n\n{}, {}\n{}\n{:?}\n{:?}\n{:?}\n{:?}\n{:?}",
                    i,
                    j,
                    fm.describe(),
                    fm.indicator(),
                    fm.grid_def(),
                    fm.prod_def(),
                    fm.prod_def().forecast_time().unwrap().to_string(),
                    fm.repr_def(),
                );
                let dc = Grib2SubmessageDecoder::from(fm).unwrap();

                let dv = dc.dispatch().unwrap();

                let val = dv.take(10).collect::<Vec<_>>();

                (strn, val)
            })
            .collect::<Vec<_>>();

        // set up decoder
        let f2 = f2
            .iter()
            .map(|lyr| {
                let ((i, j), fm) = lyr;
                let strn = format!(
                    "\n\nCorrect\n\n{}, {}\n{}\n{:?}\n{:?}\n{:?}\n{:?}\n{:?}",
                    i,
                    j,
                    fm.describe(),
                    fm.indicator(),
                    fm.grid_def(),
                    fm.prod_def(),
                    fm.prod_def().forecast_time().unwrap().to_string(),
                    fm.repr_def(),
                );
                let dc = Grib2SubmessageDecoder::from(fm).unwrap();

                let dv = dc.dispatch().unwrap();

                let val = dv.take(10).collect::<Vec<_>>();

                (strn, val)
            })
            .collect::<Vec<_>>();

/*
Code to pair up messages together for printing purposes
*/


        f2.into_iter().zip(f1.into_iter()).for_each(|(f, l)| {
            println!("{}", f.0);
            println!("{:?}", f.1);
            println!("{}", l.0);
            println!("{:?}", l.1);
        });

        // ensure the test fails
        panic!()
    }

Expected Behavior

Parse the values properly.

Actual Behavior

Produce values that are wildly inaccurate and nowhere near the values produced by the file that is parsed correctly. Here is example output from the aforementioned message that I tested with:

Correct

Grid:                                   Latitude/longitude
  Number of points:                     1038240
Product:                                Analysis or forecast at a horizontal level or in a horizontal layer at a point in time
  Parameter Category:                   Momentum
  Parameter:                            u-component of wind
  Generating Proceess:                  Forecast
  Forecast Time:                        0
  Forecast Time Unit:                   Hour
  1st Fixed Surface Type:               Isobaric surface
  1st Scale Factor:                     0
  1st Scaled Value:                     20000
  2nd Fixed Surface Type:               code '255' is not implemented
  2nd Scale Factor:                     0
  2nd Scaled Value:                     0
Data Representation:                    Grid point data - complex packing and spatial differencing
  Number of represented values:         1038240

Indicator { discipline: 0, total_length: 581327 }
GridDefinition { payload: [0, 0, 15, 215, 160, 0, 0, 0, 0, 6, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 160, 0, 0, 2, 209, 0, 0, 0, 0, 255, 255, 255, 255, 5, 93, 74, 128, 0, 0, 0, 0, 48, 133, 93, 74, 128, 21, 113, 89, 112, 0, 3, 208, 144, 0, 3, 208, 144, 0] }
ProdDefinition { payload: [0, 0, 0, 0, 2, 2, 2, 0, 81, 0, 0, 0, 1, 0, 0, 0, 0, 100, 0, 0, 0, 78, 32, 255, 0, 0, 0, 0, 0] }
"0 [h]"
ReprDefinition { payload: [0, 15, 215, 160, 0, 3, 195, 167, 184, 23, 0, 0, 0, 1, 9, 0, 1, 0, 98, 88, 209, 154, 255, 255, 255, 255, 0, 0, 105, 56, 0, 4, 0, 0, 0, 1, 1, 0, 0, 0, 71, 7, 2, 2] }
[-24.74382, -24.64382, -24.64382, -24.543821, -24.44382, -24.44382, -24.34382, -24.24382, -24.24382, -24.14382]


Incorrect

Grid:                                   Latitude/longitude
  Number of points:                     1038240
Product:                                Analysis or forecast at a horizontal level or in a horizontal layer at a point in time
  Parameter Category:                   Temperature
  Parameter:                            Temperature
  Generating Proceess:                  Forecast
  Forecast Time:                        0
  Forecast Time Unit:                   Hour
  1st Fixed Surface Type:               Isobaric surface
  1st Scale Factor:                     0
  1st Scaled Value:                     30
  2nd Fixed Surface Type:               code '255' is not implemented
  2nd Scale Factor:                     Missing
  2nd Scaled Value:                     255
Data Representation:                    Grid point data - complex packing and spatial differencing
  Number of represented values:         1038240

Indicator { discipline: 0, total_length: 675339 }
GridDefinition { payload: [0, 0, 15, 215, 160, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5, 160, 0, 0, 2, 209, 0, 0, 0, 0, 255, 255, 255, 255, 133, 93, 74, 128, 0, 0, 0, 0, 48, 5, 93, 74, 128, 21, 113, 89, 112, 0, 3, 208, 144, 0, 3, 208, 144, 64] }
ProdDefinition { payload: [0, 0, 0, 0, 0, 0, 2, 255, 16, 0, 0, 0, 1, 0, 0, 0, 0, 100, 0, 0, 0, 0, 30, 255, 255, 0, 0, 0, 255] }
"0 [h]"
ReprDefinition { payload: [0, 15, 215, 160, 0, 3, 70, 174, 52, 0, 0, 0, 0, 2, 8, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 89, 0, 4, 0, 0, 0, 1, 1, 0, 0, 0, 64, 6, 2, 2] }
[252.28, 252.28, 252.28, 252.28, 252.28, 252.28, 252.28, 252.28, 252.28, 252.28]

Additional Context

Additionally, the Parameter Category and Parameter are seemingly incorrectly parsed as they should both be Momentum, u-component of wind and the incorrect file produces Temperature, Temperature.