Hi Rob:

I havent seen an example of this before. Can you send me both files?

What file format is the external file? Why do you do it that way?
Does the "metadata file" use an absolute path address or assumed
relative to the metadata file or ?

John

On 8/21/2013 9:04 AM, Rob V. van Nieuwpoort wrote:

Hi all,

I have an application that stores metadata in the HDF5 format, but the
real "raw" data in a separate file. The HDF5 file refers to the raw
file. I would like to use this library to parse the metadata. I don't
have to read the external raw file, just the metadata in the HDF5.
However, NetcdfFile.open bails out with this stack trace:

***_UNPROCESSED MESSAGE type = ExternalDataFiles(7) raw = 7
java.io.IOException: java.lang.UnsupportedOperationException:
***_UNPROCESSED MESSAGE type = ExternalDataFiles(7) raw = 7
at ucar.nc2.NetcdfFile.open(NetcdfFile.java:425)
at ucar.nc2.dataset.NetcdfDataset.openOrAcquireFile(NetcdfDataset.java:699)
at ucar.nc2.dataset.NetcdfDataset.openFile(NetcdfDataset.java:560)
at ucar.nc2.NCdumpW.print(NCdumpW.java:168)
at ucar.nc2.NCdumpW.main(NCdumpW.java:982)
Caused by: java.lang.UnsupportedOperationException: ****UNPROCESSED
MESSAGE type = ExternalDataFiles(7) raw = 7
at ucar.nc2.iosp.hdf5.H5header$HeaderMessage.read(H5header.java:2714)
at
ucar.nc2.iosp.hdf5.H5header$DataObject.readMessagesVersion2(H5header.java:2420)
at ucar.nc2.iosp.hdf5.H5header$DataObject.(H5header.java:2306)
at ucar.nc2.iosp.hdf5.H5header$DataObject.(H5header.java:2180)
at ucar.nc2.iosp.hdf5.H5header.getDataObject(H5header.java:2027)
at ucar.nc2.iosp.hdf5.H5header.access$600(H5header.java:70)
at ucar.nc2.iosp.hdf5.H5header$DataObjectFacade.(H5header.java:2073)
at ucar.nc2.iosp.hdf5.H5header.readGroupNew(H5header.java:3920)
at ucar.nc2.iosp.hdf5.H5header.access$900(H5header.java:70)
at ucar.nc2.iosp.hdf5.H5header$H5Group.(H5header.java:2153)
at ucar.nc2.iosp.hdf5.H5header$H5Group.(H5header.java:2118)
at ucar.nc2.iosp.hdf5.H5header.makeNetcdfGroup(H5header.java:472)
at ucar.nc2.iosp.hdf5.H5header.makeNetcdfGroup(H5header.java:473)
at ucar.nc2.iosp.hdf5.H5header.read(H5header.java:215)
at ucar.nc2.iosp.hdf5.H5iosp.open(H5iosp.java:128)
at ucar.nc2.NetcdfFile.(NetcdfFile.java:1521)
at ucar.nc2.NetcdfFile.open(NetcdfFile.java:813)
at ucar.nc2.NetcdfFile.open(NetcdfFile.java:422)
... 4 more

Now, I don't need netcdf to understand about the external file, but it
would be nice if it would just ignore it or maybe log a warning. Or,
even better, support the ExternalDataFiles.

Is there any way to work around this? Or maybe it is relatively easy to
patch?

Kind regards,

Rob van Nieuwpoort
Netherlands eScience center
[email protected] mailto:[email protected]

—
Reply to this email directly or view it on GitHub
#27.

from thredds.

Hi John,

Thanks for your quick reply!
I attached the HDF5 file, I hope that works. The external file that is referenced is called “BeamFormedData.raw”, however it is too large to send by mail (for this small test it is 71MB). I could put it in a dropbox if you need it?
It is just raw 32-bit float data. We split it up in .h5 for the meta data, and .raw for the actual data, because the raw files can be large (think terabytes-petabytes ☺ ).
The path addressed is relative, so only “BeamFormedData.raw”. I don’t actually have to read that data, I just want to parse the meta data.
This is LOFAR telescope data by the way (http://www.lofar.org).

Cheers,

Rob

From: John Caron [mailto:[email protected]]
Sent: woensdag 21 augustus 2013 22:28
To: Unidata/thredds
Cc: Rob van Nieuwpoort
Subject: Re: [thredds] java.lang.UnsupportedOperationException:****UNPROCESSED MESSAGE type = ExternalDataFiles(7) raw = 7 (#27)

Hi Rob:

I havent seen an example of this before. Can you send me both files?

What file format is the external file? Why do you do it that way?
Does the "metadata file" use an absolute path address or assumed
relative to the metadata file or ?

John

On 8/21/2013 9:04 AM, Rob V. van Nieuwpoort wrote:

Hi all,

I have an application that stores metadata in the HDF5 format, but the
real "raw" data in a separate file. The HDF5 file refers to the raw
file. I would like to use this library to parse the metadata. I don't
have to read the external raw file, just the metadata in the HDF5.
However, NetcdfFile.open bails out with this stack trace:

***_UNPROCESSED MESSAGE type = ExternalDataFiles(7) raw = 7
java.io.IOException: java.lang.UnsupportedOperationException:
***_UNPROCESSED MESSAGE type = ExternalDataFiles(7) raw = 7
at ucar.nc2.NetcdfFile.open(NetcdfFile.java:425)
at ucar.nc2.dataset.NetcdfDataset.openOrAcquireFile(NetcdfDataset.java:699)
at ucar.nc2.dataset.NetcdfDataset.openFile(NetcdfDataset.java:560)
at ucar.nc2.NCdumpW.print(NCdumpW.java:168)
at ucar.nc2.NCdumpW.main(NCdumpW.java:982)
Caused by: java.lang.UnsupportedOperationException: ****UNPROCESSED
MESSAGE type = ExternalDataFiles(7) raw = 7
at ucar.nc2.iosp.hdf5.H5header$HeaderMessage.read(H5header.java:2714)
at
ucar.nc2.iosp.hdf5.H5header$DataObject.readMessagesVersion2(H5header.java:2420)
at ucar.nc2.iosp.hdf5.H5header$DataObject.(H5header.java:2306)
at ucar.nc2.iosp.hdf5.H5header$DataObject.(H5header.java:2180)
at ucar.nc2.iosp.hdf5.H5header.getDataObject(H5header.java:2027)
at ucar.nc2.iosp.hdf5.H5header.access$600(H5header.java:70)
at ucar.nc2.iosp.hdf5.H5header$DataObjectFacade.(H5header.java:2073)
at ucar.nc2.iosp.hdf5.H5header.readGroupNew(H5header.java:3920)
at ucar.nc2.iosp.hdf5.H5header.access$900(H5header.java:70)
at ucar.nc2.iosp.hdf5.H5header$H5Group.(H5header.java:2153)
at ucar.nc2.iosp.hdf5.H5header$H5Group.(H5header.java:2118)
at ucar.nc2.iosp.hdf5.H5header.makeNetcdfGroup(H5header.java:472)
at ucar.nc2.iosp.hdf5.H5header.makeNetcdfGroup(H5header.java:473)
at ucar.nc2.iosp.hdf5.H5header.read(H5header.java:215)
at ucar.nc2.iosp.hdf5.H5iosp.open(H5iosp.java:128)
at ucar.nc2.NetcdfFile.(NetcdfFile.java:1521)
at ucar.nc2.NetcdfFile.open(NetcdfFile.java:813)
at ucar.nc2.NetcdfFile.open(NetcdfFile.java:422)
... 4 more

Now, I don't need netcdf to understand about the external file, but it
would be nice if it would just ignore it or maybe log a warning. Or,
even better, support the ExternalDataFiles.

Is there any way to work around this? Or maybe it is relatively easy to
patch?

Kind regards,

Rob van Nieuwpoort
Netherlands eScience center
[email protected]:[email protected] mailto:[email protected]

—
Reply to this email directly or view it on GitHub
#27.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/27#issuecomment-23047543.

from thredds.

Hi Rob:

Im modifying the code to put out a warning but not fail when it sees a message type it doesnt understand.

This will be in the 4.5.0 release.

John

****UNPROCESSED MESSAGE type = ExternalDataFiles(7) raw = 7

Hi John,

Thanks for your quick reply!
I attached the HDF5 file, I hope that works. The external file that is referenced is called “BeamFormedData.raw”, however it is too large to send by mail (for this small test it is 71MB). I could put it in a dropbox if you need it?
It is just raw 32-bit float data. We split it up in .h5 for the meta data, and .raw for the actual data, because the raw files can be large (think terabytes-petabytes ☺ ).
The path addressed is relative, so only “BeamFormedData.raw”. I don’t actually have to read that data, I just want to parse the meta data.
This is LOFAR telescope data by the way (http://www.lofar.org).

Cheers,

Rob

from thredds.

OK, great news, thanks John!

Rob

From: John Caron [mailto:[email protected]]
Sent: dinsdag 20 mei 2014 23:25
To: Unidata/thredds
Cc: Rob van Nieuwpoort
Subject: Re: [thredds] java.lang.UnsupportedOperationException:****UNPROCESSED MESSAGE type = ExternalDataFiles(7) raw = 7 (#27)

Hi Rob:

Im modifying the code to put out a warning but not fail when it sees a message type it doesnt understand.

This will be in the 4.5.0 release.

John

****UNPROCESSED MESSAGE type = ExternalDataFiles(7) raw = 7

Hi John,

Thanks for your quick reply!
I attached the HDF5 file, I hope that works. The external file that is referenced is called “BeamFormedData.raw”, however it is too large to send by mail (for this small test it is 71MB). I could put it in a dropbox if you need it?
It is just raw 32-bit float data. We split it up in .h5 for the meta data, and .raw for the actual data, because the raw files can be large (think terabytes-petabytes ☺ ).
The path addressed is relative, so only “BeamFormedData.raw”. I don’t actually have to read that data, I just want to parse the meta data.
This is LOFAR telescope data by the way (http://www.lofar.org).

Cheers,

Rob

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/27#issuecomment-43685200.

from thredds.