Giter VIP home page Giter VIP logo

opendap / bes Goto Github PK

View Code? Open in Web Editor NEW
13.0 11.0 19.0 750.79 MB

The BES framework, which forms the basis for the Hyrax server

License: GNU Lesser General Public License v3.0

Shell 3.23% HTML 3.43% Perl 0.05% C++ 76.75% C 11.97% Makefile 1.39% M4 1.75% Scilab 0.08% Batchfile 0.09% Ruby 0.01% Python 0.55% Awk 0.02% CMake 0.45% Lex 0.07% PowerShell 0.02% Lua 0.01% DIGITAL Command Language 0.02% Fortran 0.03% Yacc 0.10% nesC 0.01%
c-plus-plus hyrax dap

bes's People

Contributors

blackone-sudo avatar captainkirk99 avatar dh-opendap avatar edwardhartnett avatar ethanrd avatar hemphilc avatar hyoklee avatar jgallagher59701 avatar kneumiller avatar kyang2014 avatar lewismc avatar ndp-opendap avatar opoplawski avatar ptype avatar rmorris2342 avatar rrimer07 avatar rwmjones avatar sawer297 avatar travis-ci-opendap avatar tsgouros avatar udayopen avatar vskorole avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

bes's Issues

'make' error to build from source

Hi,
I am trying to build the bes module from source using Build Steps guide using hyrax-dependencies. Previously it worked fine but with latest git code pull it gives error while running make command.

**DmrppArray.cc: In member function ‘void dmrpp::DmrppArray::multi_finish(CURLM*, std::vector<dmrpp::H4ByteStream, std::allocator<dmrpp::H4ByteStream> >*)’:
DmrppArray.cc:475: error: ‘curl_multi_wait’ was not declared in this scope
libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I../.. -I/hyrax-git/include/libdap -I/usr/include/libxml2 -I../../dap -I../../dispatch -DMODULE_NAME=\"dmrpp_module\" -DMODULE_VERSION=\"0.4.0\" -g3 -O0 -Wall -W -Wcast-align -MT DmrppModule.lo -MD -MP -MF .deps/DmrppModule.Tpo -c DmrppModule.cc -o DmrppModule.o >/dev/null 2>&1
libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I../.. -I/hyrax-git/include/libdap -I/usr/include/libxml2 -I../../dap -I../../dispatch -DMODULE_NAME=\"dmrpp_module\" -DMODULE_VERSION=\"0.4.0\" -g3 -O0 -Wall -W -Wcast-align -MT H4ByteStream.lo -MD -MP -MF .deps/H4ByteStream.Tpo -c H4ByteStream.cc -o H4ByteStream.o >/dev/null 2>&1
make[4]: *** [DmrppArray.lo] Error 1
make[4]: *** Waiting for unfinished jobs....
libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I../.. -I/hyrax-git/include/libdap -I/usr/include/libxml2 -I../../dap -I../../dispatch -DMODULE_NAME=\"dmrpp_module\" -DMODULE_VERSION=\"0.4.0\" -g3 -O0 -Wall -W -Wcast-align -MT DmrppCommon.lo -MD -MP -MF .deps/DmrppCommon.Tpo -c DmrppCommon.cc -o DmrppCommon.o >/dev/null 2>&1
libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I../.. -I/hyrax-git/include/libdap -I/usr/include/libxml2 -I../../dap -I../../dispatch -DMODULE_NAME=\"dmrpp_module\" -DMODULE_VERSION=\"0.4.0\" -g3 -O0 -Wall -W -Wcast-align -MT DmrppByte.lo -MD -MP -MF .deps/DmrppByte.Tpo -c DmrppByte.cc -o DmrppByte.o >/dev/null 2>&1
libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I../.. -I/hyrax-git/include/libdap -I/usr/include/libxml2 -I../../dap -I../../dispatch -DMODULE_NAME=\"dmrpp_module\" -DMODULE_VERSION=\"0.4.0\" -g3 -O0 -Wall -W -Wcast-align -MT DmrppFloat64.lo -MD -MP -MF .deps/DmrppFloat64.Tpo -c DmrppFloat64.cc -o DmrppFloat64.o >/dev/null 2>&1
libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I../.. -I/hyrax-git/include/libdap -I/usr/include/libxml2 -I../../dap -I../../dispatch -DMODULE_NAME=\"dmrpp_module\" -DMODULE_VERSION=\"0.4.0\" -g3 -O0 -Wall -W -Wcast-align -MT DmrppInt16.lo -MD -MP -MF .deps/DmrppInt16.Tpo -c DmrppInt16.cc -o DmrppInt16.o >/dev/null 2>&1
libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I../.. -I/hyrax-git/include/libdap -I/usr/include/libxml2 -I../../dap -I../../dispatch -DMODULE_NAME=\"dmrpp_module\" -DMODULE_VERSION=\"0.4.0\" -g3 -O0 -Wall -W -Wcast-align -MT DmrppFloat32.lo -MD -MP -MF .deps/DmrppFloat32.Tpo -c DmrppFloat32.cc -o DmrppFloat32.o >/dev/null 2>&1
mv -f .deps/DmrppModule.Tpo .deps/DmrppModule.Plo
mv -f .deps/DmrppCommon.Tpo .deps/DmrppCommon.Plo
mv -f .deps/H4ByteStream.Tpo .deps/H4ByteStream.Plo
mv -f .deps/DmrppInt16.Tpo .deps/DmrppInt16.Plo
mv -f .deps/DmrppByte.Tpo .deps/DmrppByte.Plo
mv -f .deps/DmrppFloat64.Tpo .deps/DmrppFloat64.Plo
libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I../.. -I/hyrax-git/include/libdap -I/usr/include/libxml2 -I../../dap -I../../dispatch -DMODULE_NAME=\"dmrpp_module\" -DMODULE_VERSION=\"0.4.0\" -g3 -O0 -Wall -W -Wcast-align -MT DmrppRequestHandler.lo -MD -MP -MF .deps/DmrppRequestHandler.Tpo -c DmrppRequestHandler.cc -o DmrppRequestHandler.o >/dev/null 2>&1
mv -f .deps/DmrppFloat32.Tpo .deps/DmrppFloat32.Plo
mv -f .deps/DmrppRequestHandler.Tpo .deps/DmrppRequestHandler.Plo
make[4]: Leaving directory `/hyrax-git/bes/modules/dmrpp_module'
make[3]: *** [all-recursive] Error 1
make[3]: Leaving directory `/hyrax-git/bes/modules/dmrpp_module'
make[2]: *** [all-recursive] Error 1
make[2]: Leaving directory `/hyrax-git/bes/modules'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/hyrax-git/bes'
make: *** [all] Error 2**

NdArray not 1-D in covjson response from Hyrax

According to the specification:

https://covjson.org/spec/#ndarray-objects

an NdArray shall be one-dimensional.

This is not true in the current covjson response in Hyrax. For example:

https://snpp-omps.gesdisc.eosdis.nasa.gov/opendap/SNPP_OMPS_Level3/OMPS_NPP_NMTO3_L3_DAILY.2/2012/OMPS-NPP_NMTO3-L3-DAILY_v2.1_2012m0126_2017m0227t060834.h5.covjson?ColumnAmountO3[140:1:150][35:1:45],Latitude[140:1:150],Longitude[35:1:45]

shows a 2-D array for range "ColumnAmountO3", although the JSON files shows it is an NdArray type.

error: 'class libdap::DDS' has no member named 'is_dap4_projected'

DapUtils.cc: In function 'void dap_utils::throw_for_dap4_typed_vars_or_attrs(libdap::DDS*, const std::string&, unsigned int)':
DapUtils.cc:174:13: error: 'class libdap::DDS' has no member named 'is_dap4_projected'
  174 |     if(dds->is_dap4_projected(inventory)){
      |             ^~~~~~~~~~~~~~~~~
DapUtils.cc: In function 'void dap_utils::throw_for_dap4_typed_attrs(libdap::DAS*, const std::string&, unsigned int)':
DapUtils.cc:189:41: error: 'class libdap::AttrTable' has no member named 'has_dap4_types'
  189 |     if(das->get_top_level_attributes()->has_dap4_types("/",inventory)){
      |                                         ^~~~~~~~~~~~~~
DapUtils.cc: In function 'uint64_t dap_utils::crsaibv_process_variable(libdap::BaseType*, uint64_t, std::vector<std::__cxx11::basic_string<char> >&)':
DapUtils.cc:316:35: error: 'class libdap::BaseType' has no member named 'width_ll'; did you mean 'width'?
  316 |             uint64_t vsize = var->width_ll(true);
      |                                   ^~~~~~~~
      |                                   width
DapUtils.cc: In function 'uint64_t dap_utils::compute_response_size_and_inv_big_vars(const libdap::D4Group*, uint64_t, std::vector<std::__cxx11::basic_string<char> >&)':
DapUtils.cc:384:37: error: 'const class libdap::D4Group' has no member named 'groups'; did you mean 'std::vector<libdap::D4Group*> libdap::D4Group::d_groups'? (not accessible from this context)
  384 |     for (const auto child_grp: grp->groups()) {
      |                                     ^~~~~~
In file included from DapUtils.cc:34:
/usr/include/libdap/D4Group.h:60:22: note: declared private here
   60 |     vector<D4Group*> d_groups;
      |                      ^~~~~~~~
make[3]: *** [Makefile:815: DapUtils.lo] Error 1

The get_dmrpp script needs to start indexing at 1 when picking a random variable

In the get_dmrpp script's dap4_array_test function, it selects a random number in order to test a variable of a given type. It calls dap4_array_dim_counter which selects the variable, but that function assumes that the match_num variable starts at 1. So if 0 is passed, it won't select a random variable and will result in an error.

roi() etc. server-side functions shall not prepend to variable and attribute names

At the moment (as of 1.13.3) using server-side functions can result in modified variable and attribute names. For instance the following CDL snippet after running roi() with a fileout netcdf response. This breaks the other-wise CF-compliance of data coming out of BES. Prepending strings like "row_subset." is not desired.

float roi_subset.Latitude(FakeDim0, nXtrack) ;
roi_subset.Latitude:_FillValue = -1.267651e+30f ;
roi_subset.Latitude:missing_value = -1.267651e+30f ;
roi_subset.Latitude:scale_factor = 1. ;
roi_subset.Latitude:add_offset = 0. ;
roi_subset.Latitude:title = "Latitude of the center of the groundpixel" ;
roi_subset.Latitude:units = "degrees_north" ;
roi_subset.Latitude:UniqueFieldDefinition = "Aura-Shared" ;
roi_subset.Latitude:origname = "Latitude" ;
roi_subset.Latitude:fullnamepath = "/HDFEOS/SWATHS/ColumnAmountNO2/Geolocation Fields/Latitude" ;

gctp.c:257:81: error: passing argument 8 of ‘hinv_init’ from incompatible pointer type [-Wincompatible-pointer-types]

After fixing libdap, I get further compilation errors:

make[6]: Entering directory '/home/rjones/d/fedora/bes/rawhide/bes-3.21.0-46/modules/hdf5_handler/gctp/src'
gcc -DHAVE_CONFIG_H -I. -I../../../..  -I./../include/ -I/usr/include/cfitsio -I/usr/include/tirpc -Wno-vla -I/include -fpic -Wno-implicit-function-declaration -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-U_FORTIFY_SOURCE,-D_FORTIFY_SOURCE=3 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64   -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -fno-omit-frame-pointer -mno-omit-leaf-frame-pointer  -c -o gctp.o gctp.c
gctp.c: warning: -Wno-implicit-function-declaration detected - is this intentional ?
gctp.c: In function ‘gctp’:
gctp.c:257:81: error: passing argument 8 of ‘hinv_init’ from incompatible pointer type [-Wincompatible-pointer-types]
  257 |       hinv_init((int)*insys,(int)*inzone,dummy,(int)*indatum,fn27,fn83,&iflgval,(int *)hinv_trans);
      |                                                                                 ^~~~~~~~~~~~~~~~~
      |                                                                                 |
      |                                                                                 int *
In file included from ./../include/cproj_prototypes.h:32,
                 from ./../include/cproj.h:33,
                 from gctp.c:33:
./../include/gctp_prototypes.h:288:108: note: expected ‘int (**)(double,  double,  double *, double *)’ but argument is of type ‘int *’
  288 | int hinv_init(int insys, int inzone, double *inparm, int indatum, char *fn27, char *fn83, int *iflg, int (*hinv_trans[])(double, double, double*, double*));
      |                                                                                                      ~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
gctp.c:260:81: error: passing argument 8 of ‘hinv_init’ from incompatible pointer type [-Wincompatible-pointer-types]
  260 |      hinv_init((int)*insys,(int)*inzone,inparm,(int)*indatum,fn27,fn83,&iflgval,(int *)hinv_trans);
      |                                                                                 ^~~~~~~~~~~~~~~~~
      |                                                                                 |
      |                                                                                 int *
./../include/gctp_prototypes.h:288:108: note: expected ‘int (**)(double,  double,  double *, double *)’ but argument is of type ‘int *’
  288 | int hinv_init(int insys, int inzone, double *inparm, int indatum, char *fn27, char *fn83, int *iflg, int (*hinv_trans[])(double, double, double*, double*));
      |                                                                                                      ~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
gctp.c:319:84: error: passing argument 8 of ‘hfor_init’ from incompatible pointer type [-Wincompatible-pointer-types]
  319 |       hfor_init((int)*outsys,(int)*outzone,dummy,(int)*outdatum,fn27,fn83,&iflgval,(int *)hfor_trans);
      |                                                                                    ^~~~~~~~~~~~~~~~~
      |                                                                                    |
      |                                                                                    int *
./../include/gctp_prototypes.h:284:112: note: expected ‘int (**)(double,  double,  double *, double *)’ but argument is of type ‘int *’
  284 | int hfor_init(int outsys, int outzone, double *outparm, int outdatum, char *fn27, char *fn83, int *iflg, int (*hfor_trans[])(double, double, double *, double *));
      |                                                                                                          ~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
gctp.c:322:85: error: passing argument 8 of ‘hfor_init’ from incompatible pointer type [-Wincompatible-pointer-types]
  322 |      hfor_init((int)*outsys,(int)*outzone,outparm,(int)*outdatum,fn27,fn83,&iflgval,(int *)hfor_trans);
      |                                                                                     ^~~~~~~~~~~~~~~~~
      |                                                                                     |
      |                                                                                     int *
./../include/gctp_prototypes.h:284:112: note: expected ‘int (**)(double,  double,  double *, double *)’ but argument is of type ‘int *’
  284 | int hfor_init(int outsys, int outzone, double *outparm, int outdatum, char *fn27, char *fn83, int *iflg, int (*hfor_trans[])(double, double, double *, double *));
      |                                                                                                          ~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
make[6]: *** [Makefile:665: gctp.o] Error 1

This is with:

libdap-3.21.0.27-1.fc41.x86_64
bes-3.21.0-46

Typo in dmrpp README.

Please change "fro" to "from"

These data files come fro a variety of sources. Some from THG and other were generated
using the mkChunkedTestData.py python script located in the parent directory.

When gdal not present, configure does not complain, but build fails (same with hdf5)

Building without gdal, I see that configure.ac correctly determined that gdal was not present:

configure: Looking for GDAL.
checking for gdal-config... no
configure: WARNING: gdal-config does not exist or it is not an exectuable file
no

However, the build then breaks:

DapFunctions.cc:29:10: fatal error: gdal.h: No such file or directory
   29 | #include <gdal.h>   // needed for scale_{grid,array}
      |          ^~~~~~~~
compilation terminated.

This is because DapFunctions includes gdal.h in an unprotected way:

#include "config.h"

#include <iostream>

#include <gdal.h>   // needed for scale_{grid,array}

#include <ServerFunctionsList.h>

I tried wrapping this in ifdefs for HAVE_GDAL, and that helps, but eventually we fail like this:

In file included from DapFunctions.cc:55:
ScaleGrid.h:72:13: error: ‘GDAL_GCP’ was not declared in this scope
   72 | std::vector<GDAL_GCP> get_gcp_data(libdap::Array *x, libdap::Array *y, int sample_x = 1, int sample_y = 1);
      |             ^~~~~~~~
ScaleGrid.h:72:21: error: template argument 1 is invalid
   72 | std::vector<GDAL_GCP> get_gcp_data(libdap::Array *x, libdap::Array *y, int sample_x = 1, int sample_y = 1);
      |                     ^
ScaleGrid.h:72:21: error: template argument 2 is invalid

So either more code needs to be protected, or gdal needs to be required, and configure.ac should error out if gdal is not found.

Similarly, if HDF5 is not found, the build proceeds, then errors like this:

make[5]: Entering directory '/home/ed/bes/modules/dmrpp_module'
g++ -DHAVE_CONFIG_H -I. -I../..  -I../.. -I../../dispatch -I../../dap -I../../http -I/usr/include/libdap -I/usr/include/libxml2 -I/usr/local/opt/openssl/include -DMODULE_NAME=\"dmrpp_module\" -DMODULE_VERSION=\"1.1.2\"  -I./../hdf5_handler -I/include -Wno-vla-extension -g -O2 --pedantic --std=c++11 -MT build_dmrpp-build_dmrpp.o -MD -MP -MF .deps/build_dmrpp-build_dmrpp.Tpo -c -o build_dmrpp-build_dmrpp.o `test -f 'build_dmrpp.cc' || echo './'`build_dmrpp.cc
build_dmrpp.cc:40:10: fatal error: H5Ppublic.h: No such file or directory
   40 | #include <H5Ppublic.h>
      |          ^~~~~~~~~~~~~
compilation terminated.
ma

CoverageJSON output invalid

This refers to http://test.opendap.org/opendap/AIRS/QC_AIRS-1.2011.02.13.090.L2.RetStd.v5.2.2.0.G11045052855.hdf.covjson.

As far as I see, all the "parameters" listed in the result, e.g. AIRSXTrack or StdPressureLay are not parameters but should be domain axes. The actual parameters, looking at http://test.opendap.org/opendap/AIRS/QC_AIRS-1.2011.02.13.090.L2.RetStd.v5.2.2.0.G11045052855.hdf.html, like O3_Resid_Ratio or TCldTopStd are all missing.

Just looking at one of the ranges objects of the output makes this mismatch obvious:

      "AIRSXTrack": {
        "type": "NdArray",
        "dataType": "integer", 
        "axisNames": ["y", "x"],
        "shape": [3],
        "values": [0, 1, 2]
      },

Here the axes are given as y and x (which are defined correctly earlier in the "domain" part), however the shape and values do not match that.

@lewismc @jonblower

fileout_covjson module can return invalid JSON reponse

The fileout_covjson can/does return invalid JSON in OPeNDAP Hyrax (1.16.0). Specifically, if we take SPURS_wg_ASL2.nc.covjson as an example

  • -nan is present in the domain/axes/x/values which is not permitted in JSON syntax
  • no comma is present after the domain/axes object
{
  "type": "Coverage",
  "domain": {
    "type" : "Domain",
    "domainType": "Grid",
    "axes": {
      "x": {
        "values": [...]
      },
      "y": {
        "values": [...]
      },
      "t": {
        "values": ["2018-01-01T00:12:20Z"]
      }
    }    <<<<<<<<<<<<<<<<<< COMMA MISSING HERE
    "referencing": [{
      "coordinates": ["t"],
      "system": {
        "type": "TemporalRS",
        "calendar": "Gregorian"
      }
    },
  • additional commas are present in parameter descriptions
...
    "parameters": {
      "lon_std": {
        "type": "Parameter",
        "description": {
          "en": "Standard deviation for longitude",  <<<<<<<< UNNECESSARY COMMA
        },
        "unit": {
          "label": {
            "en": "degrees_east"
          }
        },
...
  • An additional parameter is present at the end of the parameters block
...
        },
        "observedProperty": {
          "label": {
            "en": "Number of surface current direction measurements used"
          }
        }
      }
    }, <<<<<<<<<<< UNNECESSARY COMMA
  },
  "ranges": {
    "lon_std": {
      "type": "NdArray",
  • the JSON closing syntax is invalid
...
    "cur_dir_ndata": {
      "type": "NdArray",
      "dataType": "float", 
      "axisNames": ["t", "y", "x"],
      "shape": [1],
      "values": [...]
    }
    }
  }] <<<<<<<<<<< CLOSING SQUARE BRACKET AND ADDITIONAL CURLY BRACE RESULT IN INVALID JSON SYNTAX
}

There appear to both other issues as well but I will address them in a separate ticket.

Difficulties adjusting to different versions of HDFEOS2 library for HDF4 files

The HDFEOS2 library is for NASA HDF4 files, like MODIS.

As discussed in this issue (SpatioTemporal/STAREmaster#111), there are two different versions of HDFEOS2 that should be accomodated, 2.19 and 2.20. Unfortunately there is some const changes which interact badly with the hdf4_handler file HDFEOS2.cc.

In HDFEOS2.cc there is code that passes some of the HDFEOS2 functions as function pointers to a function that calls the HDFEOS2 function. For example:

    vector<string> gridlist;
    if (!Utility::ReadNamelist(file->path.c_str(), GDinqgrid, gridlist)) {
        delete file;
        throw1("Grid ReadNamelist failed.");
    }

This is calling the Utility::ReadNamelist() function, which is also in HDFEOS2.cc:

// Read name list from the EOS2 APIs and store names into a vector
bool Utility::ReadNamelist(const char *path,
                           int32 (*inq)(const char *, char *, int32 *),
                           vector<string> &names)
{
    char *fname = const_cast<char *>(path);
    int32 bufsize;
    int numobjs;

    if ((numobjs = inq(fname, NULL, &bufsize)) == -1) return false;
    if (numobjs > 0) {
        vector<char> buffer;
        buffer.resize(bufsize + 1);
        if (inq(fname, &buffer[0], &bufsize) == -1) return false;
        HDFCFUtil::Split(&buffer[0], bufsize, ',', names);
    }
    return true;
}

Note that the second (unnamed) parameter to ReadNamelis() is a function pointer.

This ReadNamelist function is called only 3 times, for functions GDinqgrid(), SWinqswath(), and PTinqpoint().

The challenge here is that one of these functions, SWinqswath(), has a different first parameter in 2.19 and 2.20. In 2.20 it adds a const.

Bad rpath added to build_dmrpp - not portable linkage

I'm looking to update the Fedora bes package to 3.20.10 but am running into the following issue:

+ /usr/lib/rpm/check-rpaths
*******************************************************************************
*
* WARNING: 'check-rpaths' detected a broken RPATH OR RUNPATH and will cause
*          'rpmbuild' to fail. To ignore these errors, you can set the
*          '$QA_RPATHS' environment variable which is a bitmask allowing the
*          values below. The current value of QA_RPATHS is 0x0000.
*
*    0x0001 ... standard RPATHs (e.g. /usr/lib); such RPATHs are a minor
*               issue but are introducing redundant searchpaths without
*               providing a benefit. They can also cause errors in multilib
*               environments.
*    0x0002 ... invalid RPATHs; these are RPATHs which are neither absolute
*               nor relative filenames and can therefore be a SECURITY risk
*    0x0004 ... insecure RPATHs; these are relative RPATHs which are a
*               SECURITY risk
*    0x0008 ... the special '$ORIGIN' RPATHs are appearing after other
*               RPATHs; this is just a minor issue but usually unwanted
*    0x0010 ... the RPATH is empty; there is no reason for such RPATHs
*               and they cause unneeded work while loading libraries
*    0x0020 ... an RPATH references '..' of an absolute path; this will break
*               the functionality when the path before '..' is a symlink
*          
*
* Examples:
* - to ignore standard and empty RPATHs, execute 'rpmbuild' like
*   $ QA_RPATHS=$(( 0x0001|0x0010 )) rpmbuild my-package.src.rpm
* - to check existing files, set $RPM_BUILD_ROOT and execute check-rpaths like
*   $ RPM_BUILD_ROOT=<top-dir> /usr/lib/rpm/check-rpaths
*  
*******************************************************************************
ERROR   0002: file '/usr/bin/build_dmrpp' contains an invalid  '/builddir/build/BUILD/bes-3.20.10/dap/.libs' in [/builddir/build/BUILD/bes-3.20.10/dap/.libs:/builddir/build/BUILD/bes-3.20.10/dispatch/.libs:/usr/lib64/bes]
ERROR   0002: file '/usr/bin/build_dmrpp' contains an invalid  '/builddir/build/BUILD/bes-3.20.10/dispatch/.libs' in [/builddir/build/BUILD/bes-3.20.10/dap/.libs:/builddir/build/BUILD/bes-3.20.10/dispatch/.libs:/usr/lib64/bes]

This appears to arise due to build_drmpp being linked to a module:

make[4]: Entering directory '/builddir/build/BUILD/bes-3.20.10/modules/dmrpp_module'
/bin/sh ../../libtool  --tag=CXX   --mode=link g++ -Wno-vla-extension -Wno-inconsistent-missing-override  -g -O2 --pedantic --std=c++11 -L../../dap -static -ldap_module -dynamic -L/usr/lib64/libdap -o build_dmrpp build_dmrpp-DMRpp.o build_dmrpp-DmrppCommon.o build_dmrpp-Chunk.o build_dmrpp-CurlHandlePool.o build_dmrpp-DmrppByte.o build_dmrpp-DmrppArray.o build_dmrpp-DmrppFloat32.o build_dmrpp-DmrppFloat64.o build_dmrpp-DmrppInt16.o build_dmrpp-DmrppInt32.o build_dmrpp-DmrppInt64.o build_dmrpp-DmrppInt8.o build_dmrpp-DmrppUInt16.o build_dmrpp-DmrppUInt32.o build_dmrpp-DmrppUInt64.o build_dmrpp-DmrppStr.o build_dmrpp-DmrppStructure.o build_dmrpp-DmrppUrl.o build_dmrpp-DmrppD4Enum.o build_dmrpp-DmrppD4Group.o build_dmrpp-DmrppD4Opaque.o build_dmrpp-DmrppD4Sequence.o build_dmrpp-DmrppTypeFactory.o build_dmrpp-DmrppParserSax2.o build_dmrpp-DmrppMetadataStore.o build_dmrpp-CredentialsManager.o build_dmrpp-AccessCredentials.o build_dmrpp-NgapS3Credentials.o build_dmrpp-SuperChunk.o build_dmrpp-awsv4.o build_dmrpp-DMZ.o  build_dmrpp-DmrppRequestHandler.o build_dmrpp-build_dmrpp.o build_dmrpp-h5common.o -L../../dispatch -lbes_dispatch -L../../http -lbes_http -lcurl -luuid -ltirpc  -L. -lhdf5 -ldl -ldapserver -ldap  -ldapclient -ldap  -L/usr/local/opt/openssl/lib -lcrypto -lxml2   -lz -lcurl -luuid -ltirpc
*** Warning: Linking the executable build_dmrpp against the loadable module
*** libdap_module.so is not portable!
libtool: link: g++ -Wno-vla-extension -Wno-inconsistent-missing-override -g -O2 --pedantic --std=c++11 -dynamic -o build_dmrpp build_dmrpp-DMRpp.o build_dmrpp-DmrppCommon.o build_dmrpp-Chunk.o build_dmrpp-CurlHandlePool.o build_dmrpp-DmrppByte.o build_dmrpp-DmrppArray.o build_dmrpp-DmrppFloat32.o build_dmrpp-DmrppFloat64.o build_dmrpp-DmrppInt16.o build_dmrpp-DmrppInt32.o build_dmrpp-DmrppInt64.o build_dmrpp-DmrppInt8.o build_dmrpp-DmrppUInt16.o build_dmrpp-DmrppUInt32.o build_dmrpp-DmrppUInt64.o build_dmrpp-DmrppStr.o build_dmrpp-DmrppStructure.o build_dmrpp-DmrppUrl.o build_dmrpp-DmrppD4Enum.o build_dmrpp-DmrppD4Group.o build_dmrpp-DmrppD4Opaque.o build_dmrpp-DmrppD4Sequence.o build_dmrpp-DmrppTypeFactory.o build_dmrpp-DmrppParserSax2.o build_dmrpp-DmrppMetadataStore.o build_dmrpp-CredentialsManager.o build_dmrpp-AccessCredentials.o build_dmrpp-NgapS3Credentials.o build_dmrpp-SuperChunk.o build_dmrpp-awsv4.o build_dmrpp-DMZ.o build_dmrpp-DmrppRequestHandler.o build_dmrpp-build_dmrpp.o build_dmrpp-h5common.o  -L../../dap /builddir/build/BUILD/bes-3.20.10/dap/.libs/libdap_module.so -L/usr/lib64/libdap -L../../dispatch /builddir/build/BUILD/bes-3.20.10/dispatch/.libs/libbes_dispatch.so -L../../http /builddir/build/BUILD/bes-3.20.10/http/.libs/libbes_http.a -L../dispatch ../../dispatch/.libs/libbes_dispatch.so -lbz2 -lpthread -L/usr/local/opt/openssl/lib -L. -lhdf5 -ldl -ldapserver -ldapclient -ldap -lcrypto -lxml2 -lz -lcurl -luuid -ltirpc -Wl,-rpath -Wl,/builddir/build/BUILD/bes-3.20.10/dap/.libs -Wl,-rpath -Wl,/builddir/build/BUILD/bes-3.20.10/dispatch/.libs -Wl,-rpath -Wl,/usr/lib64/bes
make[4]: Leaving directory '/builddir/build/BUILD/bes-3.20.10/modules/dmrpp_module'

which libtool notes is "not portable".

autoreconf warnings about subdir-objects setting

Getting some warnings about the subdir-objects setting.

This setting causes object files to be always placed in the same build dir as their source, instead of the build dir of the Makefile.am that is using them. (This is the same directory unless the Makefile.am includes source files from other directories, as happens in several places in bes.) This is documented here: https://www.gnu.org/software/automake/manual/html_node/Program-and-Library-Variables.html

IIRC this is a result of changes in automake defaults. In olden times, subdir-objects was on by default, but now it is off by default.

This is fixed by adding subdir-objects to the list of options for AM_INIT_AUTOMAKE in configure.ac.

 functions/unit-tests/Makefile.am:140: warning: source file '../RangeFunction.cc' is in a subdirectory,
functions/unit-tests/Makefile.am:140: but option 'subdir-objects' is disabled
automake: warning: possible forward-incompatibility.
automake: At least a source file is in a subdirectory, but the 'subdir-objects'
automake: automake option hasn't been enabled.  For now, the corresponding output
automake: object file(s) will be placed in the top-level directory.  However,
automake: this behaviour will change in future Automake versions: they will
automake: unconditionally cause object files to be placed in the same subdirectory
automake: of the corresponding sources.
automake: You are advised to start using 'subdir-objects' option throughout your
automake: project, to avoid future incompatibilities.
functions/unit-tests/Makefile.am:132: warning: source file '../scale_util.cc' is in a subdirectory,
functions/unit-tests/Makefile.am:132: but option 'subdir-objects' is disabled
functions/unit-tests/Makefile.am:136: warning: source file '../scale_util.cc' is in a subdirectory,
functions/unit-tests/Makefile.am:136: but option 'subdir-objects' is disabled
modules/dmrpp_module/Makefile.am:78: warning: source file '$(srcdir)/../hdf5_handler/h5common.cc' is in a subdirectory,
modules/dmrpp_module/Makefile.am:78: but option 'subdir-objects' is disabled
modules/dmrpp_module/unit-tests/Makefile.am:127: warning: source file '$(top_srcdir)/modules/read_test_baseline.cc' is in a subdirectory,
modules/dmrpp_module/unit-tests/Makefile.am:127: but option 'subdir-objects' is disabled
modules/dmrpp_module/unit-tests/Makefile.am:130: warning: source file '$(top_srcdir)/modules/read_test_baseline.cc' is in a subdirectory,
modules/dmrpp_module/unit-tests/Makefile.am:130: but option 'subdir-objects' is disabled

NaN values of attributes get converted to 0.0 when passed through opendap

I have a .nc file with a variable with an attribute missing_value = NaNf. When I download this file from opendap in netcdf3 or netcdf4 format I get missing_value = 0.0f in the resulting file.

I expect to get the same NaNf as in the original file

Here's a script to reproduce the issue (requires bash, docker, wget and ncdump)
https://gist.github.com/samoylovfp/ba1df8ef41f27c44ddeed9f2d4076fc9

I suspect that istringstream is the culprit in these places

https://github.com/OPENDAP/bes/blob/6724e44/modules/fileout_netcdf/FONcAttributes.cc#L389
https://github.com/OPENDAP/bes/blob/6724e44/modules/fileout_netcdf/FONcAttributes.cc#L1030

Because it cannot parse NaN:

https://ideone.com/bhTntI
https://stackoverflow.com/questions/11420263/is-it-possible-to-read-infinity-or-nan-values-using-input-streams

in hdf4_handler many annoying warnings: dynamic exception specifications are deprecated in C++11

In modules/hdf4_handler/HDFEOS2.cc there are many annoying warnings like this:

HDFEOS2.h:675:51: warning: dynamic exception specifications are deprecated in C++11 [-Wdeprecated]
  675 |                 std::vector < Dimension * >&dims) throw (Exception);
      |                                                   ^~~~~

According to this StackOverflow question, these dynamic exceptions can be simply removed, and they will become an error in C++17, so should be removed:
https://stackoverflow.com/questions/50254641/dynamic-exception-specifications-are-deprecated

I have experimented and removing the throws specification in the function declaration and the header seems to work fine and resolve the warning. I will submit a PR, but would like to see the CI working first...

ODR violations when modules loaded into besstandalone

In the dmrpp_module/testsuite/bes.conf.in file both the runtime loadable modules 

  • libdmrpp_module.so
  • libbuilddmrpp_module.so

are loaded into besstandalone from the bes.conf file in the modules/dmrpp_module /testsuite directory. This causes ODR violations to be detected by ASAN. The first error encountered is the redefinition of http/HttpError.cc 

The obvious fix here is to simply eliminate the use of libbuilddmrpp_module.so from the bes.conf.in because it's not used in these tests. I removed it and ASAN was happy and all the tests behaved as expected.

But it is an outstanding question as to why this ODR violation happened given that we load lots of modules. Is this something we need to address in how we structure our build/deployment?

Inquiring minds want to know.

Build failure with hdf5 1.12.1

Trying to build 3.20.11 for Fedora rawhide with hdf5-1.12-1:

/bin/sh ../../libtool  --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H -I. -I../..  -I gctp/include -I. -I../../dispatch -I../../dap -I/usr/include/libdap -I/usr/include/libxml2  -DMODULE_NAME=\"hdf5_handler\" -DMODULE_VERSION=\"1.16.6\" -I/usr/include/cfitsio -I/usr/include/tirpc -I/include  -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1  -m64  -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection --std=c++11 -c -o HDF5Array.lo HDF5Array.cc
libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I../.. -I gctp/include -I. -I../../dispatch -I../../dap -I/usr/include/libdap -I/usr/include/libxml2 -DMODULE_NAME=\"hdf5_handler\" -DMODULE_VERSION=\"1.16.6\" -I/usr/include/cfitsio -I/usr/include/tirpc -I/include -O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection --std=c++11 -c HDF5Array.cc  -fPIC -DPIC -o .libs/HDF5Array.o
HDF5Array.cc: In member function 'bool HDF5Array::m_array_of_reference_new_h5_apis(hid_t, hid_t)':
HDF5Array.cc:840:84: error: request for member 'data' in 'rbuf', which is of pointer type 'H5R_ref_t*' (maybe you meant to use '->' ?)
  840 |         if(H5Dread(dset_id,H5T_STD_REF,mem_space_id,file_space_id,H5P_DEFAULT,rbuf.data())<0)
      |                                                                                    ^~~~
HDF5Array.cc:849:67: error: request for member 'data' in 'rbuf', which is of pointer type 'H5R_ref_t*' (maybe you meant to use '->' ?)
  849 |         H5R_type_t ref_type = H5Rget_type((const H5R_ref_t *)rbuf.data());
      |                                                                   ^~~~
make[5]: *** [Makefile:862: HDF5Array.lo] Error 1

Unpacking source package 3.14.0 gives strange message.

I get the following messages on CentOS:

tar: Ignoring unknown extended header keyword `SCHILY.dev`
tar: Ignoring unknown extended header keyword `SCHILY.ino`
tar: Ignoring unknown extended header keyword `SCHILY.nlink`

I think the source package is made from Mac OS X and causes the above output.

Is the 'CoverageCollection' a misinterpretation of NASA data in the covjson response in Hyrax

In our test server of Hyrax 1.15.1, if I request a covjson file for a granule, most returns as 'CoverageCollection'. Some of these are globally gridded data with multiple observations (arrays) sharing a single pair of latitude and longitude coordinates. For example

https://snpp-omps.gesdisc.eosdis.nasa.gov/opendap/SNPP_OMPS_Level3/OMPS_NPP_NMTO3_L3_DAILY.2/2012/OMPS-NPP_NMTO3-L3-DAILY_v2.1_2012m0126_2017m0227t060834.h5.covjson

Should the granule be interpreted as a 'Coverage' or a 'CoverageCollection'?

make check does an install?

make check fails like this. Apparently make check is trying to do an install?

make[3]: Entering directory '/home/ed/bes/cmdln/tests'
 /bin/mkdir -p '/usr/local/bes-3.20.7/share/mds'
/bin/mkdir: cannot create directory ‘/usr/local/bes-3.20.7’: Permission denied

Zarr support

Hello, do you have a plan to develop a module to read Zarr files with N5?

I've started looking at the code, but I can't see if the work required seems huge or not. If I want to try to develop it, which module should I use as an example? NetCDF? Thanks for your help.

Cannot install experimental bes .deb package from Hyrax 1.11 release (Ubuntu 14.04)

I downloaded bes_3.14.0-2.NASA_x86_64.deb from OPeNDAP's experimental debian package index, but ran into a dependency issue when trying to install it with gdebi. Apparently it requires the package libbz2, which is not available in the default repos for Ubuntu 14.04 (Trusty Tahr).

There is, however, a package named libbz2-1.0 available for Trusty Tahr; maybe the dependency could be changed to this in the published .deb?

Below is my output showing the failed installation:

~$ sudo gdebi bes_3.14.0-2.NASA_x86_64.deb
Reading package lists... Done
Building dependency tree
Reading state information... Done
Building data structures... Done
Building data structures... Done
This package is uninstallable
Dependency is not satisfiable: libbz2

The development package, bes-devel_3.14.0-2.NASA_x86_64.deb, installed without issue.

"Product of dimension sizes exceeds the maximum DAP2 size"

I tested the support of dimension name change in the NcML module in Hyrax 1.15.4 (I am not sure if anything has changed in 1.16 - thanks if so!). It worked well until I noticed this error message, showed up when a changed dimension name was referred to in an element that attempted to define the values of a new variable. Defining new variable is one of the functionalities in NcML, and was a working functionality in the previous releases of the module. So I'd like to report it as a backward compatibility issue.

"Hyrax - Bad Request (400)
NCMLModule ParseError: at *.ncml line=881: Product of dimension sizes exceeds the maximum DAP2 size of 2147483647 (2^31-1)!"

I can send you the offending .ncml file if this is indeed a new problem.

test ppt/unit-tests/extT.cc fails due to missing return code

When doing a make -j check, I see this warning:

extT.cc: In member function ‘int extT::check_extensions(int, std::map<std::__cxx11::basic_string<char>, std::__cxx11::basic_string<char> >&, std::map<std::__cxx11::basic_string<char>, std::__cxx11::basic_string<char> >&)’:
extT.cc:106:5: warning: no return statement in function returning non-void [-Wreturn-type]
  106 |     }
      |     ^

And the test fails.

Looking in the code, I see that the int function is returning without a value. I added a "return 0" and now the test works.

libncml_module.so is not being installed when building hyrax from source

When I use the suggested procedure to build hyrax and dependencies from https://github.com/OPENDAP/hyrax
(source spath.sh; ./hyrax_clone.sh; ./hyrax_build.sh),
libncml_module.so fails its checks and is not being installed.

My guess is that it is because of a missing "rpath" to the icu library, which was built from source as a dependency and was not installed in a standard location. At least running "ldd" on libncml_module.so shows the missing dependency. Manually copying libncml_module.so to build/lib/bes and adding build/deps/lib to LD_LIBRARY_PATH makes the ncml module work.

build failed of bes 3.21.0-46 on Fedora

Making all in xmlcommand
make[2]: Entering directory '/home/ruby/rpmbuild/SOURCES/bes-3.21.0/xmlcommand'
Making all in .
make[3]: Entering directory '/home/ruby/rpmbuild/SOURCES/bes-3.21.0/xmlcommand'
depbase=`echo BESXMLInterface.lo | sed 's|[^/]*$|.deps/&|;s|\.lo$||'`;\
/bin/sh ../libtool  --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H -I. -I..  -I../dap -I../dispatch -I/usr/include/libxml2 -DWITH_GZFILEOP  -I/include  -g -O2  --pedantic --std=c++14 -MT BESXMLInterface.lo -MD -MP -MF $depbase.Tpo -c -o BESXMLInterface.lo BESXMLInterface.cc &&\
mv -f $depbase.Tpo $depbase.Plo
depbase=`echo BESXMLCommand.lo | sed 's|[^/]*$|.deps/&|;s|\.lo$||'`;\
/bin/sh ../libtool  --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H -I. -I..  -I../dap -I../dispatch -I/usr/include/libxml2 -DWITH_GZFILEOP  -I/include  -g -O2  --pedantic --std=c++14 -MT BESXMLCommand.lo -MD -MP -MF $depbase.Tpo -c -o BESXMLCommand.lo BESXMLCommand.cc &&\
mv -f $depbase.Tpo $depbase.Plo
depbase=`echo BESXMLUtils.lo | sed 's|[^/]*$|.deps/&|;s|\.lo$||'`;\
/bin/sh ../libtool  --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H -I. -I..  -I../dap -I../dispatch -I/usr/include/libxml2 -DWITH_GZFILEOP  -I/include  -g -O2  --pedantic --std=c++14 -MT BESXMLUtils.lo -MD -MP -MF $depbase.Tpo -c -o BESXMLUtils.lo BESXMLUtils.cc &&\
mv -f $depbase.Tpo $depbase.Plo
depbase=`echo BESXMLDefaultCommands.lo | sed 's|[^/]*$|.deps/&|;s|\.lo$||'`;\
/bin/sh ../libtool  --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H -I. -I..  -I../dap -I../dispatch -I/usr/include/libxml2 -DWITH_GZFILEOP  -I/include  -g -O2  --pedantic --std=c++14 -MT BESXMLDefaultCommands.lo -MD -MP -MF $depbase.Tpo -c -o BESXMLDefaultCommands.lo BESXMLDefaultCommands.cc &&\
mv -f $depbase.Tpo $depbase.Plo
depbase=`echo BESXMLShowCommand.lo | sed 's|[^/]*$|.deps/&|;s|\.lo$||'`;\
/bin/sh ../libtool  --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H -I. -I..  -I../dap -I../dispatch -I/usr/include/libxml2 -DWITH_GZFILEOP  -I/include  -g -O2  --pedantic --std=c++14 -MT BESXMLShowCommand.lo -MD -MP -MF $depbase.Tpo -c -o BESXMLShowCommand.lo BESXMLShowCommand.cc &&\
mv -f $depbase.Tpo $depbase.Plo
depbase=`echo BESXMLShowErrorCommand.lo | sed 's|[^/]*$|.deps/&|;s|\.lo$||'`;\
/bin/sh ../libtool  --tag=CXX   --mode=compile g++ -DHAVE_CONFIG_H -I. -I..  -I../dap -I../dispatch -I/usr/include/libxml2 -DWITH_GZFILEOP  -I/include  -g -O2  --pedantic --std=c++14 -MT BESXMLShowErrorCommand.lo -MD -MP -MF $depbase.Tpo -c -o BESXMLShowErrorCommand.lo BESXMLShowErrorCommand.cc &&\
mv -f $depbase.Tpo $depbase.Plo
libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -I../dap -I../dispatch -I/usr/include/libxml2 -DWITH_GZFILEOP -I/include -g -O2 --pedantic --std=c++14 -MT BESXMLInterface.lo -MD -MP -MF .deps/BESXMLInterface.Tpo -c BESXMLInterface.cc  -fPIC -DPIC -o .libs/BESXMLInterface.o
libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -I../dap -I../dispatch -I/usr/include/libxml2 -DWITH_GZFILEOP -I/include -g -O2 --pedantic --std=c++14 -MT BESXMLDefaultCommands.lo -MD -MP -MF .deps/BESXMLDefaultCommands.Tpo -c BESXMLDefaultCommands.cc  -fPIC -DPIC -o .libs/BESXMLDefaultCommands.o
libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -I../dap -I../dispatch -I/usr/include/libxml2 -DWITH_GZFILEOP -I/include -g -O2 --pedantic --std=c++14 -MT BESXMLCommand.lo -MD -MP -MF .deps/BESXMLCommand.Tpo -c BESXMLCommand.cc  -fPIC -DPIC -o .libs/BESXMLCommand.o
libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -I../dap -I../dispatch -I/usr/include/libxml2 -DWITH_GZFILEOP -I/include -g -O2 --pedantic --std=c++14 -MT BESXMLUtils.lo -MD -MP -MF .deps/BESXMLUtils.Tpo -c BESXMLUtils.cc  -fPIC -DPIC -o .libs/BESXMLUtils.o
libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -I../dap -I../dispatch -I/usr/include/libxml2 -DWITH_GZFILEOP -I/include -g -O2 --pedantic --std=c++14 -MT BESXMLShowErrorCommand.lo -MD -MP -MF .deps/BESXMLShowErrorCommand.Tpo -c BESXMLShowErrorCommand.cc  -fPIC -DPIC -o .libs/BESXMLShowErrorCommand.o
libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -I../dap -I../dispatch -I/usr/include/libxml2 -DWITH_GZFILEOP -I/include -g -O2 --pedantic --std=c++14 -MT BESXMLShowCommand.lo -MD -MP -MF .deps/BESXMLShowCommand.Tpo -c BESXMLShowCommand.cc  -fPIC -DPIC -o .libs/BESXMLShowCommand.o
In file included from BESXMLCommand.cc:37:
BESXMLCommand.h:88:32: error: 'xmlNode' has not been declared
   88 |     virtual void parse_request(xmlNode *node) = 0;
      |                                ^~~~~~~
make[3]: *** [Makefile:758: BESXMLCommand.lo] Error 1
make[3]: *** Waiting for unfinished jobs....
In file included from BESXMLShowCommand.h:36,
                 from BESXMLDefaultCommands.cc:42:
BESXMLCommand.h:88:32: error: 'xmlNode' has not been declared
   88 |     virtual void parse_request(xmlNode *node) = 0;
      |                                ^~~~~~~
BESXMLShowCommand.h:46:32: error: 'xmlNode' has not been declared
   46 |     virtual void parse_request(xmlNode *node);
      |                                ^~~~~~~
In file included from BESXMLDefaultCommands.cc:43:
BESXMLShowErrorCommand.h:46:32: error: 'xmlNode' has not been declared
   46 |     virtual void parse_request(xmlNode *node);
      |                                ^~~~~~~
In file included from BESXMLDefaultCommands.cc:44:
BESXMLSetContextCommand.h:46:32: error: 'xmlNode' has not been declared
   46 |     virtual void parse_request(xmlNode *node);
      |                                ^~~~~~~
In file included from BESXMLDefaultCommands.cc:45:
BESXMLSetContainerCommand.h:61:32: error: 'xmlNode' has not been declared
   61 |     virtual void parse_request(xmlNode *node);
      |                                ^~~~~~~
In file included from BESXMLDefaultCommands.cc:46:
BESXMLDefineCommand.h:57:62: error: 'xmlNode' has not been declared
   57 |     void handle_container_element(const std::string &action, xmlNode *node, const std::string &vallues,
      |                                                              ^~~~~~~
BESXMLDefineCommand.h:72:32: error: 'xmlNode' has not been declared
   72 |     virtual void parse_request(xmlNode *node);
      |                                ^~~~~~~
In file included from BESXMLDefaultCommands.cc:47:
BESXMLGetCommand.h:56:32: error: 'xmlNode' has not been declared
   56 |     virtual void parse_request(xmlNode *node);
      |                                ^~~~~~~
In file included from BESXMLDefaultCommands.cc:48:
BESXMLDeleteContainerCommand.h:46:32: error: 'xmlNode' has not been declared
   46 |     virtual void parse_request(xmlNode *node);
      |                                ^~~~~~~
In file included from BESXMLDefaultCommands.cc:49:
BESXMLDeleteContainersCommand.h:46:32: error: 'xmlNode' has not been declared
   46 |     virtual void parse_request(xmlNode *node);
      |                                ^~~~~~~
In file included from BESXMLDefaultCommands.cc:50:
BESXMLDeleteDefinitionCommand.h:46:32: error: 'xmlNode' has not been declared
   46 |     virtual void parse_request(xmlNode *node);
      |                                ^~~~~~~
In file included from BESXMLDefaultCommands.cc:51:
BESXMLDeleteDefinitionsCommand.h:46:32: error: 'xmlNode' has not been declared
   46 |     virtual void parse_request(xmlNode *node);
      |                                ^~~~~~~
In file included from BESXMLDefaultCommands.cc:53:
ShowPathInfoCommand.h:41:32: error: 'xmlNode' has not been declared
   41 |     virtual void parse_request(xmlNode *node);
      |                                ^~~~~~~
In file included from BESXMLDefaultCommands.cc:54:
ShowBesKeyCommand.h:47:32: error: 'xmlNode' has not been declared
   47 |     virtual void parse_request(xmlNode *node);
      |                                ^~~~~~~
In file included from BESXMLDefaultCommands.cc:58:
XMLSetContextsCommand.h:38:32: error: 'xmlNode' has not been declared
   38 |     virtual void parse_request(xmlNode *node);
      |                                ^~~~~~~
make[3]: *** [Makefile:758: BESXMLDefaultCommands.lo] Error 1
In file included from BESXMLShowErrorCommand.h:36,
                 from BESXMLShowErrorCommand.cc:35:
BESXMLCommand.h:88:32: error: 'xmlNode' has not been declared
   88 |     virtual void parse_request(xmlNode *node) = 0;
      |                                ^~~~~~~
BESXMLShowErrorCommand.h:46:32: error: 'xmlNode' has not been declared
   46 |     virtual void parse_request(xmlNode *node);
      |                                ^~~~~~~
In file included from BESXMLShowErrorCommand.cc:36:
BESXMLUtils.h:49:26: error: 'xmlNode' has not been declared
   49 |     static void GetProps(xmlNode *node, std::map<std::string, std::string> &props);
      |                          ^~~~~~~
BESXMLUtils.h:50:29: error: 'xmlNode' has not been declared
   50 |     static void GetNodeInfo(xmlNode *node, std::string &name, std::string &value, std::map<std::string, std::string> &props);
      |                             ^~~~~~~
BESXMLUtils.h:51:12: error: 'xmlNode' does not name a type
   51 |     static xmlNode * GetFirstChild(xmlNode *node, std::string &child_name, std::string &child_value,
      |            ^~~~~~~
BESXMLUtils.h:53:12: error: 'xmlNode' does not name a type
   53 |     static xmlNode * GetNextChild(xmlNode *child_node, std::string &next_name, std::string &next_value,
      |            ^~~~~~~
BESXMLUtils.h:55:12: error: 'xmlNode' does not name a type
   55 |     static xmlNode * GetChild(xmlNode *node, const std::string &child_name, std::string &child_value,
      |            ^~~~~~~
BESXMLShowErrorCommand.cc:66:6: error: variable or field 'parse_request' declared void
   66 | void BESXMLShowErrorCommand::parse_request(xmlNode *node)
      |      ^~~~~~~~~~~~~~~~~~~~~~
BESXMLShowErrorCommand.cc:66:44: error: 'xmlNode' was not declared in this scope
   66 | void BESXMLShowErrorCommand::parse_request(xmlNode *node)
      |                                            ^~~~~~~
BESXMLShowErrorCommand.cc:66:53: error: 'node' was not declared in this scope
   66 | void BESXMLShowErrorCommand::parse_request(xmlNode *node)
      |                                                     ^~~~
In file included from BESXMLUtils.cc:33:
BESXMLUtils.h:49:26: error: 'xmlNode' has not been declared
   49 |     static void GetProps(xmlNode *node, std::map<std::string, std::string> &props);
      |                          ^~~~~~~
BESXMLUtils.h:50:29: error: 'xmlNode' has not been declared
   50 |     static void GetNodeInfo(xmlNode *node, std::string &name, std::string &value, std::map<std::string, std::string> &props);
      |                             ^~~~~~~
BESXMLUtils.h:51:12: error: 'xmlNode' does not name a type
   51 |     static xmlNode * GetFirstChild(xmlNode *node, std::string &child_name, std::string &child_value,
      |            ^~~~~~~
BESXMLUtils.h:53:12: error: 'xmlNode' does not name a type
   53 |     static xmlNode * GetNextChild(xmlNode *child_node, std::string &next_name, std::string &next_value,
      |            ^~~~~~~
BESXMLUtils.h:55:12: error: 'xmlNode' does not name a type
   55 |     static xmlNode * GetChild(xmlNode *node, const std::string &child_name, std::string &child_value,
      |            ^~~~~~~
BESXMLUtils.cc: In static member function 'static void BESXMLUtils::XMLErrorFunc(void*, const char*, ...)':
BESXMLUtils.cc:57:5: error: 'va_start' was not declared in this scope
   57 |     va_start( args, msg );
      |     ^~~~~~~~
BESXMLUtils.cc:60:5: error: 'va_end' was not declared in this scope
   60 |     va_end(args); // Added jhrg 9/17/15
      |     ^~~~~~
BESXMLUtils.cc: At global scope:
BESXMLUtils.cc:74:6: error: variable or field 'GetProps' declared void
   74 | void BESXMLUtils::GetProps(xmlNode *node, map<string, string> &props)
      |      ^~~~~~~~~~~
BESXMLUtils.cc:74:28: error: 'xmlNode' was not declared in this scope
   74 | void BESXMLUtils::GetProps(xmlNode *node, map<string, string> &props)
      |                            ^~~~~~~
BESXMLUtils.cc:74:37: error: 'node' was not declared in this scope
   74 | void BESXMLUtils::GetProps(xmlNode *node, map<string, string> &props)
      |                                     ^~~~
BESXMLUtils.cc:74:63: error: expected primary-expression before '&' token
   74 | void BESXMLUtils::GetProps(xmlNode *node, map<string, string> &props)
      |                                                               ^
BESXMLUtils.cc:74:64: error: 'props' was not declared in this scope
   74 | void BESXMLUtils::GetProps(xmlNode *node, map<string, string> &props)
      |                                                                ^~~~~
BESXMLUtils.cc:109:6: error: variable or field 'GetNodeInfo' declared void
  109 | void BESXMLUtils::GetNodeInfo(xmlNode *node, string &name, string &value, map<string, string> &props)
      |      ^~~~~~~~~~~
BESXMLUtils.cc:109:31: error: 'xmlNode' was not declared in this scope
  109 | void BESXMLUtils::GetNodeInfo(xmlNode *node, string &name, string &value, map<string, string> &props)
      |                               ^~~~~~~
BESXMLUtils.cc:109:40: error: 'node' was not declared in this scope
  109 | void BESXMLUtils::GetNodeInfo(xmlNode *node, string &name, string &value, map<string, string> &props)
      |                                        ^~~~
BESXMLUtils.cc:109:53: error: expected primary-expression before '&' token
  109 | void BESXMLUtils::GetNodeInfo(xmlNode *node, string &name, string &value, map<string, string> &props)
      |                                                     ^
BESXMLUtils.cc:109:54: error: 'name' was not declared in this scope; did you mean 'tzname'?
  109 | void BESXMLUtils::GetNodeInfo(xmlNode *node, string &name, string &value, map<string, string> &props)
      |                                                      ^~~~
      |                                                      tzname
BESXMLUtils.cc:109:67: error: expected primary-expression before '&' token
  109 | void BESXMLUtils::GetNodeInfo(xmlNode *node, string &name, string &value, map<string, string> &props)
      |                                                                   ^
BESXMLUtils.cc:109:68: error: 'value' was not declared in this scope
  109 | void BESXMLUtils::GetNodeInfo(xmlNode *node, string &name, string &value, map<string, string> &props)
      |                                                                    ^~~~~
BESXMLUtils.cc:109:95: error: expected primary-expression before '&' token
  109 | void BESXMLUtils::GetNodeInfo(xmlNode *node, string &name, string &value, map<string, string> &props)
      |                                                                                               ^
BESXMLUtils.cc:109:96: error: 'props' was not declared in this scope
  109 | void BESXMLUtils::GetNodeInfo(xmlNode *node, string &name, string &value, map<string, string> &props)
      |                                                                                                ^~~~~
BESXMLUtils.cc:142:1: error: 'xmlNode' does not name a type
  142 | xmlNode *
      | ^~~~~~~
BESXMLUtils.cc:169:1: error: 'xmlNode' does not name a type
  169 | xmlNode *
      | ^~~~~~~
BESXMLUtils.cc:195:1: error: 'xmlNode' does not name a type
  195 | xmlNode *
      | ^~~~~~~
make[3]: *** [Makefile:758: BESXMLShowErrorCommand.lo] Error 1
make[3]: *** [Makefile:758: BESXMLUtils.lo] Error 1
In file included from BESXMLShowCommand.h:36,
                 from BESXMLShowCommand.cc:35:
BESXMLCommand.h:88:32: error: 'xmlNode' has not been declared
   88 |     virtual void parse_request(xmlNode *node) = 0;
      |                                ^~~~~~~
BESXMLShowCommand.h:46:32: error: 'xmlNode' has not been declared
   46 |     virtual void parse_request(xmlNode *node);
      |                                ^~~~~~~
In file included from BESXMLShowCommand.cc:36:
BESXMLUtils.h:49:26: error: 'xmlNode' has not been declared
   49 |     static void GetProps(xmlNode *node, std::map<std::string, std::string> &props);
      |                          ^~~~~~~
BESXMLUtils.h:50:29: error: 'xmlNode' has not been declared
   50 |     static void GetNodeInfo(xmlNode *node, std::string &name, std::string &value, std::map<std::string, std::string> &props);
      |                             ^~~~~~~
BESXMLUtils.h:51:12: error: 'xmlNode' does not name a type
   51 |     static xmlNode * GetFirstChild(xmlNode *node, std::string &child_name, std::string &child_value,
      |            ^~~~~~~
BESXMLUtils.h:53:12: error: 'xmlNode' does not name a type
   53 |     static xmlNode * GetNextChild(xmlNode *child_node, std::string &next_name, std::string &next_value,
      |            ^~~~~~~
BESXMLUtils.h:55:12: error: 'xmlNode' does not name a type
   55 |     static xmlNode * GetChild(xmlNode *node, const std::string &child_name, std::string &child_value,
      |            ^~~~~~~
BESXMLShowCommand.cc:61:6: error: variable or field 'parse_request' declared void
   61 | void BESXMLShowCommand::parse_request(xmlNode *node)
      |      ^~~~~~~~~~~~~~~~~
BESXMLShowCommand.cc:61:39: error: 'xmlNode' was not declared in this scope
   61 | void BESXMLShowCommand::parse_request(xmlNode *node)
      |                                       ^~~~~~~
BESXMLShowCommand.cc:61:48: error: 'node' was not declared in this scope
   61 | void BESXMLShowCommand::parse_request(xmlNode *node)
      |                                                ^~~~
make[3]: *** [Makefile:758: BESXMLShowCommand.lo] Error 1
In file included from BESXMLInterface.cc:39:
BESXMLCommand.h:88:32: error: 'xmlNode' has not been declared
   88 |     virtual void parse_request(xmlNode *node) = 0;
      |                                ^~~~~~~
In file included from BESXMLInterface.cc:40:
BESXMLUtils.h:49:26: error: 'xmlNode' has not been declared
   49 |     static void GetProps(xmlNode *node, std::map<std::string, std::string> &props);
      |                          ^~~~~~~
BESXMLUtils.h:50:29: error: 'xmlNode' has not been declared
   50 |     static void GetNodeInfo(xmlNode *node, std::string &name, std::string &value, std::map<std::string, std::string> &props);
      |                             ^~~~~~~
BESXMLUtils.h:51:12: error: 'xmlNode' does not name a type
   51 |     static xmlNode * GetFirstChild(xmlNode *node, std::string &child_name, std::string &child_value,
      |            ^~~~~~~
BESXMLUtils.h:53:12: error: 'xmlNode' does not name a type
   53 |     static xmlNode * GetNextChild(xmlNode *child_node, std::string &next_name, std::string &next_value,
      |            ^~~~~~~
BESXMLUtils.h:55:12: error: 'xmlNode' does not name a type
   55 |     static xmlNode * GetChild(xmlNode *node, const std::string &child_name, std::string &child_value,
      |            ^~~~~~~
BESXMLInterface.cc: In member function 'virtual void BESXMLInterface::build_data_request_plan()':
BESXMLInterface.cc:88:5: error: 'xmlDoc' was not declared in this scope
   88 |     xmlDoc *doc = nullptr;
      |     ^~~~~~
BESXMLInterface.cc:88:13: error: 'doc' was not declared in this scope
   88 |     xmlDoc *doc = nullptr;
      |             ^~~
BESXMLInterface.cc:89:5: error: 'xmlNode' was not declared in this scope
   89 |     xmlNode *root_element = nullptr;
      |     ^~~~~~~
BESXMLInterface.cc:89:14: error: 'root_element' was not declared in this scope
   89 |     xmlNode *root_element = nullptr;
      |              ^~~~~~~~~~~~
BESXMLInterface.cc:90:14: error: 'current_node' was not declared in this scope
   90 |     xmlNode *current_node = nullptr;
      |              ^~~~~~~~~~~~
BESXMLInterface.cc:95:9: error: 'xmlSetGenericErrorFunc' was not declared in this scope
   95 |         xmlSetGenericErrorFunc((void *) &parseerrors, BESXMLUtils::XMLErrorFunc);
      |         ^~~~~~~~~~~~~~~~~~~~~~
BESXMLInterface.cc:99:53: error: 'XML_PARSE_NONET' was not declared in this scope
   99 |                             nullptr /* encoding */, XML_PARSE_NONET /* xmlParserOption */);
      |                                                     ^~~~~~~~~~~~~~~
BESXMLInterface.cc:98:15: error: 'xmlReadMemory' was not declared in this scope
   98 |         doc = xmlReadMemory(d_xml_document.c_str(), (int)d_xml_document.size(), "" /* base URL */,
      |               ^~~~~~~~~~~~~
BESXMLInterface.cc:117:24: error: 'xmlDocGetRootElement' was not declared in this scope
  117 |         root_element = xmlDocGetRootElement(doc);
      |                        ^~~~~~~~~~~~~~~~~~~~
BESXMLInterface.cc:147:39: error: 'XML_ELEMENT_NODE' was not declared in this scope
  147 |             if (current_node->type == XML_ELEMENT_NODE) {
      |                                       ^~~~~~~~~~~~~~~~
BESXMLInterface.cc:228:9: error: 'xmlFreeDoc' was not declared in this scope
  228 |         xmlFreeDoc(doc);
      |         ^~~~~~~~~~
BESXMLInterface.cc:229:9: error: 'xmlCleanupParser' was not declared in this scope
  229 |         xmlCleanupParser();
      |         ^~~~~~~~~~~~~~~~
BESXMLInterface.cc:233:5: error: 'xmlFreeDoc' was not declared in this scope
  233 |     xmlFreeDoc(doc);
      |     ^~~~~~~~~~
BESXMLInterface.cc:245:5: error: 'xmlCleanupParser' was not declared in this scope
  245 |     xmlCleanupParser();
      |     ^~~~~~~~~~~~~~~~
make[3]: *** [Makefile:758: BESXMLInterface.lo] Error 1
make[3]: Leaving directory '/home/ruby/rpmbuild/SOURCES/bes-3.21.0/xmlcommand'
make[2]: *** [Makefile:799: all-recursive] Error 1
make[2]: Leaving directory '/home/ruby/rpmbuild/SOURCES/bes-3.21.0/xmlcommand'
make[1]: *** [Makefile:773: all-recursive] Error 1
make[1]: Leaving directory '/home/ruby/rpmbuild/SOURCES/bes-3.21.0'
make: *** [Makefile:548: all] Error 2

detailed build log: https://gist.github.com/topazus/a369486a6b061e9c3a8aac31c015bf71

ASAN finds "misaligned address" problems in the fileout_netcdf module tests.

When compiled with --enable-asan the fileout_netcdf handler generates 4 test failures

207: bescmd/vdata_test.hdf.dmrpp.bescmd              FAILED (testsuite.at:320)
210: bescmd/vdata_packed_linked_blocks.hdf.dmrpp.bescmd FAILED (testsuite.at:323)
216: bescmd/PointFile.hdf.dmrpp.bescmd               FAILED (testsuite.at:329)
217: bescmd/vdata_test.hdf.bescmd                    FAILED (testsuite.at:333)
218: bescmd/vdata_packed_linked_blocks.hdf.bescmd    FAILED (testsuite.at:334)

These errors are all of the same nature: runtime error: load of misaligned address

Is this a design issue arising from using void * or is this a usage issue in the way that the fileout_netcdf is allocating memory?

Or is this problem caused by allocating variable memory as byte/char rather than unit32_t etc.??

Examine and fix if feasible.

Here is the verbose test output for the failed tests:

[-bash: ~/OPeNDAP/hyrax/bes/modules/fileout_netcdf/tests] ./testsuite -v 207 210 216 217 218
## -------------------------------------------------------------- ##
## bes 3.21.0 test suite: modules/fileout_netcdf/tests testsuite. ##
## -------------------------------------------------------------- ##
207. testsuite.at:320: testing bescmd/vdata_test.hdf.dmrpp.bescmd ...
COMMAND: besstandalone  -c bes.nc4.grp.conf -i bescmd/vdata_test.hdf.dmrpp.bescmd
./testsuite.at:320: besstandalone -c $abs_builddir/$bes_conf -i $input > test.nc
--- /dev/null	2024-05-30 11:28:15
+++ /Users/ndp/OPeNDAP/hyrax/bes/modules/fileout_netcdf/tests/testsuite.dir/at-groups/207/stderr	2024-05-30 11:28:16
@@ -0,0 +1,25 @@
+Float32.cc:187:13: runtime error: load of misaligned address 0x611000006a86 for type 'dods_float32' (aka 'float'), which requires 4 byte alignment
+0x611000006a86: note: pointer points here
+ 8e 3f 00 00 7b 14  8e 3f 41 61 30 3f 80 00  00 40 00 00 00 40 0e 14  7b 00 01 40 0e 14 7b 42  62 31
+             ^ 
+SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Float32.cc:187:13 in 
+DmrppStructure.cc:154:26: runtime error: load of misaligned address 0x611000006a99 for type 'dods_uint16' (aka 'unsigned short'), which requires 2 byte alignment
+0x611000006a99: note: pointer points here
+ 7b 14 0e  40 00 01 40 0e 14 7b 42  62 31 3f 80 00 00 40 00  00 00 40 55 1e b8 00 02  40 55 1e b8 43
+              ^ 
+SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior DmrppStructure.cc:154:26 in 
+DmrppStructure.cc:154:26: runtime error: load of misaligned address 0x611000006a99 for type 'dods_uint16' (aka 'unsigned short'), which requires 2 byte alignment
+0x611000006a99: note: pointer points here
+ 7b 14 0e  40 00 01 40 0e 14 7b 42  62 31 3f 80 00 00 40 00  00 00 40 55 1e b8 00 02  40 55 1e b8 43
+              ^ 
+SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior DmrppStructure.cc:154:26 in 
+DmrppStructure.cc:154:17: runtime error: store to misaligned address 0x611000006a99 for type 'dods_uint16' (aka 'unsigned short'), which requires 2 byte alignment
+0x611000006a99: note: pointer points here
+ 7b 14 0e  40 00 01 40 0e 14 7b 42  62 31 3f 80 00 00 40 00  00 00 40 55 1e b8 00 02  40 55 1e b8 43
+              ^ 
+SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior DmrppStructure.cc:154:17 in 
+Int16.cc:180:13: runtime error: load of misaligned address 0x611000006a99 for type 'dods_int16' (aka 'short'), which requires 2 byte alignment
+0x611000006a99: note: pointer points here
+ 7b 14 0e  40 01 00 40 0e 14 7b 42  62 31 3f 80 00 00 40 00  00 00 40 55 1e b8 00 02  40 55 1e b8 43
+              ^ 
+SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Int16.cc:180:13 in 
207. testsuite.at:320:  FAILED (testsuite.at:320)

210. testsuite.at:323: testing bescmd/vdata_packed_linked_blocks.hdf.dmrpp.bescmd ...
COMMAND: besstandalone  -c bes.nc4.grp.conf -i bescmd/vdata_packed_linked_blocks.hdf.dmrpp.bescmd
./testsuite.at:323: besstandalone -c $abs_builddir/$bes_conf -i $input > test.nc
--- /dev/null	2024-05-30 11:28:15
+++ /Users/ndp/OPeNDAP/hyrax/bes/modules/fileout_netcdf/tests/testsuite.dir/at-groups/210/stderr	2024-05-30 11:28:17
@@ -0,0 +1,5 @@
+Float32.cc:187:13: runtime error: load of misaligned address 0x60300002a0d6 for type 'dods_float32' (aka 'float'), which requires 4 byte alignment
+0x60300002a0d6: note: pointer points here
+ 8e 3f 00 00 7b 14  0e 40 00 01 3f 8e 14 7b  00 00 40 0e 14 7b 00 01  00 00 00 00 00 00 00 00  02 02
+             ^ 
+SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Float32.cc:187:13 in 
210. testsuite.at:323:  FAILED (testsuite.at:323)

216. testsuite.at:329: testing bescmd/PointFile.hdf.dmrpp.bescmd ...
COMMAND: besstandalone  -c bes.nc4.grp.conf -i bescmd/PointFile.hdf.dmrpp.bescmd
./testsuite.at:329: besstandalone -c $abs_builddir/$bes_conf -i $input > test.nc
--- /dev/null	2024-05-30 11:28:15
+++ /Users/ndp/OPeNDAP/hyrax/bes/modules/fileout_netcdf/tests/testsuite.dir/at-groups/216/stderr	2024-05-30 11:28:18
@@ -0,0 +1,5 @@
+Float64.cc:185:13: runtime error: load of misaligned address 0x61400000465c for type 'dods_float64' (aka 'double'), which requires 8 byte alignment
+0x61400000465c: note: pointer points here
+  43 4f 00 00 9a 99 99 25  1c 35 86 41 40 7f 5c 29  40 90 a3 d7 40 16 66 66  40 21 47 ae 43 4f 32 00
+              ^ 
+SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Float64.cc:185:13 in 
216. testsuite.at:329:  FAILED (testsuite.at:329)

217. testsuite.at:333: testing bescmd/vdata_test.hdf.bescmd ...
COMMAND: besstandalone  -c bes.nc4.grp.conf -i bescmd/vdata_test.hdf.bescmd
./testsuite.at:333: besstandalone -c $abs_builddir/$bes_conf -i $input > test.nc
--- /dev/null	2024-05-30 11:28:18
+++ /Users/ndp/OPeNDAP/hyrax/bes/modules/fileout_netcdf/tests/testsuite.dir/at-groups/217/stderr	2024-05-30 11:28:18
@@ -0,0 +1,10 @@
+Float32.cc:187:13: runtime error: load of misaligned address 0x611000006f86 for type 'dods_float32' (aka 'float'), which requires 4 byte alignment
+0x611000006f86: note: pointer points here
+ 8e 3f 00 00 7b 14  8e 3f 41 61 30 00 00 80  3f 00 00 00 40 7b 14 0e  40 01 00 7b 14 0e 40 42  62 31
+             ^ 
+SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Float32.cc:187:13 in 
+Int16.cc:180:13: runtime error: load of misaligned address 0x611000006f99 for type 'dods_int16' (aka 'short'), which requires 2 byte alignment
+0x611000006f99: note: pointer points here
+ 7b 14 0e  40 01 00 7b 14 0e 40 42  62 31 00 00 80 3f 00 00  00 40 b8 1e 55 40 02 00  b8 1e 55 40 43
+              ^ 
+SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Int16.cc:180:13 in 
217. testsuite.at:333:  FAILED (testsuite.at:333)

218. testsuite.at:334: testing bescmd/vdata_packed_linked_blocks.hdf.bescmd ...
COMMAND: besstandalone  -c bes.nc4.grp.conf -i bescmd/vdata_packed_linked_blocks.hdf.bescmd
./testsuite.at:334: besstandalone -c $abs_builddir/$bes_conf -i $input > test.nc
--- /dev/null	2024-05-30 11:28:19
+++ /Users/ndp/OPeNDAP/hyrax/bes/modules/fileout_netcdf/tests/testsuite.dir/at-groups/218/stderr	2024-05-30 11:28:19
@@ -0,0 +1,5 @@
+Float32.cc:187:13: runtime error: load of misaligned address 0x60300002a8e6 for type 'dods_float32' (aka 'float'), which requires 4 byte alignment
+0x60300002a8e6: note: pointer points here
+ 8e 3f 00 00 7b 14  0e 40 01 00 7b 14 8e 3f  00 00 7b 14 0e 40 01 00  00 00 00 00 00 00 00 00  03 02
+             ^ 
+SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Float32.cc:187:13 in 
218. testsuite.at:334:  FAILED (testsuite.at:334)

## ------------- ##
## Test results. ##
## ------------- ##

ERROR: All 5 tests were run,
5 failed unexpectedly.
## -------------------------- ##
## testsuite.log was created. ##
## -------------------------- ##

Please send `modules/fileout_netcdf/tests/testsuite.log' and all information you think might help:

   To: <[email protected]>
   Subject: [bes 3.21.0] testsuite: 207 210 216 217 218 failed

You may investigate any problem if you feel able to do so, in which
case the test suite provides a good starting point.  Its output may
be found below `modules/fileout_netcdf/tests/testsuite.dir'.

[-bash: ~/OPeNDAP/hyrax/bes/modules/fileout_netcdf/tests] 

Build with hdfeos2 fails due to -fPIC warning, but library was built with -fPIC

Building without HDFEOS2 works fine.

Building with it gives this -fPIC error message. However, I'm sure that the HDFEOS library I'm pointing to was built with -fPIC. Also @jgallagher59701 reported this to me as well, but I thought it was because he was pointing to a non-relocatable library. But perhaps not...

/bin/bash ../../libtool  --tag=CXX   --mode=link g++ -I../../hdfclass -g -O2 --pedantic --std=c++11 -avoid-version -module  -o libhdf4_module.la -rpath /usr/local/bes-3.20.7/lib/bes hdfeos.tab.lo lex.hdfeos.lo HDFArray.lo HDFGrid.lo HDFSequence.lo HDFUInt32.lo HDFByte.lo HDFInt16.lo HDFStr.lo HDFUrl.lo HDFFloat32.lo HDFInt32.lo HDFStructure.lo HDFFloat64.lo HDFUInt16.lo HDFTypeFactory.lo HDFCFUtil.lo BESH4MCache.lo HDFCFStr.lo HDFEOS2.lo HDFEOS2CFStr.lo HDFSP.lo HDFSPArrayGeoField.lo HDFSPArrayAddCVField.lo HDFSPArrayMissField.lo HDFSPArray_RealField.lo HDFSPArray_VDField.lo HDFCFStrField.lo HDFEOS2CFStrField.lo HDFEOS2Array_RealField.lo HDFEOS2ArrayGridGeoField.lo HDFEOS2ArraySwathGeoField.lo HDFEOS2ArrayMissField.lo HDFEOS2ArraySwathDimMapField.lo HDFEOS2ArraySwathGeoDimMapExtraField.lo HDFEOS2ArraySwathGeoMultiDimMapField.lo HDFEOS2GeoCFProj.lo HDFEOS2GeoCF1D.lo HE2CF.lo misr_init.lo misrinv.lo dhdferr.lo hdfdesc.lo hc2dap.lo hdfutil.lo HDF4RequestHandler.lo HDF4Module.lo   hdfclass/libhdfclass.la -L/usr/local/hdfeos2.19_fPIC/lib -lhdfeos -lGctp  -lmfhdf -ldf -ljpeg -lz -lsz -ldapserver -ldap -ldapclient -ldap  -lcurl -luuid 
libtool: link: g++  -fPIC -DPIC -shared -nostdlib /usr/lib/gcc/x86_64-linux-gnu/9/../../../x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/9/crtbeginS.o  .libs/hdfeos.tab.o .libs/lex.hdfeos.o .libs/HDFArray.o .libs/HDFGrid.o .libs/HDFSequence.o .libs/HDFUInt32.o .libs/HDFByte.o .libs/HDFInt16.o .libs/HDFStr.o .libs/HDFUrl.o .libs/HDFFloat32.o .libs/HDFInt32.o .libs/HDFStructure.o .libs/HDFFloat64.o .libs/HDFUInt16.o .libs/HDFTypeFactory.o .libs/HDFCFUtil.o .libs/BESH4MCache.o .libs/HDFCFStr.o .libs/HDFEOS2.o .libs/HDFEOS2CFStr.o .libs/HDFSP.o .libs/HDFSPArrayGeoField.o .libs/HDFSPArrayAddCVField.o .libs/HDFSPArrayMissField.o .libs/HDFSPArray_RealField.o .libs/HDFSPArray_VDField.o .libs/HDFCFStrField.o .libs/HDFEOS2CFStrField.o .libs/HDFEOS2Array_RealField.o .libs/HDFEOS2ArrayGridGeoField.o .libs/HDFEOS2ArraySwathGeoField.o .libs/HDFEOS2ArrayMissField.o .libs/HDFEOS2ArraySwathDimMapField.o .libs/HDFEOS2ArraySwathGeoDimMapExtraField.o .libs/HDFEOS2ArraySwathGeoMultiDimMapField.o .libs/HDFEOS2GeoCFProj.o .libs/HDFEOS2GeoCF1D.o .libs/HE2CF.o .libs/misr_init.o .libs/misrinv.o .libs/dhdferr.o .libs/hdfdesc.o .libs/hc2dap.o .libs/hdfutil.o .libs/HDF4RequestHandler.o .libs/HDF4Module.o  -Wl,--whole-archive hdfclass/.libs/libhdfclass.a -Wl,--no-whole-archive  -L/usr/local/hdf-4.2.15_fPIC/lib -L/usr/local/hdfeos2.19_fPIC/lib /usr/local/hdfeos2.19_fPIC/lib/libhdfeos.a /usr/local/hdfeos2.19_fPIC/lib/libGctp.a /usr/local/hdf-4.2.15_fPIC/lib/libmfhdf.a /usr/local/hdf-4.2.15_fPIC/lib/libdf.a /home/ed/hyrax/build/deps/lib/libmfhdf.a -L/home/ed/hyrax/build/deps/lib /home/ed/hyrax/build/deps/lib/libdf.a -ljpeg -lz -lsz -ldapserver -ldapclient -ldap /usr/lib/x86_64-linux-gnu/libcurl.so -luuid -L/usr/lib/gcc/x86_64-linux-gnu/9 -L/usr/lib/gcc/x86_64-linux-gnu/9/../../../x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/9/../../../../lib -L/lib/x86_64-linux-gnu -L/lib/../lib -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib -L/usr/lib/gcc/x86_64-linux-gnu/9/../../.. -lstdc++ -lm -lc -lgcc_s /usr/lib/gcc/x86_64-linux-gnu/9/crtendS.o /usr/lib/gcc/x86_64-linux-gnu/9/../../../x86_64-linux-gnu/crtn.o  -g -O2   -pthread -Wl,-soname -Wl,libhdf4_module.so -o .libs/libhdf4_module.so
/usr/bin/ld: /usr/local/hdfeos2.19_fPIC/lib/libhdfeos.a(EHapi.o): relocation R_X86_64_PC32 against symbol `EHXtypeTable' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: final link failed: bad value
collect2: error: ld returned 1 exit status
make[4]: *** [Makefile:750: libhdf4_module.la] Error 1
make[4]: Leaving directory '/home/ed/bes/modules/hdf4_handler'
make[3]: *** [Makefile:888: all-recursive] Error 1
make[3]: Leaving directory '/home/ed/bes/modules/hdf4_handler'
make[2]: *** [Makefile:504: all-recursive] Error 1
make[2]: Leaving directory '/home/ed/bes/modules'
make[1]: *** [Makefile:759: all-recursive] Error 1
make[1]: Leaving directory '/home/ed/bes'
make: *** [Makefile:542: all] Error 2

BES Cache

Our 1.13.1 server in operation had the BES cache filled in the last couple days. Based on the bes.conf spec below I wonder if the server should have started purging cached files when it reached 500MB? The cache is a 100GB area.

BES.UncompressCache.dir=/tmpdata/bes
BES.UncompressCache.prefix=bes_cache
BES.UncompressCache.size=500
BES.Uncompress.Retry=2000
BES.Uncompress.NumTries=10

bescmd test failures on ubuntu

I get this when I do:

autoreconf -i && CXXFLAGS="-fpermissive -Wno-deprecated" ./configure --with-hdf4=/usr/local/hdf-4.2.15_fPIC --with-hdf5=/usr/local/hdf5-1.10.7 --with-hdfeos2=/usr/local/hdfeos_fPIC --prefix=/usr/local/bes-3.20.7 --enable-developer && make -j && sudo make install && sudo make -j check

This is on the branch that includes the --enable-developer fix (#444 )

## ----------------------------------------------- ##
## bes 3.20.7 test suite: bes/dap/tests testsuite. ##
## ----------------------------------------------- ##
  1: BESCMD $abs_srcdir/bescmd/agg_array_dds_one_ce.bescmd ok
  2: BESCMD $abs_srcdir/bescmd/agg_array_dods_one_ce.bescmd FAILED (testsuite.at:8)
  3: BESCMD $abs_srcdir/bescmd/function_result_unwrap_dds.bescmd ok
  4: BESCMD $abs_srcdir/bescmd/function_result_unwrap_dods.bescmd FAILED (testsuite.at:12)
  5: BESCMD $abs_srcdir/bescmd/function_result_unwrap_dap.bescmd expected failure (testsuite.at:13)
  6: BESCMD $abs_srcdir/bescmd/dap4_ce_function-01.bescmd expected failure (testsuite.at:15)
  7: BESCMD $abs_srcdir/bescmd/fnoc.das.bescmd       ok
  8: BESCMD $abs_srcdir/bescmd/fnoc.ddx.bescmd       ok
  9: BESCMD $abs_srcdir/bescmd/fnoc.dmr.bescmd       ok

## ------------- ##
## Test results. ##
## ------------- ##

ERROR: All 9 tests were run,
4 failed (2 expected failures).
## -------------------------- ##
## testsuite.log was created. ##
## -------------------------- ##

Please send `dap/tests/testsuite.log' and all information you think might help:

   To: <[email protected]>
   Subject: [bes 3.20.7] testsuite: 2 4 failed

You may investigate any problem if you feel able to do so, in which
case the test suite provides a good starting point.  Its output may
be found below `dap/tests/testsuite.dir'.

DAP4: url with `.dap.csv?...` does not display tabular data correctly on the browser

Description of problem

I tried to visualize data on the browser in tabular form. With DAP2 everything seems to work as expected, but with DAP4 things don't go as planned.

What I did

DAP2 works

Pasting this url: http://test.opendap.org/opendap/data/nc/sst.mnmean.nc.gz.ascii?sst[3][4][5] on a browser shows:

Dataset: sst.mnmean.nc
sst.lon, 10
sst.sst[sst.time=19813][sst.lat=80], -51

DAP4 does not

If I paste the dap4 url: http://test.opendap.org/opendap/data/nc/sst.mnmean.nc.gz.dap.csv?dap4.ce=/sst[3][4][5] onto a browser it downloads the sst.mnmean.nc.gz.csv file. Looking at its contents with a simple text editor

Dataset: sst.mnmean.nc
/sst[0][0], 

With pandas:

Screenshot 2024-04-24 at 9 13 45 PM

That is pretty confusing.

If I set the url instead as http://test.opendap.org/opendap/data/nc/sst.mnmean.nc.gz.dap.csv?dap4.ce=/sst[3:4][4:5][5:6], it downloads a csv with the sst.mnmean.nc.gz.csv file with values now:

Dataset: sst.mnmean.nc
/sst[0][0], -51, -123
/sst[0][1], 175, 143
/sst[1][0], 50, -18
/sst[1][1], 242, 206

Expected behavior?

With DAP4 I was expected to see the browser data in tabular form as in DAP2 case. Instead:
a) Downloads data rather than visualize on browser.
b) When inspecting the contents of the csv file, the data seems incorrect.

'Phony_dim' in .nc4 downloads of 3IMERGM data from Hyrax 1.13.3

Although the (HDF5) handler in Hyrax 1.13.3 correctly presents the dimensions, a .nc4 download changes them to 'phony_dim_' in download file - this is in fact observed in other data like Merra2 as well. The 3IMERGM data is located here:

https://gpm1.gesdisc.eosdis.nasa.gov/data/s4pa/GPM_L3/GPM_3IMERGM.03/

The other issue with 3IMERG is that an 'nc_' is prepended to the download filename, a very undesirable feature to us. I recognize that both issues could be related to the netcdf library.

DMR XML generates invalid type for attribute.

In [1], you can see

    DATA_GRANULE_PR_CAL_COEF {
        Float32 Vdata_field_transCoef -0.5199999809;

For the above attribute in [2], you will get

   <Attribute name="DATA_GRANULE_PR_CAL_COEF" type="Container">
        <Attribute name="Vdata_field_transCoef" type="Byte">
            <Value>-0.5199999809</Value>
        </Attribute>

The type should be "Float32", not "Byte".

[1] https://eosdap.hdfgroup.org:8080/opendap/data/NASAFILES/hdf4/1B21_CSI.990906.10217.KORA.6.HDF.das

[2] https://eosdap.hdfgroup.org:8080/opendap/data/NASAFILES/hdf4/1B21_CSI.990906.10217.KORA.6.HDF.dmr.xml

Missing #include <cstring> in http/ directory

diff -ur bes-3.21.0-46.old/http/CurlUtils.cc bes-3.21.0-46.new/http/CurlUtils.cc
--- bes-3.21.0-46.old/http/CurlUtils.cc	2024-01-23 23:28:51.000000000 +0000
+++ bes-3.21.0-46.new/http/CurlUtils.cc	2024-02-23 13:19:11.556704797 +0000
@@ -27,6 +27,7 @@
 #include <unistd.h>
 #include <fcntl.h>
 #include <ctime>
+#include <cstring>
 
 #include <curl/curl.h>
 
diff -ur bes-3.21.0-46.old/http/RemoteResource.cc bes-3.21.0-46.new/http/RemoteResource.cc
--- bes-3.21.0-46.old/http/RemoteResource.cc	2024-01-23 23:28:51.000000000 +0000
+++ bes-3.21.0-46.new/http/RemoteResource.cc	2024-02-23 13:19:39.630855718 +0000
@@ -28,6 +28,7 @@
 
 #include <cstdio>
 #include <unistd.h>
+#include <cstring>
 
 #include <sstream>
 #include <string>
diff -ur bes-3.21.0-46.old/http/tests/remote_resource_tester.cc bes-3.21.0-46.new/http/tests/remote_resource_tester.cc
--- bes-3.21.0-46.old/http/tests/remote_resource_tester.cc	2024-01-23 23:28:51.000000000 +0000
+++ bes-3.21.0-46.new/http/tests/remote_resource_tester.cc	2024-02-23 13:20:12.830034191 +0000
@@ -6,6 +6,7 @@
 
 #include <iostream>
 #include <string>
+#include <cstring>
 
 #include <sys/stat.h>
 #include <unistd.h>

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.