ngageoint / six-library Goto Github PK

To enhance the python build, six.sidd is much needed now.

From Matthew Dellinger 10/4/2016

I'm attempting to write a method that will take any CPHD file and return complex float data with scales (if any) applied. When I attempt to use the cphd::Wideband::read method that applies scaling, it works correctly for all floating point files (with and without scaling) and for 8-bit and 16-bit integer data with scaling. However,
I've got a 16-bit integer CPHD file that was created without any scaling. When I run this file through, the const mem::BufferViewstd::complex& data argument is filled with integer data, rather than floating point data.

This is occurring because in at Wideband.cpp:403, we take the else branch (we don't need to scale), so we read the (integer) data into the output buffer, byteswap it, and then return it to the user. The result is a buffer typed as complex float that is 50% full of complex 16-bit integer data, and 50% uninitialized. For floating point data, this codepath works fine, since the data is already formatted as complex float.

We're currently working around this by using 1.0 + DBL_EPSILON as a constant scale for all reads if the data is unscaled and the type isn't float. That ensures that we always go down the scaling path, but isn't significant enough to change our data in any meaningful way.

I'm not sure what the proper behavior for this method is. Perhaps the method should either upscale the data to float or throw an error? The simplest solution would be to change line 403 from

if (needToScale)

to

// We can only directly read data that is already complex float
if (needToScale && mElementSize != 8)

Which would force all reads that needed promoted to float through the scaling path. That would take a slight performance hit for unscaled integer data, because you'd be doing an extra per-sample lookup and multiply by 1. If you want, I can work up a patch for a third else if case that would call a byteSwapAndPromote() function that would do the type promotion without the scaling.

Looks like the casting from nitf::Field to size_t appears to be missing / broken.

[596/722] cxx: modules/c++/six/source/ReadControlFactory.cpp -> target/modules/c++/six/source/ReadControlFactory.cpp.1.o
../modules/c++/six/source/NITFReadControl.cpp: In member function 'void six::NITFReadControl::load(nitf::IOInterface&, const std::vector<std::basic_string<char> >&)':
../modules/c++/six/source/NITFReadControl.cpp:152:77: error: conversion from 'nitf::Field' to 'size_t {aka long unsigned int}' is ambiguous
                         subheaderLength(subheader.getSubheaderFieldsLength());
....
packages/six-library/externals/nitro/modules/c++/nitf/include/nitf/Field.hpp:424:5: note:   no known conversion for implicit 'this' parameter from 'char*' to 'size_t {aka long unsigned int}'

When trying to compile on OSX 10.15.3 the compilation throws the error:

Build failed -> task in 'mt-c++' failed (exit status 1): {task 140367451013520: cxx ThreadGroup.cpp -> ThreadGroup.cpp.1.o} ['/usr/local/bin/g++', '-fPIC', '-std=c++11', '-Wall', '-Wno-deprecated-declarations', '-O3', '-m64', '-I/Users/darren/Development/six-library/target/externals/coda-oss/modules/c++/mt/include', '-I/Users/darren/Development/six-library/externals/coda-oss/modules/c++/mt/include', '-I/Users/darren/Development/six-library/externals/coda-oss/modules/c++/include', '-I/Users/darren/Development/six-library/target/externals/coda-oss/modules/c++/math/include', '-I/Users/darren/Development/six-library/externals/coda-oss/modules/c++/math/include', '-I/Users/darren/Development/six-library/target/externals/coda-oss/modules/c++/mem/include', '-I/Users/darren/Development/six-library/externals/coda-oss/modules/c++/mem/include', '-I/Users/darren/Development/six-library/target/externals/coda-oss/modules/c++/sys/include', '-I/Users/darren/Development/six-library/externals/coda-oss/modules/c++/sys/include', '-I/Users/darren/Development/six-library/target/externals/coda-oss/modules/c++/str/include', '-I/Users/darren/Development/six-library/externals/coda-oss/modules/c++/str/include', '-I/Users/darren/Development/six-library/target/externals/coda-oss/modules/c++/except/include', '-I/Users/darren/Development/six-library/externals/coda-oss/modules/c++/except/include', '-I/Users/darren/Development/six-library/target/externals/coda-oss/modules/c++/config/include', '-I/Users/darren/Development/six-library/externals/coda-oss/modules/c++/config/include', '-I/Users/darren/Development/six-library/target/externals/coda-oss/modules/c++/types/include', '-I/Users/darren/Development/six-library/externals/coda-oss/modules/c++/types/include', '-D_FILE_OFFSET_BITS=64', '-D_LARGEFILE_SOURCE', '-DSWIG_PYTHON_SILENT_MEMLEAK', '-DPYTHONDIR="/usr/local/lib/python2.7/site-packages"', '-DPYTHONARCHDIR="/usr/local/lib/python2.7/site-packages"', '-DHAVE_PYTHON_H=1', '-DNITF_DEFAULT_PLUGIN_PATH="/usr/local/share/nitf/plugins"', '-DDEFAULT_SCHEMA_PATH="/usr/local/conf/schema/six"', '-D_LARGEFILE_SOURCE=1', '-DSWIG_PYTHON_SILENT_MEMLEAK=1', '-DHAVE_PYTHON_H', '../externals/coda-oss/modules/c++/mt/source/ThreadGroup.cpp', '-c', '-o', 'externals/coda-oss/modules/c++/mt/source/ThreadGroup.cpp.1.o']

It looks like CPUAffinityInitializerLinux is not being defined. In ...mt/CPUAdffinityInitialiser.h :
`#include <mt/AbstractCPUAffinityInitializer.h>

#if defined(WIN32)
#include <mt/CPUAffinityInitializerWin32.h>
namespace mt
{
typedef CPUAffinityInitializerWin32 CPUAffinityInitializer;
}
#endif

#if !defined(APPLE_CC)
#if defined(__linux) || defined(linux)
#include <mt/CPUAffinityInitializerLinux.h>
namespace mt
{
typedef CPUAffinityInitializerLinux CPUAffinityInitializer;
}
#endif
#endif
`
so no define if using APPLE_CC

Values uIAXLL and uIAYLL in the CPHD XML should be printed in radians per meter according to Table 11-3 Scene Geographic Coordinates in the CPHD standard. Currently they are printed as degrees per meter. The issue can be seen in six/modules/c++/cphd/source/CPHDXMLParser.cpp on lines 199 and 200.

I was able to fix the issue by creating new methods in six/modules/c++/six/SICommonXMLParser called createLatLonRadians

XMLElem SICommonXMLParser::createLatLonRadians(
        const std::string& name,
        const std::string& uri,
        const LatLon& value,
        XMLElem parent) const
{
    XMLElem e = newElement(name, uri, parent);
    createDouble("Lat", getSICommonURI(), value.getLatRadians(), e);
    createDouble("Lon", getSICommonURI(), value.getLonRadians(), e);
    return e;
}

XMLElem SICommonXMLParser::createLatLonRadians(
        const std::string& name,
        const LatLon& value,
        XMLElem parent) const
{
    return createLatLonRadians(name, getDefaultURI(), value, parent);
}

I then modified lines 199 and 200 in CPHDXMLParser.cpp to use this new method.

I'm trying to build the code with the --shared option set to Waf, but compilation fails with a problem related to the libxerces.def file. Static building works well, on the other hand.
This problem is the same both under Windows and Linux.
Since the library is released under LGPL v3.0, we can only use this software as a dynamic library within a commercial, closed-source project.
Otherwise, the inability to generate a .dll/.so distribution of six-library quite contradicts the statement "[six-library] allows applications that are not open source to make use of the library without penalty".

Ideally there would be sample data that allowed wider understanding of SICD. Possibly it could be synthesised from redistributable data in the AFRL data sets (https://www.sdms.afrl.af.mil/).

I want to implement the six library in existing software to read/write cphd files

I've successfully installed using ./waf install already into the default /usr/local destination

I've copied six-library/six/modules/c++/cphd03/tests/print_cphd_header.cpp to a localdir ~/localdir.

When I try to compile it alone I get the following:

me@machine:~/localdir$ g++ print_cphd_header.cpp -std=c++11 -o go                                     
/tmp/ccN3Jfoz.o: In function `main':
print_cphd_header.cpp:(.text+0x17a): undefined reference to `sys::Path::basename(std::string const&, bool)'
print_cphd_header.cpp:(.text+0x2b1): undefined reference to `logging::setupLogger(std::string const&, std::string const&, std::string const&, std::string const&, unsigned long, unsigned long)'
print_cphd_header.cpp:(.text+0x353): undefined reference to `cphd03::CPHDReader::CPHDReader(std::string const&, unsigned long, std::shared_ptr<logging::Logger>)'
print_cphd_header.cpp:(.text+0x39f): undefined reference to `cphd03::operator<<(std::ostream&, cphd03::FileHeader const&)'
print_cphd_header.cpp:(.text+0x3b7): undefined reference to `cphd03::operator<<(std::ostream&, cphd03::Metadata const&)'
print_cphd_header.cpp:(.text+0x3cf): undefined reference to `cphd03::operator<<(std::ostream&, cphd03::VBM const&)'
/tmp/ccN3Jfoz.o: In function `__static_initialization_and_destruction_0(int, int)':
print_cphd_header.cpp:(.text+0x6ea): undefined reference to `sys::MutexPosix::MutexPosix()'
print_cphd_header.cpp:(.text+0x6f9): undefined reference to `sys::MutexPosix::~MutexPosix()'
/tmp/ccN3Jfoz.o: In function `cphd03::FileHeader::~FileHeader()':
print_cphd_header.cpp:(.text._ZN6cphd0310FileHeaderD2Ev[_ZN6cphd0310FileHeaderD5Ev]+0x13): undefined reference to `vtable for cphd03::FileHeader'
collect2: error: ld returned 1 exit status

How do I explicitly link libraries etc. to get the file to compile on its own? For example I've tried -L/usr/local/lib in the g++ command to no avail

To reproduce:

• Make sure you're on Windows
• Make sure the SIX_SCHEMA_PATH environment variable isn't set
• install/bin/round_trip_six <nitf> <output pathname>

Should see an exception. Looks like the backslashes in the default schema path need to be escaped

the CSM API is defined such that image coordinates have their integral location at the upper left corner of the image pixel. I believe the underlying SIX library code uses the center of the pixel as the integral pixel location. This implies that when projecting coordinates to the ground, the SICD and SIDD CSM implementations need to offset the input pixel locations by (-0.5,-0.5).

In order for the Python modules to work (at least on Windows), __init__.py files are needed in install/lib/site-packages/coda and install/lib/site-packages/pysix. Currently, these are not present after a vanilla waf install command.

Version: v3.1.3 (or current master, commit 9be2a2a)
Build system: CMake
Problem: Build failure when tests are disabled

Our projects currently use NITRO for NITF support and SIX for SICD support, and we're looking at upgrading to SIX v3.1.3. We don't necessarily need the tests to be built/installed as part of our build process, so I disabled them by passing -D CODA_BUILD_TESTS=OFF to cmake when configuring my build.

Unfortunately, that breaks the build process due to these lines:

CMake is attempting to add compile definitions to a target, six_test_xml_control, that doesn't exist if the tests are disabled.

It's a small fix and I'd be happy to submit a PR.

SIXSensorModel::getValidHeightRange() should return the minimum and maximum height for which the model is valid. For some sensor models, these values does have much impact. For active sensors, the MINIMUM height is especially important, as it is entirely possible that the range arc does not intersect the Earth for heights below a certain value.

Similarly, csm::RasterGM is always supposed to provide results relative to "full image" coordinates. If a particular SICD / SIDD comprises a sub-image, and the metadata can only accomodate that sub-image, that should be reflected in getValidImageRange(). Otherwise, getValidImageRange() should reflect the size of the full image. Currently both of these methods are coded to return values from (-99999, 99999) .

In the CPHD XML, TropoSRP is supposed to come before AmpSF. The VectorParameters section of the XML string has them in the incorrect order.

Simple fix: in CPHDXMLControl.cpp:Line445, switch the block of code dealing with the AmpSF size with the following block for the TropoSRP size.

I'm trying to build SIX without Microsoft Visual studio and am running into this issue

c:\dev\six-library-master>python waf configure
Setting top to                           : c:\dev\six-library-master
Setting out to                           : c:\dev\six-library-master\target
Platform                                 : win32
Checking for 'msvc' (c compiler)         : not found
Checking for 'msvc'                      : Warning: cound not find x64 msvc, looking for others
Checking for 'msvc' (c compiler)         : not found
could not configure a c compiler!
(complete log in c:\dev\six-library-master\target\config.log)

I've looked into the python build scripts and the second c compiler associated with windows is gcc, which I have sourced in my path:

c:\dev\six-library-master>echo %PATH%
C:\ProgramData\Oracle\Java\javapath;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Program Files\Anaconda3;C:\Program Files\Anaconda3\Scripts;C:\Program Files\Anaconda3\Library\bin;C:\Program Files\Intel\WiFi\bin\;C:\Program Files\Common Files\Intel\WirelessCommon\;**C:\cygwin64\bin;C:\cygwin64\lib**;C:\Program Files\Intel\WiFi\bin\;C:\Program Files\Common Files\Intel\WirelessCommon\;C:\Users\Matthew\AppData\Local\Microsoft\WindowsApps;;C:\Program Files (x86)\Microsoft VS Code\bin

(I'm in the process of downloading MSVS, so that's why they've been added to my path) Any idea why the script seems to be searching for msvc twice instead of moving onto gcc? In compiler_c.py, it has

c_compiler={'win32':['msvc','gcc'],'cygwin':['gcc'],'darwin':['gcc'],'aix':['xlc','gcc'],'linux':['gcc','icc'],'sunos':['suncc','gcc'],'irix':['gcc','irixcc'],'hpux':['gcc'],'gnu':['gcc'],'java':['gcc','msvc','icc'],'default':['gcc'],}

So I'm confused why msvc pops up twice in the output. Any help would be greatly appreciated!

Hi,

If you load a CPHD file that does not have include an antenna structure (which is defined as optional in the specification) and then try to print an element of it, it crashes with a SEGV. I would expect it to return None.

from pysix.cphd import CPHDReader
import multiprocessing
nthreads = multiprocessing.cpu_count()

reader = CPHDReader('myfile.cphd', nthreads)

mdata = reader.getMetadata()

print mdata.antenna.numRcvAnt

Just crashes with a SEGV.

Thanks,

Matt

For the python module, a number of nitf-related methods are not typemapped correctly in the SWIG file six_sicd.i. As a result, the methods can't be called if the underlying C++ takes a nitf::Uint32 or if the method returns a nitf::Uint32, the return type is opaque.

In particular ImageSegment.getBandInfo can't be called:

(pdb) record.getImageSegment(0).getSubheader().getBandInfo(0)
*** TypeError in method ' ImageSubheader_getBandInfo', argument 2 of type 'nitf::Uint32'

And ImageSegment.getBandCount() returns an opaque type:

(pdb) record.getImageSegment(0).getSubheader().getBandCount()
<SWIG object of type 'nitf::uint32 *' at 0x7f03c6e77210>

This appears to be because the input and output typemaps for nitf::Uint32 are defined after the nitf-related includes

Moving those typemaps before the nitf includes in the .i and then regenerating the bindings via a recompilation with -DENABLE_SWIG=ON fixes the issue:

(pdb) record.getImageSegment(0).getSubheader().getBandInfo(0)
<SWIG Object of type 'nitf::BandInfo *' at 0x7fc491bb67b0>
(pdb) record.getImageSegment(0).getSubheader().getBandCount()
2

(Note: nitf::BandInfo is also opaque because it's not included in the .i file, but that's a separate issue)

The pointer for the CPHD phase history data is not offset to the starting position of each channel of data. This results in the first channel being written for each channel specified in the XML data. The issue can be see on line 341 of six/modules/c++/source/CPHDWriter.cpp.

I was able to fix the offset issue by changing the line to be:

writeCPHDDataImpl(reinterpret_cast<const sys::ubyte*>(data) + (channel * numElements * mElementSize), numElements);

I'm unsure if line 331 also needs to modified for the correct offset of each compressed phase history channel. I haven't had the chance to check it.

We have a really old Linux server running CentOS 6 and the latest compiler we can run on it is 4.4.7 which doesn't support C++11 standard. Is it possible to get a version of SIX 3.0 that doesn't need this requirement?

With each SIX release going forward, provide precompiled 64-bit binaries in these permutations...

• Linux RHEL 6 with gcc 4.4.7 and Python 2.7
• Linux RHEL 6 with gcc 4.4.7 and Python 3.4
• Linux RHEL 6 with gcc 4.4.7 and Python 3.5
• Windows with Visual Studio 2015 and Python 2.7
• Windows with Visual Studio 2015 and Python 3.4
• Windows with Visual Studio 2015 and Python 3.5

This will largely be work of setting up a Jenkins job and installing the appropriate flavors of Anaconda to support. Note that our Linux machine by default is likely going to use gcc 4.9 so you'll have to force it to use the /usr/bin version of gcc and g++.

Go ahead and build these for the last SIX release (2.2.1). I believe GitHub will let you just attach binaries to that release. If not, let me know and I can put together a spot on AWS S3 where you can put them up and we'll link to them with the release. We don't want to check them in as that'll bloat the size of the repo.

Add a utility method that will parse a ComplexData object from a given filename. That file can be formatted as either a standard SICD NITF or a SICD XML text file (a single method to support both).

@asylvest @JonathanMeans

Parsing SICD XML containing polarizations of 'UNKNOWN' cause an "Unsupported conversion" exception. I first ran into this at DualPolarizationType six::toType(), but skimming through the code it looks like 'UNKNOWN' is missing from most of the various polarization routines.

The external directory contains links to CSM3.0.3 as zip files but the 3.0.3 version is not in sync with the official 3.0.3 maintained in GitHub : See github.com/sminster/csm for the official version. Nothing in the SIXX library actually exploits any of the new features in 3.0.3 (vs 3.0.1), but if you are going to have a zipped copy, it should at least be in agreement with the official version.

Hi,
I had no trouble building the package on an Ubuntu VM but am having trouble building the library on OS-X (macports - gcc48).

To get started, I had to modify projects/csm/wscript

osxRegex = r'darwin'

section to allow it to configure correctly.

However, I run into the following error when building the package:

[ 94/722] cxx: externals/coda-oss/modules/c++/tiff/source/Header.cpp -> target/externals/coda-oss/modules/c++/tiff/source/Header.cpp.1.o
../externals/coda-oss/modules/c++/sys/source/OSUnix.cpp: In member function 'virtual void sys::OSUnix::getMemInfo(size_t&, size_t&) const':
../externals/coda-oss/modules/c++/sys/source/OSUnix.cpp:279:45: error: '_SC_PHYS_PAGES' was not declared in this scope
     long long totalNumPages = sysconfCaller(_SC_PHYS_PAGES);
                                             ^
../externals/coda-oss/modules/c++/sys/source/OSUnix.cpp:280:45: error: '_SC_AVPHYS_PAGES' was not declared in this scope
     long long availNumPages = sysconfCaller(_SC_AVPHYS_PAGES);
                                             ^
../externals/coda-oss/modules/c++/tiff/source/Common.cpp: In function 'void tiff::split(sys::Uint64_T, sys::Uint32_T&, sys::Uint32_T&)':
../externals/coda-oss/modules/c++/tiff/source/Common.cpp:57:44: warning: dereferencing type-punned pointer will break strict-aliasing rules [-Wstrict-aliasing]
     numerator = ((sys::Uint32_T *)&value)[0];

Any suggestions?

Thanks
Piyush

Hi,
Is there a copy of the CPHD interface control document ? or something that explains the format of the CPHD file please?

Add test_streaming_write.cpp and test_streaming_sicd_write.py

As far as I can tell the only way to access the wideband data from python is to read the entire image into memory. It would be great to have a way to pull out a small chip without allocating space for the entire image. Or should we be using Nitro directly to deal with non-metadata?

Now that I've gotten SIX to build and install, I'm at a loss in getting any of the test programs to run. Is there a developers guide for getting up and running in Visual Studio? Would it be possible to get a MSVS 2008 and older compatible .sln to start off of? Or at least a guide to start a new project (or existing project like I tried and failed from C:\...\six-library-master\six\modules\c++\samples) that can run these test programs?

Apologies for the vagueness of this report, the computer where I found this issue doesn't have GitHub access. I'm trying to call the static method six::NITFReadControl::getDataType on a Nitro nitf::Record object. When I use that method on a well-formed SICD or SIDD, it always returns six::DataType::NOT_SET unless I create a NITFReadControl first, in which case it returns the correct DataType. Note that I don't need to call the static method through the object, I just need to create one. I am using the latest master branch.

Most of the sample programs under modules/c++/samples require the schema path and/or plugin path to be set. Oftentimes we were too lazy to add a command-line argument for this so we made the user set an environment variable. The times when we did add a command-line argument, we usually didn't default it to a reasonable value.

Take a look at crop_sicd. There, we did default both values based on the known directory structure (i.e. we know install/share/nitf/plugins is going to be where the NITRO plugins are and install/conf/schema/six is where the schemas are going to be). But we let the user override it if they want to.

Update all the programs in this directory to do the same thing... but only add the option if it's relevant for that program. Either figure that out empirically by running them without setting environment variables or trace through to see what they're doing to figure it out. High level, the NITRO plugin path is only needed when writing NITFs (it's not needed for XML parsing or reading NITFs). The schema path is needed whenever the XML is parsed but is not needed if it's just extracted from a NITF without being parsed.

This would clean up things like the weirdness in SICDWriteControl

Hi,

It appears that the python wrapper for CPHD does not have a CPHDWriter class in it - or have I missed something? This makes writing out CPHD files from python impossible?

Thanks,

Matt

This issue was also submitted to CODA-OSS and can be found here:
mdaus/coda-oss#486 (comment)

Building CODA with CMake fails on Centos 7 due to an include order issue with Python.h.

The error I received is as follow:

...
[ 39%] Built target math.poly_test_llsq
[ 40%] Built target math.poly_test_OneD_templates
[ 40%] Built target math.poly_test_fixed
[ 40%] Built target math.poly_test_poly
[ 40%] Built target math.poly_test_inner_poly
[ 40%] Built target math.poly_test_1d_poly
[ 40%] Built target math.poly_test_2d_poly
[ 40%] Built target math.poly_test_fixed_1d_poly
[ 40%] Built target math.poly_test_fixed_2d_poly
[ 40%] Built target numpyutils-c++
[ 40%] Building CXX object externals/coda-oss/modules/c++/numpyutils/CMakeFiles/numpyutils_test_num_elements.dir/unittests/test_num_elements.cpp.o
In file included from /usr/include/python2.7/pyconfig.h:6:0,
                 from /usr/include/python2.7/Python.h:8,
                 from /home/admin/CodeProjects/six-library-SIX-3.1.3/externals/coda-oss/modules/c++/numpyutils/include/numpyutils/numpyutils.h:27,
                 from /home/admin/CodeProjects/six-library-SIX-3.1.3/externals/coda-oss/modules/c++/numpyutils/unittests/test_num_elements.cpp:24:
/usr/include/python2.7/pyconfig-64.h:1191:0: error: "_POSIX_C_SOURCE" redefined [-Werror]
 #define _POSIX_C_SOURCE 200112L
 ^
In file included from /usr/include/c++/4.8.2/x86_64-redhat-linux/bits/os_defines.h:39:0,
                 from /usr/include/c++/4.8.2/x86_64-redhat-linux/bits/c++config.h:2097,
                 from /usr/include/c++/4.8.2/cmath:41,
                 from /home/admin/CodeProjects/six-library-SIX-3.1.3/externals/coda-oss/modules/c++/include/TestCase.h:29,
                 from /home/admin/CodeProjects/six-library-SIX-3.1.3/externals/coda-oss/modules/c++/numpyutils/unittests/test_num_elements.cpp:23:
/usr/include/features.h:168:0: note: this is the location of the previous definition
 # define _POSIX_C_SOURCE 200809L
 ^
In file included from /usr/include/python2.7/pyconfig.h:6:0,
                 from /usr/include/python2.7/Python.h:8,
                 from /home/admin/CodeProjects/six-library-SIX-3.1.3/externals/coda-oss/modules/c++/numpyutils/include/numpyutils/numpyutils.h:27,
                 from /home/admin/CodeProjects/six-library-SIX-3.1.3/externals/coda-oss/modules/c++/numpyutils/unittests/test_num_elements.cpp:24:
/usr/include/python2.7/pyconfig-64.h:1213:0: error: "_XOPEN_SOURCE" redefined [-Werror]
 #define _XOPEN_SOURCE 600
 ^
In file included from /usr/include/c++/4.8.2/x86_64-redhat-linux/bits/os_defines.h:39:0,
                 from /usr/include/c++/4.8.2/x86_64-redhat-linux/bits/c++config.h:2097,
                 from /usr/include/c++/4.8.2/cmath:41,
                 from /home/admin/CodeProjects/six-library-SIX-3.1.3/externals/coda-oss/modules/c++/include/TestCase.h:29,
                 from /home/admin/CodeProjects/six-library-SIX-3.1.3/externals/coda-oss/modules/c++/numpyutils/unittests/test_num_elements.cpp:23:
/usr/include/features.h:170:0: note: this is the location of the previous definition
 # define _XOPEN_SOURCE 700
 ^
cc1plus: all warnings being treated as errors
make[2]: *** [externals/coda-oss/modules/c++/numpyutils/CMakeFiles/numpyutils_test_num_elements.dir/unittests/test_num_elements.cpp.o] Error 1
make[1]: *** [externals/coda-oss/modules/c++/numpyutils/CMakeFiles/numpyutils_test_num_elements.dir/all] Error 2
make: *** [all] Error 2

The issue can be found in

This issue can be fixed by either changing the include order to:

#include <numpyutils/numpyutils.h>
#include <TestCase.h>

or by adding Python.h before the other includes:

#include <Python.h>
#include <TestCase.h>
#include <numpyutils/numpyutils.h>

This Jira post for Mongo Python Driver pointed me to the solution:
https://jira.mongodb.org/browse/PYTHON-2260

The CSM API is intended to work with full image coordinates. It appears that, as currently implemented, if presented with an image chip, the SICD sensor model offsets the input pixel coordinate based on the FirstRow and FirstCol as defined in the XML. This is not correct. These would be appropriate values to return via the overloaded getImageStart() method. Tools exploiting CSM are expected to handle the conversion from chipped coordinates to full image coordinates.

If you run round_trip_six from a clean build without the NITF_PLUGIN_PATH environment variable set, you get this warning:

Warning: Unable to find plugin path.
Specify plugin location by setting environment variable NITF_PLUGIN_PATH, or by building the library from source

This is misleading. SIX builds the NITRO plugins statically, so it doesn't matter whether it can find the (dynamic) plugins directory.

The CMake build process will fail building shared libraries when it attempts to link test_j2k_create. This can be fix by adding C Math to the DEPS list in the J2K CMakelists.txt.

The issues can be found here:

I was able to fix the issue by modifying the line to:

    DEPS ${J2K_LIB} nitf-c nrt-c m

I don't have any experience with Waf build systems so I'm not sure if the issue also exists in the wscript.

The CPHD writer internally creates a FileHeader object and uses that when writing to disk. This means a user does not have an opportunity to fill out the classification and release information. An override should be created that allows the user to pass in a filled out FileHeader and use that instead.

If one builds the CSM plugin from the source code, it can't be used without the availability of the schema files. It would be very helpful for these to be included here.

Hi,
Is there an archive of example products generate using SIX/SICD/SIDD? I'm interested in finding workingexamples that converts data from current missions like TerraSAR-X, COSMO-SkyMed, Sentinel-1 or Radarsat-2 into one of the SIX compatible formats. In particular, I'm interested in using Level-1 SLC products for interferometric applications and in evaluating the completeness of metadata fields for such applications.

Piyush

The 'if' block is empty. if the underlying nitf_TRE_setField fails(for say a non-existent field, or an overrun of the field's length) the error will be ignored and the field not set.

Add methods to scene::ProjectionPolynomialFitter and six::sicd::Utilities that make use of the SICD valid data polygon to determine the output plane and associated slant plane coordinates that are used to fit the projection polynomials.

The NITF BandInfo portion of the Image Segment Subheader is not exposed in the six sicd Python bindings and thus values in the BandInfo can't be accessed.

I'm not sure what six-library's stance is regarding exposure of NITF elements to Python, but it would be convenient to access all NITF data via one API rather than having to try to glue together different solutions, e..g opening the file both in nitro's and six-library's python bindings.

This is fairly straightforward to expose simply by %include nitf/BandInfo.hpp; and then dealing with LookupTable's lack of a default constructor, either by defining one in the .i file, or ignoring the offending method via %ignore nitf::BandInfo::getLookupTable;

See also #352 regarding access to BandInfo in the first place.

Not able to compile any of the sample c++ code after installing six-library.

cd six-library/six/modules/c++/samples
g++ -std=c++11 extract_cphd_xml.cpp

I get undefined reference to various classes no matter which sample i try to compile, e.g. below.

Grateful for any help.

/tmp/ccEra1O8.o: In function `main':
extract_cphd_xml.cpp:(.text+0xc5): undefined reference to `cli::ArgumentParser::ArgumentParser()'
extract_cphd_xml.cpp:(.text+0x108): undefined reference to `cli::ArgumentParser::setDescription(std::string const&)'
extract_cphd_xml.cpp:(.text+0x225): undefined reference to `cli::ArgumentParser::addArgument(std::string const&, std::string const&, cli::Action, std::string const&, std::string const&, int, int, bool)'
extract_cphd_xml.cpp:(.text+0x3c7): undefined reference to `cli::ArgumentParser::addArgument(std::string const&, std::string const&, cli::Action, std::string const&, std::string const&, int, int, bool)'
extract_cphd_xml.cpp:(.text+0x569): undefined reference to `cli::ArgumentParser::addArgument(std::string const&, std::string const&, cli::Action, std::string const&, std::string const&, int, int, bool)'
extract_cphd_xml.cpp:(.text+0x70b): undefined reference to `cli::ArgumentParser::addArgument(std::string const&, std::string const&, cli::Action, std::string const&, std::string const&, int, int, bool)'
extract_cphd_xml.cpp:(.text+0x891): undefined reference to `cli::ArgumentParser::addArgument(std::string const&, std::string const&, cli::Action, std::string const&, std::string const&, int, int, bool)'
extract_cphd_xml.cpp:(.text+0x952): undefined reference to `cli::ArgumentParser::parse(int, char const**)'
extract_cphd_xml.cpp:(.text+0xda5): undefined reference to `sys::Path::basename(std::string const&, bool)'
extract_cphd_xml.cpp:(.text+0xe43): undefined reference to `sys::OSUnix::getNumCPUs() const'
extract_cphd_xml.cpp:(.text+0xe67): undefined reference to `cphd::CPHDReader::CPHDReader(std::string const&, unsigned long, std::vector<std::string, std::allocator<std::string> > const&, std::shared_ptr<logging::Logger>)'
extract_cphd_xml.cpp:(.text+0xea4): undefined reference to `cphd::CPHDXMLControl::CPHDXMLControl(logging::Logger*, bool)'
extract_cphd_xml.cpp:(.text+0xeda): undefined reference to `cphd::CPHDXMLControl::toXMLString(cphd::Metadata const&, std::vector<std::string, std::allocator<std::string> > const&, bool)'
extract_cphd_xml.cpp:(.text+0xf38): undefined reference to `io::FileOutputStreamOS::FileOutputStreamOS(std::string const&, int)'
extract_cphd_xml.cpp:(.text+0xfbb): undefined reference to `cphd::CPHDXMLControl::~CPHDXMLControl()'
extract_cphd_xml.cpp:(.text+0x1033): undefined reference to `cli::ArgumentParser::~ArgumentParser()'
extract_cphd_xml.cpp:(.text+0x165d): undefined reference to `cphd::CPHDXMLControl::~CPHDXMLControl()'
extract_cphd_xml.cpp:(.text+0x16fd): undefined reference to `cli::ArgumentParser::~ArgumentParser()'
/tmp/ccEra1O8.o: In function `__static_initialization_and_destruction_0(int, int)':
extract_cphd_xml.cpp:(.text+0x18a0): undefined reference to `sys::MutexPosix::MutexPosix()'
extract_cphd_xml.cpp:(.text+0x18af): undefined reference to `sys::MutexPosix::~MutexPosix()'
/tmp/ccEra1O8.o: In function `except::Throwable::toString() const':
extract_cphd_xml.cpp:(.text._ZNK6except9Throwable8toStringEv[_ZNK6except9Throwable8toStringEv]+0xfa): undefined reference to `except::operator<<(std::ostream&, except::Trace const&)'
/tmp/ccEra1O8.o: In function `except::Exception::Exception(except::Context const&)':
extract_cphd_xml.cpp:(.text._ZN6except9ExceptionC2ERKNS_7ContextE[_ZN6except9ExceptionC5ERKNS_7ContextE]+0x33): undefined reference to `except::Throwable::Throwable(except::Context)'
/tmp/ccEra1O8.o: In function `sys::TimeStamp::local() const':
extract_cphd_xml.cpp:(.text._ZNK3sys9TimeStamp5localEv[_ZNK3sys9TimeStamp5localEv]+0x19): undefined reference to `sys::LocalDateTime::LocalDateTime()'
extract_cphd_xml.cpp:(.text._ZNK3sys9TimeStamp5localEv[_ZNK3sys9TimeStamp5localEv]+0x5e): undefined reference to `sys::DateTime::format(std::string const&) const'
/tmp/ccEra1O8.o: In function `sys::OSUnix::OSUnix()':
extract_cphd_xml.cpp:(.text._ZN3sys6OSUnixC2Ev[_ZN3sys6OSUnixC5Ev]+0x14): undefined reference to `sys::AbstractOS::AbstractOS()'
extract_cphd_xml.cpp:(.text._ZN3sys6OSUnixC2Ev[_ZN3sys6OSUnixC5Ev]+0x1f): undefined reference to `vtable for sys::OSUnix'
/tmp/ccEra1O8.o: In function `sys::OSUnix::~OSUnix()':
extract_cphd_xml.cpp:(.text._ZN3sys6OSUnixD2Ev[_ZN3sys6OSUnixD5Ev]+0x13): undefined reference to `vtable for sys::OSUnix'
extract_cphd_xml.cpp:(.text._ZN3sys6OSUnixD2Ev[_ZN3sys6OSUnixD5Ev]+0x1f): undefined reference to `sys::AbstractOS::~AbstractOS()'
/tmp/ccEra1O8.o: In function `io::StandardOutStream::StandardOutStream()':
extract_cphd_xml.cpp:(.text._ZN2io17StandardOutStreamC2Ev[_ZN2io17StandardOutStreamC5Ev]+0x1f): undefined reference to `vtable for io::StandardOutStream'
/tmp/ccEra1O8.o: In function `cphd::Data::~Data()':
extract_cphd_xml.cpp:(.text._ZN4cphd4DataD2Ev[_ZN4cphd4DataD5Ev]+0x13): undefined reference to `vtable for cphd::Data'
/tmp/ccEra1O8.o: In function `cphd::FileHeader::~FileHeader()':
extract_cphd_xml.cpp:(.text._ZN4cphd10FileHeaderD2Ev[_ZN4cphd10FileHeaderD5Ev]+0x13): undefined reference to `vtable for cphd::FileHeader'
/tmp/ccEra1O8.o: In function `cli::Argument* cli::Argument::setDefault<bool>(bool)':
extract_cphd_xml.cpp:(.text._ZN3cli8Argument10setDefaultIbEEPS0_T_[_ZN3cli8Argument10setDefaultIbEEPS0_T_]+0x3f): undefined reference to `cli::Argument::setDefault(cli::Value*, bool)'
/tmp/ccEra1O8.o: In function `cli::Argument* cli::Argument::setDefault<char const*>(char const*)':
extract_cphd_xml.cpp:(.text._ZN3cli8Argument10setDefaultIPKcEEPS0_T_[_ZN3cli8Argument10setDefaultIPKcEEPS0_T_]+0x3f): undefined reference to `cli::Argument::setDefault(cli::Value*, bool)'
/tmp/ccEra1O8.o: In function `bool cli::Value::at<bool>(unsigned long) const':
extract_cphd_xml.cpp:(.text._ZNK3cli5Value2atIbEET_m[_ZNK3cli5Value2atIbEET_m]+0x60): undefined reference to `str::format(char const*, ...)'
extract_cphd_xml.cpp:(.text._ZNK3cli5Value2atIbEET_m[_ZNK3cli5Value2atIbEET_m]+0x192): undefined reference to `bool str::toType<bool>(std::string const&)'
/tmp/ccEra1O8.o: In function `std::string cli::Value::at<std::string>(unsigned long) const':
extract_cphd_xml.cpp:(.text._ZNK3cli5Value2atISsEET_m[_ZNK3cli5Value2atISsEET_m]+0x6a): undefined reference to `str::format(char const*, ...)'
extract_cphd_xml.cpp:(.text._ZNK3cli5Value2atISsEET_m[_ZNK3cli5Value2atISsEET_m]+0x1a9): undefined reference to `std::string str::toType<std::string>(std::string const&)'
/tmp/ccEra1O8.o: In function `sys::LocalDateTime::~LocalDateTime()':
extract_cphd_xml.cpp:(.text._ZN3sys13LocalDateTimeD2Ev[_ZN3sys13LocalDateTimeD5Ev]+0x13): undefined reference to `vtable for sys::LocalDateTime'
extract_cphd_xml.cpp:(.text._ZN3sys13LocalDateTimeD2Ev[_ZN3sys13LocalDateTimeD5Ev]+0x1f): undefined reference to `sys::DateTime::~DateTime()'
collect2: error: ld returned 1 exit status

I'm trying to use an example from SIX test_round_trip:

me@machine:~/six-library/target/six/modules/c++/cphd/tests$ ./test_round_trip  -t 6 /scratch/cphd_mess/tester.cphd  out.cphd    
ParseException: usage: ./test_round_trip [-t NUM] CPHD CPHD XSD    
missing required argument: [schema] 

I need a "schema". What/where is that?

With both current git tip and the 2.1.3 release, I see problems building with a target --prefix that the build user cannot access because it's trying to install the j2k_config.h header file.

./waf configure --prefix=/noaccess
./waf build

in build
bld.recurse()
File "/home/kjurka/git/six-library/externals/coda-oss/build/build.py", line 63, in recurse
os.walk(self.path.abspath()).next()[1]))
File "/home/kjurka/git/six-library/externals/coda-oss/build/.waf-1.7.14-6b92f3c95d8c1bad60826036bf26d63f/waflib/Context.py", line 125, in recurse
user_function(self)
File "/home/kjurka/git/six-library/externals/nitro/modules/c/j2k/wscript", line 84, in build
makedirs(installDir)
File "/usr/local/anaconda-2.3/lib/python2.7/os.py", line 150, in makedirs
makedirs(head, mode)
File "/usr/local/anaconda-2.3/lib/python2.7/os.py", line 150, in makedirs
makedirs(head, mode)
File "/usr/local/anaconda-2.3/lib/python2.7/os.py", line 157, in makedirs
mkdir(name, mode)
OSError: [Errno 13] Permission denied: '/noaccess'

This installation should only happen during the "install" stage, not "build"

Add a test showing the new six::NITFReadControl::load() overloading that takes in an io::SeekableInputStream works (#122). You can just turn around and create an io::FileInputStream and then call this method if you want.

@clydestanfield

It looks like PDF manual only covers waf, but more recent work seems to be on CMake. Which is preferred?

I'm trying to build ngaSix. I did this successfully a few months ago, but now the install is complaining. I possibly installed a new version of python in the interim. I first did this:

  `  ./waf configure --prefix=install

`
That seemed to be successful. But then the install failed:

./waf install --targets=six-python
Waf: Entering directory `/home/userjb/src/ngaSIXLibrary/target'
Waf: Leaving directory `/home/userjb/src/ngaSIXLibrary/target'
Could not find a task generator for the name 'six-python'

What am I doing wrong?