nasa / gmsec_api Goto Github PK

In building I get the error below. Please help. I cannot find the file being referenced on the system. I am using the most recent zip from the repo.

In file included from ../activemq/src/gmsec_activemq.cpp:17:
../activemq/include/CMSConnection.h:27:10: fatal error: cms/Connection.h: No such file or directory
#include <cms/Connection.h>

From what I can see, yesterday some changes were made to the templates, and API 4.9.1 (among others) were re-built and replaced the existing published releases.

I am now seeing the following error in init:

gov.nasa.gsfc.gmsec.api.GMSEC_Exception: [13,104,0]: Template Directory information failed to load: [13,107,0] : DIRECTORY is missing definition for LEVEL-1 HEADER
	at gov.nasa.gsfc.gmsec.api.jni.gmsecJNI.new_ConnectionManager(Native Method) ~[gmsec-api-4.9.1.jar:?]
	at gov.nasa.gsfc.gmsec.api.jni.mist.JNIConnectionManager.<init>(JNIConnectionManager.java:113) ~[gmsec-api-4.9.1.jar:?]
	at gov.nasa.gsfc.gmsec.api.mist.ConnectionManager.<init>(ConnectionManager.java:111) ~[gmsec-api-4.9.1.jar:?]

As an aside, I would not have expected released artifacts to change once released...

Ran the Java interface to GMSEC API 4.6-RH7 with the -Xcheck:jni to make sure it is making valid JNI calls. Used a ConnectionManager to make a request with a call back. It turns out that GMSEC API was not correctly using JNI.

This is the output from -Xcheck:jni:

WARNING in native method: JNI call made with exception pending
FATAL ERROR in native method: Bad global or local ref passed to JNI

The is the stack of the thread that caused the seg fault:

Stack: [0x00007f3145cf7000,0x00007f31464f7000],  sp=0x00007f31464f5860,  free space=8186k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libc.so.6+0x37bc7]  abort+0x297
V  [libjvm.so+0x71aa8b]  jniCheck::validate_handle(JavaThread*, _jobject*)+0xab
V  [libjvm.so+0x71ae25]  jniCheck::validate_class(JavaThread*, _jclass*, bool) [clone .constprop.176]+0x15
V  [libjvm.so+0x71be15]  checked_jni_GetStaticFieldID+0x75
C  [libgmsec_jni.so+0xb7d30]  JNIEnv_::GetStaticFieldID(_jclass*, char const*, char const*)+0x38
C  [libgmsec_jni.so+0xba67f]  gmsec::api::jni::convertEvent(JNIEnv_*, gmsec::api::Connection::ConnectionEvent)+0x109
C  [libgmsec_jni.so+0xf54f2]  gmsec::api::jni::CxxConnMgrReplyCallbackProxy::onEvent(gmsec::api::mist::ConnectionManager&, gmsec::api::Status const&, gmsec::api::Connection::ConnectionEvent)+0x3bc
C  [libGMSECAPI.so+0x37e1c5]  gmsec::api::mist::internal::CMReplyCallback::onEvent(gmsec::api::Connection&, gmsec::api::Status const&, gmsec::api::Connection::ConnectionEvent)+0x45
C  [libGMSECAPI.so+0x344cca]  gmsec::api::internal::InternalConnection::replyEvent(gmsec::api::ReplyCallback*, gmsec::api::Status const&, gmsec::api::Connection::ConnectionEvent)+0x68
C  [libGMSECAPI.so+0x36742f]  gmsec::api::internal::RequestShared::sendRequests()+0x20d
C  [libGMSECAPI.so+0x367008]  gmsec::api::internal::RequestShared::run()+0x3cc
C  [libGMSECAPI.so+0x366af0]  gmsec::api::internal::runRequestThread(gmsec::api::util::StdSharedPtr<gmsec::api::internal::RequestShared>)+0x20
C  [libGMSECAPI.so+0x34d465]  gmsec::api::util::StdThread::FunctionInvoker<void (*)(gmsec::api::util::StdSharedPtr<gmsec::api::internal::RequestShared>), gmsec::api::util::StdSharedPtr<gmsec::api::internal::RequestShared> >::_invoke()+0x35
C  [libGMSECAPI.so+0x3e3210]  gmsec::api::util::StdThread::Invoker::invoke()+0xa6
C  [libGMSECAPI.so+0x3e1c3f]  gmsec::api::util::StdThread::process0(void*)+0x21
C  [libpthread.so.0+0x7ea5]  start_thread+0xc5

The ConfigFile class allows for a user to define a Message definition, and I suppose a Message definition could be used to define a list of standard fields that can be applied to a MessageFactory object, however this seems counterintuitive. After all, the user would need to lookup the message, then acquire the field iterator, and then add each field to a list, which can then finally be passed to MessageFactory setStandardFields().

It would be nice if the ConfigFile could ingest XML content representing a list of standard fields, and then have ConfigFile provide a method to return these fields in a List (of Field objects). For example:

<DEFINITIONS>
    <STANDARD-FIELDS name="My Mission Faves">
        <FIELD NAME="DOMAIN1" TYPE="STRING" HEADER="T">MY-DOM-1</FIELD>
        <FIELD NAME="DOMAIN2" TYPE="STRING" HEADER="T">MY-DOM-2</FIELD>
        <FIELD NAME="MISSION-ID" TYPE="STRING" HEADER="T">MY-MISSION</FIELD>
        ...
    </STANDARD-FIELDS>
</DEFINITIONS>

ConfigFile cf = new ConfigFile();
cf.load("MyDefinitions.xml");
messageFactory.setStandardFields( cf.getStandardFields("My Mission Faves") );

Came across this issue when attempting to make my application robust to the connection dropping out and reconnecting. Restarting the heartbeat thread and sending a log message that the connection was established deadlocked the application. May be related to API-5587, but I don't know the details of that issue.

Steps to reproduce:

1. Create a ConnectionManager with an active bolt connection
2. Register a handler for CONNECTION_SUCCESSFUL_EVENT events. The handler will start a new thread that publishes a message and then publish a message of its own.
3. kill the bolt bus.
4. restart the bolt bus

Expected result:

Step 4 triggers the handler registered in step 2 which causes two messages to be published on the bus.

Actual result:

Application deadlocks. Zero messages are published.

Analysis:

The thread dispatching the CONNECTION_SUCCESSFUL_EVENT holds the mutex in the bolt wrapper. (Shared.cpp line 81 and 111) Control passes to the event handler while that mutex is held (Shared.cpp line 146, line 89). This means that at this point only the thread handling the event is allowed to send messages. Hopefully it doesn't try to acquire any more locks... but it will. First the thread started by handler attempts to publish a message, acquiring the write mutex (InternalConnection.cpp line 703) and then waiting on the bolt mutex (Shared.cpp line 251). So that thread is stuck until the event handler returns. The the event handler thread attempts to publish its message and waits on the write mutex (InternalConnection.cpp line 703). So the event handler thread is stuck until the other thread finishes publishing its message. A deadlock.

Recommendation:

Ensure all mutexes are released before passing control to code you don't control.

Other option is to provide specific documentation about what actions are allowed to be taken from call backs, but that will be buggy.

The design of having several layers of mutexes seems quite complex and prone to bugs. Perhaps reducing the number of mutexes in InternalConnection and Shared could avoid these types of bugs in the future. Though I don't know what tradeoffs were made in the design.

System information:

Using the RHEL 7 build of GMSEC_API 4.6.

As a user I would expect that NASA distributed binaries for RHEL 7 are compiled correctly for RHEL 7 and likewise binaries distributed for RHEL 8 are compiled correctly for RHEL 8. The binaries in [1] are not correctly compiled for RHEL 7 or 8 because:

1. They use the same so names as existing RHEL provided libraries, which causes problems when ld attempts to find the correct library to load.
2. They recompile APR and OpenSSL instead of using the RHEL provide packages apr and openssl-libs. This has several detrimental effects:
   1. This causes security vulnerabilities because the NASA provided binaries are not updated in a timely manner. See #32
   2. Performance is worse because multiple copies of the same library need to be loaded into memory.
   3. Larger size on disk to to needing multiple copies of the same libraries.

See the Red Hat [2] and Fedora [3] packaging guides.

[1] https://github.com/nasa/GMSEC_API/releases/download/API-4.9.1-release/SUPPORT.zip

[2] https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/packaging_and_distributing_software/index

[3] https://docs.fedoraproject.org/en-US/packaging-guidelines/

When using a CA certificate file and activemq GMSEC_API ignores the CA certificate file unless sec-ssl-truststore-key is set to a non-empty string. See [1]. This is confusing for users since activemq-cpp does not have the capability to use a CA certificate file key. See [2].

Recommended fix is to pass along the sec-ssl-truststore value iff it is not empty, regardless of the value of sec-ssl-truststore-key.

[1] https://github.com/nasa/GMSEC_API/blob/master/wrapper/activemq/src/CMSConnection.cpp#L687
[2] https://github.com/apache/activemq-cpp/blob/d6f76ede90d21b7ee2f0b5d4648e440e66d63003/activemq-cpp/src/main/decaf/internal/net/ssl/openssl/OpenSSLContextSpi.cpp#L238

As a Java developer I expect NASA's code to never segfault and kill the JVM. NullPointerExceptions are fine, segfaults are not.

Using GMSEC API version 4.6 I can produce a segfault every time by calling getSubject() on a received REPLY message. I can call getSubject() on received PUBLISH or REQUEST messages without any problems. Reproducer code and stack trace top are below.

I assume the problem is not actually in std::string::c_str(). My theory is that the issue is due to using reinterpret_cast<Message*>(jlong) in the native code on some long that is not the address of a message. How NASA's Java interface got an invalid pointer is beyond me.

Current thread (0x00007f08c0012800):  JavaThread "main" [_thread_in_native, id=289137, stack(0x00007f08c6ee0000,0x00007f08c6fe0000)]

siginfo: si_signo: 11 (SIGSEGV), si_code: 128 (SI_KERNEL), si_addr: 0x0000000000000000

Register to memory mapping:

RAX=0x771abb87bdb6526b is an unknown value
RBX={method} {0x00007f0878441d68} 'Message_GetSubject' '(JLgov/nasa/gsfc/gmsec/api/jni/JNIMessage;)Ljava/lang/String;' in 'gov/nasa/gsfc/gmsec/api/jni/gmsecJNI'
RCX=0x00007f08c6fdbeb0 is pointing into the stack for thread: 0x00007f08c0012800
RDX=0x00007f0828002930 is an unknown value
RSP=0x00007f08c6fdbc38 is pointing into the stack for thread: 0x00007f08c0012800
RBP=0x00007f08c6fdbc50 is pointing into the stack for thread: 0x00007f08c0012800
RSI=0x00007f08c6fdbea0 is pointing into the stack for thread: 0x00007f08c0012800
RDI=0x771abb87bdb6526b is an unknown value
R8 =0x00007f087850b928 is pointing into metadata
R9 =0x0000000000000009 is an unknown value
R10=0x00007f08b10183fb is at code_begin+859 in an Interpreter codelet
method entry point (kind = native)  [0x00007f08b10180a0, 0x00007f08b1018a40]  2464 bytes
R11=0x00007f08c7a2ca00: <offset 0xa2ca00> in /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so at 0x00007f08c7000000
R12=0x0000000000000000 is an unknown value
R13={method} {0x00007f0878441d68} 'Message_GetSubject' '(JLgov/nasa/gsfc/gmsec/api/jni/JNIMessage;)Ljava/lang/String;' in 'gov/nasa/gsfc/gmsec/api/jni/gmsecJNI'
R14=0x00007f08c6fdbec0 is pointing into the stack for thread: 0x00007f08c0012800
R15=0x00007f08c0012800 is a thread

Stack: [0x00007f08c6ee0000,0x00007f08c6fe0000],  sp=0x00007f08c6fdbc38,  free space=1007k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libstdc++.so.6+0xf3c64]  std::string::c_str() const+0x4
C  [libGMSECAPI.so+0x314cea]  gmsec::api::Message::getSubject() const+0x1c
C  [libgmsec_jni.so+0x1083e3]  Java_gov_nasa_gsfc_gmsec_api_jni_gmsecJNI_Message_1GetSubject+0x6b
j  gov.nasa.gsfc.gmsec.api.jni.gmsecJNI.Message_GetSubject(JLgov/nasa/gsfc/gmsec/api/jni/JNIMessage;)Ljava/lang/String;+0
j  gov.nasa.gsfc.gmsec.api.jni.JNIMessage.getSubject()Ljava/lang/String;+5
j  gov.nasa.gsfc.gmsec.api.Message.getSubject()Ljava/lang/String;+4

public class SigsegvTest {


    private static BoltServer boltServer;

    @BeforeAll
    public static void setup() throws InterruptedException {
        // load GMSEC_API native code
        GmsecLoader.loadGmsec();
        // start bolt server
        boltServer = new BoltServer();
        new Thread(boltServer, "boltServer").start();
        boltServer.waitUntilStarted();
    }

    @AfterAll
    public static void cleanup() {
        boltServer.stop();
        GmsecLoader.cleanGmsec();
    }

    @TempDir
    Path schemaDir;

    @Test
    public void testSegfault() throws GMSEC_Exception, IOException, InterruptedException {
        // setup
        Log.registerHandler(new LogHandler() {
            @Override
            public void onMessage(LogEntry logEntry) {
                System.err.format("%s.%09d [%s] %s:%s %s\n",
                        logEntry.time.getSeconds(), logEntry.time.getNanoseconds(),
                        logEntry.level,
                        logEntry.fileName, logEntry.lineNumber, logEntry.message);
            }
        });
        Log.setReportingLevel(LogLevel.VERBOSE);
        Config config = new Config();
        config.addValue("connectiontype", "gmsec_bolt");
        config.addValue("server", "localhost");
        config.addValue("mw-multi-resp", "true");
        config.addValue("GMSEC-MSG-CONTENT-VALIDATE", "false");
        config.addValue("GMSEC-MSG-CONTENT-VALIDATE-ALL", "false");
        config.addValue("GMSEC-MSG-CONTENT-VALIDATE-SEND", "false");
        config.addValue("GMSEC-MSG-CONTENT-VALIDATE-RECV", "false");
        config.addValue("GMSEC-VALIDATION-LEVEL", "NO_ENFORCEMENT");
        config.addValue("GMSEC-SCHEMA-LEVEL", "0");
        config.addValue("GMSEC_LEGACY_SCHEMA_FILES", "true");
        config.addValue("GMSEC-SPECIFICATION-VERSION", "201600");
        // I wish I could turn off message validation entirely, but I can't
        // so fake a directory of schemas
        Files.createDirectories(schemaDir.resolve("2016.00"));
        try (Writer out = Files.newBufferedWriter(schemaDir.resolve("2016.00").resolve(".DIRECTORY.xml"))) {
            out.write("<DIRECTORY NAME=\"2016.00\"><SCHEMA NAME=\"HEADER\" LEVEL-0=\"GMSEC\" DEFINITION=\"\"/></DIRECTORY>");
        }
        try (Writer out = Files.newBufferedWriter(schemaDir.resolve("2016.00").resolve("GMSEC.HEADER.xml"))) {
            out.write("<SCHEMA NAME=\"GMSEC.HEADER\"></SCHEMA>");
        }
        config.addValue("GMSEC-SCHEMA-PATH", schemaDir.toAbsolutePath().toString());
        String subject = "GMSEC.1.2.3.4.5.6";
        List<Message> messages = new ArrayList<>();

        // subscribe
        final ConnectionManager cm1 = new ConnectionManager(config);
        cm1.initialize();
        cm1.startAutoDispatch();
        Thread.sleep(500);
        cm1.subscribe("GMSEC.+", new ConnectionManagerCallback() {
            @Override
            public void onMessage(ConnectionManager connectionManager, Message message) {
                // save request message
                messages.add(message);
                try {
                    if (message.getKind().equals(MessageKind.REQUEST)) {
                        // create and send two replies
                        Message reply = new Message(message.getSubject(), MessageKind.REPLY);
                        connectionManager.reply(message, reply);
                        /*
                        reply = new Message(message.getSubject(), MessageKind.REPLY);
                        connectionManager.reply(message, reply);
                         */
                    }
                } catch (GMSEC_Exception e) {
                    throw new RuntimeException(e);
                }
            }
        });
     
        // send request

        try {
            final ConnectionManager cm2 = new ConnectionManager(config);
            cm2.initialize();
            Thread.sleep(500);
            Message request = new Message(subject, MessageKind.REQUEST);
            cm2.request(
                    request,
                    60000,
                    new ConnectionManagerReplyCallback() {
                        @Override
                        public void onReply(ConnectionManager connectionManager,
                                            Message request,
                                            Message reply) {
                            // save message
                            messages.add(reply);
                        }

                        @Override
                        public void onEvent(ConnectionManager connectionManager,
                                            Status status,
                                            ConnectionEvent connectionEvent) {
                            System.err.println("" + status + " " + connectionEvent);
                        }
                    },
                    -1);

        } catch (GMSEC_Exception | InterruptedException e) {
            throw new RuntimeException(e);
        }

        // inelegant wait for messages to arrive
        Thread.sleep(5000);

        // verify saved messages
        // check waited long enough for the messages to arrive
        assertThat(messages.size(), is(2));
        // check received request message has a subject
        assertThat(messages.get(0).getSubject(), not(nullValue()));
        // check received reply message has a subject
        // this causes a SIGSEGV
        assertThat(messages.get(1).getSubject(), not(nullValue()));
    }

}

File [1] contains openssl binaries. Since it was compiled around June 8, 2021 many vulnerabilities have been found in the binaries that NASA is distributing. See [2]. It appears the NASA distributed binaries are vulnerable to: CVE-2023-4807, CVE-2023-3817, CVE-2023-3446, CVE-2023-2650, CVE-2023-0465, CVE-2023-0464, CVE-2023-0286, etc. Some of these the OpenSSL project considers high severity.

See also #31 for fixing the build of the support binaries to avoid redistributing OpenSSl.

[1] https://github.com/nasa/GMSEC_API/releases/download/API-4.9.1-release/SUPPORT.zip

[2] https://www.openssl.org/news/vulnerabilities-1.1.1.html