dfdlschemas / mil-std-2045 Goto Github PK

We have test data where the header indicates VMF data, but the optional message_standard_version field is not present.

This is not well formed and should be a parse error, or a least a validation error.

Related ticket #26

Many improvements to this schema are on the daffodil-dev branch because they depend on Daffodil 3.3.0-SNAPSHOT branch for fixes.

Once Daffodil 3.3.0 is released this can be merged to master and tagged version 1.0.0.

This field is defined to be 2 bits, but is only 1 bit.

Verified in revisions C and D1 specs.

A test that exercises this bit is obviously not here, so we need to add such a test.

We should increase the version number of the schema for this fix.

Some messages have no payload and are acknowledgements.

A test illustrating one of these would be helpful.

Replaces (https://issues.apache.org/jira/browse/DAFFODIL-1335)

The format has flags that indicate that the payload is uncompressed or compressed with a variety of schemes. Only uncompressed is supported and this should be verified with an assert.

mil-std-2045 makes use of this trick for strings that have DEL characters delimiting them when they are less than their max length:

<xs:group name="tString50">
xs:sequence
<xs:element name="value" type="xs:string" dfdl:lengthKind="pattern"
dfdl:lengthPattern="[^\x7F]{0,49}(?=\x7F)|.{50}"/>
<xs:sequence
dfdl:terminator="{if (fn:string-length(./value) eq 50)
then '%WSP*;'
else '%DEL;'
}"/>

</xs:sequence>
</xs:group>

This is incorrect because %WSP*; is incorrect.

This data will not parse as a string of length 50, not followed by a DEL, but will instead absorb the space characters after the text:

"abcdeabcdeabcdeabcdeabcdeabcdeabcdeabcdeabcdeabcde "

(That's 50 characters followed by 10 spaces.)

If binary data happens to follow an element of this type tString50, and the bits of that binary data just so happen to match the bit patterns of whitespace characters, then this format will erroneously absorb them. For example, if a tString50 of length 50 is followed by a 32 bit integer having value 0x20202020, then those 4 bytes will be absorbed by the WSP*.

The intent of the dfdl:terminator expression is if the string length is 50, to NOT have a terminator region, i.e., %ES; is the correct value to be used here, not %WSP*;.

The problem is that in the DFDL spec, %ES; alone as the only string literal for a terminator (or any delimiter) is not allowed.

This constraint in the DFDL spec is probably incorrect - in the above case, we are NOT scanning for delimiters as in dfdl:lengthKind 'delimited' so it does no harm to say the terminator is '%ES;' alone. This topic has been raised with the DFDL Workgroup mailing list for discussion (on 2018-10-01).

Until this is resolved, I suggest these possible workarounds.... which may or may not work.

Replace '%WSP*; above with "%ES;%ES;" or "%ES; %ES;". Both meet the letter of the constraint in the DFDL spec., and so may bypass Daffodil checking. But they mean the same thing as just '%ES;' by itself.

This can be done like this:

<xs:choice>
  <xs:sequence dfdl:hiddenGroupRef="tns:emptyFutureUseGroup"/><!-- 5 bits all 0 -->
  <xs:group ref="tns:futureUseGroup2"/>
</xs:choice>

This will also allow removal of the not-needed enveloping element futureUseGroup1 (or 2, or 3) and allow the future use elements to appear at the top level without this.

Daffodil has experimental feature support to decode numeric enums into their symbolic strings.

Something like this will eventually be incorporated into the DFDL standard, and having this would make this schema far more useful.

So add the dfdlx:repType and dfdlx:repValues (to the enumerations of the strings) so this does symbolic decode.

One variant of a mil-std-2045 message is where the UMF (user message format) field is "0001" which means "binary file". In this case the data after the header is the contents of a binary file. The file name of this is given in another field (which must be present).

We need a test of this.

Uses of this DFDL Schema need to be able to combine this header with definitions of what the contents of that binary file content contains. That is, this schema cannot take the position that this binary file content is to be represented as some hexBinary element. The binary file content is actually beyond the end of the header; hence, isn't part of this schema.

There are many dates/times/datetimes in this schema, and it would really help if these were not parsed as individual fields, but were converted into xs:date, xs:time, or xs:dateTime fields. So as to avoid this:

             <year>
              <value>19</value>
              </year>
              <month>
                <value>1</value>
              </month>
              <day>
                <value>22</value>
              </day>
              <hour>
                <value>4</value>
              </hour>
              <minute>
                <value>50</value>
              </minute>
              <second>
                <value>10</value>
              </second>

And instead get:
<dateTime>2019-01-22T04:50:10</dateTime>

The enum for this element is different in rev C than in rev D1.

There simply needs to be different enums used depending on the revision of the header.

As with the VMF versioning which uses a different enum for C and for D1 which are not consistent with each other, the same issue arises for XML_VMF.

The enum messsageStandardVersionXML_VMFEnum must be split into a C and D1 variant, in a manner similar to what was done for regular messageStandardVersionVMFEnum_C and messageStandardVersionVMFEnum_D1, with corresponding messageStandardVersionVMFGroup.

Many hard limits like 16 are in the spec.

Ideally nothing should be 'unbounded'.

The checkConstraints asserts and other error situations should produce meaningful messages.

The schema should enforce a internal constraints on message type.

A 2045 message header can have an array of message_handling_groups

Each message_handling_group has a UMF indicating if the data is Link16, Binary File, VMF, NITF, etc.

For some of those there is a more granular message_standard_version which tells us which specific version of that standard.

Following that there is an optional vmf_message_identification_group, which obviously applies only to VMF, but this is not enforced. This contains the K-series message ID information (K01.02.03 type identification), which is not held in a VMF-message proper so must come from this header.

Following that there is an optional file_name which is NOT applicable to VMF, but to binary files, and possibly other types.
(E.g., if NITF data is carried it is not clear if there is an associated file name, or just a collection of bytes for an anonymous NITF data block.)

Right now, the message type can indicate VMF, but the data can still contain a file-name as if it was for a binary file.

rename future fields so that they are unique (future1, future2, future3) so JSON won't have repeated keys.

The variable-length strings idiom can be improved with a assert (test Kind pattern) and by adding a max-length facet to each string.

Update paths for schemaLocation attributes. Absolute paths should start with "/", relative sibling files should have no paths.

See test_2045_D1_all_fields_x2u

For example, recipient_address_group and information_address_group each have a maxOccurs which should be 16.

This schema is old and pre-dates the introduction of DFDL features like choiceDispatchKey.

Presence indicators should be done via choiceDIspatchKey or occursCountKind 'expresion' to eliminate the overhead of backtracking.

It also makes this a better example of how to best handle presence flags in DFDL.

message_precedence_codes - The enum for this element is different in rev C than in rev D/D1.