empty-repeat.xml.txt

to catch issue with randomize() usage

Differences:

• Enketo will keep going and show all XPath errors in form. ODK Validator exits after the first error.

No false errors/warnings for:

• src attribute on instance element
• unknown bind attributes such as [length,requiredMsg]
• /path/to/non-existing/node (? check if this is actually an issue)
• any valid XPath 1.0 syntax including e.g. fancy axes

strictly speaking it's allowed, but better to not do (for data exports)

Enketo Core will not pick up those namespace declarations when it builds its namespace resolver.

explore https://github.com/nexe/nexe

Seems rather cool. Won't require installation of nodeJS.

figure out why

There is a bunch of magic that enketo-core does before sending to the Xpath evaluator.

• fix dependency issues (e.g. enketo-config)
• change how jquery is added to formModel to avoid jQuery error with window and document or somehow run formModel inside jsdom context
• remove require('./plugins') from Form-model.js. It is not used and causes an error in validator.
• make sure the 'native' XPath evaluator is removed/disabled
• use enketo-core's evaluate function

skeleton.xml.txt

Thanks @Xiphware!

This is to not output false warnings for:

<data>
     <repeat>
           <a/>
    </repeat>
   <repeat>
           <a/>
    </repeat>
</data>

These could later be replaced by XML schema:

• root is h:html
• root has 1 h:head child
• root has 1 h:body child
• h:head has 1 model child
• model contains at least 1 instance child
• first instance has 1 child (only)
• first instance child has id attribute (does this actually matter?)
• contains meta or orx:meta and instanceID or orx:instanceID (although this check is also in Enketo Core
• binding without calculate should have corresponding body element (finds logical groups that don't have a ref attribute) These could be a warning, and an error in case they also have a relevant.

No spaces, and maybe no special characters

kobotoolbox/enketo-express#1035

E.g.:

• node a has calculation b + 1
• node b has calculation a + 1

Or:

• node a has calculation b + 1
• node b has calculation c + 1
• node c has calculation a + 1

Suggested implementation:

1. iterate through all calculation expressions to build a map of all calculation nodes with the nodes they reference (using xform._extractNodeReferences).
2. then, for each calculation, do a recursive check of all dependent nodes (and their dependencies etc)
3. check if any of the dependencies in 2 is the node itself

E.g. using https://docs.travis-ci.com/user/deployment/releases/ for Linux and MacOS that is triggered when a commit is pushed that has a git tag.

I started a script in feature/build-script to be use for macos and linux

Would be good to identify the errors in the message as well, either by a prefix or a code

"XML Parse Error: blabla"
"Enketo Loading Error: blablab"
"XPath Error: blalb"

see form xpath-fails.xml

by just not validating its use

Rationale:

• Commcare/Enketo-style external data
• Access to native XPath functions already supported in enketo: starts-with(), ends-with(), contains(), substring-after(), substring-before(), translate(), normalize-space(), floor(), ceiling(), last(), etc
• Access to all native XPath 1.0 axes that greatly increase the power of XForms, e.g. we can prevent duplicates in repeats using the preceding-sibling axis.
• Ability to add agreed new OpenRosa functions, such as randomize(), without dependencies.
• Ability to add custom features to client-specific enketo-validate ports.

Some ideas to do this:

first version:

• check if it's valid XML

• check it conforms to the XForm format (an XML Schema would be wonderful - maybe Dimagi has one or ODK Validate uses one already?) - required sections, required attributes on each node type, doesn't have to be hugely complex to be very useful

• run each relevant/calculate/constraint XPath expression inside a try/catch statement and see if causes an error, using Enketo's XPath evaluator. If needed, add tolerance for references to external instances (that are empty because they're external).

later:

• for more descriptive errors we could do a separate check for validity of nodeNames perhaps
• check if all bind nodesets exist in primary instance (apparently, ODK Validate doesn't do this ref. formhub/uuid behaviour where an incorrect binding nodeset path doesn't trigger errors and somehow gets 'corrected' in ODK Collect)
• check if all instance elements referred to anywhere in instance(ID)/path/to/node exist in model
• check if all itext elements referred to anywhere in exist in model
• (maybe) perform XSLT transformation which could fail or output error messages that are built into the XSL sheet
• similarly, attempt to parse and initialize the whole form.

These would all be simple to do, but to provide useful feedback messages and to make validation as complete (though better) than ODK Validate would probably take quite a lot of additional work.

False negatives:

• using indexed-repeat() to get the value of the same node in a different repeat instance
• using a predicate that includes ".", e.g. count(//text1[normalize-space(.)=“yes”])

This will convert /path/to/node syntax to ${node}

Currently the error message is utterly useless (in ODK Validate it is much better):

Errors:
Unexpected close tag Line: 94, Column: 38, Char: >

e.g. for the following <instance> (PROCEDURE is not closed):

    <instance>
        <Last_group_does_not_display id="PSS" version="1">
          <PROCEDURE>
            <DOP/>
            <SURG/>
            <PROC_NOTE/>
            <PROC_GRID jr:template="">
              <PROC/>
              <LATER/>
              <TMLOC_BLANK/>
              <PMLOC/>
              <SLNDLOC/>
              <CALC_LOC/>
            </PROC_GRID>
            <CALC_SPEC_TTL1/>
            <CALC_SHOW1/>
            <CALC_SHOW2/>
            <CALC_SPEC_TTL2/>
          <SPEC>
            <GPT>
              <TUMTYP/>
              <PM_TUMLENGTH/>
              <PM_TUMWIDTH/>
              <PM_TUMUNITS/>
              <PM_LGMLENGTH/>
              <PM_LGMWIDTH/>
              <PM_LGMUNITS/>
              <PM_MGMLENGTH/>
              <PM_MGMWIDTH/>
              <PM_MGMUNITS/>
              <PM_LENGTH/>
              <PM_WIDTH/>
              <PM_UNITS/>
              <ENTRIES>
                <PM jr:template="">
                  <PM_SPECIM/>
                  <PM_SPECIMHIST/>
                  <PM_GSS/>
                  <PM_MEDIALAT/>
                  <PM_SLICEMETH/>
                  <PM_ANTDEEP/>
                  <PM_SLICEMETH2/>
                  <PM_SUPINF/>
                  <PM_SLICEMETH3/>
                  <PM_SPECSLICE/>
                  <PM_TOTSLICE/>
                  <PM_TOTSLICEMETH/>
                  <PM_SLICENOTE/>
                  <PM_SLICEPL/>
                  <PM_SLICENUM/>
                  <PM_BLOCKNUM/>
                  <PM_SLICEPL2/>
                  <PM_SLICENUM2/>
                  <PM_BLOCKNUM2/>
                  <PM_PERFLAB/>
                  <PM_LYMNTISS/>
                  <PM_NOTE/>
                </PM>
                <TM jr:template="">
                  <TM_SPECIM/>
                  <TM_SPECIMHIST/>
                  <TM_GSS/>
                  <TM_MEDIALAT/>
                  <TM_SLICEMETH/>
                  <TM_ANTDEEP/>
                  <TM_SLICEMETH2/>
                  <TM_SUPINF/>
                  <TM_SLICEMETH3/>
                  <TM_TOTSLICE/>
                  <TM_TOTSLICEMETH/>
                  <TM_SLICENOTE/>
                  <TM_SLICEPL/>
                  <TM_SLICENUM/>
                  <TM_BLOCKNUM/>
                  <TM_SLICEPL2/>
                  <TM_SLICENUM2/>
                  <TM_BLOCKNUM2/>
                  <TM_PERFLAB/>
                  <TM_LYMNTISS/>
                  <TM_NOTE/>
                </TM>
              </ENTRIES>
            </GPT>
          </SPEC>
          <LYMPHNODES>
            <LYMNDISS/>
          </LYMPHNODES>
          <meta>
            <instanceID/>
          </meta>
        </Last_group_does_not_display>
      </instance>

Not sure. Ideally we should keep fixing bugs in nested repeats. However, it is quite a pain, very expensive. Data analysis is also not going to be much fun with nested repeats for users, so a warning might be helpful?

external data, media resources

The spec info is here: http://getodk.github.io/xforms-spec/#uris but in addition we consider http(s):// URLs valid too.

aliasify and browserify at least, I think

1. command-line
2. NodeJS module
3. web (perhaps)

getodk/xforms-spec#184 (comment)

Since this is not supported in Enketo.

Installing enketo-validate on CentOS v7, with the latest change:

[root@formservice enketo-validate]# npm install
npm WARN lifecycle [email protected]~install: cannot run in wd %s %s (wd=%s) [email protected] browserify src/FormModel.js > build/FormModel-bundle.js && browserify -g aliasify src/FormModel.js > build/FormModel-bundle-oc.js /root/applications/enketo-validate

Had to do: sudo npm install --unsafe-perm

probably no difference but worth checking for required, relevant, constraint

No longer create a separate binary for OC, but let the --oc flag do both:

1. load the OC XPath evaluator
2. validate additional custom XForm rules

The binary in Mac OS increases from 54.7 MB to 55.3 MB so this increase is negligible.

If loaded cleverly, it should not slow down the tests either.

stderr is treated as warning if result is valid (0)

https://github.com/XLSForm/pyxform/blob/master/pyxform/odk_validate/__init__.py#L116-L121

Not sure if pyxform/ODK Validate can make a distinction between warnings and errors if result is invalid.

The prepublish build script should actually be run as an install script.

For some reason this install (rollup) script fails when using npm install enketo-validate in an app.

Therefore, I've temporary published the built files to /build. I don't like this, so this should be solved properly some day.

• invalid nodename, e.g 1a and a b => parse error?
• missing closing tag
• missing namespace declaration for used namespace prefix
• function call with too few arguments
• function call with too many arguments
• bloody jr:choicename call => just pass whatever
• instance('data') for nonexisting instance => error

libxmljs (exposed via libxslt) has the capability to do such validation

@MartijnR

We can see that the latest version of pyxform offers command line options to choose how the XForm XML is validated.usage: xls2xform [-h] [--json] [--skip_validate] [--odk_validate] [--enketo_validate] [--no_pretty_print] and it appears that ODK validation will not pass some of the functions described in https://docs.opendatakit.org/form-operators-functions such as count-non-empty().

We are hoping to better understand why there are two validators and which we should be using. Then, if required we would change the Survey123 conversion API to use the most appropriate one.

Any information would be appreciated. Thanks

Helpful if enketo-validate is run after pyxform --skip-validate conversion