We use YYYY.x.x for Collect and Central because they’re applications and it’s hard to define what breaking changes are in that context.

We use semantic versioning for libraries. This is a library so it feels like it makes sense to use semantic versioning.

Sources of breaking changes for web-forms package:

• engine: what forms are supported, how computation is performed
• frontend: look and feel
• interface exposed to host

We probably want to make the interface between the host and web forms stable.

We want to avoid confusion about what's actually in use (versions of sub packages)

Let's start by deciding what the interface to the host is and we can figure out a longer term versioning scheme longer term

As a form designer
I want to write some text without a value
So that I can provide information or guidance to form users

This is the note type in XLSForm. If there is no value, there should be no control or vertical space for a value.

Typically a note in XLSForm only has a label. But because XLSForm makes note an alias to type string with the readonly bind attribute set to true(), everything that's allowed for a string is allowed for a note. Form showing what could be done with the XLSForm note type.

Partly motivated by #34, and definitely by #29, now is a good time to consider/potentially rethink package names. Currently, the (weak) assumption has been that packages published from this monorepo will be published under an @odk scope in the NPM registry. And an exception has been made for tree-sitter-xpath (because that's the naming convention suggested for tree-sitter grammars).

I'd like to consider, instead:

• A registry scope of @odk-web-forms, which would be specific enough to allow much more general naming of sub-packages.
• Include the tree-sitter grammar in this scope for consistency: I believe some other grammars are scoped, I don't think it's unreasonable for ours to be. I also still want to keep the use of tree-sitter open to reconsideration, and don't want to continue to make special accommodations for it that we don't need to.
• Favor shorter, simpler names for sub-packages.

Taking what we have now plus the split discussed in #34, we could have:

• @odk-web-forms/tree-sitter-xpath
• @odk-web-forms/common (I'm still open to other naming options for this, just... not "utils" please)
• @odk-web-forms/xpath (and I think scoped this way, I'd be more inclined to think about it as less of a generic XPath implementation, and more tailored to our use cases)
• @odk-web-forms/xforms-engine
• @odk-web-forms/ui-vue (in a previous edit, I suggested @odk-web-forms/view, but I think this name better deals with the homonym issue, and reflects the possibilities we discussed about other potential clients of the engine)

The ODK XForms spec was redressed to align with XPath so we don't need to support the legacy behavior.

• How to display?
  • Collect has project-level configuration:
    

User story

As an application developer
I want to embed an ODK web form regardless of the structure and frameworks of my application

But maybe initially we focus on Central which uses Vue

Possible options

• Javascript function (similar to Enketo Core)
• Vue component (similar to JSONForms Vue binding)
• Web Component (similar-ish to Fore but we'd expose a single component representing an entire form; possibly related to entire Fore form as Web Component)

@sadiqkhoja points out here that Vue recommends exporting Vue components

User story

As the developer of an application
I want to embed one or more ODK Web Forms
so that users of my application can use a rendered form from a form definition and other settings

Requirements

• #76
• #23
• #43
• #30
• #31
• #36
• #44

In @odk/web-forms there is currently an informal separation between what we're calling the "engine" (largely contained in packages/odk-web-forms/src/lib/xforms) and the UI (large contained in packages/odk-web-forms/src/components). As we've discussed the migration of the UI to Vue, we've also discussed these related goals:

• Formalize the engine/UI boundary
• Make explicit that we intend the engine to be web framework-agnostic
  • In so doing, potentially continue to maintain the current Solid UI in tandem with Vue
• Treat use of reactivity in the engine (currently also using Solid) as an implementation detail
  • This use of reactivity is explicitly not considered part of the interface contract at the engine/UI boundary
  • Present use of Solid for the purpose is not necessarily under reconsideration in the scope of the Vue migration

This raises important questions about the interface of the boundary itself. We discussed the importance of this yesterday, and will be discussing design in more detail today. I wanted to take the opportunity to attempt to frame this discussion, and to introduce some prior art for consideration.

Assumptions

First I'll lay out a few assumptions about goals for this interface. Some of these are restating assumptions we've been aligning on in recent days/weeks, the rest fall out of these discussions (so hopefully nothing here is particularly controversial!):

1. Engine responsible for XForms domain logic: The responsibility for business logic pertaining to XForms (i.e. the ODK XForms spec)—e.g. parsing forms into a particular runtime schema, form-defined computation, dependency resolution and synchronization—is the domain of the engine. Pushing these aspects of business logic out to any particular UI client would be challenging to maintain and test, and would risk drift between multiple prospective UI clients.

2. UI responsible for reflecting and interacting with form state: The responsibility of any given UI client—in terms of the engine's business logic—is to present form structure and state to a user, and to provide means for the user to interact with the form to manipulate its state.

3. Engine-UI interface responsible for conveying updates: Implied by 1 and 2:

   • the engine has a responsibility not just to initiate form state, but to update state as the dependencies of a given computation are updated;
   • a UI client, in turn, has a responsibility of reflecting its presentation of such updates, and updating its interactive affordances accordingly
   • the interface between the two, then, must provide a means to convey updates

4. Reads and state updates are most pressing design concern: While we should thoughtfully address all aspects of the engine-UI interface, reading state—especially reading engine-driven updates to state—are the most pressing unanswered question in the current interface. In the interest of making important short-term progress (and enabling effective iteration from there), it's likely prudent to limit the scope of this design effort to this read/updates concern. Expanding that scope to reconsider the current write interface may also be wise if the write interface is directly impacted.

5. Encpasulation as interface contract: The interface between engine and UI should not overly expose inner workings of how business logic is achieved, and that includes any particular notion of internal reactivity which might be used to that end. Notwithstanding Hyrum's Law, any such implementation detail should be treated as incidental and subject to change.

   More concretely: just because the engine may currently use Solid reactivity to perform updates internally does not mean that Solid reactivity is the interface UI clients use to consume updates.

6. Encapsulation as design guidance: It isn't a hard requirement, but likely a very good guidance, to intentionally obscure these sorts of implementation details, and in particular to intentionally distinguish aspects of the engine-UI interface concerned with reads/updates from the present reactive internals.

7. Prior art as design guidance: In contrast, we are not inventing a wheel from whole cloth. Prior art being an excellent source of inspiration for interface design, it might be prudent to adopt interface patterns from another existing reactive implementation. In particular, it might be worth exploring where there is overlap between individual UI frameworks' interfaces to the outside world—which is to say, common themes for embedding non-framework functionality into idiomatic framework code.

Lastly, I think this is more a requirement than an assumption, but it's one we didn't discuss at length and one I think we should keep in mind:

8. Form values are polymorphic: while the current implementation largely ignores this by implementing values as strings, there's some hint of it in #14 (but I'd caution that I'm referencing it more as an example of the requirement, less as an example of prior art). In any case, the XForms domain deals with values of a variety of data types, and in some cases collections of values (as in #14). I don't know if we need to address this requirement directly in the scope of this particular design, but we should at least keep it in mind, as it will have implications for client UIs.

Prior art

With those assumptions stated, I think it might be good to seed the design discussion with some reference to prior art that might inspire us. For each instance of prior art, I'll include a minimal pseudo-code example of how it might be applied.

Callbacks

Callbacks are a built-in language feature, and a persistently common idiom for receiving updates over time. Example:

interface EngineNode<T> {
	getValue: () => T;
	onUpdate: (callback: (updatedValue: T) => void) => void;
}

In this example there is still a distinct method to get the value. While a callback API can be executed synchronously, it's difficult to convey this in the interface itself. Many recent callback-based APIs provide a similar companion API for synchronous retrieval, often superceding the callback when a value is present. (Example: MutationObserver's takeRecords).

For this example, and several of the others, we might also consider an update-receiving method which returns void as an opportunity to combine the more likley use cases (get initial value, and any subsequent updates). I think this kind of combination can be confusing if not designed and documented carefully, but I'm calling it out as a permutation which could apply to several prior art examples.

Callbacks are also likely to appear as part of other interfaces. Realistically, nearly all of the other ways to express "receive updates over time" will be specializations of this underlying language feature, or specific conventions around it.

Events (DOM)

A browser-native idiom also commonly used to convey updates over time is the DOM Event type, with associated dispatch and listener APIs.

interface UpdatedValueEvent<T> extends CustomEvent {
	data: T;
}

interface EngineNode<T> {
	getValue: () => T;

	// or `addEventListener(...)`, etc
	on: (
		eventName: 'update',
		callback: (event: UpdatedValueEvent<T>) => void
	) => void;

	// or `dispatchEvent(...)`, etc
	trigger(event: UpdatedValueEvent<T>): void;
}

This tends to look like callbacks with a little bit more ceremony. A distinguishing feature of DOM events is the ability for them to propagate from a given EventTarget to another (such as the concept of "bubbling" in the actual DOM node tree). A corresponding downside of this feature is that it can be challenging to follow events to their source, both at runtime (call stacks become more convoluted, and may be broken by asynchrony) and in source code (static analysis of e.g. method calls often does not extend to these sorts of event dispatch and propagation).

Observables (user-land)

Observables are a concept used in many reactivity libraries (such as RxJS) or supported for compatiblity by others (such as Solid, which uses roughly the compatibility interface defined below).

interface Subscriber<T> {
	next?: (updatedValue: T) => void;
	complete?: () => void;
	error?: (error: unknown) => void;
}

interface EngineNode<T> {
	getValue: () => T;
	subscribe: (subscriber: Subscriber<T>): void;
}

Observables have semantics which overlap with another language feature, iterators. For the purposes of this discussion, we can also consider this a lense on how we might "abuse iterators" (and/or generators), as I quipped on our call yesterday.

In contrast with a basic callback, Subscriber has an object interface, capable of receiving additional information beyond value updates:

• complete upon termination of value updates over time (possibly analogous in our usage to the removal of a repeat instance; this also further likens the concept to iterators)
• error called when an error condition occurs over time

We could also conceivably expand this subscriber/observer interface to convey other semantic information.

In any case, this example is a helpful reminder that we'll want to consider aspects of state-over-time besides values, validation and termination among them.

Observables (future-spec)

There have been some efforts to standardize Observable, in the past as a potential JavaScript language feature (TC39), and more recently as a potential web platform feature (WICG). This more recent effort seems to have some real momentum (as it's backed by Google). RxJS is mentioned as a reference implementation, and Solid as a compatible interface, so another example would be redundant.

This is called out separately from user-land Observables specifically because it's an active standardization effort, and because it provides additional API discussion we might consider pertinent.

Subscribe-on-read ("signals", Vue's ref, etc)

Discussed a bit above is the concept of combining the interface for synchronous (e.g. first) read, and updates over time. This concept has been popularized in part by Solid's term "signal"—which other frameworks like Preact and Angular have adopted. Other frameworks provide a similar mechanism by a different name: Vue uses ref. The actual interface varies, but the gist is that accessing a value also establishes a subscription to future updates to that value.

Not to bias any particular framework's implemtnation or terminology, the following example uses a contrived name and interface for the concept:

interface SubscribingReader<T> {
	read: () => T;
	write: (value: T) => T;
}

interface EngineNode<T> {
	value: SubscribingReader<T>;
}

In order to actually implement subscribe-on-read, a system with this sort of runtime-defined reactivity must provide some means to establish a scope (or context, but not to be confused with what many UI frameworks call "context" for other uses) where subscriptions are tracked. Some solutions implicitly couple this scoping to other APIs with related concerns. Others, also implicitly, achieve scoping with a compiler (usually while addressing other concerns like templating). There are also more explicit APIs for establishing a tracking scope, but these tend to be less common, and generally more complex from an interface perspective.

Et cetera

I've tried to highlight some of the more common themes above, but of course this is not exhaustive. A very interesting resource I came upon while getting up to date on the Observable standardization efforts is A General Theory of Reactivity (GTOR). There are other areas of prior art apart from what's termed "reactivity", though it's unclear whether we'd find value in exploring those for our purposes.

• Visual design
• Canvas can be drawn on
• Canvas can be cleared
• Strokes can be undone
• Strokes can be colored
• Maintains resolution on small screens
• Works with default
• Works with edits

To consider:

• What is the relationship between this appearance and signature, draw appearances
• Is annotation done inline always or on an isolated screen in mobile mode like Collect/Enketo?
• Consider signature pad, used by Enketo. Is there Enketo code to consider?
• https://markerjs.com/
• http://fabricjs.com/
• https://github.com/wawjqyh/vue-image-annotation

Later:

• Works with pages on any page (hopefully not a relevant concern in this world but making explicit due to experience with Enketo)
• Works in grid

User Story

As a ODK project manager
I want to specify simple style customizations
So that my forms can match my branding

(Decide whether in scope/a separate issue)
As the developer of an application that uses ODK Web Forms
I want to specify a completely custom style
So that my forms can match the host application

Questions

• Do we want to make it easy for people to opt out of Material/specify their own design language?
• What colors? Do we generate palette?
• Is passing in something like a logo image a theming concern?
• Do we allow for a preamble and suffix? Or is that a responsibility of the host app?
• Are we comfortable deferring the scope to whatever component framework we pick?
• If we defer to the component framework, should we still subset what we expose?
• Maybe styling is expressed as a CSS blob

Volta, etc

Skip:

• ones that relate to functionality that's not intended to be supported yet (e.g. events and actions)
• ones that are tightly-coupled to the JavaRosa internal representation

Put in separate test package so we're not inheriting all of the test structures from JR

Maybe wire up something to show alignment level with JR

How can we establish a link between the tests between the two libraries?

Collect's implementation: https://github.com/getodk/collect/blob/e49eae16657d6677abde44e4475c1f08d3ccf9ae/collect_app/src/main/java/org/odk/collect/android/utilities/HtmlUtils.java#L23

• It includes some subtleties such as enketo/enketo#24

Documentation: https://docs.getodk.org/form-styling/#markdown

Enketo allows disabling style: https://github.com/enketo/enketo/blob/37a7af1886743c7e20ddcb9f3b0f8708d184cd88/packages/enketo-transformer/README.md?plain=1#L85

• @eyelidlessness: "I think it’s something we might want, eg maybe for like summary views. Either way I think if we’re gonna parse it, that would happen in the engine, and then we tailor the representation to what clients need and what makes the most sense for all of the common concerns around markdown. Structured data feels much safer to me than serializing it in the engine, both in terms of security and client flexibility"

Possible subtleties around using dynamic values in links: enketo/enketo#804

Consider using https://github.com/remarkjs/remark

Per this review comment:

Might be good to augment with some of the design considerations from the PR description and some things we discussed about deferring to native like the current Enketo evaluator does vs not

I decided to merge and come back to this later, but I agree it would be good to have. I think in general we'll want the same in other READMEs and perhaps even something at the root.

Breaking out an issue to track this discussion thread on #14.

Current (albeit incomplete) behavior

The intent of the behavior in #14 (which is incomplete, as called out in the comment) is this:

Given a form like:

<instance>
  <data>
    <sel />
    <fil />
  </data>
</instance>
<instance id="sel-opts">
  <root>
    <opt><val>option a</val></opt>
    <opt><val>option b</val></opt>
  </root>
</instance>
<!-- ... -->
<select ref="/data/sel">
  <itemset nodeset="instance('sel-opts')/root/opt[contains(val, /data/fil)]">
    <value ref="val" />
  </itemset>
</select>
<input ref="/data/fil" />

If a user...

1. Answers /data/sel, selecting "option a"
2. Answers /data/fil, entering "b" (causing "option a" to be removed from /data/sel's available items)

Then the form state is updated so that the prior selection of "option a" is:

• Cleared from /data/sel
• Retained in memory until¹ another selection is performed with the filtered options for /data/sel

At this point, the form's instance state is correct, but it allows either:

Next (A), if a user...

3. Answers /data/sel again, selecting "option b"

   Then the form state is updated so that:

   • /data/sel is set to "option b"
   • The previous selection of "option a" will¹ no longer be retained

4. Clears /data/fil (restoring "option a" as an available option for /data/sel)

   Then the form state for /data/sel is unchanged from step 3

... OR ...

Next (B), if a user...

3. Clears /data/fil (restoring "option a" as an available option for /data/sel) without any further user action on /data/sel following step 1's affirmative selection of "option a"

   Then the form state is restored to its state after step 1 (i.e. the value of /data/sel is again set to "option a")

Alternative behavior

Per the discussion thread, the alternative would be not to retain any memory that "option a" had been chosen (step 1), once that option is filtered out (step 2). Also per the discussion thread, this is the behavior in JavaRosa. Furthermore, it would be fair to say it is the obvious default behavior. It is unquestionably the behavior we'd implement if we skip asking the question posed by this issue.

It must have slipped my mind that emscripten is an external dependency for tree-sitter-xpath. I'd originally hoped to address this by making it a package dependency. I believe that may be possible (despite the tree-sitter documentation suggesting otherwise), but if I recall I deferred addressing it before other tooling dust had settled. Either way it should be addressed, either by inclusion in package install or by explicit documentation.

Depends on #82

User Story

As an application developer
I want to get access to filled forms from web forms
So that I can submit them to a server or otherwise process them

As a form end user
If my submission fails, I want to know what happened (e.g. I'm offline, the server isn't responding, etc)
So that I can take appropriate action

As a user I should see a language selector if the Form definition has multiple languages and I should be able change the language of the form using that selector

Use existing @getodk organization/scope.

Rename current ui-vue package to web-forms. If there are other frontends in the future, they can have more specific names. It's likely we wouldn't be the ones creating/maintaining other web frontends. We'll likely have other kinds of frontends like Scenario.

This is something I’ve intentionally deferred because it intersects with tooling which has already eaten up a lot of time and energy. But it should be tracked, and some initial observations are worth dropping here for a starting point.

• The crypto-js library is quite large, and probably not the ideal solution for digest anyway. Any solution should likely also be optional, as it is in ORXE. I’ve looked at other options, but that’s best for another issue.
• The Temporal polyfill is gargantuan. It’s also poorly set up for bundle size optimization like tree shaking (this can be improved by importing the parts in use directly from source, but it’s worth reconsidering the polyfill in general depending on the standard’s progress)
• The web-tree-sitter WASM size is large-ish, and tree-sitter-xpath isn’t tiny either. These would benefit from aggressive caching.
• In general the best tooling-side approach to bundle size optimization is an open question. Besides deferring for pragmatic progress reasons, it also makes sense to hold until other stories are more clear (like offline, PWA if we pursue that, etc)

I’m filing quickly on mobile ahead of our check in, but briefly:

• tree-sitter adds a fair bit of tooling complexity
• And page weight
• It’s unclear that we benefit from its most compelling distinct features like incremental parsing
• While the grammar is stable, it’s a constant build time drag in CI
• Other tools might be better suited for eg TypeScript integration (as in type gen for aspects of the syntax we’re concerned with)

I briefly explored Ohm over the weekend and it’s a pretty compelling alternative option (albeit with some reservations around maturity). Beyond its TS support, I was encouraged by how easy it was to build a cromulent XPath grammar, and how much more closely it matches the grammar from the spec.

User Story

As a form designer
I want to be able to define form fields that use attached media
So that I can show images, play audio, play videos, and query xml/csv/json files

In Enketo, grid can be applied at the form level to modify the layout of an entire form: https://blog.enketo.org/gorgeous-grid/ It was designed primarily to make printable forms and forms for computer data entry and works really well for those cases.

Users choose it because the find the layout and styling visually appealing.

It also makes repeats more compact and somewhat easier to use:

Sometimes it would be helpful to be able to insert a grid into a form that has a top-to-bottom layout. We have discussed also having a grid appearance to satisfy that need.

The main challenge with Enketo's current grid is that because it's a "theme", a lot of customizations that would make sense to apply to both (e.g. colors) must be specified separately.

Blocked by #28

To embed in Central, we’ll need to talk about it publicly.

We’ll want to describe high-level drivers, likely timeline, feedback wanted

Highlight effort to port JavaRosa tests

@lognaturel had mentioned differing behavior between Collect and Enketo, and linked kobotoolbox/enketo-express#193 for context. My suspicion is the behavior will already be aligned, but it would be good to test just to be sure.

As encouraged by @getodk/central for easier switching between projects.

• #45
• Pick where we'll get styled components from

Acceptance Criteria:

• A Vue library that can be integrated with any Vue Application
• It should be able to render: Form Title, Group, Repeat, Text (single line, default appearance)
• It should be able to compute "relevance" and "calculate"
• All components should have automated tests
• It should be able to raise event on Submit/Send button
• Abides by the eslint rules of the project

Not in scope - separate issues

• Language selection #58
• Select_one Select_many and other question types #59
• Theme selection #43
• Layout grid #16
• Layout pages #33
• Nothing related to debugging (show irrelevant questions as disabled)

• What js/Vue tooling
• Transifex to match other ODK tools
• Bootstrap with Collect/Central/Enketo strings?
• Define strings globally like Collect/Enketo or have local option like with Central?

Strings from other ODK components:

• Enketo
• Collect
• Central

User story

As a form designer
I want to be able to specify form fields that capture files
So that they can be accessed from the host application

Todo

• Decide on file naming. Collect uses UNIX timestamps to try to guarantee uniqueness across all media, could still be conflicts in a big deployment. Enketo uses the original filename and appends the hour and minute. Conflicts are very likely. We should decide on a default naming scheme that limits risk of conflicts and ideally conveys some human-readable info. Users would like to name the files.

See these failures, both of which succeeded after clearing their caches and rerunning. The same occurred in #10, but I forgot to make note of it at the time.

Related to the engine/client interface work serving (#45 and the fuller design coming from that), we would benefit from clear/browsable documentation of the interface. TypeDoc is probably a good starting point (and it's incorporated in my client implementation working branch). We should also clarify as part of the README that this (at least) is a specific area of documentation we consider a user-facing requirement/deliverable with ongoing maintenance expectations¹.

Blocked by #20
Blocked by #29

E.g if the preview button is alt-clicked

• Like the current Enketo-based preview, it's opened in a full screen view.
  • So it's easy to share with mobile users, etc
• Use the same EnketoID to build a link. Maybe instead of /-/preview/enketoID we use /wf/preview/enketoID and make that public so it can be tried on a range of devices (especially mobile) This feels like it was an accidental feature
• Clicking/tapping the send button shows a validation summary and reminder it's a preview only

• Always show guidance hints
• Consider configuration
  • Where?
  • Hide non-relevant vs. gray out
  • Show some formulas
  • Make more compact in some way

• Project goals and status
• Why XForms and not XLSForms?
• Why not use JavaRosa?
• Why not use Enketo Core?
• Why typescript?
• Why not browser XPath parser and evaluator?
• Why tree-sitter?
• What are related projects? Fore, Orbeon Forms; maybe provide brief contrast
• High-level decisions around engine implementation
• Why a strong separation between engine and frontend?
• Why use a frontend framework?
• Why Vue?
• Is the default one question per screen? Multiple? We don't know yet

See this form for ways that different group and repeat features can combine and affect styling.

We are doing user research to better understand needs around visual grouping so this is only preliminary styling aimed at making the different concepts and cases clear.

• Enketo and Collect have different appearances
  • Can we align with Collect? What would the default question type without appearances look like? A dialog? Inline?
• Can we use capture accuracy? Other thresholds? https://docs.getodk.org/form-question-types/#geopoint-widget
• How configurable should the mapping source be?
• What mapping engine? E.g. Google Maps, Mapbox, Leaflet...

User story

As the manager of an ODK project
I want to use Web Forms to edit existing data
So that the edit experience can happen in the same context as the data capture

Notes

• #37 (OpenRosa spec) is needed for edits
• odk-instance-first-load should NOT fire
• Some question types, notably geo questions, would benefit from a different edit experience
• There's the possibility for unexpected behavior, especially around non-relevant values which won't be in the instance. Non-relevant repeat instances are particularly an issue because positions may have shifted (see #137 (comment) for some related ideas)

User Story

As the developer of an application that uses ODK Web Forms
I want to set the default ODK Web Form language to the language of my host application/context
So that my user can have a consistent language experience

Notes

This is distinct from the form definition default language which it should always override.

As a user I should be able to use select_one and select_many question types with default appearance.

I'd like to explore the possibility of decoupling the @odk/xpath API from the XML/browser DOM API. Specifically for the purposes of this issue, the decoupling would involve accepting a different type for contextNode with the following considerations:

• Provide a significantly more limited interface, restricted to only those semantics of a node which satisfy the implemented XPath semantics. This interface would likely still be "a DOM", in the sense of representing a traversable tree of nodes consistent with the XPath node types, but it wouldn't necessarily be "the DOM".

  A simpler interface would allow us to more easily reason about potentially substituting node implementations. And being able to more easily reason about this, we could better evaluate if and how to reap other potential benefits (discussed below).

  It's an open question whether this simpler interface should attempt to maximize compatibility with "the DOM" (i.e. so that an unmodified XML/browser DOM reference is assignable), but my gut instinct is that it should not be a requirement though it may be nice to have. If we pursue this, and choose not to maximize compatibility, we'd still have the option of providing a compatibility layer e.g. wrapping the underlying XML/browser DOM APIs.

• Minimize incidental complexities of different interfaces for different node types. This is a source of significant incidental complexity in the evaluation implementation as well, and it's highly error prone. A trivial example of this kind of incidental complexity is the lack of consistent, direct access to certain properties, such as ownerDocument. Another good example is the combinatory explosion if slightly different ways to access node data (textContent, innerText, children, childElements, data, value, ...).

• Require (whether enforced or not) certain guarantees which aren't assumed in the XML/browser DOM compatible API we currently provide. Top of mind here:

  • Require that the entire context is read-only for the entire scope of a given evaluation. This would unlock a wide variety of potential performance improvements without baking in a lot of domain-specific assumptions.

  • Require that evaluation is always performed against a context node which is fully intact/connected to its root. This would overlap both with API simplification and performance.

• Provide specific affordances for additional optimizations, e.g. the ability to designate certain aspects of a context tree permanently immutable.

• Make callable aspects of the API explicit, either by choosing to express them as methods rather than as property accessors, or by designating them as getters with a clear indication that they are anticipated to have an associated runtime behavior. In the most minimal version of this, it would help us reason about where certain accesses are more expensive than others.

  In a broader vision, this would allow us to provide a reactive implementation of nodes, e.g. allowing a reactive graph of dependencies to be established automatically as they're accessed while evaluating an expression referencing them. This in turn could potentially allow for other valuable use cases, such as producing an evaluation history log which can be referenced directly back to e.g. instance nodes—both valuable for direct debugging, as well as potentially enriching bug reports from users.

Currently, when using XPath paths with the shortcut syntax for position equality is used with JR's Scenario, position is 0-indexed to match the internal representation: https://github.com/getodk/javarosa/pull/750/files#diff-f9d2e62c99f4070a02e4db45b699a79ce1a7cec1f6d6d3082e6406a5c9bf8650R147

User-facing position is 1-based.

Unfortunately, I don't think the change in JR is entirely trivial but it seems worth it.

Motivation

Despite overwhelming similarities and copious documented references between the XPath and ODK XForms specifications (as well as ODK XForms both in real world use and in existing test suites), there are crucial semantic differences in how certain node-set references are expected to be resolved. These semantic differences depend on structural information about an XForm which is out of band from its ultimate evaluation context.

There may be other cases, but the most prominent case which comes to mind is how dependencies are resolved in expressions from repeats and their descendants. Take this example, an abbridged version of a test fixture ported from JavaRosa:

<h:html>
  <h:head>
    <model>
      <instance>
        <data id="repeat-calcs">
          <repeat>
            <position/>
            <position_2/>
          </repeat>
        </data>
      </instance>
      <bind nodeset="/data/repeat/position" calculate="position(..)"/>
      <bind nodeset="/data/repeat/position_2" calculate="/data/repeat/position * 2"/>
    </model>
  </h:head>
  <h:body>
    <repeat nodeset="/data/repeat" />
  </h:body>
</h:html>

Strictly adhering to XPath semantics, the calculation of position_2 will always produce 2. This is because the reference to the absolute path /data/repeat/position will resolve the first matching node, in document order.

The expectation in ODK XForms is that the same path will resolve to the node nearest its context node. In a second repeat instance, the path reference should resolve to the position node in the same repeat parent. More concretely:

<repeat>
  <position><!-- position(..) = 1 --></position>
  <position_2><!-- /data/repeat[1]/position * 2 = 2 --></position_2>
</repeat>
<repeat>
  <position><!-- position(..) = 2 --></position>
  <position_2><!-- /data/repeat[2]/position * 2 = 4 --></position_2>
</repeat>

The closest semantic equivalent in XPath, per spec, would be as if the expression were relative:

<repeat>
  <position><!-- position(..) = 1 --></position>
  <position_2><!-- ../position * 2 = 2 --></position_2>
</repeat>
<repeat>
  <position><!-- position(..) = 2 --></position>
  <position_2><!-- ../position * 2 = 4 --></position_2>
</repeat>

A tempting solution is to rewrite such expressions in exactly this way, and then evaluate them according to standard XPath semantics. I've explored this, and it is promising. But I have a couple of concerns with this approach:

• It's unclear to me how broadly this kind of a semantic mapping applies. Every exception, every permutation of special consideration for certain form structures and expression specifics, is a potential explosion of complexity and bugs.
• It introduces a huge indirection between a given XForm's actual expressions and the expressions that actually get evaluated. It's already challenging to maintain and debug a custom interpreter inside a host language.

Having mulled this for the last few weeks, I think a more appropriate way to address this semantic discrepancy is to build on the same mechanisms already used to address other XPath/XForms semantic discrepancies: function implementations, namespacing, and shadowing/overrides.

I believe the function implementation interface is, or is very nearly, well suited to handle this particular issue and its many permutations we'll likely encounter as we fill out support for repeats. I also believe this will have tremendous potential benefits for performance optimization (more on that below).

Proposal summary

In brief, this is a proposal to:

1. Extend the capabilities of XPath function implementations to support evaluation of all supported XPath syntax… as if each part of the XPath syntax were a function call.
2. Migrate internal implementation of non-function expression syntax to utilize said function implementation support.
3. Provide interfaces for function implementations to conditionally override, supplement, and defer to, their internal/default counterparts.
4. In support of the above, improve the pertinent APIs themselves for better ergonomics and maintainability of XPath function implementations generally.

I'll address the last of these first, then the others in order.

Improved function implementation APIs

The existing FunctionImplementation API (and its derivatives) has proven successful enough to support a full reimplementation of the current Enketo evaluator's scope. But it has a few really glaring weaknesses:

• Type safety around arity: every FunctionImplementation's host implementation (as in the function which is actually called in the JavaScript runtime) is typed as variadic, with zero required arguments. The vast majority use the unsafe ! (non-null assertion) operator extensively to work around this limitation.

• Static and runtime handling of parameter value types: currently, parameter type information is unused (even if it is present). Parameters which should always be (or be cast to) a particular type must be manually cast in each host function. More problematic, functions operating on nodes rely on manual runtime type validation in each host function as well. The latter is a particular footgun for this proposal.

• Parameter ergonomics: parameters are always evaluated lazily—which is necessary in a few cases (e.g. if(false(), explode(), kittens()) must return kittens without exploding them or anything else), and potentially good for performance in a few others (e.g. short-circuiting on NaN values)—but mostly it's just a lot of unnecessary ceremony.

• Return types: the base FunctionImplementation class has no support for specifying a return type at all, and certainly no means of enforcing it. Its typed subclasses are somewhat an improvement. This is currently less pressing an issue, but it's one I'd like to bring along for the ride. And it's a potential opportunity for other improvements, like automatic documentation generation, and possibly adding certain runtime checks e.g. to ensure a given function implementation valid to substitute/supplement another.

• Ambiguity around use of context: every host function currently takes a context parameter, and must use it to resolve each other parameter due to the aforementioned pervasive lazy evaluation. This makes it unclear when and how functions actually operate explicitly on their calling context.

• Alignment between signature definition and actual host function signature: there really isn't any, largely due to the other issues discussed above.

I believe all of these can be addressed with judicious use of a few complex TypeScript types, and careful consideration in particular around how/whether to keep evaluation lazy. Examples in the sections below, while pseudocode, will be at least partly based on prototyping around these API improvements.

Migrate internal non-function syntax to function implementations

Each non-function aspect of an XPath expression could be expressed as a function call. For instance, this expression:

//foo[position() = 2]

… could be expressed something like this psuedo-code:

(xpath/absolute-path
  (xpath/step
    (xpath/axis :descendant-or-self)
    (xpath/node-type :node))
  (xpath/step
    (xpath/axis :child)
    (xpath/name-test "foo")
    (xpath/predicate
      (xpath/=
        (fn/position)
        (xpath/number "2")))))

I chose to make this example lisp-y, because it can concisely express concepts like namespacing and special keywords, and because the intent is to show the idea of syntax-as-functions in the abstraact without overly tying it to how that would be implemented here.

I also took some liberties to simplify some aspects of the example. Notably absent is the concept of "context", which should be a parameter to several of the function calls. Some of the functions as expressed here would also have a more complex signature than the present FunctionImplementation API can express (or would with the improvements discussed above, though I'm open to accommodating that if we prefer).

But for the purposes of this proposal, implementations for the functions involved might look something like (again, this is pseudo-code):

const absolutePath = fn(
  SYNTAX_NS,
  'absolute_path',

  // A syntax function's parameters could be:
  // - the evaluation context
  // - ...its child syntax nodes
  parameters(context, variadic(syntaxNode('step'))),

  // Explicit return annotation.
  returns(nodeSet),

  // Signature inferred from above `parameters` (and checked against `returns`)
  (context, ...steps) => {
    let currentContext = context;

    for (const step of steps) {
      // Each `step` here is a syntax node.
      //
      // Dispatch of syntax functions could be handled internally to simplify
      // host function usage (more on dispatch below).
      //
      // Syntax producing a node-set is frequently part of a sub-expression, and
      // frequently produces a new context for evaluation of the next sub-
      // expression, so this can also be handled internally.
      currentContext = currentContext.callSyntaxFn(step);
    }

    // Functions specifying a return type could be free to return either the
    // specified value type directly, or the appropriate "result" box type.
    // In this case, a node-set's result type is also a context, so the context
    // is an appropriate return value here, but it would also be valid to return
    // `currentContext.nodes` or just an `Iterable<Node>` in general.
    return currentContext;
  }
);

// The idea here is that many parts of the syntax are inherently "polymorphic",
// but individual syntax function implementations could opt to handle only a
// specific subset of the potential syntax (like a method overload).
const descendantOrSelfNodeStep = fn(
  SYNTAX_NS,
  'step',
  parameters(
    context,
    // This function implementation *only* handles this axis...
    syntaxNode('axis', 'descendant-or-self'),
    // ... and this node type test
    syntaxNode('node_type', 'node'),
    variadic(syntaxNode('predicate'))
  ),
  returns(nodeSet),

  // As described above, a node-set returning function may return `Iterable<Node>`
  // which means it can also be implemented as a generator.
  function* descendantOrSelfNodes(context, axis, nodeType, ...predicates) {
    // Context is already iterable; this is (for instance) how position is handled
    for (let currentContext of context) {
      // Which would allow it to be handled as a function as well.
      for (const predicate of predicates) {
        currentContext = currentContext.callSyntaxFn(predicate);
      }

      // This being a descendant-or-self axis, we first yield "self"...
      for (const node of currentContext.nodes) {
        yield node;

        // ... and then recurse for descendants.
        for (const child of node.childNodes) {
          const childContext = currentContext.fromNode(child);

          // Option 1: The calling context could retain a reference to the
          // currently evaluated syntax node, to facilitate such recursion.
          yield* childContext.callSyntaxFn(context.syntaxNode);

          // Option 2: this function is callable directly
          yield* descendantOrSelfNodes(childContext, axis, nodeType, ...predicates);
        }
      }
    }
  }
);

const namedChildStep = fn(
  SYNTAX_NS,
  'step',
  parameters(
    context,
    syntaxNode('axis', 'child'),
    syntaxNode('name_test', 'unqualified'),
    variadic(syntaxNode('predicate'))
  ),
  returns(nodeSet),
  function* (context, axis, nameTest, ...predicates) {
    // This is *sometimes* an oversimplification due to complexities in
    // namespaces, but not always. More on that below.
    const selector = `:scope > ${nameTest.text}`;

    for (const node of context.contextNodes) {
      const namedChildren = (node as MaybeElementNode).querySelectorAll?.(selector) ?? [];

      // We could also simplify predicate filtering with a dedicated API
      yield* context.applyPredicates(namedChildren, predicates);

      // Alternately, `step` implementations could defer handling predicates
      // entirely by simply omitting them from their `parameters` signature.
    }
  }
);

const position = fn(
  FN_NS,
  'position',
  parameters(context),
  returns(number),
  (context) => {
    // The current implementation defers to an internal `context` API, which
    // we could continue to do. An alternative option will help illustrate the
    // next section.
    return context.contextPosition;
  }
);

const eqExprNumberNumber = fn(
  SYNTAX_NS,
  'eq_expr',
  // Note: this is *only* an implementation of `eq_expr` where both operands
  // are evaluated as numbers. In this proposal as it stands, we'd provide
  // separate implementations for each permutation of operand types.
  parameters(number, number),
  returns(boolean),
  (lhs, rhs) => lhs === rhs
);

Provide interfaces for function implementations to conditionally override, supplement, and defer to, their internal/default counterparts.

This section is going to be somewhat handwavy. I've thought about the API aspects of this least of all. The general idea is that a consumer of the primary Evaluator can provide specialized syntax/function implementations suitable for specific conditions, and fall back to standard behavior when those specific conditions are not met.

I think the best way to illustrate this is to come right back to the motivation for this proposal. Assuming we have:

• A context node position_2
• In an XForms primary instance
• Whose path we already know as (or can look up/map back to) /data/repeat[2]/position_2
• An expression to evaluate in the form /data/repeat/position * 2

We can provide a function overriding absolute_path something like:

declare const state: EntryState;

const xformsAwareAbsolutePath = fn(
  SYNTAX_NS,
  'absolute_path',
  // Let's say the `string` parameter here is the full text of the absolute path.
  parameters(context, string),
  returns(nodeSet),
  function* (context, path) {
    let found: Node | null = null;

    for (const node of context.contextNodes) {
      found = state.getNodeState(node)?.getReferencedNode(path);

      if (found != null) {
        break;
      }
    }

    // If lookup failed, fall back to other (e.g. internal) implementations
    if (found == null) {
      return fallback();
    }

    // Otherwise
    yield found;
  }
);

Fallback?

Calling fallback() here is doing a lot of work to carry this example (and again, it's pseudo-code to illustrate the idea, not an actual API proposal), without much explanation of how it would work. Basically it's kinda sorta like a continuation, where it instructs the evaluator to try the next candidate syntax function capable of handling the same syntax node for the given syntax node in the given context.

To make explicit something that's only been implied up to this point: currently (as of #13), functions are looked up by name (local, optionally namespaced) in each of the FunctionLibrarys available to the current Evaluator context. Once a matching function is found, it's called, and that's that. This proposal would depend on potentially finding multiple implementations:

• for a given function name
• of a compatible signature

... calling each in turn until it does not "fall back"—essentially until one of the candidate functions returns a value satisfying its returns annotation.

(All matching implementations "falling back" would be considered an error.)

Other cases will be more complicated, of course. For XForms semantics, we'll need to handle at least path references with predicates between steps. I believe elements of the above provide some direction for how this might be handled within the scope of this proposal. I'm hesitant to go too much deeper into examples for this use case without prototyping further, with more team visibility and discussion.

But to fill out some other ideas alluded above…

Performance opportunity: named child steps

Name tests, as handled internally in @odk/xpath, are extraordinarily general in order to satisfy some edge cases around the XPath and XML namespace specs that we don't really need to accommodate in most (if not all) XForms usage. An earlier example included an "oversimplification" of such a name test, but with this proposal that simplification can be employed only in conditions we know it safe.

Suppose we take an upfront step to ensure the XForms namespace is used as the default namespace, and can ignore any more complicated namespace testing for unqualified names within a form entry's primary instance:

declare const state: EntryState;

const xformsAwareNamedChildStep = fn(
  SYNTAX_NS,
  'step',
  parameters(
    context,
    syntaxNode('axis', 'child'),
    syntaxNode('name_test', 'unqualified'),
    variadic(syntaxNode('predicate'))
  ),
  returns(nodeSet),
  function* (context, _, name, ...predicates) {
    let found: Node | null = null;

    for (const node of context.contextNodes) {
      found = state.getNodeState(node)?.getChildNamed(name.text);

      if (found != null) {
        break;
      }
    }

    // If lookup failed, fall back to other (e.g. internal) implementations
    if (found == null) {
      return fallback();
    }

    // Otherwise
    yield* context.applyPredicates([found], predicates);
  }
)

Performance opportunity: arbitrary static subtrees

Itemsets populated by instance()/... expressions are by far the biggest performance issue in the web-forms work so far. This is largely because the same expressions and sub-expressions are evaluated repeatedly, redundantly, without any notion of their static-ness. An example also feels redundant at this point, it would be a slight variation on the previous two. In any case, I believe that a similar lookup/fallback approach for secondary instance data would be a significant improvement to performance in this case… particularly combined with application of the approach for primary instance references in those same itemset predicates.

Per spec, randomize accepts an optional seed. The current implementation currently checks that the seed is an integer, and throws (a string, a tangentially related my-bad) if it's not. This was derived from the implementation in OpenRosa XPath Evaluator, so I'd expect this bug affects Enketo as well. Which is somewhat surprising, because I discovered the bug by using XLSForms' prominent example for stable-randomization.

Steps to reproduce

1. Use the aforementioned XLSForms example in a form

Expected behavior

1. A form using a decimal/float seed to randomize will stably randomize options, without error
2. Even if there is an error due to an actually invalid seed value, it will be surfaced to the user in some meaningful way
3. Possibly some sort of recovery behavior for certain invalid seed values (like NaN or ""?), but this would potentially involve spec clarification

Observed behavior

(From memory, happy to repro again if needed) Form load fails without explanation.

Likely has ramifications around default font sizes and general theming

Blocked by #36

User Story

As an application developer
I want edited instances to follow the ODK XForms spec for deprecatedID
So that I can treat edits differently from new submissions