Developer experience and rough workflow about action-input HOT 21 CLOSED

@johnshaughnessy and I wrote up a quick mock of what we were envisioning from the discussions we had previously had with @fernandojsg. It's somewhat different from what you have outlined, and probably a bit more low level. Some of the features outlined above (like an evented api, automatic toggling of action maps based on orientation/mode changes, etc) would still make sense to build on top of the API we are proposing. One core difference is that everything is represented as reading from and writing to semantic paths.

https://gist.github.com/netpro2k/7f91598d32476f97d24285c6d77f17d6

We still need to work through a quick mock of what the internals of Action.tick() would look like, namely how things are read from input sources and how filters are resolved. High level every frame we fill a big map of semantic paths starting with reading from each input source, piping this through each filter which reads from and writes to this map, and then ultimately having the application read from this map through the get/fill accessors (which may also care about the previous frame's map for rising/falling edge detection)

We also have not thought through at all how information like gltf models for the different input devices may be exposed, or how information like the localization provided in the action definition might come into play.

Also while writing this up we jotted down some questions we came up with but didn't fully resolve yet.

• Should the polling API include both methods for getting the current action state as well as edge transitions (what we have outlined above), or should this purely be handled with filters?
• What do we name the state transition functions? (we called it getBoolOn above, but it could be most verbosly described as getBoolNewlyOnThisFrame)
• Do filter instances need names or should they just be refereed to by instance id? (In practice these bindings are not going to be generated by hand, but it may be useful to be able to give them meaningful names)
• Should filters have shorthand for single source/destination?
• Should filters be able to trigger vibrations (ex, virtual_dpad may want to pulse on virtual button presses)?
• What does the vibration/haptics/output API look like?
• Do we include filters with the lib? Which ones? How are others loaded?
• How are binding definitions loaded and how does the lib help activate the correct binding definition for the users input devices?

from action-input.

I think that our concepts might not be too far from each other, but there may be a difference in goals and scope. I've been thinking that the parts in your gist (action definitions, bindings, filters) would be defined in JSON and then used to initialize ActionMappers.

One thing where I need to dig in and get clarity is on how (if?) we can avoid requiring app writers to know the difference between specific hardware like Vive or Rift. I wonder if we can have semantic paths for common types of input, like maybe the library writers (that's us) link device specific paths like /devices/vive_controller/left/thumbpad/touch to less specific but still useful paths like '/input/left-hand/pointing`.

My goal is that in the common case app writers can use input paths that still work as new hardware shows up. So, in a couple of years when Foo Co. ships a controller, the lib can be updated to detect that hardware and link /devices/foo_controller/left/button_0 (or whatever) to '/input/left-hand/pointing`.

I also haven't fully thought through action map switching. I feel like the lib should make it possible to easily switch between action maps, but the logic for switch has to live (as you wrote) at a layer above this library. Like, app frameworks like React or Potassium could have some default behavior for mode switching between flat, portal, and immersive, but scenario specific switches like between play mode and menu mode are app specific.

from action-input.

I also imagine that action/binding definitions would exist in json files, but the fetching and loading of these is a separate API concern from actually registering/activating them, so I think the low level methods we showed here are required either way (and are probably useful to expose if app developers want to handle this loading themselves, for example fetching them from their own database).

The ergonomics and common patterns on each device are such that if I am writing a sufficiently complex app I probably also want to provide configs for all the major devices. That said I do think there is value in providing an "idealized" device that people can write a default config against as a fallback for devices they aren't explicitly targeting or devices that come out in the future. I am a bit afraid of this being a rabbit hole and never getting it to be quite right.

The most important thing for "real" future proofing though is providing mechanisms for users to easily set up their own mappings and share those mappings with other users of the application (this becomes particularly complicated if we want to support custom filter code...)

from action-input.

As promised, here is a very rough idea of how the internal architecture for the above mock usage might look:

https://gist.github.com/netpro2k/16e2f79c8774d1f1959216ca9018d487

This is very pseudocode-ey and I take a bunch of liberties all over so happy to jump on a Vidyo call to walk through it (if we do, we should record it). I also quickly stubbed out how an evented model would be added on top of it.

High level, every Action.tick() we fill a new (probably pooled somehow) InputFrame by first having all the active InputSources write their values to the InputFrame, then that InputFrame gets passed to each active Filter, which writes out it's values. At this point the InputFrame has been filled with all the new values needed for the given frame. We then run through all the bindings copying values from their source to their destination in the InputFrame so that we can just directly query them. A reference to the previous InputFrame is always kept around so that methods like Action.getBoolOn() can handle rising/falling edges. To fire events all you need to do is look at the current and previous InputFrames and fire events on edges/changes.

from action-input.

Ok, I've looked through your code examples and have some low level questions (e.g. why use a singleton Action) which I think we can ignore for now. In general, I think we're mostly thinking the same thing for how information flows and gets mapped to actions.

But, I want to work on how we could have app writers create binding maps that work on semantic paths that are abstracted away from specific hardware so that they have a chance to work as new hardware is shipped.

How about this for an information flow?

gamepads[0].buttons[0].pressed

is linked by a lib-provided GamepadInputSource to

/device/vive/controller/left/thumbpad/pressed

is linked by a lib-provided abstraction map to hw-agnostic input path

/input/hand/left/thumb/pressed

is bound by an app-provided binding map to

/filter/long-on, 2 seconds

which emits

/action/gameplay/in/double-jump

which is served via polling API and emitted via event by the ActionManager

from action-input.

why use a singleton Action

I am fully glossing over details about instantiation, Action might be a singleton, or an instance created, or take a reference to a session or context, etc. Just focused on high level concepts and dataflow in these examples.

/device/vive/controller/left/thumbpad/pressed is linked by a lib-provided abstraction map to hw-agnostic input path /input/hand/left/thumb/pressed

Yep exactly! I think having some bindings built into the library for things like this makes a ton of sense. There might also be paths like /device/vive/any/... for getting inputs from any Vive want, or /devices/vive/primary/... for getting the users primary hand (not something we really have the ability to know now, but I could imagine this coming form the runtime at some point, or perhaps saved user preferences.

That said, I would still push (I guess in our documentation) that these abstracted mappings should be just the minimum thing a developer should provide default bindings for, and that ideally they would also make device specific bindings for the hardware they care to support.

Also you could imagine other InputSources like LeapMotionInputSource, KeyboardInputSource, MouseInputSource, HMDInputSource, etc, all writing to some /device/... path,

from action-input.

I'm still thinking about the mapping of hw to hw-agnostic input paths.

Controls that are used by only one finger get mapped that way, indexed to handle more than one control per finger.

handy vive input map

Pose:
/device/vive/controller/left/pose -> /input/hand/left/pose

Thumb controls:
/device/vive/controller/left/button/0/touched -> /input/hand/left/thumb/0/touched
/device/vive/controller/left/button/0/pressed -> /input/hand/left/thumb/0/pressed
/device/vive/controller/left/axis/0 -> /input/hand/left/thumb/0/x-axis
/device/vive/controller/left/axis/1 -> /input/hand/left/thumb/0/y-axis

Index finger controls:
/device/vive/controller/left/button/3/pressed -> /input/hand/left/finger-1/0/pressed
/device/vive/controller/left/button/3/value -> /input/hand/left/finger-1/0/value

Middle finger controls:
/device/vive/controller/left/button/2/pressed -> /input/hand/left/finger-2/0/pressed
/device/vive/controller/left/button/2/value -> /input/hand/left/finger-2/0/value

handy xbox input map

Thumb controls:
/device/xbox/controller/button/11/pressed -> /input/hand/right/thumb/0/pressed
/device/vive/controller/axis/2 -> /input/hand/left/thumb/0/x-axis
/device/vive/controller/axis/3 -> /input/hand/left/thumb/0/y-axis
/device/xbox/controller/button/a -> /input/hand/right/thumb/1
/device/xbox/controller/button/b -> /input/hand/right/thumb/2
/device/xbox/controller/button/y -> /input/hand/right/thumb/3
/device/xbox/controller/button/x -> /input/hand/right/thumb/4
/device/xbox/controller/button/menu -> /input/hand/right/thumb/5

from action-input.

I like the @netpro2k and @johnshaughnessy proposal, it's basically the same way I was imagining the API.
@TrevorFSmith I understand the feeling of having an abstraction from the specific HW model, but I beliefve both use cases will fit here, for "simple" experiences where you just care about a point or select actions, a laser ray and so on and showing a simple model for all the devices it will be enough to have a hight level of abstraction.
But for more advanced experiences where you really need to show the model, tooltips, use all the buttons and so on it's valuable to be able to identify what controller you have and map accordingly, similar to how steam controller process works. In that case I would see more useful using xbox/controller/button/a than the more abstract input/hand/right/thumb/2

from action-input.

@fernandojsg Yes, I am thinking that the hardware specific paths like 'xbox/controller/button/a' are still available, even though they are also linked to 'input/hand/right/thumb/2' so that the app developer can decide whether or not to use the hardware-agnostic paths. But, I would like to work on how to make it so that using the hardware-agnostic paths is what most app devs choose so that their apps continue to work as hardware changes.

from action-input.

Maybe we'll find that the hardware-agnostic paths just aren't good enough for really nice applications, but I hope that we can find ways to prevent what's happening now, where experiences only work on the hardware that the developer owns and often stops working as new hardware comes out.

from action-input.

But if you look at steam, they're hardware specific mappings but they're forward compatible because of the input to action mapping itself that lets the community or the developer define a new mapping for new hardware so I feel we could get the same.

from action-input.

Yeah the thing that prevents experiences from breaking later is that users can write and share mappings for new devices. I am realizing this also probably means in many cases also creating new InputSources, though maybe this is solved by just heavily encouraging pulling the library from our CDN, or also providing mechanisms for users to load external InputSources... hmm

from action-input.

How do you envision users making and sharing bindings for web apps? This seems really unlikely to me. Installable games seem like a very different use case.

from action-input.

I do believe that we can update the library to map new hardware to hardware agnostic paths, and if the apps are using our CDN then their app magically keeps working with new hardware. This seems much more likely than a theoretical user binding sharing network.

from action-input.

Just spitballing, but imagine maybe a (on by default but can be disabled) config button or keybind that would bring up the configuration UI we are proposing. In addition to creating a binding on the fly, or importing a binding file from disk/url, you might also be able to search for other bindings users have shared for the current site (obvious privacy concerns we need to think about here). This would require us (or some other party) to be hosting these configurations, but that doesn't seem too insane solong as its optional.

from action-input.

Perhaps a system like that could work for big, immersive experiences like a FPS where there are a lot of technical people who are willing to put in time doing something like making a binding, but it seems like a bad fit for the great majority of applications that will be mainly flat and just mix in portal and immersive modes for specific features of the site.
For example, nytimes.com will be mostly flat, but they might build a special topic area like for the 2018 Winter olympics with some VR features. Most of the people who show up won't be technical and they won't go looking for bindings that the nytimes didn't create. Ten years from now, when someone is visiting the archives and wants to use the immersive feature, it should still work.
Same for most entertainment sites like NBC.com, netflix.com, etc.
And then there's the long tail of awesome sites created by artists, writers, whatever. Maybe they make a little experience that lets their users go inside something they created. There's not going to be a big audience of techies who will create and share custom bindings, but we still want their apps to keep working for more than the lifetime of the hardware that the creator used to make it.
So, I think it's crucial for this library that we do some thinking and working toward the goal of making web apps not bound to a single type of hardware and to give the library maintainers a chance to map in new hardware and have it work somewhat well in old apps.

from action-input.

That said, I am also up for thinking about what it would take to make a system for easily and safely sharing configs. LIke, are the configs per-domain? Per-page? Per unique URL, including parameters? Linked by an id that the web app developer has puts in a meta tag? Can we safely share filters?

There are a lot of questions!

from action-input.

Most of the people who show up won't be technical and they won't go looking for bindings that the nytimes didn't create

The idea is that there will be a small number of people who will create, share, and upvote bindings so that there is a reasonable default available for everyone else. If even one binding exists for a user's device for an application, that can be used as the default.

are the configs per-domain? Per-page? Per unique URL, including parameters? Linked by an id that the web app developer has puts in a meta tag?

Good question... If we are hosting and sharing binding defs, perhaps the ID-in-the-meta-tag approach is best.

from action-input.

I'm looking around the proposal to try to match the lower level implementation and I've some questions for @netpro2k and @johnshaughnessy:

In my prototype I was handling the buttons state changes at the lower level itself, and generating events when button down, up or change occurs. The main idea behind it is that, as we discussed initially, we could use that lower level API by itself without any of the extra action/mapping functionality on top of that, basically as a nicer abstraction over the current input apis (as shown here https://twitter.com/fernandojsg/status/963856765812117504)
But in your code the change status and even the query of the state of the input is done directly on the input source which is already in a middle/high level of abstraction and there we already will need semantic paths and so on. Is it that on purpose? If so, should we focus this implementation on the high level goal and forget about reusing the lower level as an stand alone library? Or should we try to match these both keeping the standalone low level adapted to your proposal?
If the later, one option could be to fill the inputFrame directly with the input data from the lower level, but we'll have two ways where we'll generate the "pressdown" event, on your code and on the lower level bindings. Should we use the lower level binding for that purpose instead?

I'll try to come out with an example of both proposals in the meantime but I would like to hear your thoughts on that

from action-input.

TL;DR: should we focus on how we would like to use the higher level API and implement the lower level just to match that use, or should we think also on using the lower level as standalone and use it on the hightlevel as a third party component?

from action-input.

My general plan was to have two separate wadges of functionality: one that maps low level hardware events into semantic events (e.g. button click to "jump" or recognized hand gesture to "open menu") and one that most apps talk to that provides an event API and a polling API for actions.

I think the next thing I'm going to do is to stub out the APIs of both wadges and submit a PR for discussion.

from action-input.