conversence / hyperknowledge Goto Github PK

Why do I want event source inheritance?

Use case 1: I want to test a hypothesis, so I make a "branch" of my Source as a different Source, and run what-if scenarios in that Forked Source, while still accessing all the statements in the parent Source.

Use case 2: I want to create a conversation space that can refer to statements both agents have already made in a public thread.

Use case 3: I want to create a community space with aggregate beliefs (for some value of aggregation.)

Current naive implementation: The handler gets the event and the current (source-local) projection state of all projections that the event refers to. Problem: if the event is inherited, I can see many cases where I want the folding function to know more than this.

Use case 4: If a statement has been made locally that shadows a statement in an inherited thread, and the statement is revised in the inherited thread, I may want to (4a) keep shadowing the value (priority to local values) or (4b) to mark the projection as conflicting (since the shadowing may not be valid for the new statement.)

What is the minimum information that the handler needs access to for this use cases? Ideally it should not need knowledge of past events, so the information must be encoded in the projection.

First stab: The projection could contain the timestamp of the last local event, and a dict giving, for each source directly inherited, the last known event and the last known event that was resolved. If the last known (unresolved) event is later than the last local event, there is a known conflict, and a need for resolution. (It could be automatic in simple cases.)

I could also see a case for the handler of inherited events also having access to the projection state in the inherited thread and how it was affected by that same event. (So at least the state after, and maybe also before.) Note that inherited projection state may well be using a different schema (e.g. aggregation awareness may exist only in the containing thread.)

Currently in a single table with an "obsolete" column for snapshots.
There should be multiple history storage cases, defined as part of the projection schema:

1. No history (current state only)
2. Separate snapshots: Have both a current state and a snapshot history table (in another database schema?)
3. Integrated snapshot (as now: a single table with an obsolete column)
4. Full history: integrated snapshot, record every change, allows simplest time travel possible.

In cases 1 and 2, this allows foreign keys.
Case 4 allows can be used for easy time travel on (eg) name bindings.

Each event's payload, as serialized in the event's time's context, should be serialized as a struct and given a SHA-based identity. This allows recursive statements.
There is a lot of imprecision on recursive statements, as the event is serialized with prefix whose URL expansion may vary in time. I think this imprecision is acceptable.

Each source should have a projection processor that can advance independently from other sources.
Break up the projection processor in multiple processes.

Allow to re-process a source when the handlers have changed. Moving towards time travel.
Depends on #4.

Remember each HK schema by its SHA. (Eventually connect with IPFS, but out of this scope.)
The server has a global state, where each prefix is linked both with a URL and (optionally, independently) a HK schema at any given time. Allow time travel. This defines a "context", which can be expressed as a simplified json-ld context. Extend this json-ld context to have the signature of the prefix' schema. (Use "prefix@sha" namespace for this...)
The events will use this time travel to refer to the context at the time of their creation. Optionally, incoming events could refer to the context's timestamp to ensure that they are using up-to-date definitions.
This means that there can be only one HK context definition per prefix at any time; eventually allow context merging to maintain this invariant.
(Note: Projections always use the current schema. Not sure I want to allow projection time travel, except maybe in snapshots.)
All this should eventually allow handlers to work on earlier event definitions safely.

Note: This is completely in flux. I am now thinking of schema definition element events on the event stream, with the issue that the event inheritance semantics is really undefined. (I should disable event inheritance for some event types?) The schemas elements become immutable, and the events could keep pointing to those by immutable identity. The range should still be a name (Term or UUID) that is bound to a schema element at time $t$ for a given source.