While the most common read pattern for clients will be to read at the end of the log, from time to time new clients will show up that want to sync over the full history of the stream.
As I explained in #40, the issue in open fragments is the maximum size of a fragment has an exponential impact on the necessary bandwidth and processing in the client. This would argue for creating smaller fragments, however this also has downsides, as smaller fragments would mean more requests to the server.
This is why it would be ideal to rewrite historical fragments (say older than a day, immutable), into larger fragments.
Fetching bigger files (especially if the HTTP headers indicate relations, so this can add concurrency) is much more efficient than many smaller ones (connection setup, higher compression rate,...), but has a drawback in the current form: no tree:Node streaming parser exists, essentially requiring the entire graph (of one page) to be parsed in memory.
When compacting historical fragments into these larger graphs, this could be an issue.
This is why I would suggest a default way of structuring the data in a page, such that a stream aware parser can stream parse the document, and emit members as they are processed.
This would significantly reduce the memory requirements in the case of large fragments.

The layout of a page (say, using turtle serialization as it offers best compression) could look something like this:

• First the stream membership statements, required to find the tree members
• Then the members, one by one, ordered first by object id, then timestamp path
  ( this allows for member skipping if the client is interested in latest state only, reducing the number of upserts on the database the stream is projected in.)
• Last the relation pointing to the next page

As all member triples are 'grouped', the parser can read one member at the time.

As the document would be a normal RDF file, and the only semantics added are there to support the streaming behavior, this should be completely backwards compatible for clients that don't support streaming tree parsing.
The capability could be indicated by a statement on the view.

When outputting hypermedia URIs that will be dereferenced by the client, as an LDES server, it would be best (rfc 2119 SHOULD?) to make those relative URIs.

This will allow for transparent proxy mappings without the need for proxy to rewrite any content.
As relative URIs are interpreted relative to the URL of the document they occur on, this will effectively resolve any potential proxy issues.

ldes:timestampPath is now optional. If however order is important, this MUST be set. Clarify this in the spec.

Hello @pietercolpaert,

https://tree.linkeddatafragments.org/linked-data-event-streams/
cannot be reached anymore


Thanks

There is currently a proposal to support named graph serializations like TriG and JSON-LD.

The proposal would be to allow something like this:

ex:C1 a ldes:EventStream ;
      tree:member ex:Graph1, ex:Graph2 .

ex:Graph1 prov:generatedAtTime "2022-12-25T12:00" ;
                    foaf:primaryTopic ex:Streetname1 .

ex:Graph1 {
   ex:Streetname1 rdfs:label "Station Street" .
}

ex:Graph2 prov:generatedAtTime "2023-12-25T12:00" ;
                    foaf:primaryTopic ex:Streetname1 .

ex:Graph2 {
   ex:Streetname1 rdfs:label "Station Square" .
}

Current proposal would be that the EventSource view must have relations with tree:path equal to the ldes:timestampPath

It would be easier for readers to have access to a set of examples of how to use LDES. Also we might need to add additional comments to the examples in the spec for clarity.

Although the ldes:versionKey predicate allows defining which properties to use as a version key, there's an implicit recommendation to use dct:isVersionOf because it's the only one mentioned in the examples.

The Prov-O ontology contains another useful predicate that could be used to link the version object to the main entity: specializationOf. Given how Prov-O is a W3C recommendation, and it might get used to describe projections as well, it may be interesting to consider this one instead of dct:isVersionOf.

https://shapetrees.github.io/specification/spec possible moved to https://shapetrees.org/TR/specification/

One of the notes currently ends "You can indicate that the new ldes:EventStream is derived from".

First of all, I'm really new to the spec, so I'm sure I'll make some wrong assumptions here...
But I have leaned that the easiest way to learn something on the internet is to make a statement, and someone will try to prove you wrong.

So here it goes:
I under the impression that the design of the event stream was focused on being bulk-loadable in a triplestore as-is, in such a way that the data can be queried without further processing; the state is present in the stream itself.
Although I think this is a really interesting feature, its biggest disadvantage is that the semantics of the event metadata and the entity data get mixed.
I'll try to argue here for an alternative approach, where the event stream is seen as more as transaction log.

So this is how I understood the spec; what I thought to be the requirements that where behind the definition of spec:

• Events are appended to the end of the stream
• Events are not business events, but the representation of the full state of an entity.
  (I assume this because of the design where the stream is queryable in and by itselve, more a growing graph.)
• Entities can be updated
• Old version, that no longer contribute to the state of the system can be cleaned up, if the stream decides to do so.

So now for what I see to be the disadvantages to the current LDES approach:

• The event data and metadata are mixed. Because of this, the stream semantics might need to be updated when the datamodel of the producer evolves.
  There is a risk of having semantic collision between carrier and message, if for instance a DCAT Datasat would be carried over a stream, the use of isVersionOf would conflict. This is worked around with by the ldes:timestampPath and ldes:versionOfPath, but gives room to errors.
• No order to the events can lead to inconsistency in the materialised state derived from the stream. Using time for ordering in a distributed system will lead to errors at some point.
• Deletion of objects is implicit, through removing members from the stream.
• Lack of transaction semantics. Since the state of objects is carried over, it might be that multiple objects need to be changed at once in order to remain adherent to the defined shapes.
  (This might not be relevant if the stream consist of sensor data, where each entity is an instance of the same class, but if the source is a business application the need will arise eventually).
• Due to the approach to versioning, the graph of all events can't be used directly, but a subgraph needs to be derived by the indirection through the tree:member property of the event stream instance.
  This seems to imply that for each update two pages: the first containing the stream definition and the last one with the latest event.
  In case the stream grows to terrabytes of data, the reprocessing of the (huge) EventStream instance could become an issue.
• Another issue I see with the versioning is the mix between application versioning (an entity received updates through time, and I want to see previous versions of it), and infrastructure concerns (I want to be able to do point-in-time restores if the application update messes up the database).

So, after this, I would like to propose another event representation that tries to address these issues(Example 1 of spec):

ex:C1 a ldes:EventStream ;
    tree:shape ex:shape1.shacl .

[ 
    a ldes:Event ; # Could be implied if pages are dedicated?
    ldes:sequence "1"^xs:integer ;
    ldes:commitTime "2021-01-01T00:00:00Z"^^xsd:dateTime ; # Used as metadata in case a point-in-time recovery of the stream needs to be performed, and the recovery point needs to be determined).

    ldes:key <ex:Observation1> ; # The subject of the object represented.
    ldes:value [ # value is optional, if not present this is a tombstone event (delete).
        a sosa:Observation ;
        sosa:resultTime "2021-01-01T00:00:00Z"^^xsd:dateTime ;
        sosa:hasSimpleResult "..." .
    ]
]

This also allows for representing the stream in a key-value ledger model as used by Apache Kafka.
The semantics for materialising this stream in a triplestore would be simple.
For each event in stream:

• Delete all triples in DB with the event key as subject (ignoring nested blank nodes for a moment).
• Take the ldes:value of the event, and set the subject of the triples to the event:key. Insert into DB

Under this model, versioning of objects can be done at the application level using the isVersionOf approach if the versions concerns the application (e.g. multiple versions are kept so users can roll back to a previous version of an entity), as well by the log retention policy algorith that is chosen on the stream level, which only concern is to be able to move back in time (for instance for point in time recovery), and doesn't need to concern about application versioning.

A retention policy could keep the full state (decided by the producer what should make up the state), as well as the transaction log for a certain time.
(following the idea of how topic log compaction is done in Apache Kafka)

Additional properties could be used to signal transaction boundaries (if retention policies apply, the boundaries can only be used when reading the head of the log, where messages are sequential).
e.g. ldes:trx_id "1"
If the stream delivery method can't guarantee that events are committed to the stream representation in an atomic way, a transaction event count could be added to the first event part of a transaction, to indicate the number of events that should be read as part of the transaction.

I was hoping to read what the prefixes used in the examples refer to: sosa, adms, prov, ...

Search forms are a convenience tool, but certainly not a must or even a should. Let’s move that example to the back.

When a client requires hard guarantees on consistency, the logic described in the RDF Dataset Canonicalization could be used to provided hashes of the state that should be reached after applying a fragment, or even better, a transaction.
This becomes relevant in cases where LDES is used as a replication protocol for named graphs (the client should have an exact copy of the named graph the publisher intended). For instance, consistency could be lost if a client is offline longer than allowed by the retention period, which could result in missed delete operations (tombstone events). If a checksum mismatch is detected, the client must restart replication from the start of the log to arrive at consistent state.

Reference: https://www.w3.org/TR/rdf-canon/

Discovering an LDES from 1 specific subject

Proposed solution: a new predicate ldes:eventStream to be introduced, linking a subject to one or more EventStreams that contains, or will contain, updates about this subject in the future.

?subject <https://w3id.org/ldes#eventStream> <target> .

This would imply that <target> a ldes:EventStream . and dereferencing the target results in an RDF resource describing an LDES. This LDES may contain members that link to the ?subject.

Discovering from an existing tree:Collection

When the same data collection has updates, a collection may also have precisely one event stream for updates about its members. We propose a ldes:versionStream predicate for this.

<collection> <https://w3id.org/ldes#versionStream> [
           dcat:accessUrl <target> ;
           ldes:versionKey (dcterms:isVersionOf) 
]

The predicate ldes:versionKey on top of the VersionStreamDescription will explain how its members can be mapped to specific versions.

Publishes a dataset often entails adding some additional statements (e.g., dct:isVersionOf), but this data then becomes indistinguishable from the original data elements. This can be an issue for operations such as version materializes, which should yield the current version of each concept, as it would appear without this LDES-specific metadata.

The specification currently describes how to add a version key to a retention policy, but this cannot be used for collections without a retention policy. Furthermore, the version key only specifies which predicate is used to link the version URI to the concept URI, but there's often another predicate that is used to assign a timestamp to this version. This timestamp metadata would also be useful for other issues, such as #16.

I would propose to move the versioning metadata (the version key and timestamp predicate) to the Collection description, and possibly make it mandatory. Perhaps it can become part of the shape description.

The idea of this retention policy is that it combines the benefits of snapshots with retention.
This way, if replication of state is the concern of the client, a log that is as small as possible can be offered.
This saves resources for both publisher and subscriber.

The idea of the algorithm is explained here: https://docs.confluent.io/kafka/design/log_compaction.html

While the design of some LDES-es can be really simple, just keeping a history log of all pages and members processed to keep the state will not scale to really big LDES-es.

In earlier work, we have introduced the ldes:timestampPath to point at a property that can be used to keep the state in a different way: just resume from that timestamp.

If would be good to have a standard algorithm described in the spec to explain how a client must pause and resume from a certain state.

Coined by @sandervd

Relevance:

• In the Matrix channel, this was raised by @madnificent as something that we need in the spec
• In DCAT-AP Feeds, this is left as an issue to this spec

Two parts we need:

1. A summary on top of a republished view or LDES when it was last synced (cfr. the older issue #5)
2. A detailed log of why certain member were adopted by the consumer, and why certain members have been ignored

1 can be derived from 2 as well

The ldes:retentionPolicy property is defined as:

ldes:retentionPolicy a rdf:Property ;
   rdfs:label "has retention policy"@en;
   rdfs:comment "Links to a retention policy."@en ;
   rdfs:domain tree:View ;
   rdfs:range ldes:RetentionPolicy .

But there is no class tree:View defined in the TREE vocabulary. Given that this is a LDES specific property, shouldn't its domain be ldes:EventStream instead?

the current link is to http://www.openwebfoundation.org/legal/the-owf-1-0-agreements/owfa-1-0 which yields 404

probably needs to be updated to https://www.openwebfoundation.org/the-agreements/the-owf-1-0-agreements-granted-claims/owfa-1-0

I have had this idea for quite some time, to use GIT as a way of synchronizing graphs over the network.
The data structure behind git, namely blob, tree and commit objects, could mix quite elegantly with the way LDES is structured.
(an old writing I did about this: https://github.com/sandervd/linkeddata-git )
As git has internal optimisations, such as packfiles https://git-scm.com/book/en/v2/Git-Internals-Packfiles
that can make transfer of deltas much more efficient than one by one...

Anyone interested in exploring this topic? I haven't looked into the constraints of git (the most important on would be the max size of tree objects), and at the moment I don't have a use case, I just wanted to throw the idea out for now :)

How do we indicate that we have a consistent knowledge graph across LDESes?

For instance, what if we split Linked OpenStreetMap into 3 LDESes: one for nodes, one for ways and one for relations. Then, for each LDES, multiple objects are added into one transaction. Just processing one member would not be very valuable, as you would get an inconsistent replication: the osm:Way would for example not yet have the necessary osm:Nodes to point to.

A solution would be to have a transaction system that indicates whether the transaction is still in progress, and define the bounds (probably based on a timestamp?). The set of members can then only be processed into a derived view or service from the moment the transaction is marked as completed.

Any ideas on this?

A way to circumvent transactions in LDES is to just post members at exactly the same datetime. However, we might want to introduce parts of the knowledge graph changes at different times.

Related work:

• CRDTs - see solid/vocab#69

With time-based retention policies, a tree-path is given and a duration. Based on those 2 triples, it is possible to reason which members should be deleted when a retention policy is present.

For version-based retention policies however, this is not clear. The only information given in this policy is an amount x and a versionKey, which states the collection of which a minimum of x should be preserved.

When that amount is reached, it is impossible to reason which members should be deleted. As it is not favorable to have random deletion, it might be interesting to delete the oldest members of the collection.

I suggest that, just like in time-based retention policies, deterministic deletion behavior can be achieved by adding a tree-path, which leads to a timestamp of a member. Based on that timestamp, the oldest member of the collection can be pruned due to the retention policy.

Recurring question is: how do we indicate something has been removed, created or updated? Activity Streams 2.0 does this however. Examples of AS2 applied to data management can be found in for example:

• https://www.eventnotifications.net/
• IIIF Change Discovery: https://iiif.io/api/discovery/1.0/

Issue to start a discussion on how to deal with time-series with LDES.

Pointers:

• LDES in LDP: deals with a time-fragmented LDES (stored on an LDP) using GTE relations.

JSON-LD can lower the barrier for adoption of Linked Data when given clear rules on how to implement the JSON frame. @phochste proposed to have, in this repository, also a standard JSON-LD context that can be use for a default frame

TODO

Some back-end systems are able to expose an event stream of last updated items, but don’t keep track of things that have been deleted. In that way, a retention policy should exist that describes the fact that the LDES conceptually does contains the as:Remove activity, but that it hasn’t been included, yet can be inferred from the fact that the earlier included as:Create is not anymore part of this

Use cases:

• DCAT-AP Feeds in which catalogs just want to provide a dump of their current state
• Explicitely describing what happens in the IIIF Change Discovery level 0: https://iiif.io/api/discovery/1.0/#level-0-basic-resource-list

There is a note suggesting to keep a list of already processed items. Would it be a recommendable alternative in some situations to instead keep the latest processed item as a marker of how far the stream has been processed?

As long as a fragment is being written to, the client needs to parse the entire fragment again, including the members it already processed.
In the worst case scenario, when the client polling interval is >= the member addition interval, the amount of effectively transferred and parsed members is given by (max_fragment_size * (max_fragment_size + 1) /2), where max_fragment_size is the max. number of members allowed in a fragment.
Say 5 members are allowed per page, and the client is polling faster than member are written to the fragment (and etags are used to avoid needless processing), than the amount of effectively transferred and parsed members = 1 + 2 + 3 + 4 + 5 = 15.
If the max_fragment_size is set to 250 members, 31375 members are parsed to process 250 members.
At this point the efficiency of both data transfer and parsing compute has dropped to 0,80% =((max_fragment_size * (max_fragment_size + 1) /2)/max_fragment_size)) *100, leaving 99,2% of resources spend of algorithmic overhead.

These 31375 members, have to be transferred over the web and parsed, hence a lot of bandwidth and cpu cycles are wasted.
In order to mitigate this, I would propose to introduce some semantics in LDES, so that a server can indicate to a client that it follows the 'delayed flush strategy'.
So how would this work? A server would only write out fragments that are immutable. It can do this by buffering writes for a maximum time (let's go with 10 seconds for the argument). After this time (or when the max fragment size is reached), all buffered members are flushed as an immutable fragment. This results in potentially smaller fragments being written, which is fine for those reading the at the end of the log.
As a client knows (from the view definition) that fragments are written out in an atomic fashion, it only needs to request the fragment once.
If no relations are found in the fragment, the client can then fall back to polling using HEAD requests, when the relations are present in the HTTP headers.

Of course, this means that data can be delayed during the buffer window, but the trade-off would be reasonable to make.
The profile could be announced on the view with a simple statement.

Status indication

When replicating a stream on your own system when, for example, building other tree:views on top of them with different fragmentations, you MAY track the progress of how much elements from the original stream are processed into your own system.

This should become possible using an ldes:elementsProcessed as follows:

<Collection> a ldes:EventStream ;
    hydra:totalItems 500 ;
    tree:view :View1 .
:View1 a tree:Node ;
    ldes:elementsProcessed 250 .

I couldn't directly understand example 3. Is it implied that there are three documents, each consisting of the triples where the subject is <C1>, <?page=1>, <?page=2>?

What does ... consist of - will all observations be individually declared as members of <C1> and will they be referenced by the first of the documents?

Section: https://github.com/SEMICeu/LinkedDataEventStreams#the-linked-data-event-streams-specification:

• First alinea: create a hyperlink on RDF to https://www.w3.org/RDF/
• Fourth alinea: create hyperlinks on Activity Streams to https://www.w3.org/TR/activitystreams-core/, on DCAT-AP to https://joinup.ec.europa.eu/collection/semantic-interoperability-community-semic/solution/dcat-application-profile-data-portals-europe, on LDP to https://www.w3.org/TR/ldp/ or Shape Trees to https://shapetrees.org/.

Section: https://github.com/SEMICeu/LinkedDataEventStreams#build-the-spec :

• create a hyperlink on bikeshed to the bikeshed documentation GitHub pages link https://tabatkins.github.io/bikeshed/

Section: https://github.com/SEMICeu/LinkedDataEventStreams#acknowledgements

• create a hyperlink on SEMIC to https://joinup.ec.europa.eu/collection/semantic-interoperability-community-semic

Intuitively, a "stream" refers to a collection of items ordered by the time they were published in the collection. Thus, the stream grows at its end. The specification seems to consider these an important but not the only relevant type of "streams". (There is a note saying "A 1-dimensional fragmentation based on creation time of the immutable objects is probably going to be the most interesting and highest priority fragmentation for an LDES" but then continuing "sometimes the back-end of an LDES server cannot guarantee that objects will be published chronologically".)

Should these two separate cases be motivated, perhaps already in the introduction?

LDES members are mangled with the LDES metadata such as versionOf, timestamp, etc. which is a bit against the philosophy of the OSI model which assumes your have an envelope with metadata where the actual member data in resides.

Problem

LDES member data is mixed with their metadata. For example:

<MemberJefke#8> a foaf:Person;
  foaf:name "Jefke";
  foaf:age "45";
  dct:created "2023-01-01T00:00:00.00Z";
  dct:versionOf <MemberJefke>;
.

In the example, the member describes about a foaf:Person but also mixes the metadata within when the member was created.
This gives a weird view to this person for example, you could assume that it was created on 2023-01-01 but its age is 45 which is not possible. The bigger problem here is that metadata about the member is integrated into the member instead of separately where it points to the member, for example:

<MemberJefkeMetadata#8> 
  dct:created "2023-01-01T00:00:00.00Z";
  dct:versionOf <MemberJefke>;
  dct:subject <MemberJefke#8>;
.
<MemberJefke#8> a foaf:Person;
  foaf:name "Jefke";
  foaf:age "45";
.

Similar approaches

• W3C Acitivity Streams does the same by relating to the object it is describing with as:object.
• OSI model encapsulates data as well like this with metadata around it. This is possible in multiple layers even.

CC: @sandervd

Introduce term

A client that is already synchronized to an event stream may be disappointed if the data publisher decides to deprecate the fragmentation it's subscribed to, because it may just look like the event stream is no longer updated.

If root nodes can be cached, clients may never discover the new APIs at all. So should the specification forbid setting root nodes as immutable?

Add a third kind of retention policy: Point-In-Time Retention Policy in which data generated before a specific time is removed

As proposed in #32

The ldes:timestampPath should be carefully assessed as it’s not always the case that the timestamp on the version object is a change in the real-world on top of the real-world entity

The LatestVersionSubset gives the impression that only version (the latest) will be kept.
On this site it gives a clearer explanation that it allows multiple versions, that the amount is to be defined by the policy.

I suggest updating the https://w3id.org/ldes spec to make clear more than one version of a state object can be kept

First tell "why" you would be making version subsets, because an eventstream in itself is versioned, so why make subsets?

In order to indicate you only keep 1 specific version, e.g., the latest, in your stream
-> 1 specific version of what?
-> 1 specific version for every object

What's the definition of "ldes:amount"? numberOfLatestVersionsPerObject?

How do we tackle other use cases, e.g., what if you only keep the versions between a certain time window and region?

https://semiceu.github.io/LinkedDataEventStreams/#introduction
You can indicate that the new ldes:EventStream is derived from another ldes:EventStream.
Is there a specific ontology suggested here to indicate this derivation?
Thanks

It would be interesting to be able to point to an optional json-ld context on the eventstream, that can be used to apply to LDES members to JSON-ify them.
This would allow to use any serialization within a data pipeline, transforming the data as you go, while being able to create a JSON document from the member as needed.

Issue to start a discussion of possible interaction alternatives between LDES and DCAT-AP

Currently, the expected behavior is as follows:

1. When a Cache-control: max-age=x and Age: Y has been set, poll the page again in X-Y seconds, or when it is immutable, don’t poll it ever again.
2. When there are is no Cache-control header present, poll the page according to a ldes client specific strategy, that could be influenced by a configuration flag

In the reference implementation of the LDES client, there is a flag on a pollingInterval. This is a problem as it requires specific configuration for a specific kind of LDES.

There should be an algorithm in the LDES spec that indicates when you MUST poll again, even when caching headers are not set. This thus can first elaborate on how to interpret max-age and age, but I would like to have an LDES specific property on a tree:Node to either overwrite the max-age, or set the polling interval when it’s not set.

# If set, you can ignore HTTP headers and not visit this page again
<> ldes:timeToLive "immutable" .
# If this is set, the time to live of the page is 60 seconds. It may however be possible that the page was generated 59 seconds ago. For more fine-grained control, you will need to check HTTP headers to then check the `Age` header.
<> ldes:timeToLive 60 . 

This may also be useful for hosting LDESes in systems in which the caching headers are not easy to change.

When there is no timeToLive, or max-age present, we should have a fall-back of a default value. I’m unsure what this default value should be then.