Document current state of interoperable content types about websub HOT 18 CLOSED

I would love to see this as a separate specification layered on top of the basic notification/subscription model. I have long felt that the Atom-centricity of PuSH was a weakness, and I'm really pleased that this iteration of spec work has split the two aspects apart.

RabbitHub (at least when I wrote it; @brc859844 has long since taken over the project) used fat pings only and did nothing Atom- or RSS-specific at all. It was "content neutral", if you like.

from websub.

The way I read this section 6 then all notifications should be fat pings and thus there's no need for Superfeedr to define whether it's a fat ping or not because PuSH as far as it's concerned is always a fat ping. Relevant section:

A content distribution request is an HTTP [RFC7231] POST request from hub to the subscriber's callback URL with the payload of the notification. This request must have a Content-Type corresponding to the type of the topic. The hub may reduce the payload to a diff between two consecutive versions if its format allows it.

It has even experimented with digging IndieWeb content in those fat pings: https://blog.superfeedr.com/indieweb-microformats-fragments-subscriptions/

(As far as I remember fat pings was an important reason PuSH was created – to avoid the thundering herd problem that thin pings can result in – when e.g. all subscribers of a blog at once decides to fetch that new post they received a ping about)

from websub.

Section 6 doesn't actually require that there be any content delivered in the notification. In practice, thin pings aren't a problem for small-ish websites (you can often get more traffic tweeting out a link to your site, or posting it on Hackernews/Reddit, than results from subscribers of your PuSH feed). Sites that know they have a large subscriber base can always use a hub that supports fat pings.

I still haven't been able to find any documentation on the actual payload that any hub sends for a fat ping. The goal of this issue was to clearly document (with examples) the current interoperable combinations of fat/thin pings and content types. We know that even PubSubHubbub 0.4 has been used for content types other than Atom/RSS, so let's document them.

from websub.

Here's the spec of what Superfeedr sends in its fat pings: https://documentation.superfeedr.com/schema.html

The specification is for feed content and it says the following about other types of payload:

If you’re subscribing to arbitrary content resources, we will send you the exact content of the resource to which you’ve subscribed.

from websub.

So just to be clear, regardless of what content type is being subscribed to, Superfeedr will send an Atom feed in the payload? And the payload is actually a complete Atom feed. Does it diff the feed to send only the new element?

There also seems to be an option of receiving the payload in a JSON format https://documentation.superfeedr.com/schema.html#json

Does this mapping of other content types to Atom fall under what's allowed by section 6?

This request must have a Content-Type corresponding to the type of the topic.

It would seem that if Superfeedr is sending a JSON document as the notification, but the topic was an Atom feed, then this is not technically allowed by the spec. Should we update the spec to allow hubs to convert the notification to a different content type than the publisher sends?

from websub.

@aaronpk It's not touching non-feed content, only feed content is touched and it normalizes different feed formats to a single format which I think should be allowed if the hub wants to do it.

Would be best to have @julien51 explain more about what and why Superfeedr does or doesn't do.

(On the main question – is Google's hub updated to PuSH 0.4? Because it does fat pings as well in its 0.3 version at least and is/were one of the most widely used hubs. Also: What does the WordPress.com one from Automattic do? It's available as open source as well. I think it does fat pings as well)

from websub.

Google's says it supports 0.4: https://pubsubhubbub.appspot.com/

Can you find any documentation of the payloads these hubs send? I haven't been able to at all.

from websub.

First a couple correction to what's written above: Superfeedr only sends fat pings. Google's hub also only sends fat pings when you're subscribed to it... Wordpress's hub also sends fat pings... etc.
The 0.3 spec actually applied to both RSS AND Atom.

About fat pings and why they are important:
There are many reason why fat pings are an essential part of the spec. The most frequent one is that it allows publishers to have conservative caching mechanism. Using fat pings means that the subscriber does not have to fetch the resources themselves and hence are immune to caching delays. Other reasons involve problems like thundering herds where all subscribers would hit the same resource on the publisher's end at the same time.

Now, to clarify Superfeedr's behavior on public hubs (does not require a superfeedr account): you can only subscribe to "feeds" types of resources (RSS or Atom) provided that these resources point to a hub at Superfeedr. We will always normalize the data to Atom for easier consumption and we will always send fat pings.
For our "default" hub, which requires the use of a Superfeedr account. You can subscribe to anything provided that it's accessible via an HTTP url. This includes feeds, which we will optionally normalize to Atom or JSON. This also includes "fragment" subscriptions in which you subscribe to only a subset of an HTTP document. The default behavior is to send the full content of the published HTTP resource, using the same content type.

from websub.

I'm sorry, I didn't intend this thread to be a debate about fat vs thin pings. I am specifically looking for documentation on the types of pings and content types supported by various implementations. Like I originally said, I was not able to find any documentation on the actual payload Superfeedr, Wordpress or Google send, so please drop links in this thread if you know where that lives. The point of this thread was to suggest creating a section in the spec that documents what actual implementations have done to date.

from websub.

Here's documentation on what GitHub returns from their hub, which is JSON fat pings: https://developer.github.com/v3/repos/hooks/#response-format

from websub.

Here's an example of the ambiguity of saying "the notification is a diff" for RSS/Atom. Does the hub send a new RSS feed with a single item? (The top-level XML object is the feed) Or is it just the one item? (The top-level XML object is <item>). I'm looking for specific documentation on what the existing hubs actually send in order to clarify this.

from websub.

It probably follows the mechanism used in PuSH 0.3 in most cases, and it has two mechanisms – one if there's just a single source feed: https://pubsubhubbub.github.io/PubSubHubbub/pubsubhubbub-core-0.3.html#contentdistribution
And another if there are multiple sources feeding into the same PuSH: https://pubsubhubbub.github.io/PubSubHubbub/pubsubhubbub-core-0.3.html#aggregatedistribution

from websub.

Closing this as I believe we moved into a direction of full content fat pings in the spec (no diffing). Feel free to re-open if you disagree.

I also feel this is a small step backward but it will let us leap forward. When implementations are overwhelmingly asking for diffs, then we can revisit.

from websub.

I agree, now that the spec makes it clear that fat pings are required, this isn't as important to document in the spec. Once we get to the point of trying to make diffing interoperable we can revisit this.

from websub.

We discussed this during the F2F meeting:

• For RSS and Atom, we will add a sentence like 0.3 had, describing how to deliver partial feeds with only the new items.
• For other content types, the hub MUST NOT modify the document that it retrieved from the topic URL.

The complexity added by creating diffing mechanisms for various content types is not worth the tradeoff on the bandwidth potentially saved.

from websub.

http://www.w3.org/2016/11/17-social-irc#T19-35-40

from websub.

I can work on a PR for this.

from websub.

Modulo discussion in #68 as to whether websub is even appropriate, the ResourceSync system has both, though pings with diffs are much rarer. See: http://www.openarchives.org/rs/1.0.9/resourcesync#PatchCon

from websub.