Comments (18)
I would love to see this as a separate specification layered on top of the basic notification/subscription model. I have long felt that the Atom-centricity of PuSH was a weakness, and I'm really pleased that this iteration of spec work has split the two aspects apart.
RabbitHub (at least when I wrote it; @brc859844 has long since taken over the project) used fat pings only and did nothing Atom- or RSS-specific at all. It was "content neutral", if you like.
from websub.
The way I read this section 6 then all notifications should be fat pings and thus there's no need for Superfeedr to define whether it's a fat ping or not because PuSH as far as it's concerned is always a fat ping. Relevant section:
A content distribution request is an HTTP [RFC7231] POST request from hub to the subscriber's callback URL with the payload of the notification. This request must have a Content-Type corresponding to the type of the topic. The hub may reduce the payload to a diff between two consecutive versions if its format allows it.
It has even experimented with digging IndieWeb content in those fat pings: https://blog.superfeedr.com/indieweb-microformats-fragments-subscriptions/
(As far as I remember fat pings was an important reason PuSH was created – to avoid the thundering herd problem that thin pings can result in – when e.g. all subscribers of a blog at once decides to fetch that new post they received a ping about)
from websub.
Section 6 doesn't actually require that there be any content delivered in the notification. In practice, thin pings aren't a problem for small-ish websites (you can often get more traffic tweeting out a link to your site, or posting it on Hackernews/Reddit, than results from subscribers of your PuSH feed). Sites that know they have a large subscriber base can always use a hub that supports fat pings.
I still haven't been able to find any documentation on the actual payload that any hub sends for a fat ping. The goal of this issue was to clearly document (with examples) the current interoperable combinations of fat/thin pings and content types. We know that even PubSubHubbub 0.4 has been used for content types other than Atom/RSS, so let's document them.
from websub.
Here's the spec of what Superfeedr sends in its fat pings: https://documentation.superfeedr.com/schema.html
The specification is for feed content and it says the following about other types of payload:
If you’re subscribing to arbitrary content resources, we will send you the exact content of the resource to which you’ve subscribed.
from websub.
So just to be clear, regardless of what content type is being subscribed to, Superfeedr will send an Atom feed in the payload? And the payload is actually a complete Atom feed. Does it diff the feed to send only the new element?
There also seems to be an option of receiving the payload in a JSON format https://documentation.superfeedr.com/schema.html#json
Does this mapping of other content types to Atom fall under what's allowed by section 6?
This request must have a Content-Type corresponding to the type of the topic.
It would seem that if Superfeedr is sending a JSON document as the notification, but the topic was an Atom feed, then this is not technically allowed by the spec. Should we update the spec to allow hubs to convert the notification to a different content type than the publisher sends?
from websub.
@aaronpk It's not touching non-feed content, only feed content is touched and it normalizes different feed formats to a single format which I think should be allowed if the hub wants to do it.
Would be best to have @julien51 explain more about what and why Superfeedr does or doesn't do.
(On the main question – is Google's hub updated to PuSH 0.4? Because it does fat pings as well in its 0.3 version at least and is/were one of the most widely used hubs. Also: What does the WordPress.com one from Automattic do? It's available as open source as well. I think it does fat pings as well)
from websub.
Google's says it supports 0.4: https://pubsubhubbub.appspot.com/
Can you find any documentation of the payloads these hubs send? I haven't been able to at all.
from websub.
First a couple correction to what's written above: Superfeedr only sends fat pings. Google's hub also only sends fat pings when you're subscribed to it... Wordpress's hub also sends fat pings... etc.
The 0.3 spec actually applied to both RSS AND Atom.
About fat pings and why they are important:
There are many reason why fat pings are an essential part of the spec. The most frequent one is that it allows publishers to have conservative caching mechanism. Using fat pings means that the subscriber does not have to fetch the resources themselves and hence are immune to caching delays. Other reasons involve problems like thundering herds where all subscribers would hit the same resource on the publisher's end at the same time.
Now, to clarify Superfeedr's behavior on public hubs (does not require a superfeedr account): you can only subscribe to "feeds" types of resources (RSS or Atom) provided that these resources point to a hub at Superfeedr. We will always normalize the data to Atom for easier consumption and we will always send fat pings.
For our "default" hub, which requires the use of a Superfeedr account. You can subscribe to anything provided that it's accessible via an HTTP url. This includes feeds, which we will optionally normalize to Atom or JSON. This also includes "fragment" subscriptions in which you subscribe to only a subset of an HTTP document. The default behavior is to send the full content of the published HTTP resource, using the same content type.
from websub.
I'm sorry, I didn't intend this thread to be a debate about fat vs thin pings. I am specifically looking for documentation on the types of pings and content types supported by various implementations. Like I originally said, I was not able to find any documentation on the actual payload Superfeedr, Wordpress or Google send, so please drop links in this thread if you know where that lives. The point of this thread was to suggest creating a section in the spec that documents what actual implementations have done to date.
from websub.
Here's documentation on what GitHub returns from their hub, which is JSON fat pings: https://developer.github.com/v3/repos/hooks/#response-format
from websub.
Here's an example of the ambiguity of saying "the notification is a diff" for RSS/Atom. Does the hub send a new RSS feed with a single item? (The top-level XML object is the feed) Or is it just the one item? (The top-level XML object is <item>
). I'm looking for specific documentation on what the existing hubs actually send in order to clarify this.
from websub.
It probably follows the mechanism used in PuSH 0.3 in most cases, and it has two mechanisms – one if there's just a single source feed: https://pubsubhubbub.github.io/PubSubHubbub/pubsubhubbub-core-0.3.html#contentdistribution
And another if there are multiple sources feeding into the same PuSH: https://pubsubhubbub.github.io/PubSubHubbub/pubsubhubbub-core-0.3.html#aggregatedistribution
from websub.
Closing this as I believe we moved into a direction of full content fat pings in the spec (no diffing). Feel free to re-open if you disagree.
I also feel this is a small step backward but it will let us leap forward. When implementations are overwhelmingly asking for diffs, then we can revisit.
from websub.
I agree, now that the spec makes it clear that fat pings are required, this isn't as important to document in the spec. Once we get to the point of trying to make diffing interoperable we can revisit this.
from websub.
We discussed this during the F2F meeting:
- For RSS and Atom, we will add a sentence like 0.3 had, describing how to deliver partial feeds with only the new items.
- For other content types, the hub MUST NOT modify the document that it retrieved from the topic URL.
The complexity added by creating diffing mechanisms for various content types is not worth the tradeoff on the bandwidth potentially saved.
from websub.
http://www.w3.org/2016/11/17-social-irc#T19-35-40
from websub.
I can work on a PR for this.
from websub.
Modulo discussion in #68 as to whether websub is even appropriate, the ResourceSync system has both, though pings with diffs are much rarer. See: http://www.openarchives.org/rs/1.0.9/resourcesync#PatchCon
from websub.
Related Issues (20)
- use of Link header HOT 2
- hub vs subscriber in section 8 HOT 5
- High Level Architecture HOT 5
- At risk: limiting the use of HTML <link> to the HTML <head> HOT 16
- Add Acknowledgments section HOT 1
- Temporary vs Permanent Redirect HOT 1
- Diffing for HTML with <article> HOT 3
- Alternative protocols for callbacks HOT 4
- Content Distribution topic clarification HOT 3
- How to handle removals from a list HOT 2
- https://websub.net is not available due to certificate issues HOT 1
- discovery of hub url order and https vs http HOT 2
- GitHub Code Scanning Alerts on hub.challenge when using Express.js
- Example 4 has wrong Content-type HOT 1
- Let's stop treating URLs as secrets: the removal of hub.verify_token in prior draft 0.3
- protocol diagram is hard to read HOT 1
- How to ensure publisher and subscriber data remains private when transmitted through hubs?
- Consistency Issues with WebSub HOT 4
- Expiration time is still defined as optional in section 1.
- Changing callbacks in renewal requests? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from websub.