scripting / river5 Goto Github PK

View Code? Open in Web Editor NEW

428.0 27.0 122.0 311 KB

A river-of-news RSS aggregator in JS running in Node.

License: MIT License

JavaScript 66.01% HTML 31.07% CSS 2.92%

river5's Introduction

River5

River5 is a river-of-news RSS aggregator in JavaScript running in Node, written by Dave Winer.

How River5 works

When it starts up, River5 reads files in the lists folder at the top level of the River5 folder.

Each list contains a set of URLs of feeds. The list files can be straight text, a JSON array of URLs of feeds, or a traditional OPML subscription list, with extensions .txt, .json and .opml. River5 ignores files in the lists folder that are not in one of these formats.

The feeds pointed to from the lists can be in RSS 0.9x, 1.0 or 2.0, or Atom 1.0.

Every 15 minutes, River5 reads all the feeds you're subscribed to, and routes new items to corresponding files in the rivers folder at the top level of the River5 folder. You're subscribed to a feed if it appears in one or more of your lists. The river files are a documented form of JSONP that we've been using since River3, in 2010.

To read the output of River5 on the machine it's running on, go to the home page. Assuming your river is running on the default port, 1337, you would go to this address to read the rivers. http://localhost:1337/. If it's running on a public server, just replace localhost with the domain name or IP address of the server.

Pretty much everything in this narrative is configurable.

Requirements

A current Node.js installation.

The machine running River5 ahould able to access the Internet.

It is possible to run the software on a non-public machine. Of course, you won't be able to access the home page remotely, or receive realtime notifications of feed updates. You'll also have to find a way to upload the output files in the rivers folder to a place where they can be accessed, if you want others to be able to access your river.

If you don't have Node.js installed on your machine, here are step by step instructions for installing Node.js on a Macintosh.

How to install

Download the folder from the repository to your local computer.
From the command line, go to this folder.
npm install
node river5.js

If it's working properly, you should see tons of stories scroll by in the console. River5 is reading each of the feeds in the lists for the first time. When you see things settle down, then you should open the home page of the RIver5 website, at http://localhost:1337/ and see the result of the first feed readings.

At first there will be a lot of old stories, but as the rivers run for hours and days and into weeks, you'll get the flow. It's a good way to read news.

Examples of lists

I have included several files in the lists folder to help you get started, so something actually happens the first time you run River5. You can edit, consolidate or delete them, as you wish.

myJsonFeeds.json -- a list of URLs of feeds in the JSON format that River5 understands.
myOpmlFeeds.opml -- a list of URLs of feeds in the standard OPML format for subscription lists.
myTxtFeeds.txt -- a list of URLS in a text file.
nyt.opml -- a list of a New York Times news feeds.
hn.opml -- a list with just one feed in it, the Hacker News firehose. It's a good list for testing RIver5 because it updates so frequently.

To view your rivers

Go to http://localhost:1337/.

Configuring

There's only one way to configure it, via the config.json file in the same folder as the river5.js app.

The example config.json in the folder sets the max number of items in a river to 300.

All of the config.json options and their default values are listed on this page.

For River4 users

River5 does not support S3 storage, so if that's how you're running your rivers, you should continue to use River4.

If you're running out of the filesystem, then River5 can replace River4.
Follow the installation instructions above, and before you launch river5.js, remove the files from the lists folder, and replace them with the lists from your River4 installation.
It will take a while before your rivers re-populate, so I recommend doing it in parallel with your existing River4 install.

You may want to review the Configuration settings page, there are options for setting new things, and in some cases the defaults have changed.

Where's the code?

The heart of River5 is in a Node package in the lib folder, called davereader.

To upgrade to the latest version of the software npm update at the command line in the river5 folder.

river5.js itsef is a very small file and by design almost never changes.

Examples of working rivers

Podcatch.com subscribes to my friends' podcast feeds.
mlbriver.com and nbariver.com collect news for two sports I love.
I have a River panel on the Scripting News home page.
My rivers page puts all the news I care about on a single tabbed page, with panels for the NYT, Washington Post, Guardian, Movies, Tech, baseball and basketball.

Other docs

The road to River5, reviews the previous versions of my river-of-news software going back to 1999, with a special focus on the transition from River4 to River5.
The Hello World of Rivers shows you how to create a standalone river, one that people can read without accessing your RIver5 server.
My blog post announcing River5.

Updates

v0.50 -- 5/18/17 by DW

Major release of River5, but in a way not much changed.

The lib folder is gone. The two items that were in the folder have been published using NPM, the official Node.js software distribution system. This means that they can be used in projects other than River5.

river5.js has been changed to use the new version of feedtools, called davereader (the name feedtools was already taken). It should work exactly as before.

This change will make new feed-based software possible. I've written about that on the reader site, and in a blog post.

Otherwise the changes should be totally transparent to River5 users. The software should behave exactly as it did before. Not much actually changed, it's just been reorganized to make it more useful in more situations.

Future updates will be logged in the Updates section of the reader site.

To update to the new version, follow the same instructions as for previous releases.

v0.49 -- 4/18/17 by DW

Support for podcasting. New config values, described on the config docs page.

Added buildRiver callbacks.

v0.48 -- 3/29/17 by DW

Changes for Electric River. Added three new callbacks to config that allow us to do things a bit more efficiently because the reading and configuring app is running in the same process as the server. Wrote a howto doc for people with experience running River5. Added flBuildEveryFiveSeconds to config, if true, we build changed rivers every five seconds instead of once a minute, the default. Building rivers became a lot more efficient in the change from River4 so it made sense to allow them to be rebuilt more frequently.

v0.47 -- 2/9/17 by DW

Fixed two problems reported by Andrew Shell. Added some feeds to the default set up, and configured the default home page so it only has three panels and they have intelligent labels, using the relatively new homePage configuration option. Fixed a problem with NYT feeds and cookies. We now set the User-Agent header on requests. Apparently some servers won't respond unless this header is present.

v0.46 -- 11/16/16 by DW

Refreshed the admin app, relocated it, and fixed a couple of entry-points it depends on that didn't work with CORS.

v0.45 -- 3/14/16 by DW

Fixed a bug that prevented addToRiver callbacks from working correctly. When you made a change to an item in a river it wasn't actually being reflected back in the river.

As a result of this fix, River4 callbacks should now work in River5.

v0.44 -- 2/22/16 by DW

Fixed a bug that prevented text file lists from working. It would only subscribe to the last feed in the list.

v0.43 -- 2/4/16 by DW

You can now configure the panels on the server's home page.

The new configuration feature is documented in a new section on the configuration page.

v0.42 -- 2/4/16 by DW

We accept two new formats for files in the lists folder -- straight text and JSON, in addition to the traditional OPML subscription lists.

If a file in the lists folder has a .txt extension, we assume it's just a series of lines with the URLs of feeds. It's the simplest of all the list formats.

If it has a .json format, we look for a JSON file that's just an array of URLs.

Created a new top-level lists folder containing example lists in each of the three supported formats.

v0.41 -- 2/3/16 by DW

We now maintain a new data structure for each list, riverData.json, in the same folder as the listInfo.json file.

It contains an array of river items that are part of this river. To build the riverjs file, all we have to do is loop over that array.

It's kept in memory while River5 is running, so now building a river on reads a file from the disk the first time a river is built or an item is added to it.

One of the first bennies of rebuilding the river software, it's now quite a bit faster at building rivers, and it's no longer a function of how dense the river is. In previous versions, rivers that don't update often would require us to go back a long ways into the calendar structure to build the river.

Now the calendar structure is just an archive, it's no longer used to build anything. It could be made optional.

Questions, comments?

Please post a note on the River5 mail list.

river5's People

Contributors

Stargazers

Watchers

Forkers

tedchoward vijaykiran cloudytom sidewinder pkdevboxy neuroradiology nkavanagh dc310 mjgardner purlieus pandastyle chaodoze sagarhani arafsheikh freso angelos-p srijancse-zz pulkit4tech evijit ninjacomics rahul-raturi rnavagamuwa pranith pandusonu2 supun94 anjali05 hackertron janakact digideskio riddhishb abhi-kumar ztl8702 gcolajan oddcoder g33kyaditya devyanikota shekharrajak vithulan radusqrt darthbhyrava misg prakritibhrdwj reachtarunhere abhimanyu-bitsgoa soumitraagarwal pwnfoo willasaywhat niccokunzmann skbly7 mbovan karansaxena hrishikeshgoyal tmess567 jeffreychung rakesh-verma-16 opticod jaymine jaminy joyceg vikas-parashar gasuleg ananyo2012 chiragchamoli boycaught lucaswilric marcusjmb mayeaux robertomalatesta fpcmotif whaleen keencode alanfx sparkyt benson8 jamestewartjr martindale jameslegue troutcolor hatimak lukas-hetzenecker enirate jacobwindsor oevl neonmaccca dm-media phictionalone enterstudio sportsbite rositarijtano marcno ognase chrisbodhi aramzs dannypage wzewdie aysadkozanoglu hugolundin kylekirkby samuelgoto slorba

river5's Issues

Displaying several columns in one tab/panel

Hello,

I'm interested in displaying more than one column on a single tab/panel.
Which file/folder should I be looking at to accomplish this? Thanks!

Browser tab for dashboard displays "River4 dashboard"

The tab says River4 rather than River5.

Is it possible to have a river file on a remote server?

For portability reasons I would like to have a feeds.json file that I can modify from wherever without having to do anything with my rivers5 instance.

When I add the public url as a river it says that it cannot be found, and it seem to be looking for a file in rivers with the name as the url..

Thank you!

Any thoughts on adding database support?

I love this project, thanks for your great work. I'd love to help support it as I use it. One feature that I'd personally like for my application is database support.

I was thinking an extra configuration option to tell the system to send new articles to a database instead of (or in addition to) the JSON files that are currently maintained.

The advantage to this would be persistence, and it would allow users to create more complex frontends.

Just curious, do you have any thoughts on this feature? I realize this may break too far from its intended use case, so maybe it's not appropriate, but I wanted to ask before I start work on a PR.

Thanks again.

how to add new feeds directly to another backend?

Many thanks for great work. I have a use case and I don't know how to implement it.

I have a backend as a service (Parse / Firebase) and i want that every time a new feed exists, it should be added to my backend.

so i don't need river5 web page. i want callback to directly call a web service in my backend every time there is a new data

Thank you for support

Adding a .gitignore file

What do you think about adding a .gitignore file?

I'd be happy to submit a PR for this.

How do I display the RSS feed chronoligically

I've added a few RSS feeds, but I want them to be displayed from most recent to oldest, and not the opposite.

I've considered the config file but couldn't find an attribute to do that.

River title can't include an apostrophe

Putting an apostrophe in the title of a river (e.g."title": "Larry's Feed") in config.json stops the river from being read. I don't know if this is inherent to config.json parsing. Might be worth noting this is the documentation.

Fetching feeds for the first time only shows the last item in the feed.

I fetched a fresh copy of the repo and stripped the lists/ and config down to a single feed (lists/hn.opml) and let the system run. It reports to the console that it collects all the items. The rivers/ folder (once it is manufactured) only has the last item. The view only shows the last item correspondingly. It should show all the items.

I tried with both flAddItemsFromNewSubs set and unset in the config.json.

{
"maxRiverItems": 300,
"flAddItemsFromNewSubs": true,
"homePage": {
"panels": [
{
"title": "Hacker News",
"river": "hn.js"
}
]
}
}

Homepage HTML problem for Firefox on Ubuntu

Pointing the Firefox browser (44.0.2) running on Ubuntu (15.10) at a river server (either river4 or river5 on localhost or a remote server) displays only a (mostly) unformatted "Stuff" menu.

It works correctly when using the Chromium browser (48.0.2564.116) or the same version of Firefox on Windows.

The web console on the failing browser complains about a

ReferenceError: $ is not defined

regarding the line near the end of the home page html that reads

$(document).ready (function () {

I opened the file

http://rss2.io/code/feedtools/misc/serverhomepage.html

in Intellij IDEA. I believe this is the page being retrieved and served for the home page. The IDE led me through three lines it believes are incorrect.

The <head> section is not being closed properly due to a spurious </style> tag (at about line 242) before the </head> tag.
What appears to be a template place-holder, [%config%], at about line 54 is not enclosed in quotes. I assume that is the error since the template on line 36 does have quotes and is not flagged.

Near the end of the file the last piece of script before the closing </body> tag

    <script>
    $(document).ready (function () {
        startup ();
        });
    </script>

does not have the document argument enclosed in quotes. This is the line the browser was complaining about. Every other such statement in the html does have quotes surrounding the argument.

I made the suggested changes and saved a local copy. However, I apparently cannot understand how to construct a file URI

urlServerHomePageSource: "file:///home/david/projects/river5/misc/editedserverhomepage.html"

in feedtools.js to serve the edited html or I would have tested the changes myself.

The problem is real, if weird, but I don't know enough about Javascript or jQuery to know if the changes make any difference.

"undefined" in title of many items in scripting.com/river.html

I don't know if this is the right place to report this, but many of the items in scripting.com/river.html have "undefined (Feed)" as the "source" of the item. I append a recent screen shot.

Is this expected? (Everything else seems to work fine.) Thanks.

Windows Install no worky out of the box.

Downloaded the code, installed Node, node river5.js gives this error (pasted below)...
I have an older version of River which works fine and seems to have a lot more folders/structure to the code than this latest version here. Also the older river5.js points to
var feedtools = require ("./lib/feedtools.js");

The new version points to the davereader module, but it's not present in the downloaded code.
Is this code just meant to be an update to existing versions?

C:\river5>node river5.js
internal/modules/cjs/loader.js:892
throw err;
^

Error: Cannot find module 'davereader'
Require stack:

C:\river5\river5.js
←[90m at Function.Module._resolveFilename (internal/modules/cjs/loader.js:889:15)←[39m
←[90m at Function.Module._load (internal/modules/cjs/loader.js:745:27)←[39m
←[90m at Module.require (internal/modules/cjs/loader.js:961:19)←[39m
←[90m at require (internal/modules/cjs/helpers.js:92:18)←[39m
at Object. (C:\river5\river5.js:24:17)
←[90m at Module._compile (internal/modules/cjs/loader.js:1072:14)←[39m
←[90m at Object.Module._extensions..js (internal/modules/cjs/loader.js:1101:10)←[39m
←[90m at Module.load (internal/modules/cjs/loader.js:937:32)←[39m
←[90m at Function.Module._load (internal/modules/cjs/loader.js:778:12)←[39m
←[90m at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:76:12)←[39m {
code: ←[32m'MODULE_NOT_FOUND'←[39m,
requireStack: [ ←[32m'C:\river5\river5.js'←[39m ]

Add a license.txt file

Is it GPLv2 like some of your other code?

Tom Critchlow wants to run River5 on Glitch

Tom Critchlow wants to run River5 on Glitch. Has anyone had any success with this?

https://twitter.com/tomcritchlow/status/988452117823348736

Dave

River5 version confusion

An earlier version of River5 seemed to have a memory leak: after running for months, the amount of free RAM dwindled, and the computer got slow. Stopping River5 (with Ctl-C) freed up the RAM, and the computer was much snappier. Starting River5 again (npm start) worked as expected. I'm not reporting a memory leak against an obsolete version...

But... this made me wonder if I was up to date. And I wasn't (I was using River5 v0.4.14). So I updated and had these observations with a git pull from 0c2804e on macOS 10.13.6:

The startup line shows [email protected], but later on, the log shows River5 v0.5.9 running on port 1337. This is confusing.
It would help with diagnostics if the startup information also displayed the date/time, and possibly the version of Node (I was using 5.3.0, which may also have been part of the memory leak problem.)

Thanks!

richb$ npm start

> [email protected] start /Users/richb/github/river5
> node river5.js


{
    "enabled": true,
    "httpPort": 1337,
    "flHttpEnabled": true,
    "webSocketPort": 1338,
    "flWebSocketEnabled": true,
    "dataFolder": "data/",
    "listsFolder": "lists/",
    "riversFolder": "rivers/",
    "podcastsFolder": "podcasts/",
    "localStoragePath": "localStorage.json",
    "statsFilePath": "serverStats.json",
    "templatePath": "misc/template.html",
    "addToRiverCallbacksFolder": "callbacks/addToRiver/",
    "buildRiverCallbacksFolder": "callbacks/buildRiver/",
    "riverDataFileName": "riverData.json",
    "listInfoFileName": "listInfo.json",
    "flAddItemsFromNewSubs": true,
    "maxRiverItems": 300,
    "maxBodyLength": 280,
    "flSkipDuplicateTitles": true,
    "flRequestCloudNotify": true,
    "flMaintainCalendarStructure": false,
    "flWriteItemsToFiles": false,
    "ctMinutesBetwBuilds": 15,
    "maxConcurrentFileWrites": 100,
    "flWatchAppDateChange": false,
    "fnameApp": "lib/feedtools.js",
    "urlServerHomePageSource": "https://s3.amazonaws.com/scripting.com/code/river5/serverhomepage.html",
    "urlDashboardSource": "https://s3.amazonaws.com/scripting.com/code/river5/dashboard.html",
    "urlFeedViewerApp": "https://cdn.jsdelivr.net/npm/river5/includes/feedviewer/index.html",
    "urlFavicon": "https://cdn.jsdelivr.net/npm/river5/includes/misc/favicon.ico",
    "flIntegratedFeedViewer": true,
    "flBuildEveryFiveSeconds": false,
    "flDownloadPodcasts": false,
    "maxFileNameLength": 32,
    "maxConcurrentPodcastDownloads": 10,
    "flSaveFeedRivers": true,
    "httpUserAgent": "River5 v0.5.9",
    "homePage": {
        "panels": [
            {
                "title": "Main",
                "river": "myOpmlFeeds.js"
            },
            {
                "title": "NYT",
                "river": "nyt.js"
            },
            {
                "title": "Hacker News",
                "river": "hn.js"
            }
        ]
    }
}

River5 v0.5.9 running on port 1337.

startWebSocketServer: websockets port is 1338

River5 v0.5.9: 8:27:00 AM, 203 feeds, 0 reads, 0 stories, 0 open sockets, port: 1337.
...

mobile first?

How can I use it with my mobile browser? It's missing UI like bootstrap.

JavaScript error in main process

This error is popping up for about 80 (I may have lost count in clicking the dialog box) of the 111 feeds that are loaded in the mySubs.opml file.
Given another issue listed, I counted the https feeds and they amount to 77 .
However, quite a few https links are feeding the river...

I fail to see where the syntax, for this particular case, is mistaken:

<outline type="rss" text="Amsterdam" xmlUrl="http://www.amsterdam-advisor.com/amsterdam-travel.xml" />

These were generated from the List editor (version 0.42a)

Note that the text value is different from the screen shot to that in the opml file.
Is there a cache that needs flushing somewhere?

Digging into the folder, I notices that all these 'errors' are generating a folder with the outline tag. So something is generating them improperly...

Some include files still point to http://fargo.io

The following 4 files in the repo and on jsdelivr.net appear to have document.write calls to add scripts from http://fargo.io, not the new //cdn.jsdelivr.net locations causing https instances of river5 to not work as expected

includes/basicincludes2/beep.js
includes/riverBrowserIncludes/emojify.js
includes/riverBrowserIncludes/riverbrowser.js
includes/misc/riverbrowser.js

How to modify requests made to pages that I subscribe to?

Can i customise the User Agent headers? I want to leverage on socks5-https-client when performing the requests.

Thanks in advance.

Broken umlauts in feeds

Hello,

nice work crafting river5 and share it with us! Thank you.

Unfortunately I got a problem with umlauts. If the RSS feed contains any umlauts they will be not displayed correctly. Feeds incoming are looking good, so the encoding is changed while parsed by river5.
Did a quick look, but didn't find the right spot for now, will have a closer look later. Maybe you can give me a hint or you have the time to fix it.

Anyway thanks in advance.

feedtools.js link in documentation returns 404 page

The link to feedtools.js in the section https://github.com/scripting/river5#wheres-the-code returns a 404 page not found error now that it is no longer part of this repo.

Don't include external script only available on http

I tried to install River on my server, which use https, so my browser refused to fetch scripts from fargo.io, because of mixed content rules.

From a security point of view, it would also be better to include this scripts into the repository of river5, to prevent attacks if files on fargo.io are modified.

Displaying generated rivers on personal sites

Hello!
I recently came across your River5 project on GitHub. It has been interested in going through the code to understand all that is going on and I am attempting to include one of the generated "river.js" files on my blog website, however, I am getting a little confused.

Does anyone happen to have a video walking through the instructions on this page: https://github.com/scripting/river5/blob/master/docs/HELLOWORLD.md

I understand downloading the source code and making the necessary changes as such but not sure how or where to deploy the new code. I'll still rather new to web development.
Personally, learning from example would be ideal.
Thank you in advance!

Doesn't work with HTTPS

the default River layout includes various JavaScript resources via HTTP. These imports fail and throw a mixed-content warning when the river is hosted on HTTPS.

Needless to say, the site is not usable without JavaScript.

Solution would be to either

move all the front-end dependencies into the river5 repository so they are hosted from the same server as the Node app, or
enable HTTPS on http://fargo.io/ and http://api.nodestorage.io

I would prefer the former because I don't see why my River instance needs to depend on external websites. I assume fargo.io and nodestorage.io belong to Dave Winer, but some day he might decide to give the domains up (fargo.io already shows a retirement page).

ERR_INVALID_CALLBACK Upon Starting App

Upon starting the app, I will see several feed items being loaded through the console, but before I can even check it out on localhost, the app stops and I get the following error:

fs.js:129
throw new ERR_INVALID_CALLBACK();
^

TypeError [ERR_INVALID_CALLBACK]: Callback must be a function
at maybeCallback (fs.js:129:9)
at Object.writeFile (fs.js:1159:14)
at eval (eval at runUserScript (C:\Users\Nitro Dubz\Desktop\Web Dev\bootcamp\cryptonews\node_modules\river5\node_modules\davereader\davereader.js:2104:6), :8:5)
at doLevel (C:\Users\Nitro Dubz\Desktop\Web Dev\bootcamp\cryptonews\node_modules\river5\node_modules\davereader\davereader.js:1907:7)
at C:\Users\Nitro Dubz\Desktop\Web Dev\bootcamp\cryptonews\node_modules\river5\node_modules\davereader\davereader.js:1896:8
at suppressedCallback (fs.js:200:5)
at FSReqWrap.oncomplete (fs.js:141:20)