freiheit / discord_feedbot Goto Github PK

Moved to https://gitlab.com/ffreiheit/discord_feedbot

License: MIT License

Python 95.85% Shell 1.64% Dockerfile 2.43% Procfile 0.07%

discord discord-bot feed rss atom rss-feed atom-feed

discord_feedbot's Introduction

Discord RSS Bot

Primary location is https://gitlab.com/ffreiheit/discord_feedbot -- github.com/freiheit/discord_feedbot is a mirror and all others are forks.

[[TOC]]

Bot for taking in an RSS or Atom feed and sharing it into a Discord channel.

Designed to be very configurable.

It should be possible to use as a library in another bot. (But can simply run with same tokens instead, since it's write-only to discord)

Since this bot doesn't react to anything in channels or messages, you can have it sign into the same account as another bot, and externally it should appear to be a single bot.

Elite: Dangerous related Discord?

If your Discord server is related to Elite: Dangerous, you may be able to use my feedbot instance instead of running your own copy. Head to https://discord.gg/s97tH5Bsw6 and "Follow" the rooms you want.

(I have some other feeds, but E:D is the most thoroughly covered)

Installation / Initial Setup

See INSTALLING.md

Adding feeds

See FEEDS.md

My configuration

The configuration of my instance of feedbot (minus auth token) is here: https://gitlab.com/ffreiheit/feedbot-config

Use as Library

I have not tested at all, but I have tried to make it possible to plug this in as a library. Probably needs work. You'll need to replace your simple "client.run()" with the more complicated stuff inside the if name block instead (in order to insert the background task coroutines into async)

If you want to do live changes, you should be able to change the CONFIG variable.

I'm guessing a few changes will need to be made in order to run as a library, like maybe changing how client= gets set, to something with a setup function that takes an optional client argument, maybe? If you figure it out, please give me a pull request.

If you don't need live changes, since this bot is read-only, it can easily be running alongside another bot using same account to appear as a single user.

Frequently Asked Questions (FAQ)

FAQ items are in INSTALLING.md and FEEDS.md

Contributing

See CONTRIBUTING.md.

Chat with me

Come to https://discord.gg/s97tH5Bsw6 and talk in the #dev room.

Bug reports

Preferred avenue for bug reports is an Issue filed at https://gitlab.com/ffreiheit/discord_feedbot/-/issues

Security Bugs

Check the "confidential" checkbox when submitting an Issue at https://gitlab.com/ffreiheit/discord_feedbot/-/issues

Tech Stack

freiheit/discord_feedbot is built on the following main stack:

Python – Languages
Shell – Shells
Pylint – Code Review
GitHub Actions – Continuous Integration

Full tech stack here

Financial Support

I have a very few costs associated with this project. (under $10/month) I'm happy to donate my time. But if you want to thank me financially: Or: https://discord.com/servers/feedbot-910747606722965555

discord_feedbot's People

Contributors

Stargazers

Watchers

discord_feedbot's Issues

Multi room

Make capable of pushing feed data to multiple rooms. Should be able to specify which feeds go to which rooms.

Wrapping fieldname with ``` tags makes the bot ignore the field

Hello, I got the bot up and running, however, in my local config my feed is like this

fields = "New Post",link,**title**,```description```,".."

When i wrap the description with ``` it gets ignored, what i want is to wrap the content inside of the description within a code block, so it looks tidier, am i doing something wrong or is there another way?

Permission for Sever Ministry of Propaganda FLC

Discord Server Name: Ministry of Propaganda FLC
Contact Info: Along with Discord DM Edmund Black#0216, email: [email protected]
Room ID Number: 382301360013705227

Asyncio - SSLProto ResourceWarning: unclosed transport

After recently updating my Ubuntu box, and the feed2discord bot. It runs, but does not seem to post into channel, erroring with the following;

/usr/lib/python3.5/asyncio/sslproto.py:328: ResourceWarning: unclosed transport <asyncio.sslproto._SSLProtocolTransport object at 0x7f10f012a978>
warnings.warn("unclosed transport %r" % self, ResourceWarning)

Current version are as follows;

pip freeze
aiohttp==2.0.7
args==0.1.0
arrow==0.7.0
async-timeout==1.2.0
asyncio==3.4.3
cffi==1.8.3
chardet==3.0.2
clint==0.5.1
command-not-found==0.3
discord.py==0.16.7
feedparser==5.2.1
html2text==2016.9.19
language-selector==0.1
multidict==2.1.4
pycparser==2.17
pycurl==7.43.0
pygobject==3.20.0
PyNaCl==1.0.1
python-apt==1.1.0b1
python-dateutil==2.6.0
python-debian==0.1.27
python-systemd==231
pytz==2017.2
requests==2.13.0
six==1.10.0
ssh-import-id==5.5
ufw==0.35
unattended-upgrades==0.1
urllib3==1.13.1
websockets==3.3
ws4py==0.4.2
yarl==0.10.0

Asyncio Issue

This problem is most likely the exact same problem as stated here #18

The bot would be able to send feeds for a maybe a few min before the scirpt just halts, and won't do anything until I press any key which seems to cause the bot to formally stop.

Here is a some data: http://pastebin.com/CzzxNbur
This is with log=1. Right after the first DEBUG:__main__:warframe:sleeping for 60 seconds (line 5) nothing happened for maybe an hour until I pressed a key, then the rest of the log there showed up and stopped the bot

Here is with log=3 http://pastebin.com/8k7BTJDr

I am running Windows 10 Pro x64

Switch to OAuth

Looks like it's time to switch to the new API stuff... Get the little "bot" badge, do things the new way, etc etc...

https://discordapp.com/developers/docs/topics/oauth2

Empty pubDate in instragram feed causes TypeError.

Instagram feed using "tag" (http://widget.websta.me/rss/tag/taghere) works as a feed and displays posts in Discord channel but only with a high max_age as mentioned in #19 previously.

With "user" feed (http://widget.websta.me/rss/n/usernamehere), it displays back TypeError and doesn't post anything.

Below is debug output:

DEBUG:__main__:instagramtest:processing entries
DEBUG:__main__:instagramtest:item:processing this entry
DEBUG:__main__:instagramtest:item:checking database history for this item
INFO:__main__:instagramtest:item http://websta.me/p/1214601160999041871_329452045 $ unseen, processing:
ERROR:__main__:instagramtest:Unexpected error:
ERROR:__main__:(<class 'TypeError'>, TypeError('Tuple or struct_time argument r$ quired',), <traceback object at 0x7f3c364d6ec8>)
ERROR:__main__:instagramtest:giving up

Request to add bot

Server: Spaceman's Retreat.
Contact: zzUrbanSpaceman#7994 on Discord is probably best, or via Github. I'd rather not put an email address that will be scraped and spammed in an issue.
Room ID: 321060041345990656

Thanks!

Make usable as a library

Make usable as a library that another python bot could call.

No more global code
Everything in a Class?
Class init for setting up all the things
move the stuff that sets up all the background coroutines to a function on its own
make it possible to pass in a data structure instead parsing a config file?
find somebody that wants to actually plug this crazy code into their bot.

imp.py:32: PendingDeprecationWarning

I've been using this bot for a week. past few hours i noticed my bot having yellow/Idle status (which usually show green/online status) so i hopped to check it's error. it says

/usr/lib/python3.4/imp.py:32: PendingDeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  PendingDeprecationWarning)

rest of log is https://pastebin.com/raw/EA2hReFj
please note that all line after Traceback (most recent call last): are appeared after i do keyboard interrupt

I already tried to reinstall all python modules including it's pip (i even do OS reinstall just to make sure). the problem still persist even after i do both above but:

the bot still running properly. can do fetching rss & post it to discord
the bot on another device (debian 8 vmware) has no this imp.py:32 error

heres my current version:
~/bot/discord_feedbot# pip freeze
appdirs==1.4.3
cffi==0.8.6
chardet==2.3.0
colorama==0.3.2
cryptography==0.6.1
dnspython==1.12.0
html5lib==0.999
ndg-httpsclient==0.3.2
packaging==16.8
ply==3.4
pyasn1==0.1.7
pycparser==2.10
pycrypto==2.6.1
pyOpenSSL==0.14
pyparsing==2.2.0
requests==2.4.3
six==1.10.0
urllib3==1.9.1
ws4py==0.3.5

is there any workaround to fix this? i'd like to see my bot status set to green/online instead of yellow after this error appeared.

thank you in advance

Fix date stuff

The date stuff doesn't seem to be working right...

Script skips everything (too old)

Not sure why this keeps happening, but every time I start the feed2discord.py, it loads up fine and connects to the bot account, and every time it reads the RSS URLs (just Reddit so far), it keeps throwing back similar line to this:

INFO:main:kpopreddit:item t3_43oas1 unseen, processing:
INFO:main:kpopreddit:too old; skipping

I have no idea why it keeps saying that all the posts are old, because one was made only minutes before on /r/kpop. The bot has all the required permissions to connect and post messages in the Discord channel too. It doesn't post anything from the feed in any of the channels and eventually it closes with:

/usr/lib/python3.5/asyncio/selector_events.py:577: ResourceWarning: unclosed transport <_SelectorSocketTransport fd=10>

If it's of anything, this is how I've got the feed in feed2discord.ini:

[kpopreddit]
channels = rkpop
feed_url = https://www.reddit.com/r/kpop/.rss
fields = link
max_age = 7200

As well as the Channels section:

[CHANNELS]
rkpop = 164987645401890816

[DEFAULT]
rss_refresh_time = 500
max_age = 86400
send_typing = 1

It does the same thing for all 3 entries I have in the CHANNELS section, which are all formatted similarly (and I figure it's correct, hopefully, with the last number in the channel's URL).

This is all running on a Ubuntu 14.04 server with Python 3.5.0 installed, for reference.

Sorry if this is long, but I'm sorta new to Github and needed some help with this. Thanks!

Use as a library

Anyone tried this?

If yes do you mind posting your code/tweaks?

I would like to do that but have no idea how.

invalid literal for int() with base 10: '0.1'

ERROR:asyncio:Task exception was never retrieved
future: <Task finished coro=<background_check_feed() done, defined at feed2discord.py:330> exception=ValueError("invalid literal for int() with base 10: '0.1'",)>
Traceback (most recent call last):
  File "E:\Python34\lib\asyncio\tasks.py", line 236, in _step
    result = next(coro)
  File "feed2discord.py", line 347, in background_check_feed
    start_skew_min = FEED.getint('start_skew_min',0.1)
  File "E:\Python34\lib\configparser.py", line 1236, in getint
    fallback=fallback)
  File "E:\Python34\lib\configparser.py", line 781, in getint
    return self._get(section, int, option, raw=raw, vars=vars)
  File "E:\Python34\lib\configparser.py", line 776, in _get
    return conv(self.get(section, option, **kwargs))
ValueError: invalid literal for int() with base 10: '0.1'
Traceback (most recent call last):
  File "feed2discord.py", line 710, in <module>
    loop.run_until_complete(client.connect())
  File "E:\Python34\lib\asyncio\base_events.py", line 263, in run_until_complete
    self.run_forever()
  File "E:\Python34\lib\asyncio\base_events.py", line 236, in run_forever
    self._run_once()
  File "E:\Python34\lib\asyncio\base_events.py", line 1017, in _run_once
    event_list = self._selector.select(timeout)
  File "E:\Python34\lib\selectors.py", line 311, in select
    r, w, _ = self._select(self._readers, self._writers, [], timeout)
  File "E:\Python34\lib\selectors.py", line 302, in _select
    r, w, x = select.select(r, w, w, timeout)

I have tried multiple versions of python allredy, latest, minimum, something in between

Procfile for deployment on heroku?

Hello, I'm trying to deploy this on heroku but it needs a procfile and i can't figure it out :(

Add call functions that can customize a RSS feed URL

I'm curious if/how this could be implemented in what you have given us. You could consider it an enhancement I suppose.

For example, a user types "~track (insert string here)" and have it append that string to the end of the URL.

Game status

Is there anyway to disable the game status of the bot so it doesn't takeover the status?

per room delays for feeds

After #25, do some feedname.delay=5 type stuff and replace the normal send with a coroutine that sleeps and then sends.

HTTPError() with many feeds on same host

Running into this with a bunch of reddit search feeds. Seems to mostly just be an issue at startup.

Add a pre-feed sleep? Make it randomized? Hmm...

function clean up

Should really have most line above and below your other functions also in functions, then call those at the bottom of the script using a

if  __name__ == "__main__":
    func1()
    func2()

from stack overflow: "By doing the main check, you can have that code only execute when you want to run the module as a program and not have it execute when someone just wants to import your module and call your functions themselves."

Needs better icons

The current icon options there kinda suck... Needs better icons...

HTTP error, most likely due to Reddit maintenance crashes bot.

As said above, my bot crashed last night around the time Reddit had maintenance that lasted for a few. I guess it didn't handle the error well? It was not functioning until I restarted it a few minutes ago. I could be wrong but that is around the time it broke. Here is the log below.

INFO:__main__:/r/Saber: processing feed
DEBUG:__main__:/r/Saber:setting up extra headers for HTTP request.
DEBUG:__main__:(None, None)
DEBUG:__main__:/r/Saber:no stored lastmodified
DEBUG:__main__:/r/Saber:no stored ETag
DEBUG:__main__:/r/Saber:sending http request for https://www.reddit.com/r/Saber/new/.rss
DEBUG:__main__:<ClientResponse(https://www.reddit.com/r/Saber/new/.rss) [504 Gateway Timeout]>
<CIMultiDictProxy('Content-Type': 'text/html; charset=UTF-8', 'Content-Length': '5768', 'Accept-Ranges': 'by                                                                                                                    tes', 'Date': 'Thu, 30 Mar 2017 05:00:53 GMT', 'Via': '1.1 varnish', 'Connection': 'keep-alive', 'X-Served-By': 'cache-or                                                                                                                    d1728-ORD', 'X-Cache': 'MISS', 'X-Cache-Hits': '0', 'X-Timer': 'S1490850053.100418,VS0,VE20', 'Cache-Control': 'private,                                                                                                                     max-age=3600', 'Server': 'snooserv')>

DEBUG:__main__:/r/Saber:HTTP error: 504
ERROR:__main__:/r/Saber:Unexpected error:
ERROR:__main__:Traceback (most recent call last):
File "/var/www/vhosts/<censored>/bots/saber_rss/discord_feedbot/feed2discord.py", line 488, in backgroun                                                                                                                    d_check_feed
  raise HTTPError()
File "/usr/local/lib/python3.5/dist-packages/aiohttp/web_exceptions.py", line 77, in __init__
  body=body, text=text, content_type=content_type)
File "/usr/local/lib/python3.5/dist-packages/aiohttp/web_reqrep.py", line 832, in __init__
  super().__init__(status=status, reason=reason, headers=headers)
File "/usr/local/lib/python3.5/dist-packages/aiohttp/web_reqrep.py", line 419, in __init__
  self.set_status(status, reason)
File "/usr/local/lib/python3.5/dist-packages/aiohttp/web_reqrep.py", line 463, in set_status
  self._status = int(status)
TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'

ERROR:__main__:/r/Saber:giving up
DEBUG:__main__:/r/Saber:sleeping for 60 seconds
ERROR:asyncio:Task exception was never retrieved
future: <Task finished coro=<background_check_feed() done, defined at /var/www/vhosts/<censored>/bots/sabe                                                                                                                    r_rss/discord_feedbot/feed2discord.py:354> exception=TypeError("int() argument must be a string, a bytes-like object or a                                                                                                                     number, not 'NoneType'",) created at /var/www/vhosts/<censored>/bots/saber_rss/discord_feedbot/feed2discord.py:772>
source_traceback: Object created at (most recent call last):
File "/var/www/vhosts/<censored>/bots/saber_rss/discord_feedbot/feed2discord.py", line 772, in <module>
  loop.create_task(background_check_feed(feed, loop))
Traceback (most recent call last):
File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
  result = coro.send(None)
File "/var/www/vhosts/<censored>/bots/saber_rss/discord_feedbot/feed2discord.py", line 488, in backgroun                                                                                                                    d_check_feed
  raise HTTPError()
File "/usr/local/lib/python3.5/dist-packages/aiohttp/web_exceptions.py", line 77, in __init__
  body=body, text=text, content_type=content_type)
File "/usr/local/lib/python3.5/dist-packages/aiohttp/web_reqrep.py", line 832, in __init__
  super().__init__(status=status, reason=reason, headers=headers)
File "/usr/local/lib/python3.5/dist-packages/aiohttp/web_reqrep.py", line 419, in __init__
  self.set_status(status, reason)
File "/usr/local/lib/python3.5/dist-packages/aiohttp/web_reqrep.py", line 463, in set_status
  self._status = int(status)
TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'
/usr/local/lib/python3.5/dist-packages/aiohttp/connector.py:59: ResourceWarning: Unclosed connection Connect                                                                                                                    ion<('www.reddit.com', 443, True)>
ResourceWarning)
ERROR:asyncio:Unclosed connection
client_connection: Connection<('www.reddit.com', 443, True)>
source_traceback: Object created at (most recent call last):
File "/var/www/vhosts/<censored>/bots/saber_rss/discord_feedbot/feed2discord.py", line 778, in <module>
  loop.run_until_complete(client.connect())
File "/usr/lib/python3.5/asyncio/base_events.py", line 375, in run_until_complete
  self.run_forever()
File "/usr/lib/python3.5/asyncio/base_events.py", line 345, in run_forever
  self._run_once()
File "/usr/lib/python3.5/asyncio/base_events.py", line 1304, in _run_once
  handle._run()
File "/usr/lib/python3.5/asyncio/events.py", line 125, in _run
  self._callback(*self._args)
File "/usr/lib/python3.5/asyncio/tasks.py", line 307, in _wakeup
  self._step()
File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
  result = coro.send(None)
File "/var/www/vhosts/<censored>/bots/saber_rss/discord_feedbot/feed2discord.py", line 465, in backgroun                                                                                                                    d_check_feed
  headers=http_headers)
File "/usr/local/lib/python3.5/dist-packages/aiohttp/client.py", line 555, in __iter__
  resp = yield from self._coro
File "/usr/local/lib/python3.5/dist-packages/aiohttp/client.py", line 198, in _request
  conn = yield from self._connector.connect(req)
File "/usr/local/lib/python3.5/dist-packages/aiohttp/connector.py", line 320, in connect
  conn = Connection(self, key, req, transport, proto, self._loop)
/usr/local/lib/python3.5/dist-packages/aiohttp/client_reqrep.py:585: ResourceWarning: Unclosed response <Cli                                                                                                                    entResponse(https://www.reddit.com/r/Saber/new/.rss) [504 Gateway Timeout]>
<CIMultiDictProxy('Content-Type': 'text/html; charset=UTF-8', 'Content-Length': '5768', 'Accept-Ranges': 'by                                                                                                                    tes', 'Date': 'Thu, 30 Mar 2017 05:00:53 GMT', 'Via': '1.1 varnish', 'Connection': 'keep-alive', 'X-Served-By': 'cache-or                                                                                                                    d1728-ORD', 'X-Cache': 'MISS', 'X-Cache-Hits': '0', 'X-Timer': 'S1490850053.100418,VS0,VE20', 'Cache-Control': 'private,                                                                                                                     max-age=3600', 'Server': 'snooserv')>

ResourceWarning)
ERROR:asyncio:Unclosed response
client_response: <ClientResponse(https://www.reddit.com/r/Saber/new/.rss) [504 Gateway Timeout]>
<CIMultiDictProxy('Content-Type': 'text/html; charset=UTF-8', 'Content-Length': '5768', 'Accept-Ranges': 'by                                                                                                                    tes', 'Date': 'Thu, 30 Mar 2017 05:00:53 GMT', 'Via': '1.1 varnish', 'Connection': 'keep-alive', 'X-Served-By': 'cache-or                                                                                                                    d1728-ORD', 'X-Cache': 'MISS', 'X-Cache-Hits': '0', 'X-Timer': 'S1490850053.100418,VS0,VE20', 'Cache-Control': 'private,                                                                                                                     max-age=3600', 'Server': 'snooserv')>

source_traceback: Object created at (most recent call last):
File "/var/www/vhosts/<censored>/bots/saber_rss/discord_feedbot/feed2discord.py", line 778, in <module>
  loop.run_until_complete(client.connect())
File "/usr/lib/python3.5/asyncio/base_events.py", line 375, in run_until_complete
  self.run_forever()
File "/usr/lib/python3.5/asyncio/base_events.py", line 345, in run_forever
  self._run_once()
File "/usr/lib/python3.5/asyncio/base_events.py", line 1304, in _run_once
  handle._run()
File "/usr/lib/python3.5/asyncio/events.py", line 125, in _run
  self._callback(*self._args)
File "/usr/lib/python3.5/asyncio/tasks.py", line 307, in _wakeup
  self._step()
File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
  result = coro.send(None)
File "/var/www/vhosts/<censored>/bots/saber_rss/discord_feedbot/feed2discord.py", line 465, in backgroun                                                                                                                    d_check_feed
  headers=http_headers)
File "/usr/local/lib/python3.5/dist-packages/aiohttp/client.py", line 555, in __iter__
  resp = yield from self._coro
File "/usr/local/lib/python3.5/dist-packages/aiohttp/client.py", line 200, in _request
  resp = req.send(conn.writer, conn.reader)
File "/usr/local/lib/python3.5/dist-packages/aiohttp/client_reqrep.py", line 517, in send
  self.response._post_init(self.loop)
WARNING:asyncio:Executing <Task pending coro=<Client.connect() running at /usr/local/lib/python3.5/dist-pack                                                                                                                    ages/discord/client.py:414> wait_for=<Future pending cb=[Task._wakeup()] created at /usr/lib/python3.5/asyncio/base_event                                                                                                                    s.py:252> cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164] created at /usr/lib/python3.5/as                                                                                                                    yncio/base_events.py:367> took 0.104 seconds
WARNING:asyncio:Executing <Task pending coro=<Client.connect() running at /usr/local/lib/python3.5/dist-pack                                                                                                                    ages/discord/client.py:414> wait_for=<Future pending cb=[Task._wakeup()] created at /usr/lib/python3.5/asyncio/base_event                                                                                                                    s.py:252> cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164] created at /usr/lib/python3.5/as                                                                                                                    yncio/base_events.py:367> took 0.125 seconds
INFO:__main__:Logged in as 'Saber' ('168018476525551616')
WARNING:asyncio:Executing <Task pending coro=<Client.connect() running at /usr/local/lib/python3.5/dist-pack                                                                                                                    ages/discord/client.py:414> wait_for=<Future pending cb=[Task._wakeup()] created at /usr/lib/python3.5/asyncio/base_event                                                                                                                    s.py:252> cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164] created at /usr/lib/python3.5/as                                                                                                                    yncio/base_events.py:367> took 0.135 seconds
INFO:__main__:Logged in as 'Saber' ('168018476525551616')
WARNING:asyncio:Executing <Task pending coro=<Client.connect() running at /usr/local/lib/python3.5/dist-pack                                                                                                                    ages/discord/client.py:414> wait_for=<Future pending cb=[Task._wakeup()] created at /usr/lib/python3.5/asyncio/base_event                                                                                                                    s.py:252> cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164] created at /usr/lib/python3.5/as                                                                                                                    yncio/base_events.py:367> took 0.120 seconds
INFO:__main__:Logged in as 'Saber' ('168018476525551616')
WARNING:asyncio:Executing <Task pending coro=<Client.connect() running at /usr/local/lib/python3.5/dist-pack                                                                                                                    ages/discord/client.py:414> wait_for=<Future pending cb=[Task._wakeup()] created at /usr/lib/python3.5/asyncio/base_event                                                                                                                    s.py:252> cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164] created at /usr/lib/python3.5/as                                                                                                                    yncio/base_events.py:367> took 0.126 seconds
INFO:__main__:Logged in as 'Saber' ('168018476525551616')
WARNING:asyncio:Executing <Task pending coro=<Client.connect() running at /usr/local/lib/python3.5/dist-pack                                                                                                                    ages/discord/client.py:414> wait_for=<Future pending cb=[Task._wakeup()] created at /usr/lib/python3.5/asyncio/base_event                                                                                                                    s.py:252> cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164] created at /usr/lib/python3.5/as                                                                                                                    yncio/base_events.py:367> took 0.130 seconds
INFO:__main__:Logged in as 'Saber' ('168018476525551616')
WARNING:asyncio:Executing <Task pending coro=<Client.connect() running at /usr/local/lib/python3.5/dist-pack                                                                                                                    ages/discord/client.py:414> wait_for=<Future pending cb=[Task._wakeup()] created at /usr/lib/python3.5/asyncio/base_event                                                                                                                    s.py:252> cb=[_run_until_complete_cb() at /usr/lib/python3.5/asyncio/base_events.py:164] created at /usr/lib/python3.5/as                                                                                                                    yncio/base_events.py:367> took 0.145 seconds
INFO:__main__:Logged in as 'Saber' ('168018476525551616')

Add ETag and/or Last-Modified-Date support to be nicer to servers

http://pythonhosted.org/feedparser/http-etag.html

Multiple feeds

Make capable of handling multiple feeds.

Unable to run 'invalid syntax'

When trying to run this in Ubuntu 16.04 server, I get the following error;

File "feed2discord.py", line 311
yield from client.send_typing(channel['object'])
^
Any thoughts? I can't see anything else about it.

game played Invalid

failing to set the game played and continuing to run!

Logging

Need to do all the magic logging things, so I can see what's actually going on inside coroutines, etc...

"unclosed transport" error from asyncio.sslproto

On windows, I have this error every time discord-bot-rss try to add a rss link on Discord:
C:\Users\my_name\AppData\Local\Programs\Python\Python35-32\lib\asyncio\sslproto.py:327 ResourceWarning: unclosed transport <asyncio.sslproto._SSLProtocolTransport object at 0x04D1FAFO> warning.warn(2unclosed transport %r" % self, ResourceWarning)
(04D1FAFO changes every time the bot try to add a feed)

I have upgrade asyncio to the latest version but this issue still remains.

Edit titles (or other fields) with regular expression (or similar)

As per our discussion, it would be a good feature to be able to modify the contents of a title or description for a feed.

Example:
Original Title: #IceHockey #Livescore @ScoresPro: (SUI-) #ZSC Lions vs #SC Bern: 2-1 E/p2

New Title: @ScoresPro: (SUI-) #ZSC Lions vs #SC Bern: 2-1 End of Period 2

HTML to markdown filter.

Currently looks for a single html thing (br) and changes to markdown (newline). Non-galnet feeds will probably have more html.

Discord markdown is limited; just look for very limited list and let other html tags through? Use a full-fledged html2markdown filter?

Fix galnet special case

Galnet doesn't include url, but Guid is end of url. Make that behavior per-feed configurable and use url from feed when available.

Max items per run

Add a feature to allow specifying a maximum number of items to post from each run of a feed. This could be used as alternative to max_age to limit verbosity when setting up a new feed.

Bot doesn't work avec last pull with python3.5

Hello,

The bot doesn't work any more after the last pull.

python3.5 feed2discord.py
Error: requires python 3.4 or newer

Python 3.5.2 (default, Jun 28 2016, 08:46:01)

Any ideas?

Fix DB schema

There's some holdover from prior iterations, that doesn't really make sense (fields we no longer assume are always there, etc).

Should it just be: feed_id, item_id, pubDate ?

Date checking

Add some date/time checking logic, to only let through posts with a maximum age.

ERROR:asyncio:Task exception was never retrieved future: <Task finished coro=<actually_send_message() done, defined at feed2discord.py:310> exception=NotFound('NOT FOUND (status code: 404): 404: Not Found ',)>

"Huge" log below: (it's not really that bad)


C:\discord_feedbot-master\discord_feedbot-master>python feed2discord.py
WARNING:discord.client:PyNaCl is not installed, voice will NOT be supported
INFO:__main__:discordreddit: Starting up background_check_feed
INFO:__main__:Logged in as 'Reddit Aww Feed' ('259129586753732609')
INFO:__main__:discordreddit: processing feed
INFO:__main__:discordreddit:item t3_5ilukr unseen, processing:
INFO:__main__:discordreddit:item:fresh and ready for parsing
INFO:__main__:discordreddit:item t3_5iluod unseen, processing:
INFO:__main__:discordreddit:item:fresh and ready for parsing
INFO:__main__:discordreddit:item t3_5iluxf unseen, processing:
INFO:__main__:discordreddit:item:fresh and ready for parsing
INFO:__main__:discordreddit:item t3_5iluzp unseen, processing:
INFO:__main__:discordreddit:item:fresh and ready for parsing
INFO:__main__:discordreddit:item t3_5ilv50 unseen, processing:
INFO:__main__:discordreddit:item:fresh and ready for parsing
INFO:__main__:discordreddit:item t3_5ilvbp unseen, processing:
INFO:__main__:discordreddit:item:fresh and ready for parsing
INFO:__main__:discordreddit:item t3_5ilvgq unseen, processing:
INFO:__main__:discordreddit:item:fresh and ready for parsing
INFO:__main__:discordreddit:item t3_5ilvjh unseen, processing:
INFO:__main__:discordreddit:item:fresh and ready for parsing
ERROR:asyncio:Task exception was never retrieved
future: <Task finished coro=<actually_send_message() done, defined at feed2discord.py:310> exception=NotFound('NOT FOUND (status code: 404): 404: Not Found
',)>
Traceback (most recent call last):
  File "C:\Program Files (x86)\Python 3.5\lib\asyncio\tasks.py", line 239, in _step
    result = coro.send(value)
  File "feed2discord.py", line 321, in actually_send_message
    yield from client.send_message(channel['object'],message)
  File "C:\Program Files (x86)\Python 3.5\lib\site-packages\discord\client.py", line 1152, in send_message
    data = yield from self.http.send_message(channel_id, content, guild_id=guild_id, tts=tts, embed=embed)
  File "C:\Program Files (x86)\Python 3.5\lib\site-packages\discord\http.py", line 135, in request
    raise NotFound(r, data)
discord.errors.NotFound: NOT FOUND (status code: 404): 404: Not Found
ERROR:asyncio:Task exception was never retrieved
future: <Task finished coro=<actually_send_message() done, defined at feed2discord.py:310> exception=NotFound('NOT FOUND (status code: 404): 404: Not Found
',)>
Traceback (most recent call last):
  File "C:\Program Files (x86)\Python 3.5\lib\asyncio\tasks.py", line 239, in _step
    result = coro.send(value)
  File "feed2discord.py", line 321, in actually_send_message
    yield from client.send_message(channel['object'],message)
  File "C:\Program Files (x86)\Python 3.5\lib\site-packages\discord\client.py", line 1152, in send_message
    data = yield from self.http.send_message(channel_id, content, guild_id=guild_id, tts=tts, embed=embed)
  File "C:\Program Files (x86)\Python 3.5\lib\site-packages\discord\http.py", line 135, in request
    raise NotFound(r, data)
discord.errors.NotFound: NOT FOUND (status code: 404): 404: Not Found
ERROR:asyncio:Task exception was never retrieved
future: <Task finished coro=<actually_send_message() done, defined at feed2discord.py:310> exception=NotFound('NOT FOUND (status code: 404): 404: Not Found
',)>
Traceback (most recent call last):
  File "C:\Program Files (x86)\Python 3.5\lib\asyncio\tasks.py", line 239, in _step
    result = coro.send(value)
  File "feed2discord.py", line 321, in actually_send_message
    yield from client.send_message(channel['object'],message)
  File "C:\Program Files (x86)\Python 3.5\lib\site-packages\discord\client.py", line 1152, in send_message
    data = yield from self.http.send_message(channel_id, content, guild_id=guild_id, tts=tts, embed=embed)
  File "C:\Program Files (x86)\Python 3.5\lib\site-packages\discord\http.py", line 135, in request
    raise NotFound(r, data)
discord.errors.NotFound: NOT FOUND (status code: 404): 404: Not Found
ERROR:asyncio:Task exception was never retrieved
future: <Task finished coro=<actually_send_message() done, defined at feed2discord.py:310> exception=NotFound('NOT FOUND (status code: 404): 404: Not Found
',)>
Traceback (most recent call last):
  File "C:\Program Files (x86)\Python 3.5\lib\asyncio\tasks.py", line 239, in _step
    result = coro.send(value)
  File "feed2discord.py", line 321, in actually_send_message
    yield from client.send_message(channel['object'],message)
  File "C:\Program Files (x86)\Python 3.5\lib\site-packages\discord\client.py", line 1152, in send_message
    data = yield from self.http.send_message(channel_id, content, guild_id=guild_id, tts=tts, embed=embed)
  File "C:\Program Files (x86)\Python 3.5\lib\site-packages\discord\http.py", line 135, in request
    raise NotFound(r, data)
discord.errors.NotFound: NOT FOUND (status code: 404): 404: Not Found
ERROR:asyncio:Task exception was never retrieved
future: <Task finished coro=<actually_send_message() done, defined at feed2discord.py:310> exception=NotFound('NOT FOUND (status code: 404): 404: Not Found
',)>
Traceback (most recent call last):
  File "C:\Program Files (x86)\Python 3.5\lib\asyncio\tasks.py", line 239, in _step
    result = coro.send(value)
  File "feed2discord.py", line 321, in actually_send_message
    yield from client.send_message(channel['object'],message)
  File "C:\Program Files (x86)\Python 3.5\lib\site-packages\discord\client.py", line 1152, in send_message
    data = yield from self.http.send_message(channel_id, content, guild_id=guild_id, tts=tts, embed=embed)
  File "C:\Program Files (x86)\Python 3.5\lib\site-packages\discord\http.py", line 135, in request
    raise NotFound(r, data)
discord.errors.NotFound: NOT FOUND (status code: 404): 404: Not Found
ERROR:asyncio:Task exception was never retrieved
future: <Task finished coro=<actually_send_message() done, defined at feed2discord.py:310> exception=NotFound('NOT FOUND (status code: 404): 404: Not Found
',)>
Traceback (most recent call last):
  File "C:\Program Files (x86)\Python 3.5\lib\asyncio\tasks.py", line 239, in _step
    result = coro.send(value)
  File "feed2discord.py", line 321, in actually_send_message
    yield from client.send_message(channel['object'],message)
  File "C:\Program Files (x86)\Python 3.5\lib\site-packages\discord\client.py", line 1152, in send_message
    data = yield from self.http.send_message(channel_id, content, guild_id=guild_id, tts=tts, embed=embed)
  File "C:\Program Files (x86)\Python 3.5\lib\site-packages\discord\http.py", line 135, in request
    raise NotFound(r, data)
discord.errors.NotFound: NOT FOUND (status code: 404): 404: Not Found
ERROR:asyncio:Task exception was never retrieved
future: <Task finished coro=<actually_send_message() done, defined at feed2discord.py:310> exception=NotFound('NOT FOUND (status code: 404): 404: Not Found
',)>
Traceback (most recent call last):
  File "C:\Program Files (x86)\Python 3.5\lib\asyncio\tasks.py", line 239, in _step
    result = coro.send(value)
  File "feed2discord.py", line 321, in actually_send_message
    yield from client.send_message(channel['object'],message)
  File "C:\Program Files (x86)\Python 3.5\lib\site-packages\discord\client.py", line 1152, in send_message
    data = yield from self.http.send_message(channel_id, content, guild_id=guild_id, tts=tts, embed=embed)
  File "C:\Program Files (x86)\Python 3.5\lib\site-packages\discord\http.py", line 135, in request
    raise NotFound(r, data)
discord.errors.NotFound: NOT FOUND (status code: 404): 404: Not Found
ERROR:asyncio:Task exception was never retrieved
future: <Task finished coro=<actually_send_message() done, defined at feed2discord.py:310> exception=NotFound('NOT FOUND (status code: 404): 404: Not Found
',)>
Traceback (most recent call last):
  File "C:\Program Files (x86)\Python 3.5\lib\asyncio\tasks.py", line 239, in _step
    result = coro.send(value)
  File "feed2discord.py", line 321, in actually_send_message
    yield from client.send_message(channel['object'],message)
  File "C:\Program Files (x86)\Python 3.5\lib\site-packages\discord\client.py", line 1152, in send_message
    data = yield from self.http.send_message(channel_id, content, guild_id=guild_id, tts=tts, embed=embed)
  File "C:\Program Files (x86)\Python 3.5\lib\site-packages\discord\http.py", line 135, in request
    raise NotFound(r, data)
discord.errors.NotFound: NOT FOUND (status code: 404): 404: Not Found

Any idea what's going on here and why it's not working/throwing errors? What is it saying isn't found?

Add ability to set formatting around certain fields

Currently we have a bunch of galnet-specific things hard-coded in. When we switch to a field list for each feed, I'd still like to be able to italicize the date, bold the title, etc...

Insane Traffic

Hello,
I use your discord_feedbot on a Linux server and monitor 2 different RSS feeds with the status of my host systems and VMs. But after 2-4 h the Bot uses a lot of traffic. And I mean a real lot. Constantly 200 - 250 mbit/s. If I'm not careful that eats away a few hundreds of Gigabyte easily. Any Ideas on how to solve this?

Asyncio - Unclosed Transport

WARNING:discord.client:PyNaCl is not installed, voice will NOT be supported
INFO:__main__:ednews: Starting up background_check_feed
INFO:__main__:discordreddit: Starting up background_check_feed
INFO:__main__:galnet: Starting up background_check_feed
INFO:__main__:gazelle: Starting up background_check_feed
D:\AppData\Local\Programs\Python\Python35-32\lib\asyncio\selector_events.py:581:
 ResourceWarning: unclosed transport <_SelectorSocketTransport fd=644>
  warnings.warn("unclosed transport %r" % self, ResourceWarning)
D:\AppData\Local\Programs\Python\Python35-32\lib\site-packages\aiohttp\client.py
:88: ResourceWarning: Unclosed client session <aiohttp.client.ClientSession obje
ct at 0x035D3350>
ERROR:asyncio:Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x035D3350>
ERROR:asyncio:Task was destroyed but it is pending!
task: <Task pending coro=<background_check_feed() running at feed2discord.py:335
> wait_for=<Future pending cb=[Task._wakeup()]>>
ERROR:asyncio:Task was destroyed but it is pending!
task: <Task pending coro=<background_check_feed() running at feed2discord.py:335
> wait_for=<Future pending cb=[Task._wakeup()]>>
ERROR:asyncio:Task was destroyed but it is pending!
task: <Task pending coro=<background_check_feed() running at feed2discord.py:335
> wait_for=<Future pending cb=[Task._wakeup()]>>
ERROR:asyncio:Task was destroyed but it is pending!
task: <Task pending coro=<background_check_feed() running at feed2discord.py:335
> wait_for=<Future pending cb=[Task._wakeup()]>>

D:\Tutorials\discord_feedbot-master>python feed2discord.py
WARNING:discord.client:PyNaCl is not installed, voice will NOT be supported
INFO:__main__:ednews: Starting up background_check_feed
INFO:__main__:discordreddit: Starting up background_check_feed
INFO:__main__:galnet: Starting up background_check_feed
INFO:__main__:gazelle: Starting up background_check_feed
D:\AppData\Local\Programs\Python\Python35-32\lib\asyncio\selector_events.py:581:
 ResourceWarning: unclosed transport <_SelectorSocketTransport fd=644>
  warnings.warn("unclosed transport %r" % self, ResourceWarning)
D:\AppData\Local\Programs\Python\Python35-32\lib\site-packages\aiohttp\client.py
:88: ResourceWarning: Unclosed client session <aiohttp.client.ClientSession obje
ct at 0x03433350>
ERROR:asyncio:Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x03433350>
ERROR:asyncio:Task was destroyed but it is pending!
task: <Task pending coro=<background_check_feed() running at feed2discord.py:335
> wait_for=<Future pending cb=[Task._wakeup()]>>
ERROR:asyncio:Task was destroyed but it is pending!
task: <Task pending coro=<background_check_feed() running at feed2discord.py:335
> wait_for=<Future pending cb=[Task._wakeup()]>>
ERROR:asyncio:Task was destroyed but it is pending!
task: <Task pending coro=<background_check_feed() running at feed2discord.py:335
> wait_for=<Future pending cb=[Task._wakeup()]>>
ERROR:asyncio:Task was destroyed but it is pending!
task: <Task pending coro=<background_check_feed() running at feed2discord.py:335
> wait_for=<Future pending cb=[Task._wakeup()]>>

asyncio issue?

Use asyncio for feed fetching.

Required for scaling up to handling many feeds and many channels efficiently.

Likely tricky to combine properly with #9...

DB cleanup

Delete rows from feed_items that are older than X*max_age?

Add ability to filter out specific items by regex

Example: galnet has items titled "Galactic News: Weekly Security Digest", "Galactic News: Weekly Economic Report", "Galactic News: Weekly Expansion List", and "Galactic News: Weekly Conflict Report", and those tend not to be particularly useful...

Consider including an option to control whether items are discarded completely, or just get a different set of fields (exclude the long description and just drop a URL and title for less annoyance)

Per-room config of feeds

Thinking something like room1.fields in the feed to give a specific discord room different config. Most likely to add a "@role" type thing initially...

ERROR:asyncio:Unclosed client session

C:\Python34\discord\discord_feedbot>python feed2discord.py
WARNING:discord.client:PyNaCl is not installed, voice will NOT be supported
INFO:main:ednews: Starting up background_check_feed
INFO:main:status: Starting up background_check_feed
C:\Python34\lib\site-packages\aiohttp-1.0.5-py3.4.egg\aiohttp\client.py:88: Reso
urceWarning: Unclosed client session <aiohttp.client.ClientSession object at 0x0
000000003BBE6D8>
C:\Python34\lib\importlib_bootstrap.py:2150: ImportWarning: sys.meta_path is em
pty
ERROR:asyncio:Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x0000000003BBE6D8>
ERROR:asyncio:Task was destroyed but it is pending!
task: <Task pending coro=<background_check_feed() running at feed2discord.py:331

wait_for=>
ERROR:asyncio:Task was destroyed but it is pending!
task: <Task pending coro=<background_check_feed() running at feed2discord.py:331
wait_for=>

"fields = dc:creator" not working

ERROR:main:process_field:dc:creator:no such field

source: http://twitrss.me/

My config:

[twitter]
channels = general
feed_url = http://twitrss.me/twitter_user_to_rss/?user=twitter
fields = **dc:creator**,description,link
max_age = 3600

Can't see if it is a re-tweet or not

Config file

Move config stuff out of main code (so I can run direct from repo)

bot not running

set right token.. time zone etc etc

here's the windows cmd window

C:\Program Files\Python35\lib\site-packages\discord__init__.py:47: UserWarning:
The next major version of discord.py (v0.10.0) will have major breaking changes
that will require updating/changing your code.
Please check the migrating guide to alleviate yourself of unexpected issues.
http://discordpy.readthedocs.org/en/latest/migrating.html
It is strongly recommended to make the switch as soon as possible.

warnings.warn(_warning_message, UserWarning)
Traceback (most recent call last):
File "feed2discord.py", line 490, in
@client.async_event
File "C:\Program Files\Python35\lib\site-packages\discord\client.py", line 523, in getattr
raise AttributeError(msg.format(self.class, name))
AttributeError: '<class 'discord.client.Client'>' object has no attribute 'async_event'
C:\Program Files\Python35\lib\site-packages\aiohttp\client.py:86: ResourceWarning: Unclosed client session <aiohttp.client.ClientSession object at 0x000001DCFF8DC278>
ERROR:asyncio:Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x000001DCFF8DC278>
C:\Program Files\Python35\lib\asyncio\base_events.py:379: ResourceWarning: unclosed event loop <_WindowsSelectorEventLoop running=False closed=False debug=False>

C:\Users\sirfacealot\Documents\discord_rss_bot>

Figure out complicated item fields

Handle stuff like content:encoded that gets turned into a dictionary.
Support more complicated field names like <itunes:summary> (this might already work)
Model these more complicated fields/attributes in the config
Support nested stuff in the items (accessible via item['outerField']['innerField']).
Figure out how to get at interesting data when it is in an attribute of a field instead of contained by it (<enclosure url="..." />).

ResourceWarning: unclosed transport

I am not able to get the bot running. Getting the error messages below. I'd appreciate your help in understanding what went wrong.

D:\discord_feedbot-master>feed2discord.py
C:\Python35\lib\asyncio\selector_events.py:581: ResourceWarning: unclosed transport <_SelectorSocketTransport fd=716>
warnings.warn("unclosed transport %r" % self, ResourceWarning)
C:\Python35\lib\site-packages\aiohttp\client.py:88: ResourceWarning: Unclosed client session <aiohttp.client.ClientSession object at 0x03F79930>
ERROR:asyncio:Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x03F79930>
ERROR:asyncio:Task was destroyed but it is pending!
task: <Task pending coro=<background_check_feed() running at D:\discord_feedbot-master\feed2discord.py:402> wait_for=>
ERROR:asyncio:Task was destroyed but it is pending!
task: <Task pending coro=<background_check_feed() running at D:\discord_feedbot-master\feed2discord.py:402> wait_for=>
ERROR:asyncio:Task was destroyed but it is pending!
task: <Task pending coro=<background_check_feed() running at D:\discord_feedbot-master\feed2discord.py:402> wait_for=>