0x111 / telegram-rss-bot Goto Github PK

View Code? Open in Web Editor NEW

41.0 2.0 21.0 24 KB

RSS bot for telegram written in Go

License: MIT License

Go 90.87% Makefile 4.98% Dockerfile 0.94% Shell 3.21%

golang go telegram telegram-bot telegram-bots rss-feed rss-feed-parser

telegram-rss-bot's Introduction

telegram-rss-bot

Introduction

This is an another telegram bot for usage with RSS feeds.

First steps

To use this, you first need to register a telegram bot by reading the documentation: https://core.telegram.org/bots#3-how-do-i-create-a-bot

For this bot to work, you will need a token which authorizes you to use the telegram api.

First run

After you have the token, you should create a copy of the sample file and fill it out accordingly.

Configuration

Rename bot-config.sample.json to bot-config.json.

You can put the config file in the current folder on where the binary resides or put it in this folder if created $HOME/.telegram-rss-bot, the app should be able to find it here too if you want to have some fixed location for your configuration files.

{
  "telegram_auth_key": "token",
  "migrations": "v1",
  "telegram_api_debug": false,
  "db_path": "./bot.db",
  "log_level": "info",
  "feed_parse_amount": 5,
  "feed_post_amount": 2,
  "feed_updates_interval": 600,
  "feed_posts_interval": 400
}

telegram_auth_key: is the token which you've got from registering the bot
migrations: this will be used mostly in the future, to define which migration to run (this can change)
telegram_api_debug: if this is turned on, you will see debug messages from the telegram api on your stdout
db_path: this contains a path to the db file, if this does not exist, it will be created if the app will have permission to do that
log_level: with this, you can set the log level to display, the app is using logrus for logging, so this is accepting all the values from this url https://github.com/sirupsen/logrus#level-logging
feed_parse_amount: this represents the amount of how much of the feed items should be parsed from the provided feed url (e.g. if you provide a url, which has 10 items, then only the 5 latest will be saved to our database, you can alter this value if needed)
feed_post_amount: this represents the amount of how much of the parsed feed data should be posted to their respective channels (e.g. if you set this to 2, the bot will post every $feed_posts_interval only 2 entries, you can alter value if needed)
feed_updates_interval: this represents the interval at which rate the feeds saved in the database should be updated in seconds (60 = 60 seconds and so on)
feed_posts_interval: this represents the interval at which rate the feeds should be posted to their respective channels in seconds (60 = 60 seconds and so on)

Docker support

You can also run this application as a docker container.

Docker hub

You can pull the official docker image

docker pull ruthless/telegram-rss-bot
docker run -e TELEGRAM_AUTH_KEY="MY-TOKEN" ruthless/telegram-rss-bot

Build from source

Execute the following steps:

git clone https://github.com/0x111/telegram-rss-bot
docker build -t telegram-rss-bot:latest .
docker run --name telegram-rss-bot -e TELEGRAM_AUTH_KEY="MY-TOKEN" -d telegram-rss-bot:latest

Important

Advisory: You should respect the rate limiting of the Telegram API (More info about this: https://core.telegram.org/bots/faq#my-bot-is-hitting-limits-how-do-i-avoid-this)

Feel free to open a PR if you find some bugs or have improvements (I am sure there can be many of those :))

If you find bugs but you have no idea how to fix them, please open an issue with a detailed description on how to reproduce the bug.

telegram-rss-bot's People

Contributors

Stargazers

Watchers

telegram-rss-bot's Issues

Show feed name on messages

It's difficult to recognize what message is from which feed when subscribing to some social network service (for example, twitter which provide "rt", and it's link to the original post when using all 3rd party feed extractor I ever found).

Support public channels.

Implement /start

https://github.com/0x111/telegram-rss-bot/blob/master/commands/command.go

As a manpage.

Export feeds as opml

It's better to allow users to backup their subscribed feeds.

Feed duplication

The implementation was done in a more of personal look, so I did not think about this while development.

At the moment, we are checking if the feed already exists for current channel, this is fine, but we would need to extend this logic.
One can add the same feed url many times, this would mean the app would load the same feed n times over and over, although on a different channel. This is obviously a waste of resources.

We need a way, to determine if the feed already exists (it does not matter if on a different channel) and just reuse that, instead of loading the same feed many times, we would do this once only and improve the process.

K8s/OpenShift deployment

First of all, thank you very much for working on this and sharing with the community. I was looking for something on golang for maintaining RSS feeds on Telegram, but ended up wasting my time on not so useful things in other language.

I took your code and made changes to make it deployable on k8s/OpenShift environments. Would you be interested in a PR with such changes?

HEROKU ?

Post in order of date.

Right now, feed items are posted in the order they are seen in, posting newer items first, and older items later. It makes sense to do the reverse.

Getting more advanced, get the date of post from the feed.

Database size

Right now, we have a .db file where we store all the data in a sqlite database format. This works well, because users don't need to run a separate database server and can use a file based database basically anywhere without extra installation steps.

The database grows over time, depending on how many feeds do you have and how often they publish new entries. We need a cleanup for old records and simply remove them periodically to make the database size as small as possible.

We do not need old entries anyways for long-term. If this is a problem it could be a flag, to turn it on/off based on user preference. Someone might want to save the history maybe.

INFO[3200] Updating feeds                                feedID=1 feedUrl="https://rss.golem.de/rss.php"
2020/05/29 12:14:59 sendMessage resp: {"ok":false,"error_code":400,"description":"Bad Request: can't parse entities: Can't find end of the entity starting at byte offset 197"}
ERRO[3200] There was an error while posting the update to the feed!  chatID=43919129 error="Bad Request: can't parse entities: Can't find end of the entity starting at byte offset 197" feedPost="{7 2 Am Freitag online: WeAreDevelopers Live Week – u.a. mit GraphQL und Chatbots https://www.heise.de/news/Am-Freitag-online-WeAreDevelopers-Live-Week-u-a-mit-GraphQL-und-Chatbots-4726708.html?wt_mc=rss.red.ho.top-news.atom.beitrag.beitrag false 0001-01-01 00:00:00 +0000 UTC 43919129}"
2020/05/29 12:14:59 sendMessage resp: {"ok":false,"error_code":400,"description":"Bad Request: can't parse entities: Can't find end of the entity starting at byte offset 145"}
ERRO[3200] There was an error while posting the update to the feed!  chatID=43919129 error="Bad Request: can't parse entities: Can't find end of the entity starting at byte offset 145" feedPost="{8 2 Automatisierte Sicherheit für autonome Fahrzeuge https://www.heise.de/news/Automatisierte-Sicherheit-fuer-autonome-Fahrzeuge-4768764.html?wt_mc=rss.red.ho.top-news.atom.beitrag.beitrag false 0001-01-01 00:00:00 +0000 UTC 43919129}"
INFO[3200] Updating feeds                                feedID=2 feedUrl="https://www.heise.de/rss/heise-top-atom.xml"

Ask the user to populate feed on add.

Current state: Nothing happens when a feed is added. Every time the feed is refreshed, depending on the config, 2 items get posted to the channel, until all old items are posted.

Proposed:

After feed is added, ask the user for yes/no (buttons!) whether to get existing items, or only new items from the current moment forward (mark all items in feed 'posted').
If posting existing items, post all at once.