maybe-hello-world / digestbot Goto Github PK

View Code? Open in Web Editor NEW

16.0 5.0 4.0 337 KB

Slack bot that provide you a digest of hot workspace topics

License: MIT License

Python 98.21% Dockerfile 1.64% Shell 0.15%

digestbot's Introduction

digestbot

Bot for Slack, who send you top messages whenever you want!

What bot can do:

find and send you top messages from a given channel, pre-defined sets of channels or all channels
select messages for last N hours / days / weeks
sort messages by 3 different sorting types
create timers for you to receive a digest on schedule
create your own presets to combine channels in one group

Can I use this bot for my own space?

Sure! To deploy your bot you have to:

Create a Slack app in your workspace and get USER_SLACK_TOKEN and BOT_SLACK_TOKEN
Prepare needed environment variables in .env file (see env.template)
Deploy this bot using the docker-compose.yml file with the bot and database
- You can modify docker-compose file how you want (provide a database from another host, for example)
- Do not forget to pass environment variables to bot from .env file or in any other way
Done!

How can I help?

If you have any good ideas or noticed a bug - create a new issue, and we will talk about it! Before submitting the issue, please check ROADMAP.md - possibly we are already working on that. :)

Doesn't know what to do? Check issues and implement a solution for one.

Anyway, Pull Requests are welcome!

digestbot's People

Contributors

Stargazers

Watchers

Forkers

varan42 sergey778 zergey datahack-ru

digestbot's Issues

Slack API timeout handling

Handle timeout of connection to slack API

Slack slash commands instead of current RTM

Migrate from the custom command_parser on top of the RTM client to normal slash commands

Получение информации о тредах

прикрутить conversations.replies к скрипту и вытаскивать информацию о тредах

Migrate DB interaction to separate container with API

It's better to separate all interactions with a database into one container and provide API for other services there.
It will consolidate interaction with DB in one place, remove current problems with tables creation and migrations, and support future DB access for other services (as Q&A bot, for example)

Migrate tests to folders with corresponding services

Add users own categories

We should add users possibility to add their own categories which will be available only for them

Get more reaction weights

We need to add more ODS-specific reactions weights to reactions dictionary for better reaction_rate calculations of messages.

Handle with Retry-After responses from Slack API

If you send too many requests to Slack API sometimes it returns 429 Retry-After response. We should handle these responses and sleep (async) for given amount of seconds (or maybe return to user 'sorry, too much requests' answer).

not working with docker compose.

Hello,

I'm very exciting to find this wonderful project.
So I tried to run digestbot from my Slack Workspace.
However, the code of the Master Branch seems to run normally.
I used Docker Compose to run this, and the Commit ID of the Master branch I used is 480e903.

Below are the error log I have experienced.
Can I get help to solve this problem?

Thanks.

$ docker compose up
[+] Running 6/6
 ⠿ Network digestbot_default        Created                                                                                                                                                                    4.8s
 ⠿ Container digestbot_postgres_1   Created                                                                                                                                                                    0.0s
 ⠿ Container digestbot_dbservice_1  Created                                                                                                                                                                    0.0s
 ⠿ Container digestbot_crawler_1    Created                                                                                                                                                                    0.0s
 ⠿ Container digestbot_uiservice_1  Created                                                                                                                                                                    0.0s
 ⠿ Container digestbot_timers_1     Created                                                                                                                                                                    0.0s
Attaching to crawler_1, dbservice_1, postgres_1, timers_1, uiservice_1
postgres_1   | The files belonging to this database system will be owned by user "postgres".
postgres_1   | This user must also own the server process.
postgres_1   |
postgres_1   | The database cluster will be initialized with locale "en_US.utf8".
postgres_1   | The default database encoding has accordingly been set to "UTF8".
postgres_1   | The default text search configuration will be set to "english".
postgres_1   |
postgres_1   | Data page checksums are disabled.
postgres_1   |
postgres_1   | fixing permissions on existing directory /var/lib/postgresql/data ... ok
postgres_1   | creating subdirectories ... ok
postgres_1   | selecting dynamic shared memory implementation ... posix
postgres_1   | selecting default max_connections ... 100
postgres_1   | selecting default shared_buffers ... 128MB
postgres_1   | selecting default time zone ... UTC
postgres_1   | creating configuration files ... ok
postgres_1   | running bootstrap script ... ok
postgres_1   | performing post-bootstrap initialization ... sh: locale: not found
postgres_1   | 2021-07-13 16:30:03.590 UTC [30] WARNING:  no usable system locales were found
postgres_1   | ok
postgres_1   | syncing data to disk ... initdb: warning: enabling "trust" authentication for local connections
postgres_1   | ok
postgres_1   |
postgres_1   | You can change this by editing pg_hba.conf or using the option -A, or
postgres_1   | --auth-local and --auth-host, the next time you run initdb.
postgres_1   |
postgres_1   | Success. You can now start the database server using:
postgres_1   |
postgres_1   |     pg_ctl -D /var/lib/postgresql/data -l logfile start
postgres_1   |
postgres_1   | waiting for server to start....2021-07-13 16:30:04.347 UTC [35] LOG:  starting PostgreSQL 13.3 on x86_64-pc-linux-musl, compiled by gcc (Alpine 10.3.1_git20210424) 10.3.1 20210424, 64-bit
postgres_1   | 2021-07-13 16:30:04.348 UTC [35] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgres_1   | 2021-07-13 16:30:04.351 UTC [36] LOG:  database system was shut down at 2021-07-13 16:30:04 UTC
postgres_1   | 2021-07-13 16:30:04.353 UTC [35] LOG:  database system is ready to accept connections
postgres_1   |  done
postgres_1   | server started
postgres_1   |
postgres_1   | /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
postgres_1   |
postgres_1   | waiting for server to shut down....2021-07-13 16:30:04.454 UTC [35] LOG:  received fast shutdown request
postgres_1   | 2021-07-13 16:30:04.455 UTC [35] LOG:  aborting any active transactions
postgres_1   | 2021-07-13 16:30:04.457 UTC [35] LOG:  background worker "logical replication launcher" (PID 42) exited with exit code 1
postgres_1   | 2021-07-13 16:30:04.457 UTC [37] LOG:  shutting down
postgres_1   | 2021-07-13 16:30:04.468 UTC [35] LOG:  database system is shut down
postgres_1   |  done
postgres_1   | server stopped
postgres_1   |
postgres_1   | PostgreSQL init process complete; ready for start up.
postgres_1   |
postgres_1   | 2021-07-13 16:30:04.571 UTC [1] LOG:  starting PostgreSQL 13.3 on x86_64-pc-linux-musl, compiled by gcc (Alpine 10.3.1_git20210424) 10.3.1 20210424, 64-bit
postgres_1   | 2021-07-13 16:30:04.571 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
postgres_1   | 2021-07-13 16:30:04.571 UTC [1] LOG:  listening on IPv6 address "::", port 5432
postgres_1   | 2021-07-13 16:30:04.575 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgres_1   | 2021-07-13 16:30:04.579 UTC [47] LOG:  database system was shut down at 2021-07-13 16:30:04 UTC
postgres_1   | 2021-07-13 16:30:04.582 UTC [1] LOG:  database system is ready to accept connections
dbservice_1  | Checking for script in /app/prestart.sh
dbservice_1  | Running script /app/prestart.sh
dbservice_1  | Let the DB start: wait for 10 seconds...
crawler_1    | 07.13.2021-04:30:10 - crawler - INFO - Slack API connection successfully established.
crawler_1    | 07.13.2021-04:30:10 - crawler - INFO - Wait for 15 seconds to allow DB services to start...
uiservice_1  | Checking for script in /app/prestart.sh
uiservice_1  | Running script /app/prestart.sh
uiservice_1  | Let the DB services start: wait for 15 seconds...
timers_1     | 07.13.2021-04:30:13 - __main__ - INFO - Wait for 15 seconds to allow DB services to start...
dbservice_1  | [2021-07-13 16:30:16 +0000] [1] [INFO] Starting gunicorn 20.0.4
dbservice_1  | [2021-07-13 16:30:17 +0000] [1] [INFO] Listening at: http://0.0.0.0:80 (1)
dbservice_1  | [2021-07-13 16:30:17 +0000] [1] [INFO] Using worker: uvicorn.workers.UvicornWorker
dbservice_1  | [2021-07-13 16:30:17 +0000] [13] [INFO] Booting worker with pid: 13
dbservice_1  | [2021-07-13 16:30:17 +0000] [14] [INFO] Booting worker with pid: 14
dbservice_1  | [2021-07-13 16:30:17 +0000] [13] [INFO] Started server process [13]
dbservice_1  | [2021-07-13 16:30:17 +0000] [13] [INFO] Waiting for application startup.
dbservice_1  | [2021-07-13 16:30:17 +0000] [13] [INFO] Application startup complete.
dbservice_1  | [2021-07-13 16:30:17 +0000] [14] [INFO] Started server process [14]
dbservice_1  | [2021-07-13 16:30:17 +0000] [14] [INFO] Waiting for application startup.
dbservice_1  | [2021-07-13 16:30:17 +0000] [14] [INFO] Application startup complete.
crawler_1    | 07.13.2021-04:30:25 - crawler - INFO - Starting crawling...
uiservice_1  | [2021-07-13 16:30:25 +0000] [1] [INFO] Starting gunicorn 20.0.4
uiservice_1  | [2021-07-13 16:30:25 +0000] [1] [INFO] Listening at: http://0.0.0.0:80 (1)
uiservice_1  | [2021-07-13 16:30:25 +0000] [1] [INFO] Using worker: uvicorn.workers.UvicornWorker
uiservice_1  | [2021-07-13 16:30:25 +0000] [10] [INFO] Booting worker with pid: 10
uiservice_1  | [2021-07-13 16:30:25 +0000] [11] [INFO] Booting worker with pid: 11
uiservice_1  | [2021-07-13 16:30:26 +0000] [10] [INFO] Started server process [10]
uiservice_1  | [2021-07-13 16:30:26 +0000] [10] [INFO] Waiting for application startup.
uiservice_1  | [2021-07-13 16:30:26 +0000] [11] [INFO] Started server process [11]
uiservice_1  | [2021-07-13 16:30:26 +0000] [11] [INFO] Waiting for application startup.
uiservice_1  | {"loglevel": "info", "workers": 2, "bind": "0.0.0.0:80", "graceful_timeout": 120, "timeout": 120, "keepalive": 5, "errorlog": "-", "accesslog": "-", "workers_per_core": 1.0, "use_max_workers": null, "host": "0.0.0.0", "port": "80"}
uiservice_1  | 07.13.2021-04:30:27 - UI - INFO - Slack API connection successfully established.
uiservice_1  | [2021-07-13 16:30:27 +0000] [10] [INFO] Application startup complete.
uiservice_1  | {"loglevel": "info", "workers": 2, "bind": "0.0.0.0:80", "graceful_timeout": 120, "timeout": 120, "keepalive": 5, "errorlog": "-", "accesslog": "-", "workers_per_core": 1.0, "use_max_workers": null, "host": "0.0.0.0", "port": "80"}
uiservice_1  | 07.13.2021-04:30:27 - UI - INFO - Slack API connection successfully established.
uiservice_1  | [2021-07-13 16:30:27 +0000] [11] [INFO] Application startup complete.
dbservice_1  | {"loglevel": "info", "workers": 2, "bind": "0.0.0.0:80", "graceful_timeout": 120, "timeout": 120, "keepalive": 5, "errorlog": "-", "accesslog": "-", "workers_per_core": 1.0, "use_max_workers": null, "host": "0.0.0.0", "port": "80"}
dbservice_1  | 172.24.0.4:38336 - "PUT /message/ HTTP/1.1" 422
crawler_1    | 07.13.2021-04:30:27 - crawler - ERROR - [('http://dbservice:80/message/',), {'data': '[{"username": "U01J8RK7EHJ", "text": "test2", "timestamp": "1626191253.000700", "channel_id": "C01TEM06X28", "reply_count": 0, "reply_users_count": 0, "thread_length": 5, "link": null, "reactions_rate": 0.0}, {"username": "U01J8RK7EHJ", "text": "test", "timestamp": "1626188532.000300", "channel_id": "C01TEM06X28", "reply_count": 1, "reply_users_count": 1, "thread_length": 4, "link": null, "reactions_rate": 0.0}]'}, '{"detail":[{"loc":["body"],"msg":"value is not a valid list","type":"type_error.list"}]}']
crawler_1    | Traceback (most recent call last):
crawler_1    |   File "/app/common/extras.py", line 29, in try_request
crawler_1    |     raise ValueError(str([args, kwargs, answer.text]))
crawler_1    | ValueError: [('http://dbservice:80/message/',), {'data': '[{"username": "U01J8RK7EHJ", "text": "test2", "timestamp": "1626191253.000700", "channel_id": "C01TEM06X28", "reply_count": 0, "reply_users_count": 0, "thread_length": 5, "link": null, "reactions_rate": 0.0}, {"username": "U01J8RK7EHJ", "text": "test", "timestamp": "1626188532.000300", "channel_id": "C01TEM06X28", "reply_count": 1, "reply_users_count": 1, "thread_length": 4, "link": null, "reactions_rate": 0.0}]'}, '{"detail":[{"loc":["body"],"msg":"value is not a valid list","type":"type_error.list"}]}']
crawler_1    | 07.13.2021-04:30:27 - crawler - INFO - Messages from 5 channels parsed and sent to the database.
dbservice_1  | 172.24.0.4:38338 - "GET /message/linkless HTTP/1.1" 200
timers_1     | 07.13.2021-04:30:28 - __main__ - INFO - Starting timer processor...
dbservice_1  | 172.24.0.6:39612 - "GET /timer/nearest?time_border=2021-07-13T16%3A20%3A28.048983 HTTP/1.1" 200
dbservice_1  | 172.24.0.6:39614 - "GET /timer/overdue?time_border=2021-07-13T16%3A20%3A28.059136 HTTP/1.1" 200
dbservice_1  | 172.24.0.6:39616 - "GET /timer/count HTTP/1.1" 200

my .env

SLACK_USER_TOKEN=xoxp-1614705107874-1620869252596-2226XXXX
SLACK_BOT_TOKEN=xoxb-1614705107874-2205564551175-rur2Z7XXXX
DB_PASS=XXXX
SIGNING_SECRET=961587XXXXX

Shrink docker container size

Right now docker container size is about 800Mb, and it's rather big for such simple bot. I think it's better to shrink it somehow - install conda env without cache, use smaller image as baseline, maybe cut out conda and use just pip & requirements (in this way you maybe should somehow parse and install requirements from conda env file instead of just adding additional requirements file, but it is discussable)

Add possibilty to set your own timer for getting top every day/week/etc

We should add possibility to set your own timers (for each user) when to get top news. For example: every day etc.

Graceful shutdown of the bot

Right now bot do not gracefully shutdown despite of all my attempts to do so. I think it's possible to reproduce this bug and find out a solution how to solve it.

Add the ability to exclude persons or channels from results

Add the ability to exclude persons or channels from search results by the "top" command.
At least add the possibility for each request, better: as a user setting.

:bimorf: :)

Refactor database code

Add more holistic config parser

We need to store some configuration parameters in both configs files, env variables, and also redefine it from (maybe) other sources (like redefining YAML config variable by env, etc.).
We should find either existing config parsing library with such possibilities or write one.

Use `preset` instead of `category`

Currently, code base contains both names, we should remove category names (or preset, but I prefer to remove category) to keep it clean and maintainable

Make QnA optional in UI

Right now QnA module is deeply integrated into uiservice and it's rather inconvenient for those who want (possibly) to use this bot for their own slack without QnA.
I suggest make QnA code optional, so qna command parsing and buttons will be displayed only if qna link is presented.

Presets proposal

Preset is a set of channels with common idea or similar topics. Presets are useful when you want to get top messages from several channels and do not want to enumerate these channels every time you ask the bot.
Here in comments you can propose a preset and we will add it if it appropriate and relevant. Please also vote for presets which you want to add be added if they are proposed by someone else.

Template:
name: #channel1, #channel2, ...

Metrics

Right now no metrics are gathered at all and we can't even calculate amount of requests to the bot.
It'll be awesome to think about it and implement some lightweight solution with such metrics as: amount of requests, error/warning/exception counters, received "retry-after" amount etc.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.