Giter VIP home page Giter VIP logo

django-telegram-bot's Introduction

django-telegram-bot

Sexy Django + python-telegram-bot + Celery + Redis + Postgres + Dokku + GitHub Actions template. Production-ready Telegram bot with database, admin panel and a bunch of useful built-in methods.

⭐ graph: Sparkline

Check the example bot that uses the code from Main branch: t.me/djangotelegrambot

Features

Built-in Telegram bot methods:

  • /broadcast — send message to all users (admin command)
  • /export_users — bot sends you info about your users in .csv file (admin command)
  • /stats — show basic bot stats
  • /ask_for_location — log user location when received and reverse geocode it to get country, city, etc.

Content

How to run

Quickstart: Polling & SQLite

The fastest way to run the bot is to run it in polling mode using SQLite database without all Celery workers for background jobs. This should be enough for quickstart:

git clone https://github.com/ohld/django-telegram-bot
cd django-telegram-bot

Create virtual environment (optional)

python3 -m venv dtb_venv
source dtb_venv/bin/activate

Install all requirements:

pip install -r requirements.txt

Create .env file in root directory and copy-paste this or just run cp .env_example .env, don't forget to change telegram token:

DJANGO_DEBUG=True
DATABASE_URL=sqlite:///db.sqlite3
TELEGRAM_TOKEN=<PASTE YOUR TELEGRAM TOKEN HERE>

Run migrations to setup SQLite database:

python manage.py migrate

Create superuser to get access to admin panel:

python manage.py createsuperuser

Run bot in polling mode:

python run_polling.py 

If you want to open Django admin panel which will be located on http://localhost:8000/tgadmin/:

python manage.py runserver

Run locally using docker-compose

If you want just to run all the things locally, you can use Docker-compose which will start all containers for you.

Create .env file.

You can switch to PostgreSQL just by uncommenting it's DATABASE_URL and commenting SQLite variable.

cp .env_example .env

Docker-compose

To run all services (Django, Postgres, Redis, Celery) at once:

docker-compose up -d --build

Check status of the containers.

docker ps -a

It should look similar to this:

Try visit Django-admin panel.

Enter django shell:

docker exec -it dtb_django bash

Create superuser for Django admin panel

python manage.py createsuperuser

To see logs of the container:

docker logs -f dtb_django

Deploy to Production

Production stack will include these technologies:

  1. Postgres as main database for Django
  2. Celery + Redis + easy scalable workers
  3. Dokku as PaaS (will build app from sources and deploy it with zero downtime)

All app's services that are going to be launched in production can be found in Procfile file. It includes Django webserver (Telegram event processing + admin panel) and Celery workers (background and periodic jobs).

What is Dokku and how it works

Dokku is an open-source version of Heroku.

I really like Heroku deployment approach:

  1. you push commit to Main branch of your Repo
  2. in couple minutes your new app is running
  3. if something breaks during deployment - old app will not be shut down

You can achieve the same approach with Dokku + Github Actions (just to trigger deployment).

Dokku uses buildpacks technology to create a Docker image from the code. No Dockerfile needed. Speaking about Python, it requires requirements.txt, Procfile files to run the things up. Also files DOKKU_SCALE and runtime.txt are useful to tweak configs to make the deployed app even better. E.g. in DOKKU_SCALE you can specify how many app instances should be run behind built-in load balancer.

One disadvantage of Dokku that you should be warned about is that it can work with one server only. You can't just scale your app up to 2 machines using only small config change. You still can use several servers by providing correct .env URLs to deployed apps (e.g. DATABASE_URL) but it will require more time to setup.

Deploy using Dokku: step-by-step

I assume that you already have Dokku installed on your server. Let's also assume that the address of your server is <YOURDOMAIN.COM> (you will need a domain to setup HTTPs for Telegram webhook support). I'd recommend to have at least 2GB RAM and 2 CPU cores.

Create Dokku app

dokku apps:create dtb

You might need to added .env variables to app, e.g. to specify Telegram token:

dokku config:set dtb TELEGRAM_TOKEN=.....

Postgres and Redis

Postgres and Redis are configured as Dokku plugins on a server. They will automatically add REDIS_URL & DATABASE_URL .env vars to the app after being linked. You might need to install these Dokku plugins before. Install Postgres, install Redis.

dokku postgres:create dtb
dokku postgres:link dtb dtb

dokku redis:create dtb
dokku redis:link dtb dtb

Deploy on commit with Github Actions

Go to file .github/workflows/dokku.yml:

  1. Enter your host name (address of your server),
  2. Deployed dokku app name (in our case this is dtb),
  3. Set SSH_PRIVATE_KEY secret variable via GitHub repo settings. This private key should have the root ssh access to your server.

This will trigger Dokku's zero-downtime deployment. You would probably need to fork this repo to change file.

After that you should see a green arrow ✅ at Github Actions tab that would mean your app is deployed successfully. If you see a red cross ❌ you can find the deployed logs in Github Actions tab and find out what went wrong.

HTTPS & Telegram bot webhook

Why you need to setup webhook

Basic polling approach is really handy and can speed up development of Telegram bots. But it doesn't scale. Better approach is to allow Telegram servers push events (webhook messages) to your server when something happens with your Telegram bot. You can use built-in Dokku load-balancer to parallel event processing.

HTTPS using Letsencrypt plugin

For Telegram bot API webhook usage you'll need a https which can be setup using Letsencrypt Dokku plugin. You will need to attach a domain to your Django app before and specify a email (required by Letsencrypt) - you will receive notifications when certificates would become old. Make sure you achieved a successful deployment first (your app runs at <YOURDOMAIN.COM>, check in browser).

dokku domains:add dtb <YOURDOMAIN.COM>
dokku config:set --global DOKKU_LETSENCRYPT_EMAIL=<[email protected]>
dokku letsencrypt:enable dtb

Setup Telegram Bot API webhook URL

You need to tell Telegram servers where to send events of your Telegram bot. Just open in the browser:

https://api.telegram.org/bot<TELEGRAM_TOKEN>/setWebhook?url=https://<YOURDOMAIN.COM>/super_secter_webhook/

After deployment

You can be sure that your app is deployed successfully if you see a green arrow at the latest workflow at Github Actions tab.

You would need to create a superuser to access an admin panel at https://<YOURDOMAIN.COM>/tgadmin. This can be done using a standard way using django shell:

Open shell in deployed app

dokku enter dtb web 

Create Django super user

Being inside a container:

python manage.py createsuperuser

After that you can open admin panel of your deployed app which is located at https://<YOURDOMAIN.COM>/tgadmin.

Read app logs

dokku logs dtb -t

django-telegram-bot's People

Contributors

abolix avatar erhosen avatar fnceba avatar ialif avatar ohld avatar shamaevnn avatar verybigsad avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

django-telegram-bot's Issues

bot always answers "Welcome back", even for new users

Hi, found a small bug. Must be uppeared after resolving #16 🙂

Steps to reproduce:

  1. Start local setup via docker-compose.
  2. Send /start from a new device that is not yet in DB.
  3. You'll get message "Welcome back, {first_name}!" instead of "Sup, {first_name}!"

The reason:
Method User.get_user_and_created is called twice for /start command: first in handler_logging function, second in command_start. On the second call created is always false, so the message is always "Welcome back".

Possible solution:
In handler_logging call func first, and then do the actual logging part.

Adding linux-utils on deploy

Добрый день! Настраивал экспорт в pdf и для этого нужен бинарник wkhtmltopdf(устанавливается с помощью apt). При тестировании на локальном сервере я просто добавил его в dockerfile и он устанавливался во время сборки.
Но при деплое это почему-то не работает, как будто читается какой-то другой dockerfile.
Пробовал написать установку в procfile, но это не сработало и, как я понял, так не делается. wkhtmltopdf: apt update && apt install -y --no-install-recommends wkhtmltopdf
Как можно добавить утилиту на сервер?

Good afternoon! I set up export to pdf and for this I need the wkhtmltopdf binary (installed using apt). When testing on a local server, I just added it to the dockerfile and it was installed during the build.
But for some reason it doesn't work when it is deployed, as if some other dockerfile is being read.
I tried to write an installation in procfile, but it didn't work and, as I understood, it's not done that way. wkhtmltopdf: apt update && apt install -y --no-install-recommends wkhtmltopdf
How can I add the utility to the server?

Add global handler for handling all errors

When some error occurs, the handler must handle the error.

IF there is a variable in settings.py named CHAT_ID_FOR_LOGGING_ERRORS, then print error message in console and send it to some admin chat_id (it might be id of a user or chat).
ELSE print only error message.

Parse Locations using ArcGIS

Telegram users might send their location to the bot. We receive only (lat,lng) but we might be interested to know more details about it (e.g. country / city).

Use this snippets:

# models.py
class Arcgis(models.Model):
    location = models.OneToOneField(Location, on_delete=models.CASCADE, primary_key=True)

    match_addr = models.CharField(max_length=200, **nb)
    long_label = models.CharField(max_length=200, **nb)
    short_label = models.CharField(max_length=128, **nb)

    addr_type = models.CharField(max_length=128, **nb)
    location_type = models.CharField(max_length=64, **nb)
    place_name = models.CharField(max_length=128, **nb)

    add_num = models.CharField(max_length=50, **nb)
    address = models.CharField(max_length=128, **nb)
    block = models.CharField(max_length=128, **nb)
    sector = models.CharField(max_length=128, **nb)
    neighborhood = models.CharField(max_length=128, **nb)
    district = models.CharField(max_length=128, **nb)
    city = models.CharField(max_length=64, **nb)
    metro_area = models.CharField(max_length=64, **nb)
    subregion = models.CharField(max_length=64, **nb)
    region = models.CharField(max_length=128, **nb)
    territory = models.CharField(max_length=128, **nb)
    postal = models.CharField(max_length=128, **nb)
    postal_ext = models.CharField(max_length=128, **nb)

    country_code = models.CharField(max_length=32, **nb)

    lng = models.DecimalField(max_digits=21, decimal_places=18, **nb)
    lat = models.DecimalField(max_digits=21, decimal_places=18, **nb)

    updated_at = models.DateTimeField(auto_now=True)
    created_at = models.DateTimeField(auto_now_add=True)

    @classmethod
    def from_json(cls, j, location_id):
        arc, _ = cls.objects.get_or_create(location_id=location_id)

        if "address" not in j or "location" not in j:
            return arc

        a = j.get("address")

        arc.match_addr = a.get("Match_addr")
        arc.long_label = a.get("LongLabel")
        arc.short_label = a.get("ShortLabel")
        arc.addr_type = a.get("Addr_type")
        arc.location_type = a.get("Type")
        arc.place_name = a.get("PlaceName")
        arc.add_num = a.get("AddNum")
        arc.address = a.get("Address")
        arc.block = a.get("Block")
        arc.sector = a.get("Sector")
        arc.neighborhood = a.get("Neighborhood")
        arc.district = a.get("District")
        arc.city = a.get("City")
        arc.metro_area = a.get("MetroArea")
        arc.subregion = a.get("Subregion")
        arc.region = a.get("Region")
        arc.territory = a.get("Territory")
        arc.postal = a.get("Postal")
        arc.postal_ext = a.get("PostalExt")
        arc.country_code = a.get("CountryCode")

        l = j.get("location")
        arc.lng = l.get("x")
        arc.lat = l.get("y")

        arc.save()
        return arc
# reverse geocode lat,lng
def reverse_geocode(lat, lng):
    r = requests.post(
        "https://geocode.arcgis.com/arcgis/rest/services/World/GeocodeServer/reverseGeocode",
        params={
            'f': 'json',
            'location': f'{lng}, {lat}',
            'distance': 50000,
            'outSR': '',
        },
        headers={
            'Content-Type': 'application/json',
        }
    )
    return r.json()

Add broadcast admin command & celery task

Code samples:

def broadcast(update, context):
    u = User.get_user(update, context)
    if not u.is_admin:
        return

    text = update.message.text.replace("/broadcast", "")
    update.message.reply_text("Use /broadcast1 to broadcast text message like this:")
    return update.message.reply_text(
        text,
        parse_mode=telegram.ParseMode.MARKDOWN,
        # disable_web_page_preview=True
    )

def broadcast1(update, context):
    u = User.get_user(update, context)
    if not u.is_admin:
        return

    text = update.message.text.replace("/broadcast1", "")
    user_ids = list(User.objects.filter(is_blocked_bot=False, is_admin=False).values_list("user_id", flat=True))

    tasks.broadcast_message.delay(message=text, user_ids=user_ids)

    update.message.reply_text(f"This message will be broadcasted to all ({len(user_ids)}) users in couple minutes.")
    update.message.reply_text(text, parse_mode=telegram.ParseMode.MARKDOWN)

No hook sender verification

class TelegramBotWebhookView(View):
# WARNING: if fail - Telegram webhook will be delivered again.
# Can be fixed with async celery task execution
def post(self, request, *args, **kwargs):
if DEBUG:
process_telegram_event(json.loads(request.body))
else:
# Process Telegram event in Celery worker (async)
# Don't forget to run it and & Redis (message broker for Celery)!
# Locally, You can run all of these services via docker-compose.yml
process_telegram_event.delay(json.loads(request.body))
# e.g. remove buttons, typing event
return JsonResponse({"ok": "POST request processed"})

Anyone can send a request and imitate any actions of users who started the bot

According to the documentation, such a check is enough

  • accept incoming POSTs only from subnets 149.154.160.0/20 and 91.108.4.0/22

TypeError with arbitrary callback_data

Hi, trying to use example with arbitrary callback_data, but it gives error TypeError: expected string or bytes-like object. I just copied all functions to tgbot/handlers.py and set all handlers from main() in tgbot/dispatcher.py. Any solution?

Add bot usage logging

Log every command & params that a user used in separate table (in Django models.py). Probably not all commands should be monitored.

Important:

  1. Add flag in settings.py to enable/disable this feature. For free Heroku users this might be crucial because only 10k lines are allowed on Postgres free tier
  2. Add a useful decorator that will simplify the logging of command handler usage.
  3. Later it would be cool to show some basic usage graphs on Django Admin panel: DAU, funnels, retentions.

Add test bot & auto deploy Main brunch

I would be great if there would be some test bot that runs the code from Main branch. We need to register a test telegrambot for these purposes.

Autodeploy - I can share the server where the code would be autodeployed using Dokku and GitHub Actions.

Make migrations folder persistent

Заметил, что при деплое миграции затираются. Может проблема только у меня, но она есть. Подскажите, пожалуйста, решение.
В .gitignore я конечно же добавил папку миграций, иначе в прод улетают мои локальные миграции, что иногда может ломать прод - ведь на локалхосте что только не творится и это нельзя запускать в прод )

Celery not working

periodic tasks can only be started manually one time and never restarts by its own.
console says "Usage: celery worker [OPTIONS]
Try 'celery worker --help' for help.

Error: Invalid value for '-P' / '--pool': <class 'celery.concurrency.prefork.TaskPool'> is not one of 'prefork', 'eventlet', 'gevent', 'solo', 'processes', 'threads'."

Issue occurs only when deploying to dokku, in local containers it works just fine

'backports.zoneinfo.ZoneInfo' object has no attribute 'normalize'

При попытке запустить production-версию на виртуальной машине через dokku и github actions, билд валится с ошибкой:
" "backports.zoneinfo.ZoneInfo' object has no attribute 'normalize' ".

В интернете нашел схожую проблему : agronholm/apscheduler#461
В виде решения предлагалось понизить версию пакета tzlocal с 3.6 до 2.1
Так как эта библиотека не прописывалась в зависимостях приложения, я запустил на ВМ следующую команду:
sudo pip install --force-reinstall tzlocal==2.1
Результат нулевой :[

Пожалуйста, помогите разобраться с проблемой или дайте направление для поиска решения, буду рад любой помощи!
Логи билда прилагаю

Add deploy to heroku button

For new users it would be super useful to be able to deploy a bot to Heroku

Important:

  1. Use free Postgres plugin (10k lines for free)
  2. Use Heroku app.json to add setup .env params to Heroku app installation page (see example).
  3. Use webhook since free Heroku allows to run only one line in Procfile.

Take a look at this repo for example Heroku integration: https://github.com/ohld/telegram-support-bot

WARNINGS: tgbot.Location: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'

/Users/user/django-telegram-bot/env/lib/python3.10/site-packages/telegram/ext/dispatcher.py:252: UserWarning: Asynchronous callbacks can not be processed without at least one worker thread.
warnings.warn(
System check identified some issues:

WARNINGS:
tgbot.Location: (models.W042) Auto-created primary key used when not defining a primary key type, by default 'django.db.models.AutoField'.
HINT: Configure the DEFAULT_AUTO_FIELD setting or the TgbotConfig.default_auto_field attribute to point to a subclass of AutoField, e.g. 'django.db.models.BigAutoField'.

Bot fails on /start

Hi,
I've got an error with starting TG Bot: FOREIGN KEY constraint failed. Tried on local env with SQLite and docker-compose file. Both ways have the same result.

Steps to reproduce:

  1. clone repo
  2. create new TG Bot
  3. create .env file as described at wiki
  4. start docker-compose
  5. open TG and /start with my bot
  6. -> nothing happen in TG, but there is described error at logs

Possible root cause:
Looks like handler_logging decorator is executed ahead creating user at command_start

@handler_logging()
def command_start(update, context):
    u, created = User.get_user_and_created(update, context)

Here is an error trace:

Traceback (most recent call last):
  File ".../django-telegram-bot/venv/lib/python3.8/site-packages/telegram/ext/dispatcher.py", line 442, in process_update
    handler.handle_update(update, self, check, context)
  File ".../django-telegram-bot/venv/lib/python3.8/site-packages/telegram/ext/handler.py", line 160, in handle_update
    return self.callback(update, context)
  File "/home/dmitrii/storage/documents/devel/external/django-telegram-bot/tgbot/handlers/utils.py", line 26, in handler
    UserActionLog.objects.create(user_id=user_id, action=action, created_at=timezone.now())
  File ".../django-telegram-bot/venv/lib/python3.8/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File ".../django-telegram-bot/venv/lib/python3.8/site-packages/django/db/models/query.py", line 447, in create
    obj.save(force_insert=True, using=self.db)
  File ".../django-telegram-bot/venv/lib/python3.8/site-packages/django/db/models/base.py", line 753, in save
    self.save_base(using=using, force_insert=force_insert,
  File ".../django-telegram-bot/venv/lib/python3.8/site-packages/django/db/models/base.py", line 790, in save_base
    updated = self._save_table(
  File ".../django-telegram-bot/venv/lib/python3.8/site-packages/django/db/models/base.py", line 895, in _save_table
    results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw)
  File ".../django-telegram-bot/venv/lib/python3.8/site-packages/django/db/models/base.py", line 933, in _do_insert
    return manager._insert(
  File ".../django-telegram-bot/venv/lib/python3.8/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File ".../django-telegram-bot/venv/lib/python3.8/site-packages/django/db/models/query.py", line 1254, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File ".../django-telegram-bot/venv/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1397, in execute_sql
    cursor.execute(sql, params)
  File ".../django-telegram-bot/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 98, in execute
    return super().execute(sql, params)
  File ".../django-telegram-bot/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File ".../django-telegram-bot/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File ".../django-telegram-bot/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File ".../django-telegram-bot/venv/lib/python3.8/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File ".../django-telegram-bot/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File ".../django-telegram-bot/venv/lib/python3.8/site-packages/django/db/backends/sqlite3/base.py", line 413, in execute
    return Database.Cursor.execute(self, query, params)
django.db.utils.IntegrityError: FOREIGN KEY constraint failed

Make telegram bot Webhook secret

Right now everyone can POST event to TelegramBotWebhookView and fake Telegram events. This is a security issue so it is better to change Webhook url to contain secret value like TELEGRAM_TOKEN.

Current Webhook POST endpoint:

/super_secter_webhook

Let's change it to:

/telegram/webhook/<TELEGRAM_TOKEN>

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.