nekromoff / eia-scraper Goto Github PK

View Code? Open in Web Editor NEW

5.0 7.0 0.0 11.54 MB

EIA Scraper - European Union Environmental Impact Assessment project scraper, processor and notifier for Slovakia.

License: GNU General Public License v3.0

PHP 84.63% JavaScript 0.76% Vue 0.38% SCSS 0.84% Blade 13.40%

eia environmental-impact-assessment eu european-union scraper slovakia watchdog politics environment open-government

eia-scraper's Introduction

Environmental Impact Assessment Scraper and Notifier

EIA project scraper, processor and notifier for Slovakia.

Features

Scrapes latest EIA projects and keeps their advanced info
Sends notifications with the custom filtered results to an email

Future plans

Generate prefilled documents and uploads them to Google Drive on-the-fly
Automatic sending of the generated documents (from template), if no action taken until deadline by email and snail mail.

Dependencies

Laravel 5.4

eia-scraper's People

Contributors

Stargazers

Watchers

eia-scraper's Issues

Send one email notification for multiple projects instead of separate emails

Currently, one email notification is sent per project

Symfony\Component\DomCrawler\Crawler::extract(): Argument #1 ($attributes) must be of type array, string given, called in /var/www/sub/eia/app/Http/Controllers/EIAController.php on line 196

Zacala vyskakovat tato chyba, viete to pls fixnut? Este predtym si premazte fingerprint v systemstate, lebo inak nezbehne crawl a chybu neuvidite.

send notification only once even when multiple searches match one project

Notification email is currently sent out as many times as watcher searches match project region/district/locality.

moznost platby na webe pre zmenu typu / programu

toto suvisi so #40, ale zatial to nemam premyslene, davam to sem ako zaklad, neskor rozsirim

/sk/ duplication in URL manipulation

Report by email:

Reason yet unknown.

user login to edit existing subscriptions

Send notifications on watcher email added

Go through past projects up to certain date (e.g. -10 days).

Add Pay by Square image to email and web

to allow preset payments for the service

Introduce configuration for IMAP and Mailgun HTTP API for sending emails

+implement Mailgun API to send emails, if configured

integrate google drive

to automatically create template documents

Sekciu "Tento email dostavate..." dat v maili iba raz

zrusme u opakujucu sa sekciu: "Tento email dostavate...." pri linku na odhlasenie zo vsetkeho.

add date or name to subject in notification emails

When sending the notification email, add date or name of the EIA project to subject (to prevent Gmail / email client from grouping all EIA notifications).

Add map with existing points to the homepage

OSM + location point per each EIA project

nova funkcionalita: typ pouzivatela + pocet notifikacii

novy stlpec type do watchers

Tento vyuzijeme na odlisenie rozne typy sledovatelov - nekomercnych, komercnych, s limitom na pocet notifikacii mesacne, bez a pod. Zaroven potrebujeme evidovat pocet odoslanych notifikacii na jednotlivych watchers za urcite obdobie - nateraz dajme 3 mesiace (90 dni) retention, starsie sa mozu zmazat - tu stacia timestamps a pocet poslanych EIA notifikacii (pri viacerych spojenych do jedneho emailu sa rata kazdy EIA zaznam zvlast, napr 2 EIA projekty = pocet 2 v tabulke o odoslanych emailoch).

Nova tabulka watchers_log (?) moze mat takuto strukturu:

primary_id	watcher_id	count	timestamps
1	326	2	2021-01-20 15:45:12

Typy uzivatelov nateraz:

9999: unlimited (db migracia na zaciatku prideli kazdemu tento typ)
1: basic/trial (5 notifikacii mesacne, nasledne sa uz emaily neposielaju)
10: commercial, 10 notifikacii mesacne
...dalsie urovne, zatial nedefinovane...
100: commercial, bez obmedzenia poctu zaznamov

Pred kazdym rozoslanim notifikacii sa najprv skontroluje type vo watchers pre danu emailovu adresu a overi sa v novej tabulke so zaznamom pocet odoslanych notifikacii za poslednych 30 dni.

Ak je pocet mensi, ako limituje typ, posle sa notifikacia standardne
Ak je pocet mensi, ale zaroven je sucasnym zaslanim dosiahnuty (presiahnuty, ak dochadza k spojeniu viacerych), posle sa notifikacia standardne + upozornenie, ze pre dostavanie dalsich notifikacii si je potrebne zmenit program na vyssi
Ak je pocet vacsi ako limituje typ, notifikacia sa neposle

Add automatic removal of permanently bounced emails via webhook

https://documentation.mailgun.com/en/latest/user_manual.html#webhooks

On event type: permanent failure
Send webhook notification to URL endpoint that will remove that email from watchers (any existing record with that email).

Use signed webhook to verify Mailgun: https://documentation.mailgun.com/en/latest/user_manual.html#webhooks