Scripts to help scrape financial data from israeli bank and credit card providers, archive it and optionally send it to YNAB. Transactions can currently be stored in a Google Cloud Platform firestore database. Based on israeli-bank-scrapers.
- Create firestore account
- Place
firebase-service-account.json
file in root dir - Install all dependencies using
yarn install
- Run
yarn configure
to configure your scrapers and optional YNAB integration - That's it!
- scrape - run all scrapers for all stored configurations, and save them to firestore.
- uploadToYnab - Upload to YNAB (Valid YNAB API key required)
- startBot - Start a telegram bot. See required env vars below
- startWeb - Starts a web server that starts a scrape when receiving a POST to the root address, sending logs to a telegram user indicated by CHAT_ID env var. Does not require a running telegram bot. Good for scheduled tasks.
- configure - Create or edit user configurations.
/tx [number of days]
- print a summary of all transactions in the last few days. Default number of days is 5./scrape
- trigger a scrape and upload to YNAB.
Heroku is supported.
- Create an heroku app, and set the required env vars marked Required for heroku integration.
- Add the Puppeteer buildpack.
- ONLY_PROVIDERS - [Optional] Comma delimited list. Scraper types to run.
- ONLY_ACCOUNTS - [Optional] Comma delimited list. Scraper accounts to run.
- MONTHS_TO_SCRAPE - [Optional] Number of months to scrape/upload to ynab. Period always start from the start of the month. (i.e. Passing 0 means to just start at current month)
- BOT_TOKEN - (Required for telegram bot) - Bot token, as obtained from the telegram @botfather
- CHAT_ID - (Required for telegram bot) - Telegram user to receive logs from the telegram bot "scrape" command
- GOOGLE_CREDENTIALS - (Required for heroku integration) - A base 64 version of the
firebase-service-account.json
file. - HEROKU_ADDRESS - (Required for heroku integration + running a telegram bot) - Full address pointing to heroku app. (e.g. https://awesome-scraper.herokuapp.com )
- WEBSERVER_PORT - [Optional] When running in web server mode (or via docker), the port to use. defaults to 9000