brafdlog / caspion Goto Github PK
View Code? Open in Web Editor NEWAutomated budget tracking from Israeli financial institutions
Home Page: https://www.caspion.org
License: MIT License
Automated budget tracking from Israeli financial institutions
Home Page: https://www.caspion.org
License: MIT License
I'm trying to think about a way to force us to import only from originalBudgetTrackingApp
and not from any originalBudgetTrackingApp/*
when we are on the FrontEnd code.
Maybe this:
https://eslint.org/docs/rules/no-restricted-imports
Just keep in your mind
Originally posted by @baruchiro in #91 (comment)
Isracard and amex use id and not username. so you have to manually add those configurations. also, because of the check in
BankScraper.scrape (!credentials.username) user need to add a "dummy" username field as well
id: ,
card6Digits: <6 last digits of card>
password:
After we have a working MVP it would be a good time to stop and think what we want the UI/UX of the app to be.
Take into account:
Rename the originalBudgetTrackingApp
to library
(or whatever) and move all Vue
/Fronend related code to ui
folder or whatever.
Then I think you can create another folder for tools
like your YNAB helper. (It's not mean that this issue is assigned to @brafdlog)
Currently every error during the import or export process makes the whole process fail. In addition, there are cases where multiple error events will be emitted for the same error.
We need to decide on a consistent way to handle errors. Probably don't want to crash the whole process because one importer or exporter fail but we do want to indicate the failure clearly to the user
Is this project alive?
I would like to do a similar project.
Make a short video showing scraping and automatic update of google sheets. Also worth showing the resulting report.
Add this video to the README
After running the initial setup steps - I get a "Cannot find module './categoryCalculationScript'" error.
the "categoryCalculationScript.js" file is in the .gitignore.
is it possible the categoryCalculationScript-example.js file was supposed to be part of the git repository but is not?
To avoid rename the app/node/categoryCalculationScript-example.js
when starting to work with the project, and to prepare this project to be installable, we need first to get the categories mapping from an outer file.
In this way, we also can create a global configuration file that will hold many business names, and use it by default, unless the user want to derive or ignore it.
Because we are using vuex-persist
, and it loads the config
asynchronicity, the app can't load until the store is loaded, so we have a Splash Screen (currently just a text "SPLASH").
We need to create more beautiful or just loading spinner, or use Vue Suspense
feature (don't know if it supported in Vue2)
Project for issues we want to handle pre merge: https://github.com/brafdlog/budget-tracking/projects/4
Related issues:
Post Merge Actions:
Tag
or by pushing to master
)israeli-bank-scrapers-desktop
.configManager
? Define it for all importers?Writing tests for the main flows can be pretty easy if we mock the code that fetches transactions from the financial institutions and mock the code that sends the transactions to ynab/google sheets.
Can also use the csv exporter in tests since it requires no setup
After merging #118, need to validate the exporting to Google Spreadsheets code, includes
CI should currently run tests.
In the future can also package the application for installation
Pass sub-config to internal functions instead of the whole config. The functions should get only the relevant config, for example, the export function should get only the outputVendors
part from the Config
.
Make the build process create an installer for mac/windows linux and do the needed adjustments to make it work as an installed app (as opposed to now just being run as a yarn script from the development environment.
Create a ui that conveys clearly the current status of importing/exporting.
This will have two parts:
The current authentication with google sheets is through a service account.
This is far from ideal because it requires an annoying process and because it requires a scope (auth/spreadsheets
) that grants much more access than what we actually need.
Instead we should implement the oauth installed application flow and probably use the auth/drive.file
and create the spreadsheet from the app.
Also this should be exposed in the ui of the configuration so the user clicks a button to integrate google sheets and that starts the process.
The current json textarea was a quick and dirty solution to allow changing the config.
We want to allow users to change the configuration in a clear ui.
So, instead of the json text area we want to have ui components that allow setting all the configurations.
When we are working with the config
read and write, we are assuming that the loaded config is in the correct schema, because it saved by us or loaded from the default schema.
But if you adding a new Exporter, for example, it will cause an error since the exporter in the config is not nullable
, but because it is a new exporter, it does not exist in the existed config.
Also, if you think the app is separated into two parts- one is managing the config, and the second is giving the config to the scraping process, you don't want the user will have to create a full config with all the exporters, just for marking all the exporters as enabled=false
.
What I'm saying is that maybe all the config properties on all the levels should be nullable
.
Exporting data to a Google Spreadsheet to allow automations - can be exported on demand (and be filtered by dates and/or cards/categories) AND on a regular basis, when new transaction appears. This will provide an opportunity for further automations, such as emails.
The FINANCIAL_COMPANIES_CONFIG
in app/web/constants/financialCompaniesConfig.js
should be generated from the source list in the israeli-bank-scrapers
.
Now, if we want to enhance this list with logo and so on, we need to assign the additional info to the existed list, and handle default values when we did not configure our value yet.
In this way, you don't need to add configuration manually as #25 did.
I think it would be cleaner to add the level to the BudgetTrackingEvent itself, setting it by default to INFO and having error set it to ERROR. Then you won't need this class and can use the event publisher directly.
In the current implementation you don't have access to the full api of the emitter which is useful (see comment about the option to set the status by listening to events)
Originally posted by @brafdlog in #110 (comment)
All our current exporters require configuration (except for json that will probably not be useful for the common person).
Having a valuable exporter that works out of the box is important because it gives immediate value and will reduce the chance that people will give up during the initial setup to the complexity of it.
So, we can should create a new exporter that exports to an excel file (csv? xlsx?). This exporter will be enabled by default with some default file path.
On the ui of the excel exporter config there will be a link to open the file.
Current integration with YNAB is through their api key.
This process is annoying to set up and is less natural than the oauth flow that was built for this.
So we should set up the oauth integration with ynab and expose it in the ui of the config screen.
See YNAB docs
Currently the logic for classifying a transaction relies on the categoryCalculationScript.js
.
Most of the logic there is matching patterns of descriptions to patterns or exact matches.
In order to make this easily configurable by the user, we want to move this logic to a json configuration that will contain the list of patterns and exact matches.
After this is taken from the json, expose a ui for managing this in the config screen.
Notes:
Proposed solution:
The getCategoryNameByTransactionDescription
will fetch this json and use it to get the transaction category according to the following logic:
Look for an exact match for the transaction description string in the exactMatches
arrays.
If not found fall back to looking for substring matches in the patternMatches
arrays.
JSON structure:
{
"exactMatches": {
"electricity": ["חברת חשמל", "חיוב חשמל חודשי"],
"groceries": ["המכולת של יהונתן", "המכולת של ברוך"]
},
"patternMatches": {
"car maintenance": ["מוסך", "טסט"],
"groceries": ["מכולת"]
}
}
This json should be saved in a place where it can be used both in development and in release. There are a few options to implement this, see open questions.
Have a defaultCategoryMapping.json
file committed to git. If the user has no mapping logic defined yet, take the default and save it as the user's json for future customization.
Open questions:
Instead of using the budget-tracking
or https://github.com/brafdlog/budget-tracking
in many places in the log, we need to declare them once and use the variable all over the code.
It can be implemented by Webpack DefinePlugin
or import { name } from 'package.json
.
The user should see clear feedback about what is currently happening.
Something like the icons of the accounts being scraped and next to each an indication of the status (perhaps with an icon) pending, in progress with loader, done.
The simplest implementation for this is to fetch periodically from YNAB/Google sheets the mapping from transaction description to category and save this locally. Then use this for classifying future transactions with the same description.
An alternative implementation is to expose in the ui the transactions that are about to sent to the vendor and allow the user to classify them inside our app. This will mean we can store the mapping of description to category without fetching it from outside.
Original comment:
I want to remove all the code from the root index.ts
, to get a clear list about what the backend exporting. I have three (or more) points:
Move all the code from index.ts
to separate files in a logical order. The export functionality should be in its file, the same with the import functionality, and the method that takes from import to export in another file.
In this way, we can validate (I hope there is lint for that) the Vue
access only to the @/originalBudgetTrackingApp
, and the backend itself access only to the flow
module, and this module uses its "internal" methods to import and export.
With this structure you can understand half of the code only by its module.
The current setup is an overkill.
We need a simple setup of electron + react + webpack and also typescript (which can be added separately) .
So we can reduce allot of the boilerplate code.
Dependabot can't resolve your JavaScript dependency files.
As a result, Dependabot couldn't update your dependencies.
The error Dependabot encountered was:
Error whilst updating dependencies in package-lock.json:
Cannot read property 'match' of undefined
It looks like your lockfile has some corrupt entries with missing versions and needs to be re-generated.
You'll need to remove package-lock.json and node_modules before you run npm install.
If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.
You can mention @dependabot in the comments below to contact the Dependabot team.
Since scraping is a very integrative process it can fail from many causes. Instead of making the user retry himself, it would be good if there was a retry mechanism in the code that would retry x times if the scraping failed.
The consistent naming convention in the repo is camelCase for directories and files so I would rename:
config-manager
-> configManager
default-config
-> defaultConfig
Originally posted by @brafdlog in #54 (comment)
I think the Vue part is not full compatible with this rule. But also need to think maybe in Vue the convention is different- https://vuejs.org/v2/style-guide/#Single-file-component-filename-casing-strongly-recommended
Syncing with the main Israeli Bank Scrapers project updates will provide a better experience for the user and eliminate the need to update the software manually.
Currently the encryption key is hard coded. Ideally each user/installation would have his own encryption key.
The current financial account config looks terrible. So one thing is to make it look better - perhaps a card per financial institution?
In addition, this config only supports username and password, but there are many financial institutions that have different credentials (see here).
The config component should support the different variants of credentials so when the financial institution is chosen, allow inserting the credentials that fit that it requires.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.