brafdlog / caspion Goto Github PK

Make a short video showing scraping and automatic update of google sheets. Also worth showing the resulting report.

Add this video to the README

Since scraping is a very integrative process it can fail from many causes. Instead of making the user retry himself, it would be good if there was a retry mechanism in the code that would retry x times if the scraping failed.

Dependabot can't resolve your JavaScript dependency files.

As a result, Dependabot couldn't update your dependencies.

The error Dependabot encountered was:

Error whilst updating dependencies in package-lock.json:
Cannot read property 'match' of undefined

It looks like your lockfile has some corrupt entries with missing versions and needs to be re-generated.
You'll need to remove package-lock.json and node_modules before you run npm install.

If you think the above is an error on Dependabot's side please don't hesitate to get in touch - we'll do whatever we can to fix it.

You can mention @dependabot in the comments below to contact the Dependabot team.

• Are we should run if no importers defined? (For example, to test the exporters)
• Are we should export if no transactions fetched (empty array)?
• Are we should throw or ignore or event not start to import if no exporters?

Rename the originalBudgetTrackingApp to library (or whatever) and move all Vue/Fronend related code to ui folder or whatever.

Then I think you can create another folder for tools like your YNAB helper. (It's not mean that this issue is assigned to @brafdlog)

The current authentication with google sheets is through a service account.

This is far from ideal because it requires an annoying process and because it requires a scope (auth/spreadsheets) that grants much more access than what we actually need.

Instead we should implement the oauth installed application flow and probably use the auth/drive.file and create the spreadsheet from the app.

Also this should be exposed in the ui of the configuration so the user clicks a button to integrate google sheets and that starts the process.

After merging #118, need to validate the exporting to Google Spreadsheets code, includes

• Create a new Spreadsheet by name.
• Create a new sheet by const name.
• Read existed data from the sheet and merge with the new data.

Current integration with YNAB is through their api key.

This process is annoying to set up and is less natural than the oauth flow that was built for this.

So we should set up the oauth integration with ynab and expose it in the ui of the config screen.

See YNAB docs

The current setup is an overkill.

We need a simple setup of electron + react + webpack and also typescript (which can be added separately) .

So we can reduce allot of the boilerplate code.

After we have a working MVP it would be a good time to stop and think what we want the UI/UX of the app to be.

Take into account:

• Configuring importers
  • YNAB setup
  • Google sheets setup
• Configuring exporters
• Global configuration (like show browser)
• Import and export progress indication
• Category management and rules for automatically assigning transactions to categories
• Report after imporn export finishes

Currently every error during the import or export process makes the whole process fail. In addition, there are cases where multiple error events will be emitted for the same error.
We need to decide on a consistent way to handle errors. Probably don't want to crash the whole process because one importer or exporter fail but we do want to indicate the failure clearly to the user

I'm trying to think about a way to force us to import only from originalBudgetTrackingApp and not from any originalBudgetTrackingApp/* when we are on the FrontEnd code.

Maybe this:
https://eslint.org/docs/rules/no-restricted-imports
Just keep in your mind

Originally posted by @baruchiro in #91 (comment)

The current financial account config looks terrible. So one thing is to make it look better - perhaps a card per financial institution?
In addition, this config only supports username and password, but there are many financial institutions that have different credentials (see here).

The config component should support the different variants of credentials so when the financial institution is chosen, allow inserting the credentials that fit that it requires.

• Update the readme
• Document in Readme the up to date flow of setting up the app from scratch with importers and exporters

The user should see clear feedback about what is currently happening.

Something like the icons of the accounts being scraped and next to each an indication of the status (perhaps with an icon) pending, in progress with loader, done.

Create a ui that conveys clearly the current status of importing/exporting.

This will have two parts:

1. An accounts state in the ui code that gets updated by listening to events.
2. A component showing each account and its current status.

Original comment:
I want to remove all the code from the root index.ts, to get a clear list about what the backend exporting. I have three (or more) points:

Move all the code from index.ts to separate files in a logical order. The export functionality should be in its file, the same with the import functionality, and the method that takes from import to export in another file.
In this way, we can validate (I hope there is lint for that) the Vue access only to the @/originalBudgetTrackingApp, and the backend itself access only to the flow module, and this module uses its "internal" methods to import and export.
With this structure you can understand half of the code only by its module.

Writing tests for the main flows can be pretty easy if we mock the code that fetches transactions from the financial institutions and mock the code that sends the transactions to ynab/google sheets.
Can also use the csv exporter in tests since it requires no setup

To avoid rename the app/node/categoryCalculationScript-example.js when starting to work with the project, and to prepare this project to be installable, we need first to get the categories mapping from an outer file.

In this way, we also can create a global configuration file that will hold many business names, and use it by default, unless the user want to derive or ignore it.

I think it would be cleaner to add the level to the BudgetTrackingEvent itself, setting it by default to INFO and having error set it to ERROR. Then you won't need this class and can use the event publisher directly.

In the current implementation you don't have access to the full api of the emitter which is useful (see comment about the option to set the status by listening to events)

Originally posted by @brafdlog in #110 (comment)

The FINANCIAL_COMPANIES_CONFIG in app/web/constants/financialCompaniesConfig.js should be generated from the source list in the israeli-bank-scrapers.

Now, if we want to enhance this list with logo and so on, we need to assign the additional info to the existed list, and handle default values when we did not configure our value yet.

In this way, you don't need to add configuration manually as #25 did.

Syncing with the main Israeli Bank Scrapers project updates will provide a better experience for the user and eliminate the need to update the software manually.

Project for issues we want to handle pre merge: https://github.com/brafdlog/budget-tracking/projects/4

Related issues:

• Make the configManage use the crypto module and remove hard coded encryption key
• Make the node code expose an api as defined here
• Update the electron UI flow to work with two main flows - configure, run scraping as defined here
• Make the electron UI work the node "library" code
• Remove the Contributors part or hide it in a Menu or something #87
• Align filename #57

Post Merge Actions:

• Decide how to release (by Tag or by pushing to master)
• Copy relevant issues from israeli-bank-scrapers-desktop.
• New repo/app name?
• Check if there unused packages
• Add MacOS to tests
• Create Splash screen (until the renderer process loaded)
• Set show browser in configManager? Define it for all importers?
• Add ability to disable importer and exporter (and keep them configured)

Exporting data to a Google Spreadsheet to allow automations - can be exported on demand (and be filtered by dates and/or cards/categories) AND on a regular basis, when new transaction appears. This will provide an opportunity for further automations, such as emails.

#28

After running the initial setup steps - I get a "Cannot find module './categoryCalculationScript'" error.
the "categoryCalculationScript.js" file is in the .gitignore.
is it possible the categoryCalculationScript-example.js file was supposed to be part of the git repository but is not?

The current json textarea was a quick and dirty solution to allow changing the config.
We want to allow users to change the configuration in a clear ui.
So, instead of the json text area we want to have ui components that allow setting all the configurations.

Pass sub-config to internal functions instead of the whole config. The functions should get only the relevant config, for example, the export function should get only the outputVendors part from the Config.

Instead of using the budget-tracking or https://github.com/brafdlog/budget-tracking in many places in the log, we need to declare them once and use the variable all over the code.

It can be implemented by Webpack DefinePlugin or import { name } from 'package.json.

CI should currently run tests.

In the future can also package the application for installation

Make the build process create an installer for mac/windows linux and do the needed adjustments to make it work as an installed app (as opposed to now just being run as a yarn script from the development environment.

All our current exporters require configuration (except for json that will probably not be useful for the common person).
Having a valuable exporter that works out of the box is important because it gives immediate value and will reduce the chance that people will give up during the initial setup to the complexity of it.
So, we can should create a new exporter that exports to an excel file (csv? xlsx?). This exporter will be enabled by default with some default file path.
On the ui of the excel exporter config there will be a link to open the file.

Is this project alive?

I would like to do a similar project.

Because we are using vuex-persist, and it loads the config asynchronicity, the app can't load until the store is loaded, so we have a Splash Screen (currently just a text "SPLASH").

We need to create more beautiful or just loading spinner, or use Vue Suspense feature (don't know if it supported in Vue2)

Currently the encryption key is hard coded. Ideally each user/installation would have his own encryption key.

The consistent naming convention in the repo is camelCase for directories and files so I would rename:

config-manager -> configManager
default-config -> defaultConfig

Originally posted by @brafdlog in #54 (comment)

I think the Vue part is not full compatible with this rule. But also need to think maybe in Vue the convention is different- https://vuejs.org/v2/style-guide/#Single-file-component-filename-casing-strongly-recommended

Isracard and amex use id and not username. so you have to manually add those configurations. also, because of the check in
BankScraper.scrape (!credentials.username) user need to add a "dummy" username field as well

id: ,
card6Digits: <6 last digits of card>
password:

The simplest implementation for this is to fetch periodically from YNAB/Google sheets the mapping from transaction description to category and save this locally. Then use this for classifying future transactions with the same description.

An alternative implementation is to expose in the ui the transactions that are about to sent to the vendor and allow the user to classify them inside our app. This will mean we can store the mapping of description to category without fetching it from outside.

Currently the logic for classifying a transaction relies on the categoryCalculationScript.js.
Most of the logic there is matching patterns of descriptions to patterns or exact matches.
In order to make this easily configurable by the user, we want to move this logic to a json configuration that will contain the list of patterns and exact matches.
After this is taken from the json, expose a ui for managing this in the config screen.

Notes:

1. We need to allow people to have their own setup - both control category names and control the mapping, because this is something that is not consistent across people and across budgeting tools. In the future we may consider having this configuration per output vendor.
2. We want this to work both in development from the repository and for the released electron app.
3. When running for the first time (either in dev or released electron app) should work with a default configuration without requiring an initial setup.
4. In future consider supporting regex patterns, but not required for initial phase.
5. Write unit tests for this logic.

Proposed solution:

The getCategoryNameByTransactionDescription will fetch this json and use it to get the transaction category according to the following logic:

Look for an exact match for the transaction description string in the exactMatches arrays.

If not found fall back to looking for substring matches in the patternMatches arrays.

JSON structure:

{
  "exactMatches": {
    "electricity": ["חברת חשמל", "חיוב חשמל חודשי"],
    "groceries": ["המכולת של יהונתן", "המכולת של ברוך"]
  },
  "patternMatches": {
    "car maintenance": ["מוסך", "טסט"],
    "groceries": ["מכולת"]
  }
}

This json should be saved in a place where it can be used both in development and in release. There are a few options to implement this, see open questions.

Have a defaultCategoryMapping.json file committed to git. If the user has no mapping logic defined yet, take the default and save it as the user's json for future customization.

Open questions:

1. Should this be part of the config or separate json?
2. Where should this be stored? Could be a json file or local storage etc. See: https://medium.com/cameron-nokes/how-to-store-user-data-in-electron-3ba6bf66bc1e
3. Need to verify it works nicely with Hebrew category names and patterns.

When I try to install by running israeli-bank-scrapers-desktop Setup 0.1.3-beta.exe of unifyRepos, either by building from code or by taking the artifacts from github, it gets stuck on "Installing, please wait".
I was not yet able to find where the log files are.

#27

When we are working with the config read and write, we are assuming that the loaded config is in the correct schema, because it saved by us or loaded from the default schema.

But if you adding a new Exporter, for example, it will cause an error since the exporter in the config is not nullable, but because it is a new exporter, it does not exist in the existed config.

Also, if you think the app is separated into two parts- one is managing the config, and the second is giving the config to the scraping process, you don't want the user will have to create a full config with all the exporters, just for marking all the exporters as enabled=false.

What I'm saying is that maybe all the config properties on all the levels should be nullable.