Feature suggestion - obviously for a future release!

The user can provide a file that contains mapping information (format yet to be decided), primarily consisting of a Payee pattern and a master category:category; plus possibly additional information?
For each input transaction, the script would look up the Payee string in the mapping file. If a match is found, the output category is filled with the data from the mapping file.

This was directly inspired from this section:
https://github.com/jasonmadigan/aib-to-ynab#mapping-files

Very rough format idea:

"valid for [SECTION]s, or * for all sections";"input payee pattern, allows Regex or at least asterisk as wildcard";"output Payee string";"output category string"
"[AT Raiffeisen bank],[AT Raiffeisen VISA]";"paypa*";"PayPal";"Online spending:PayPal"
"*";"Spar*";"Spar";"Household:Groceries"

The input columns could be case insensitive; the output columns case sensitive.

Using a freshly downloaded bank file, I get this error with Python 3 (on master and also on the other branches). (discovered while testing #59)

Your specified download directory was not found: /home/torben/Downloadsx
Parsing file: elbi_umsaetze_20171025234336.csv
Using format: AT Raiffeisen Bank
Traceback (most recent call last):
  File "bank2ynab.py", line 179, in <module>
    main()
  File "bank2ynab.py", line 169, in main
    output = clean_data(file)
  File "bank2ynab.py", line 76, in clean_data
    for row in transaction_reader:
  File "/usr/lib/python3.5/codecs.py", line 321, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xdc in position 21: invalid continuation byte

Note:

• The file seems to be not in Unicode format but in Windows ISO-8859-1 format, at least according to Notepadqq (a Notepad++ clone for Linux).
• The VISA statement from the same bank, same website, same time of download, is reported as UNIX/OS X UTF-8 w/o BOM format. This file has not been tested yet because the script stops after the first file (and I haven't tried to run it without the other file present yet!).

As requested by u/fio_smiles, this may be possible. Need handling for the following main issues:

• ConfigParser to configparser
• csv delimiter as string, not unicode
• csv newline invalid

Assigning myself to this issue. It's not a high importance item, but might help prevent problems if @torbengb starts accidentally using Python 2 again...

Example row: "NL12RABO3456789012","EUR","20140318","D","3.00","","","20140318","ba","","Rest Bernoulliborg GRONINGEN","Betaalautomaat 14:42 pasnr. 004","","","","","","",""
Note date format.

• Remco32/rabobank-to-ynab-converter Gives us column header example and row example!
• BrianSetz/rabonab
• canassa/rabobank-ynab Lists filename as transactions.csv!
• eddokloosterman/php-rabobank-ynab-csv-converter Lists filename as transactions.txt.
• jtdev/ynab-converter [java] This one lists transactions.csv AND transactions.txt!
• karsalfrink/RabobankToYNAB [Python] transactions.txt again.

[NL Rabobank]
Source Filename Pattern = transactions
Source Filename Extension = .txt
Input Columns = skip,skip,Date,Outflow/Inflow Designator,Inflow,skip,Payee,skip,skip,skip,Memo,skip,skip,skip,skip,skip,skip,skip
Plugin = Need to parse the Outflow/Inflow designator. This is reasonably common, so a general plugin for this would probably be most helpful.

Master Issue #21

Sourced from andsens/ynab-csv-converter.

Row format: 'date', 'txndate', 'text', 'amount', 'balance'

Unclear if header is present.
delimiter=';'

To be filled in as required. Delete fields that match default values.
Source Filename Pattern = example_transaction_export_filename
Source CSV Delimiter = ;
Input Columns = Date,skip,Payee,Inflow,skip

From Issue #21 master list

As noted in issue #81 we need documentation about the plugin concept.

For starters that will probably be wiki. Perhaps migrate content to the GitHub "SRT" format later?

Apart from our manual testing with random files, @toyg suggested in #54 a more structured approach.

I welcome the idea but I'm not sure how to do it - from what I've seen, GitHub has several useful features built in, but it does require some effort to set up. I don't want to impose a lot of overhead on this little project of ours, so once more it boils down to "is it worth the effort?" Certainly a subjective measure, and I will be last person to object or stand in the way if you want to go ahead with it.

Let's investigate options, benefits, and required effort?

As mentioned in issue #5 it would be useful to have a central list of required modules, or dependencies. Apparently Python doesn't support the "package.json" format but there are other interesting solutions that make dependencies easy to handle.

Of course a there's a relevant Stack Overflow answer :-) but seems like this alternative would create a requirement file that contains only what the specified project needs - and skip other irrelevant bits we might have in our local environment.

(This idea was sparked mainly by the Github notification about dependencies.)

Based on this reddit comment, perhaps our code could be integrated into the YNAB Toolkit? That would be cool! Our project needs more supported formats first, and it would require some adaptations.

Specifically at the end of these pages:

• YNAB 4 support page
• nYNAB help page

There is a question of how much we can actually stretch the current approach of just reordering a few fields. For example, I'm looking at https://github.com/JamesBelchamber/ynab-jlp-csv-converter/blob/master/convert.py and apparently that bank (JLP) does not have different columns for Inflows and Outflows, but rather a single column for values that are marked as CR in another field when it's a credit. This is literally the first random script I picked from the list, chances are that there are loads of these. Special-casing this sort of thing will only get us so far.

What about adding the possibility of loading a different class that will do the record parsing and will then provide an iterator returning the necessary format? That way we can deal with untractable special cases in dedicated modules.

I'm thinking we could have a modules subfolder that contains, for example, a jlp.py file with some sort of class that provides a standard interface. Then in the config file the user could specify an option like Parsing Module = jlp. If the option is specified, the code will load the module and hand over parsing to it, expecting to get back an iterator that we can use with csv; otherwise, we do the regular parsing (in fact, we could have a default.py module that holds default parsing logic, shortening the main script). This is pretty easy in Python 3 with a bit of importlib magic, and I'm pretty sure there is a Python2 equivalent too.

As well as dealing with difficult special cases, this could potentially open the door to supporting pretty much any file format, not just csv.

So far we (well, you!) have been very good and diligent about doing changes in feature-specific branches. (I haven't been as diligent, but luckily my inputs have been mostly testing and documentation so I don't feel too guilty.)

Anyway, I've been thinking about terms like "versions" and "releases". As long as all our efforts flow back into master, I'm not sure we can ever say we have a stable release point?

You're probably aware of several different strategies. I happen to like this one and put it to you as a question: should we think about having a branching strategy? Which one? This one I suggested (cheat sheet here), or some other?

Header row: ""Datum","Naam / Omschrijving","Rekening","Tegenrekening","Code","Af Bij","Bedrag (EUR)","MutatieSoort","Mededelingen""

• AndreSomers/ING2YNAB (empty project)
• dive/ing-to-ynab-json-converter
• ernezt/ynab-ing-converterThis doesnt' look like a real program...
• Remco32/ing-to-ynab-converter I can't really interpret this one!

Plugin created. Needs further follow-up to get test file.

• JamesBelchamber/ynab-jlp-csv-converter
• clancychilds/jlp-ynab-converter

@toyg has a plugin for this in the JLP branch.

To be filled in as required. Delete fields that match default values.
Source Filename Pattern = example_transaction_export_filename
Source Filename Extension = .csv
Use Regex for Filename = False
Source CSV Delimiter = ,
Source Has Column Headers = True
Input Columns = Date,Payee,Outflow,Inflow,Running Balance
Plugin =

My testing showed that the script stops at line 71 'transaction_data = list(transaction_reader)' if the file contains special characters like üöäÜÖÄßæøåÆØÅ. Shouldn't the whole Unicode business already take care of umlauts?

The error message on the console is:

Trying format:  Raiffeisen Bank
Parsing file:  elbi_umsaetze_20171018213618.csv
Traceback (most recent call last):
  File "./transaction_csv_cleanup.py", line 160, in <module>
    main()
  File "./transaction_csv_cleanup.py", line 152, in main
    output = clean_data(file)
  File "./transaction_csv_cleanup.py", line 70, in clean_data
    transaction_data = list(transaction_reader)
  File "/usr/lib/python3.5/codecs.py", line 321, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xdf in position 3101: invalid continuation byte

When I replace all umlauts in the test file 'elbi_umsaetze_20171018213618.csv' then this error goes away.

Workaround: open the file in Notepad and use search&replace to remove the special characters. (Yes this sucks, we're working on it!)

The following issues were flagged in the main script by pycodestyle.py. It's probably worth looking at these for the purpose of readability as the code grows more complex.
This is based off the test-prep branch as it stands currently (which I think needs to be merged into master, as I mentioned in #77.).
EDIT 2: Obviously we'll need to do this every now and then, but I've brought the existing code in the core up to standard anyway. Closing this.

EDIT: Worked through these, so moved them below this line.

• bank2ynab.py:311:80: E501 line too long (102 > 79 characters)
• bank2ynab.py:328:80: E501 line too long (99 > 79 characters)
• bank2ynab.py:133:80: E501 line too long (86 > 79 characters)
• bank2ynab.py:32:1: E402 module level import not at top of file
• bank2ynab.py:33:1: E402 module level import not at top of file
• bank2ynab.py:34:1: E402 module level import not at top of file
• bank2ynab.py:35:1: E402 module level import not at top of file
• bank2ynab.py:99:80: E501 line too long (85 > 79 characters)
• bank2ynab.py:218:80: E501 line too long (93 > 79 characters)
• bank2ynab.py:220:80: E501 line too long (98 > 79 characters)
• bank2ynab.py:219:80: E501 line too long (101 > 79 characters)
• bank2ynab.py:217:80: E501 line too long (92 > 79 characters)
• bank2ynab.py:216:80: E501 line too long (86 > 79 characters)
• bank2ynab.py:214:80: E501 line too long (95 > 79 characters)
• bank2ynab.py:213:80: E501 line too long (97 > 79 characters)
• bank2ynab.py:212:80: E501 line too long (95 > 79 characters)
• bank2ynab.py:104:80: E501 line too long (113 > 79 characters)
• bank2ynab.py:105:80: E501 line too long (119 > 79 characters)
• bank2ynab.py:106:80: E501 line too long (115 > 79 characters)
• bank2ynab.py:107:80: E501 line too long (115 > 79 characters)
• bank2ynab.py:108:80: E501 line too long (116 > 79 characters)
• bank2ynab.py:109:80: E501 line too long (112 > 79 characters)
• bank2ynab.py:110:80: E501 line too long (115 > 79 characters)
• bank2ynab.py:111:80: E501 line too long (119 > 79 characters)
• bank2ynab.py:112:80: E501 line too long (112 > 79 characters)
• bank2ynab.py:113:80: E501 line too long (117 > 79 characters)
• bank2ynab.py:114:80: E501 line too long (104 > 79 characters)
• bank2ynab.py:122:80: E501 line too long (86 > 79 characters)
• bank2ynab.py:282:80: E501 line too long (141 > 79 characters)
• bank2ynab.py:296:80: E501 line too long (99 > 79 characters)
• bank2ynab.py:89:80: E501 line too long (81 > 79 characters)
• bank2ynab.py:276:13: E722 do not use bare except'
• bank2ynab.py:86:37: E128 continuation line under-indented for visual indent
• bank2ynab.py:87:37: E128 continuation line under-indented for visual indent

Sourced from andsens/ynab-csv-converter.

Row format: 'account_no', 'date', 'txn_date', 'payee1', 'payee2', 'text','amount', 'currency'

Unclear if header is present.

To be filled in as required. Delete fields that match default values.
Source Filename Pattern = example_transaction_export_filename
Input Columns = skip, Date, skip, Payee, Memo, Inflow, skip

From Issue #21 master list

My testing showed that the script stops if the imported CSV file ends with a newline.

Workaround: remove the trailing Enter.

Sourced from ma-ver-ick/ynab-converter.

File layout:
Sparkasse
Auftragskonto, Buchungstag, Valutadatum, Buchungstext, Verwendungszweck, Beguenstigter/Zahlungspflichtiger, Kontonummer, BLZ, Betrag, Waehrung, Info

To be filled in as required. Delete fields that match default values.
Source Filename Pattern = example_transaction_export_filename
Source Filename Extension = .csv
Use Regex for Filename = False
Source CSV Delimiter = ,
Source Has Column Headers = True
Input Columns = Date,Payee,Outflow,Inflow,Running Balance
Plugin =

From Issue #21 master list

As originally suggested here perhaps main() should call a new validate() function where a variety of validation checks could live?

1. "Configuration file not found [...]"
2. "Specified directory not found, attempting to find Downloads folder."

I noted that there are nearly 90 YNAB CSV projects. I want to contact each of them and ask for their CSV format. This ticket is my reminder to do that.

Edit 1: Here's a checklist in alphabetical order of all the repositories I've found (source / recent). Checked off when I have analysed them.
Edit 2: @nocalla added some subheadings to try make this list a bit more comprehensible. Try to make headings for scripts for specific banks if there are duplicates, as this will be the most efficient way for us to incorporate individual banks. Create an issue for each identified bank and use the bank format label to track - link the issue number back to the relevant entry on this list.
Edit 3: @nocalla has changed the checkmarks to mean that they have been made into an issue

I am adding the formats to our survey. (Results are here.)

Issue Created

Link to relevant issue search

• #89 Handelsbanken [SE]
  • joacand/HandelsbankenYNABConverter
  • jkauppinen/FinnishYnabCvsConverter (not entirely sure if this is the same bank)
• #90 Rabobank [NL]
  • Remco32/rabobank-to-ynab-converter
  • BrianSetz/rabonab
  • canassa/rabobank-ynab
  • eddokloosterman/php-rabobank-ynab-csv-converter
  • jtdev/ynab-converter [java]
  • karsalfrink/RabobankToYNAB [Python]
  • tjukker/YNAB-bank-statement-converters
  • rvgate/YnabConverter
• #91 ING [NL]
  • AndreSomers/ING2YNAB
  • dive/ing-to-ynab-json-converter
  • ernezt/ynab-ing-converter
  • Remco32/ing-to-ynab-converter
  • mvd81/ynabconverter [PHP]
• #92 Nordea [NO]
  • bivainis/ynab-csv-nordea
  • davols/nordea-ynab-convert
  • cosminlupu/to-ynab
  • balvig/ynab_import
• #93 JLP [UK]
  • JamesBelchamber/ynab-jlp-csv-converter
  • clancychilds/jlp-ynab-converter
• #94 Amex [International?] akirayamamoto/amexcleaner
• #95 Danksebank [DK]
  • mywonderchild/YNAB-CSV-Converter [Python]
  • trahkoi/ynab-danske-csv
  • andsens/ynab-csv-converter
• #96 Hypovereinsbank [DE] andsens/ynab-csv-converter
• #97 Nordnet [SE] andsens/ynab-csv-converter
• #98 Sparekassen Kronjylland [DK] andsens/ynab-csv-converter
• #99 Barclaycard [DE] ma-ver-ick/ynab-converter [Python]
• #100 Sparkasse [DE] ma-ver-ick/ynab-converter [Python]
• #102 Purdue Federal [US] wesmcouch/Purdue-Federal-You-Need-A-Budget-Converter-YNAB-
• #103 Budapest Bank [HU] trekdemo/bank2ynab.rb
• #104 First National Bank [ZA] toranome/fnb2ynab
• #105 Itaú Unibanco [BR]
  • willybarro/ynab-tools
  • Panthro/Itaucard2YNAB
• #106 Bancolombia [CO] Asgaroth/ynab
• #107 Alfa-Bank [RU] BatON-RW/ynab-csv-converters
• #108 Mint [Other Budgeting Software] tehmikey/YNABFileConverter
• #109 Simple.com [US] spuder/simple-ynab
• #110 Skandiabanken [SE] sklirg/SkandiabankenDoesYNAB
• #112 Easybank [AT] jbogard/EasyBankToYnab
• #113 Betterment [US] jat255/Betterment_to_YNAB
• #114 Metro Bank [UK] Thau/metro-ynab-convert
• #115 BB&T
  • sirbarrence/bbt_csv_to_ynab
  • phildopus/bbt-ynab-convert
• #116 Knab [NL] sido/knab-to-ynab
• #117 Barclays [UK]scottrobertson/barclays-to-ynab
• #118 Deutsche Sparkasse [DE]
  • Salem5/ConvertBankstatementForYNAB
  • git-commit/ynab-conversion-scripts
  • DarkSmile92/YNABConverterPy
  • heysl/csv-camt2ynab {not 100% sure this is the right bank for this one}
• #119 Siam Commercial Bank [TH] ROldford/pdf2ynab
• #120 Ally Cashback [US] philtr/csv2ofx
• #121 KB Bank [CZ] rockwood/kb
• #122 Schwab [US] & #123 TD Bank [US]RichardLitt/ynab-convert-csv
• #124 AXA Homebanking [BE] PVandenheede/AXA2YNAB
• #125 Migros Bank [CH] p-26/MigrosBankSwiftMessageParser
• #126 OCBC Bank [SG]fongming/OCBC-YNAB-Parser [Python]
• #127 Royal Bank of Scotland [UK] jcalero/ynab-rbs-converter [PHP]
• #128 UBS [CH] fahyik/UBS_YNAB
• #129 DNB [NO]
  • cronin101/DNB-2-YNAB-CSVConverter
  • Snikanes/ynab-convert
• #130 Alelo [BR] figueredo/bank2ynab
• #131 Deutschebank [DE] marfei/db-to-ynab [C#]
• #132 Deutsche Kreditbank [DE] martinth/dkb-to-ynab [Ruby]
• #133 Finnish Osuuspankki [FI] mixu-/ynab-csv [Python]
• #134 Swedbank [SE] jpsk/Swedbank-to-YNAB4-converter
• #135 Banco Bradesco [BR] pedroaxl/ynab-scripts
• #136 OTP [HU] abesto/otp2ynab
• #137 N26 [DE] DarkSmile92/YNABConverterPy
• #138 Chase [US] rennes-us/ynab-convert
• #139 Shopify [E-Commerce Platform] rennes-us/ynab-convert
• #140 Cathay [?] johnlinvc/ynab_tw
• #141 Consorsbank [DE] nicolaipischel/YnabBankDataConverter [C#]
• #142 Starmoney [Banking Software] fpoppinga/ynab-converter
• #143 Selected Polish Banks (not sure which!) [PL] mnowak/ynab-pl-converter [Java] this reads PDF files...
• #144 Personal Capital [Banking Software] mgrecar/pc_to_ynab [Ruby]
• #145 KSK [DE] Bockwurst2/ynab_csv This is all in German! I can't work it out
• #146 Comdirect [DE] Bockwurst2/ynab_csv

To Be Sorted

• this long list of country plugins for ofxstatement
• dkusidlo/ynab_csv_converter I think this works for 4 different banks. n26, taspa, diba, kospa are the names listed

Irrelevant/Empty/No bank specified

• PhantomOverride/ynab4-csv No bank specified
• ashraymehta/statement-converter No bank specified
• adnilsson/Bank2YNAB Bank not specified
• KrisMysen/terra-to-ynab [java] this is an empty stub
• pgr0ss/ynab_to_ledger this actually exports from Ynab to a program called Ledger.
• gallamine/CSV-to-YNAB empty
• mhusbyn/ynab-currency-converter this converts ynab4 files to different currency, unrelated
• markov-a/YNAB-to-SQLite3 This converts Ynab4 files to a SQLite database, unrelated.
• eqdw/ynabify General use library, no specific bank mentioned
• p-26/Mt940ToYnabConverter Empty
• esnunes/bsbr-to-ynab Empty
• asteroidb612/nynab-ynab4 Converter from nYNAB export to YNAB4 format
• rschoultz/SHB2YNAB Empty
• mika-s/FinancialExchangeConverter This is a general use one for converting csv files to a certain output format.

Already Supported

• AIB [IE] jasonmadigan/aib-to-ynab[JavaScript]
• AIB [IE] heedfull/AIBtoYNABCsvConverter[Python]

I think it could be worth having a chatgroup somewhere to coordinate in realtime. Last night I was almost undone by not realizing @nocalla was pushing to the same branch I was working on.

I’m happy with anything you guys like - telegram, whatsapp, irc...

Would it be helpful to rename the script from transaction_csv_cleanup.py to bank2ynab.py, and the config file from config.conf to bank2ynab.conf?

I believe it would help "the project" because it creates a direct correlation between the project name and the script name (like MS Excel is also called excel.exe). It would also be handy because users (me!) could put the script and config in the user-home directory or anywhere else that is already in the path, and avoid having to cd to the script location first or precede the command with the script path.

If you agree I could do the simple pull, but these are your file names so I don't want to steamroll over your choices, hence this question for your consideration.

Looking at the various converter scripts, one of the recurring conversions that have to be done is date parsing. Some banks use '-' as separator, some (for example Rabobank ) have no separator at all...

Python already has syntax for date parsing, so we should probably just have an optional parameter to surface it when necessary.

Date formats accepted by YNAB are documented here.

To kick things off, I'll start with these. I've tried to sort them in a sort of chronological order:

• nickname for this configuration (so people can have several)
• download from bank (boolean)
• download from bank URL
• bank login
• bank auth1
• bank auth2
• downloaded file path (like "/home/username/Downloads/")
• downloaded filename pattern (like "bank_export_*")
• downloaded filename extension (like ".csv")
• downloaded file CSV delimiter character (like "," or ";")
• downloaded file has column headers (boolean)
• downloaded file column names and column order
• convert (boolean)
• converted file filename pattern (prefix like "fixed_")
• converted file filename pattern (suffix like "_fixed")
• converted filename extension (like ".csv")
• remove downloaded file after conversion (boolean)

• how to get the script working
  • how to install required modules
  • how to modify the config file
  • how the script detects one or more config files, and how it finds the correct config section
  • how to take advantage of the bank formats included in the project

(see #4 (comment) for a starting point)

Situation:

• The check currently looks in the script directory for any file that endswith(".conf") (line 156). What is the reason for this choice of search pattern?

Problem:

• No error appears if the actual bank2ynab.conf is missing, if foo.conf is present.
• The search pattern would also find a file named foo.conf or even one named dont_use_this_bank2ynab.conf.
• These files would be found even if the actual bank2ynab.conf is present. Which one would then be used?

Suggestion:

• Search for the exact config file bank2ynab.conf.

This looks like a manifestation of the handling of Windows files outlined here,

Be careful using codecs.open() to read files on Windows. The docs say: """Note Files are always opened in binary mode, even if no binary mode was specified. This is done to avoid data loss due to encodings using 8-bit values. This means that no automatic conversion of '\n' is done on reading and writing.""" This means that your lines will end in \r\n and you will need/want to strip those off.

Sources:

• bivainis/ynab-csv-nordea
• davols/nordea-ynab-convert
• cosminlupu/to-ynab

2017-10-23,Kortköp 171019 GOG COM,,"-119,00","1.000,99"

To be filled in as required. Delete fields that match default values.
Source Filename Pattern = example_transaction_export_filename
Source Filename Extension = .csv
Use Regex for Filename = False
Source CSV Delimiter = ,
Source Has Column Headers = True
Input Columns = Date,Payee,Outflow,Inflow,Running Balance
Plugin =

https://github.com/scottrobertson/monzo-export
https://github.com/scottrobertson/starling-export
https://github.com/scottrobertson/tide-export
https://github.com/scottrobertson/teller-export (Barclays, Isle of Man Bank, Nationwide, Natwest, Royal Bank of Scotland, Santandar, TSB, Ulster Bank NI, HSBC UK)

source: https://www.reddit.com/r/ynab/comments/75fctx/tools_can_we_consolidate_all_our_ynab_import/do5uhua/

I can't remember my logic for the list(transaction_reader) line that's causing trouble in issue #12. Going to experiment with removing it.

python bank2ynab.py
Format: IE Bank of Ireland
Can't find: c:\users\example-username\Downloads
Trying: D:\Users\nocalla\Downloads
Parsing file: test_BOI_TransactionExport (2).csv
Parsed 28 lines
Writing file: fixed_test_BOI_TransactionExport (2).csv
Traceback (most recent call last):
File "bank2ynab.py", line 341, in
main()
File "bank2ynab.py", line 330, in main
write_data(file, output)
File "bank2ynab.py", line 280, in write_data
with CrossversionCsvWriter(new_filename, __PY2) as writer:
File "bank2ynab.py", line 44, in init
self.encoding = detect_encoding(self.file_path)
File "bank2ynab.py", line 110, in detect_encoding
with codecs.open(filepath, "r", encoding=enc) as f:
File "C:\Users\nocalla\AppData\Local\Programs\Python\Python35-32\lib\codecs.py", line 895, in open
file = builtins.open(filename, mode, buffering)
FileNotFoundError: [Errno 2] No such file or directory: 'D:\Users\nocalla\Downloads\fixed_test_BOI_TransactionExport (2).csv'

I want to improve/finish the user guide in the README.md file.

As sourced from @toyg:

Ok, I did some more testing and there is still an outstanding issue with csv not working properly in py2 when source data contains non-ascii characters, regardless of how source or target files are opened and regardless of unicode_literals (which I'd rather stay away from, anyway, since it changes a lot of things behind the scenes in py2).

This is even mentioned halfway through the examples, providing a solution (the UnicodeWriter class) that is fugly but will probably work. I'll have a look online for alternatives.

Prerequisite before sorting Pull #56

Sourced from ma-ver-ick/ynab-converter.

File layout:
Barclaycard Germany
Buchung/ Valuta, Belegdatum, Beschreibung, Kartennummer, Karteninhaber, Betrag

To be filled in as required. Delete fields that match default values.
Source Filename Pattern = example_transaction_export_filename
Source Filename Extension = .csv
Use Regex for Filename = False
Source CSV Delimiter = ,
Source Has Column Headers = True
Input Columns = Date,Payee,Outflow,Inflow,Running Balance
Plugin =

From Issue #21 master list

I'll change how my payee swap thing works to just use payee info only if memo is blank or there's no memo column. I think this is logical behaviour for a default setting and doesn't really need a switch on the configuration. Thoughts?

Sourced from akirayamamoto/amexcleaner

Input (no header):
# Date, Description, Amount, Bill Start Date
29/06/2012,Outflow description,R$ #.###,##, 01/06/2012
30/06/2012,Inflow description,- R$ #.###,##, 01/06/2012

To be filled in as required. Delete fields that match default values.
Source Filename Pattern = example_transaction_export_filename
Source Filename Extension = .csv
Use Regex for Filename = False
Source CSV Delimiter = ,
Source Has Column Headers = True
Input Columns = Date,Payee,Outflow,Inflow,Running Balance
Plugin =

From Issue #21 master list

Also found in Scottrobertson's code:

def perform_request(path)
  raise 'Cannot find config file' unless File.exists?(config_path)

Sourced from wesmcouch/Purdue-Federal-You-Need-A-Budget-Converter-YNAB-.

if row[1].include? '(Pending)' # Do not include pending transactions in the import
		next
	end
	
	if row[4][0].chr == '(' # Move outflows and inflows to proper columns
		datLength = row[4].length
		datLength = datLength - 2
		row[4] = row[4][1, datLength]
		row[5] = ""
	else
		row[5] = row[4]
		row[4] = ""
	end

To be filled in as required. Delete fields that match default values.
Source Filename Pattern = example_transaction_export_filename
Source Filename Extension = .csv
Use Regex for Filename = False
Source CSV Delimiter = ,
Source Has Column Headers = True
Input Columns = Date,Payee,Outflow,Inflow,Running Balance
Plugin =

From Issue #21 master list

This is a feature request, absolutely optional and not urgent, but there you go.

It would be cool to actually put in some logic to allow for templates from source filenames. What I mean is:

• UK Cooperative bank (and Smile online bank) generate csv files called XXXXXX_ZZZZZZZZ_YYYY_MM_DD.csv, where XXXXXX is the bank sort code, ZZZZZZZZ is the account number, and YYYY_MM_DD is the date when the file was generated.

• it would be cool if bank2ynab could understand these info, even superficially. This would probably help in any future feature that involves automatic uploading, since it might make it easier to target this or that account.

Let's fix this bad boy.

Scott Robertson has a method to push transactions into YNAB programmatically.

• Can we incorporate that into our script, in order to eliminate the manual step of drag-and-dropping the output CSV file?

If we can do that, we could all but automate the entire process, like he has done. Manually donwload a CSV file (possibly automate this later, for US/UK), then run bank2ynab which then auto-converts, auto-imports, and removes the original CSV and the converted CSV file.

source: https://github.com/scottrobertson/fintech-to-ynab#csv-imports:

You can import a CSV directly into YNAB via the CLI. To do so, please use the following:
python python/import.py --account=AccountName --path=/path/to.csv
The format should be: date, description, amount

Hi, I got a request from @toyg to add this issue here since you were looking for more formats with input examples to add to this project.

This is about support for Handelsbanken, a Swedish bank.

The file extension is .xls, but the content of the file is HTML. See attached file for input example. Note that the actual data is anonimised, so the math does not add up.

kontotransactionlist.txt
(The attached file has extension .txt since it's not allowed to attach .xls files on github, but the real file should be .xls)

Since the format is unconventional you will probably need a plugin. I created a converter some time ago which may help: https://github.com/joacand/HandelsbankenYNABConverter

It's just a quick script and it probably needs be improved and changed to fit this project. You are free to use the code in any way you see fit.

Edit from Nocalla (tentative config 15/02/2018)
[SE Handelsbanken]
Source Filename Pattern = kontotransactionlist
Source Filename Extension = .xls
Input Columns = skip,Date,Payee,Inflow,skip
Source CSV Delimiter = ;
Header Rows = 7
Plugin = handelsbanken

From Issue #21 master list

Script language: Python2.7
Operating system: Windows
OS Version: 10

Request:
I'd like to be able to specify a switch in the config to remove non-transaction data from being written into my import file.

Background:
One thing that my bank puts into the records is when interest rates change, etc, and this gives records which I end up deleting prior to or after import, if I withdraw from an ATM either I get a second row of identifying information which is irrelevant for YNAB.

Approach idea:

• add a switch into the config
• add a test condition into the writer (I've done this locally in a version I can play with and try to push if of interest) which checks for input and output blanks in the row
• bypass the writing of the row if it's a non-transaction

Thoughts?

Situation:
Users (including ourselves) make changes to the bank2ynab.conf file, not least in order to specify our personal preference for the download directory.

Problem:
Having a user-modified local config file conflicts with the config file of the package because it has the same name.

Solution suggestion:
We could deliver the config file as bank2ynab.conf.txt and require the user to actively rename or copy it to bank2ynab.conf in order to activate it. That would be part of the user guide / installation instructions and could also be checked by the suggested validate() function.
This would allow us to maintain a true "package version" while allowing users to hold local preferences in their own actual copy.

Drawback:
When a user downloads an updated release, he must actively merge the changes in the new bank2ynab.conf.txt into his existing bank2ynab.conf.

The drawmack carries a risk of mistakes, but the benefit of content separation could make it worth it. Thoughts?

Sourced from andsens/ynab-csv-converter.
Also trahkoi/ynab-danske-csv.

Row format: 'date', 'text', 'amount', 'balance', 'status', 'cleared'

Unclear if header is present.

To be filled in as required. Delete fields that match default values.
Source Filename Pattern = example_transaction_export_filename
Source Filename Extension = .csv
Use Regex for Filename = False
Source CSV Delimiter = ,
Source Has Column Headers = True
Input Columns = Date,Payee,Outflow,Inflow,Running Balance
Plugin =

From Issue #21 master list

Script language: Python2 and Python3
Operating system: Linux
OS Version: Ubuntu 16.04 LTS

What did you DO? (steps to reproduce)
grabbed bank2ynab.py from master, then ran it once with py2 and once with py3.

What did you EXPECT to happen?
successful conversion.

What ACTUALLY happened?
Error using py2:

Traceback (most recent call last):
  File "bank2ynab.py", line 451, in <module>
    b2y = Bank2Ynab(get_configs(), __PY2)
  File "bank2ynab.py", line 423, in __init__
    bank_config = fix_conf_params(config_object, section)
  File "bank2ynab.py", line 231, in fix_conf_params
    config[key] = get_config_line(conf_obj, section_name, config[key])
  File "bank2ynab.py", line 245, in get_config_line
    line = conf_obj.get(section_name, param)
  File "/usr/lib/python2.7/ConfigParser.py", line 618, in get
    raise NoOptionError(option, section)
ConfigParser.NoOptionError: No option 'plugin' in section: 'AT Raiffeisen Bank'

Error using py3:

Traceback (most recent call last):
File "/usr/lib/python3.5/configparser.py", line 786, in get
value = d[option]
File "/usr/lib/python3.5/collections/init.py", line 878, in getitem
return self.missing(key) # support subclasses that define missing
File "/usr/lib/python3.5/collections/init.py", line 870, in missing
raise KeyError(key)
KeyError: 'plugin'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "bank2ynab.py", line 451, in <module>
    b2y = Bank2Ynab(get_configs(), __PY2)
  File "bank2ynab.py", line 423, in __init__
    bank_config = fix_conf_params(config_object, section)
  File "bank2ynab.py", line 231, in fix_conf_params
    config[key] = get_config_line(conf_obj, section_name, config[key])
  File "bank2ynab.py", line 245, in get_config_line
    line = conf_obj.get(section_name, param)
  File "/usr/lib/python3.5/configparser.py", line 789, in get
    raise NoOptionError(option, section)
configparser.NoOptionError: No option 'plugin' in section: 'AT Raiffeisen Bank'

• I do not understand what the option 'plugin' means. There is no config parameter containing that word.
• I also noted that the line Use Payees for Memo was removed from the default section in .conf in master. I would say that's a mistake and it should come back (default FALSE). Right?

Sourced from andsens/ynab-csv-converter.

Row format: 'line_id', 'bogf_date', 'trns_date', 'val_date', 'trns_type','stock_name', 'instr_type', 'isin', 'quantity', 'price','interest', 'fee', 'amount', 'currency', 'buy_price','result', 'total_qty', 'saldo', 'exch_rate', 'trns_text','shred_date', 'verification_number',

Unclear if header is present.

To be filled in as required. Delete fields that match default values.
Source Filename Pattern = example_transaction_export_filename
Source Filename Extension = .csv
Use Regex for Filename = False
Source CSV Delimiter = ,
Source Has Column Headers = True
Input Columns = Date,Payee,Outflow,Inflow,Running Balance
Plugin =

From Issue #21 master list

I nominate my handsome script as a starting point! In all seriousness though, your idea of a universal config file is a good one. So if we each add our individual script and adapt it for a config maybe we can merge scripts then?

Line 111:

I've looked things up and it seems that the 'open()' call should be replaced with an 'io.open()' call instead. I believe the reason is that 'open()', on Linux, does not have the 'newline' parameter (not even in my up-to-date version '3.5.1-3'). (source, source)

Using 'io.open()' requires an import of 'io' in line 20, which in turn requires 'io' to have to be installed first using 'sudo python -m pip install unicodecsv').

But using 'io.open()' gives me all kinds of follow-up troubles, so that can't be it either. I also found sources that suggested importing 'unicodecsv' instead of 'csv' because that handles things nicer, but that too became too hairy for a Python novice as I am.

Another suggestion (and another) was to code myself out of the parameter problem while staying with the normal 'open()' but I couldn't make it work:

#    with open(new_filename, "w", newline = "") as file:
    if os.name is "nt": newlinefoo = {'newline':''}
    else: newlinefoo = {}
    with open(new_filename, "w", newlinefoo) as file: