probstj / ccgains Goto Github PK

There are some coins that have changed ticker symbols. One example is AntShares(ANS) -> NEO (NEO) which happened in 2017. Most exchanges updated their historical pricing data to reflect the new trading pair (e.g. NEO/BTC instead of ANS/BTC), so it is not possible to get pricing data.

I think the easiest way to correct for issues like this (there are a few other examples I know of besides just ANS/NEO) would be to update the ticker symbol on ingestion, but I could see some reasons to not do that, and instead create a second currency relation (e.g. pull NEO/BTC data, but rename it to ANS/BTC).

I would like to add a optional argument in the form of a dict {'old ticker': 'new ticker'}, but would appreciate suggestions on where it makes the most sense to implement this.

Hi,

first I wanted to thank you for this very nice library. I am trying to use it to do my tax declaration but there is something I don't understand: how do you consider transactions between your wallets: e.g. if you buy a 1 BTC in a website and then transfer it to your personal wallet (like a Trezor). Then you are not really selling your bitcoin.

I tried using the function th.add_missing_transaction_fees(raise_on_error=True) but I am getting an error that says that a withdrawal is lower than the previous deposit: (fake BTC data)

ValueError: The withdrawal from 2017-09-19 13|33| 14+00:00 (0.4110000 BTC, Wallet A) is lower than the first deposit (2017-09-22 21:45:43+00:00, 0.5000000 BTC, Wallet B) following it.

The problem is that the function is trying to couple the first withdrawal with the next deposit, while actually the first withdrawal corresponds to a payment (i.e. I'm not getting it back to any wallet).
Actually, right after the deposit a withdrawal is listed, which corresponds to the actual source of the deposit. This are the relevant movements: (fake BTC data)

...
20 Abhebung   19.09.2017, 13:33:14 0,00000000 BTC 0,41100000 BTC 0,00000000 BTC Wallet A
21 Einzahlung 22.09.2017, 21:45:43 0,50000000 BTC 0,00000000 0,00000000 BTC Wallet B
...
23 Abhebung   22.09.2017, 21:45:43 0,00000000 BTC 0,50096500 BTC 0,00000000 BTC Wallet A
...

So, as can be seen, the transaction No. 20 is considered to go to the No. 21, when actually the No. 23 should be the one connected to the No. 21. They are happening at the same time, whilst the other one occurred a few days earlier.

Hi, I want to use a custom trade file, in format csv, may be you have some example of csv-file and how to custom the append_csv. [ append_csv(file_name, param_locs=range(0, 11), delimiter=’, ’, skiprows=1, de-fault_timezone=None) ]
My idea it's to use your sw to calc the capital gain of trade [ future, options, stocks...], that I export the info in csv format from my broker site.
Many thanks.

I'm reviewing the library to understand whether it fits for my task of analyzing history of a trading strategy. I found some strange thing which I want to clarify.

In bags.py pay() method uses rate obtained from self.relation.get_rate() which is some historical data obtained from an exchange. It looks fine for processing non-base_currency trades (BTC:ALTCOIN markets), but it's looks strange for trades involving base_currency (BTC:USD). For such trade you can clearly see what amount of base currency was obtained by selling asset just from trade transaction, you don't need to refer to some external average rate for that. Furthermore, such method can influence "Proceeds" giving slightly different amounts of base currency which will be not equal to really obtained amount.

Though I see that detailed report actually uses real sell price for rendering first line (Sold xxx BTC for xxx BASE) {% set buy_amount = sale.buy_ratio * sale.to_pay %}, but all "proceeds", "total taxable gain" etc uses rate-derived base asset amounts.

I want to understand why such behavior was implemented, maybe I'm missing something? I actually needed to analyze trading history on XXX:YYY market and I was surprised why I cannot do that just by using trade history data only.

I'm getting the following error when running ccGains. No error reported when ccGains was run on the previous years data .

INFO    : TRADE #1
INFO    : Exchange on 2018-01-07 09:38:26+00:00: Acquired ...
INFO    : Processing trade: Exchange, 2018-01-07 09:38:26+00:00, BTC, ...
INFO    : Selling ... XRP for ... BTC at Poloniex (2018-01-07 09:38:26+00:00, fee: ... BTC)
WARNING : Date 2018-01-07 09:38:26+00:00 missing in cached data. Repeating request to API
INFO    : Fetched historical price data with request: https://poloniex.com/public?command=returnTradeHistory&currencyPair=BTC_XRP&start=1515283200&end=1515369600
INFO    : Successfully fetched 1000 trades
INFO    : Saved bags' state to /ccGains/profile-XX/precrash.json
Traceback (most recent call last):
  File "pandas/_libs/index.pyx", line 439, in pandas._libs.index.DatetimeEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 987, in pandas._libs.hashtable.Int64HashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 993, in pandas._libs.hashtable.Int64HashTable.get_item
KeyError: 1515315600000000000

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/ccGains/env3/lib/python3.6/site-packages/ccgains/bags.py", line 843, in pay
    self.relation.get_rate(dtime, currency, self.currency))
  File "/ccGains/env3/lib/python3.6/site-packages/ccgains/relations.py", line 280, in get_rate
    result *= self.historic_prices[step_key].get_price(dtime)
  File "/ccGains/env3/lib/python3.6/site-packages/ccgains/historic_data.py", line 113, in get_price
    return df.at[pd.Timestamp(dtime).floor(df.index.freq)]
  File "/ccGains/env3/lib/python3.6/site-packages/pandas/core/indexing.py", line 2270, in __getitem__
    return self.obj._get_value(*key, takeable=self._takeable)
  File "/ccGains/env3/lib/python3.6/site-packages/pandas/core/series.py", line 1187, in _get_value
    return self.index.get_value(self._values, label)
  File "/ccGains/env3/lib/python3.6/site-packages/pandas/core/indexes/datetimes.py", line 939, in get_value
    return self.get_value_maybe_box(series, key)
  File "/ccGains/env3/lib/python3.6/site-packages/pandas/core/indexes/datetimes.py", line 971, in get_value_maybe_box
    key, tz=self.tz)
  File "pandas/_libs/index.pyx", line 81, in pandas._libs.index.IndexEngine.get_value
  File "pandas/_libs/index.pyx", line 89, in pandas._libs.index.IndexEngine.get_value
  File "pandas/_libs/index.pyx", line 441, in pandas._libs.index.DatetimeEngine.get_loc
KeyError: Timestamp('2018-01-07 09:00:00+0000', tz='UTC')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "ccGains-XX-2018.py", line 386, in <module>
    main()
  File "ccGains-XX-2018.py", line 249, in main
    bf.process_trade(trade)
  File "/ccGains/env3/lib/python3.6/site-packages/ccgains/bags.py", line 1087, in process_trade
    'buy_ratio': trade.buyval / trade.sellval})
  File "/ccGains/env3/lib/python3.6/site-packages/ccgains/bags.py", line 848, in pay
    currency, self.currency, dtime))
  File "/ccGains/env3/lib/python3.6/site-packages/ccgains/bags.py", line 240, in _abort
    raise exception_type(msg)
ValueError: Could not fetch the price for currency_pair XRP_AUD on 2018-01-07 09:38:26+00:00 from provided CurrencyRelation object.

Just a question/discussion, not an issue:

Do you have any strong opinions about overall code style/format? Specifically, most other projects I work own or contribute to use linters, code formatters, static type checkers, etc. in order to help maintain consistency between contributors and also to pick up on code smell early.

If you really prefer to keep it the way it is, no problem and I'll stick with the style you already have going on.

But if you're not averse to it, I'd be happy to do the work to standardize the project. My preferences would be:

• pylinter (code inspection)
• black (code formatting - note that it enforces 88 column lines instead of the 80 that most of the code currently is at)
• mypy (static type hinting)

I think in the longer term (and I intend to keep using this library for quite a while) this would lead to better code quality, easier contributions from others, and enhance readability.

Relatedly, I would like to start writing unit tests. That's less project-wide impact, but just the same, do you have any preference on testing framework? Again, I'd be happy to get things going in the right direction if you agree, but don't want to step on toes here 😄

Let me know what you think, and thanks again for a great project!

Regards,
Anson

Hello there, nice project. I just stumbled upon it.

I see you have LIFO, FIFO, and LPFO

Typically users would want to minimize taxable gains by first eliminating their highest cost assets first, not the lowest cost assets.

In the US, we typically default to HCFO (Highest Cost First Out) that is if we are not also incorporating things like long term and short term gains and losses.

To stick with your P for price instead of C for cost, I would recommend also adding a HPFO method. I would assume the method would be a simple copy paste and change to sort decending rather than sort ascending.

Hi,

I'd like to request that support for crypto.com csv-files gets added.

Descriptions of different transactions.

• 'Crypto Earn' = interest payed on funds deposited to Crypto Earn
• 'MCO Stake Rewards' = interest payed on staked MCO
• 'EUR -> BTC' = BTC purchase using EUR

There is no need to process Crypto Earn deposits or withdrawals as they are not taxable events.

crypto_com_example.zip

I think I successfully loaded ccgains on my Linux machine but cannot get pip to work:
tomf@tomVaio:/ccgains/ccgains$ pip install .
Traceback (most recent call last):
File "/usr/bin/pip", line 9, in
from pip import main
ImportError: cannot import name main
I get the same error with
tomf@tomVaio:/ccgains/ccgains$ pip install --user .
as well as in one directory up:
tomf@tomVaio:/ccgains$ pip install --user .
To help figure out the problem here are the directory contents:
tomf@tomVaio:/ccgains$ dir
ccgains COPYING data examples README.md setup.py
contributors.txt COPYING.LESSER docs MANIFEST.in run_tests.sh tests
tomf@tomVaio:/ccgains/ccgains$ dir
bags.py historic_data.py relations.py templates
binance_util.py init.py reports.py trades.py
I removed the ccgains directories and "git cloned" again but got the same error message with pip. There are no obvious errors that I can see during "git clone" but here is confirmation, in case that helps determine the problem:
tomf@tomVaio:$ git clone https://github.com/probstj/ccgains.git
Cloning into 'ccgains'...
remote: Enumerating objects: 94, done.
remote: Counting objects: 100% (94/94), done.
remote: Compressing objects: 100% (37/37), done.
remote: Total 564 (delta 57), reused 94 (delta 57), pack-reused 470
Receiving objects: 100% (564/564), 1.16 MiB | 0 bytes/s, done.
Resolving deltas: 100% (362/362), done.
Checking connectivity... done.

Trying to get that working on Ubuntu 18, and when running the example.py, the following errors occur:

Traceback (most recent call last):
File "example.py", line 392, in
main()
File "example.py", line 252, in main
log_bags(bf)
File "example.py", line 387, in log_bags
' ' + '\n '.join(str(bags).split('\n')))
File "/home/t/.local/lib/python3.6/site-packages/ccgains/bags.py", line 388, in str
return self.to_data_frame().to_string(
File "/home/t/.local/lib/python3.6/site-packages/ccgains/bags.py", line 385, in to_data_frame
axis='columns')
File "/home/t/.local/lib/python3.6/site-packages/pandas/util/_decorators.py", line 221, in wrapper
return func(*args, **kwargs)
File "/home/t/.local/lib/python3.6/site-packages/pandas/core/generic.py", line 1329, in rename_axis
raise ValueError("Use .rename to alter labels " "with a mapper.")
ValueError: Use .rename to alter labels with a mapper.

Any ideas on how to fix this? thanks!

I have been unable to install ccGrains on MAC with Python 3.7.2

Here are the steps I am following:
git clone https://github.com/probstj/ccgains.git
cd ccgains
pip3 install .
I get the this error:

* USE_PKGCONFIG: False
.. ERROR:: Could not find a local HDF5 installation.
   You may need to explicitly state where your local HDF5 headers and
   library can be found by setting the ``HDF5_DIR`` environment
   variable or by using the ``--hdf5`` command-line option.

Looks like an error in installing the required 'tables' module.
I have tried to set the HDF5_DIR env variable, using --hdf5, and brew install hdf5
But nothing helped in fixing this issue. Any ideas?

Currently fee may be paid in buy_currency or sell_currency. I'm analyzing data from Bitshares decentralized exchange, and there are several types of fees. One of them is a fee for each transaction performed, and this fee may be paid in any asset.

Here is a withdrawal example where fee was paid in BTS core asset:

Kind,Date,Buy currency,Buy amount,Sell currency,Sell amount,Fee currency,Fee amount,Exchange,Mark,Comment
Withdrawal,2017-12-19T23:21:09,,0,STRANGE,185.0,BTS,0.21851,Bitshares,-1,1.11.105563860

To handle such situation probably an additional check into BagQueue.process_trade() may be added: if fee_currency != buy/sell currency THEN make virtual Trade object and recursively pass it into process_trade(). Similar check in Trade() which raises an exception may be changed to simply set self.feeval = 0.

I found this library (blockcypher-python [1]) that can be used to get the real fees from a given transaction. Maybe it could be integrated?

It works like this

from blockcypher import get_transaction_details, from_satoshis

tx = get_transaction_details('b44003aaad20e7dffd66c7bf7f9f9abc4498de1932476c4db7fc62f5b330f64b')
fees = from_satoshis(tx['fees'], 'btc')
print(fees)
>>> 1.014e-05

[1] https://github.com/blockcypher/blockcypher-python

First of all, thank you for this nice tool. It saved me a lot of time and I really like the reports.
However, I think the Binance Trades TPLOC is not correct.
In the Trade docstring is written that the buy_amount should exclude transaction fees but the current TPLOC included fees.

Current:

TPLOC_BINANCE_TRADES = {
'kind': 2, 'dtime': 0,
'buy_currency': lambda cols: currency_for(cols, 'buy'),
'buy_amount': lambda cols: [Decimal(cols[4]), Decimal(cols[5])][(cols[2].upper() == 'SELL')],
'sell_currency': lambda cols: currency_for(cols, 'sell'),
'sell_amount': lambda cols: [Decimal(cols[4]), Decimal(cols[5])][(cols[2].upper() == 'BUY')],
'fee_currency': 7,
'fee_amount': 6,
'exchange': 'Binance',
}

Suggestion:

TPLOC_BINANCE_TRADES = {
'kind': 2, 'dtime': 0,
'buy_currency': lambda cols: currency_for(cols, 'buy'),
'buy_amount': lambda cols: [Decimal(cols[4])-Decimal(cols[6]), Decimal(cols[5])-Decimal(cols[6])][(cols[2].upper() == 'SELL')],
'sell_currency': lambda cols: currency_for(cols, 'sell'),
'sell_amount': lambda cols: [Decimal(cols[4]), Decimal(cols[5])][(cols[2].upper() == 'BUY')],
'fee_currency': 7,
'fee_amount': 6,
'exchange': 'Binance'
}

Sample CSV data: