Hi,
I am trying to learn how to use Demeter and test some simple static strategies comparing to real-world positions and understand if my strategy works well and how close it is to real-world results.
my findings show gaps of 5-10% in collected pees calculation and also in position liquidity at the end of the test.

I am new to Python and Demeter so maybe I am doing something wrong.
is that an expected result? I saw in the code that you don't support fee calc when a tick is crossed, which may have an effect on accuracy, do you plan to support it in the future?

example:
i took this position 
usdc/eth 0.3%
24.10 7:39 - 29.10 18:00
https://etherscan.io/tx/0xe694c090428c8d104a5a9b7a6a10c9ae934de9a0df06a4de2d20719181d7a346
https://app.uniswap.org/pools/589914

demeter results:

usdc fee 150.73527 (vs 142)
eth fee 0.0792835 (vs 0.07613)

usdc in position 22488.34 (vs 25990)

eth in position 10.38382 (vs 11.02)

see the attached image for real results at the time of comparison.

I used demeter-fetch to get data from big query

[from]
chain = "ethereum" # polygon or ethereum
datasource = "big_query" # big_query or rpc or file
dapp_type = "uniswap" # uniswap or aave
[from.uniswap]
pool_address = "0x8ad599c3a0ff1de082011efddc58f1908eb6e6d8"
[from.big_query] # keep this section according to from.datasource
start = "2023-10-24"
end = "2023-10-29"

[to]
type = "minute" # minute or tick or raw
save_path = "./usdc-eth-data"
multi_process = false # process in multi_process, defaut: False

thats the strategy and test file:

from datetime import date

import pandas as pd

from demeter import UniV3Pool, Actuator, MarketInfo, UniLpMarket, TokenInfo, Strategy, EvaluatorEnum
from demeter.download import ChainType
from strategy_ploter import plot_position_return_decomposition
from _decimal import Decimal
from typing import Dict

pd.options.display.max_columns = None
pd.set_option('display.width', 5000)

class MyStrategy3(Strategy):
def init(self):
super().init()

def initialize(self):
    market: UniLpMarket = self.markets[market_key]
    market.add_liquidity(1636.04, 1935.31)
    super().__init__()

if name == "main":
usdc = TokenInfo(name="usdc", decimal=6) # declare token0
eth = TokenInfo(name="eth", decimal=18) # declare token1
pool = UniV3Pool(usdc, eth, 0.3, usdc) # declare pool
market_key = MarketInfo("uni_market3")

actuator = Actuator()  # declare actuator
broker = actuator.broker
market = UniLpMarket(market_key, pool)

broker.add_market(market)
# add balances according to this tx
broker.set_balance(eth, 11.977767765444931017)
broker.set_balance(usdc, 24275.246863)

actuator.strategy = MyStrategy3()

market.data_path = "../usdc-eth-03-data"
market.load_data(ChainType.Ethereum.name.lower(),
                 "0x8ad599c3a0ff1de082011efddc58f1908eb6e6d8", # ETH/USDC 0.3% uniswap v3
                 date(2023, 10, 25),
                 date(2023, 10, 29))
actuator.set_price(market.get_price_from_data())
actuator.run(
    evaluator=[EvaluatorEnum.MAX_DRAW_DOWN, EvaluatorEnum.ANNUALIZED_RETURNS]
)  # run test

# get result
evaluating_result: Dict[EvaluatorEnum, Decimal] = actuator.evaluating_indicator  # {<EvaluatorEnum.MAX_DRAW_DOWN: 3>: Decimal('0.04801148755391014037566625016'), <EvaluatorEnum.ANNUALIZED_RETURNS: 1>: Decimal('-0.9916890919693757317759809010')}
actuator.save_result("./res-hex",  # save path
                     account=True,  # save account status list as a csv file
                     actions=True)  # save actions as a json file and a pickle file

plot_position_return_decomposition(actuator.get_account_status_dataframe(),
                                   actuator.token_prices[eth.name],
                                   market_key)

the final status was:

net_value eth usdc uni_market3_net_value uni_market3_base_uncollected uni_market3_quote_uncollected uni_market3_base_in_position uni_market3_quote_in_position uni_market3_position_count

2023-10-29 23:59:00 46038.45896264917812562138567 0 4629.96960295986140984796065 41408.48935968931671577342502 150.7352701088900142474106575 0.07928350165347513303123274220 22488.34657493972420853693269 10.38382421897665466066069915 1

backtest-20231029-203206.action.json
backtest-20231029-203206.account.csv

How accurate can be the calculation of the commission earned by the uniswap v3 pool? (back test on historical data)

When I run the backtest and compare the result with the results of my real pool, the difference in eth is less than 0.06 percent, but in usdt the difference is 0.428%.
Is it possible to get 1 in 1 exact data?,
and if not, how to get closer to these values?
What is acceptable difference ?

Sorry for asking questions about unfinished work.

I'm trying to run some back testing of uniswap v3 pools with Deribit options. And I'm using Tardis for historical datasets. With tardis i can get the datasets you're using while unit testing your code with all necessary fields. Problem is that this csv have different to yours internal structure.

Example of my CSV:
deribit_options_chain_2024-01-01_OPTIONS.csv

exchange,symbol,timestamp,local_timestamp,type,strike_price,expiration,open_interest,last_price,bid_price,bid_amount,bid_iv,ask_price,ask_amount,ask_iv,mark_price,mark_iv,underlying_index,underlying_price,delta,gamma,vega,theta,rho
deribit,BTC-27SEP24-160000-P,1704067200027000,1704067200029840,put,160000,1727424000000000,0,2.45,,,,,,,2.4467,75.18,BTC-27SEP24,46593.7,-0.94333,0,45.67483,-6.35109,-1169.77956
deribit,BTC-27SEP24-140000-P,1704067200044000,1704067200047184,put,140000,1727424000000000,0,2.021,0.0001,1,,,,,2.0214,72.68,BTC-27SEP24,46593.7,-0.92595,0,56.21241,-7.55606,-1017.04325
deribit,BTC-27SEP24-95000-P,1704067200045000,1704067200047717,put,95000,1727424000000000,0,1.095,0.0002,1,,,,,1.0816,67.4,BTC-27SEP24,46593.7,-0.82594,0.00001,103.00981,-12.84055,-658.24389

From where do you get the historical dataset for testing deribit?
If you are also using Tardis, can you share the code that downloading that csv?

Thank you a lot, you created an amazing tool, you did a great job!

Hello there.
Currently, I'm working on a script that finds and buys call & put, daily options for specific price range.
For example, when the date is 2024.02.15 08:00 I want to buy the nearest daily put option for price 2910. Nearest is ETH-16FEB24-2900-P.

row about this option

Code:
market.buy('ETH-16FEB24-2900-P', 1)

Exception:
demeter._typing.DemeterError: insufficient order to buy, required amount is 1, available amount is 0

I think that "bids" column in .csv's that you send via dropbox is not full. Otherwise, how I can buy daily put&call options for specific price range.

Thank you for your response,
On the edge of a breakdown
Love, Alex

The function demeter.uniswap.market.UniLpMarket.__update_fee updates LP fees for each position opened.

However, if position_liquidity > pool_liquidity, the position's share is 1 so it receives all the fees generated by the pool in the current timestamp_index. This leads to a bug where for n positions opened, broker earns n * all_fees_generated_in_current_timestamp_index

I think when calculating share, liquidity of all brokker's positions should be considered as well

Is there any way to run back test with multiple markets at the same time?
Interesting in Deribit+Uniswap v3 pools

Hi, when I run the backtest for a USDC/ETH pool, the output JSON & CSV shows me buy and sell transactions with a 'price' field in the block but this field is not present in the add or remove liquidity action type blocks in the JSON. Would be great if this could be added so it can be tracked what the price of ETH was when the add or remove liquidity action type was triggered. Tried doing this myself by editing the Actuator class but did not succeed.

Hi,
I am trying to run this lib and its strategy samples.
is there more doc on the logs that are printed after backtester is done, I would like to understand the performance of a stategy and it's not very clear or explained in code/readme/manual.

Alternatively is the a project/UI tool/open source that uses the lib and you can point me to it?

For example, i get these log prints when running:

(demeter) strategy-example (master) ✗ python3 12_constant_Interval.py

2023-10-18 17:58:15,790 - INFO - data has been prepared
2023-10-18 17:58:15,791 - INFO - init strategy...
2023-10-18 17:58:15,793 - INFO - start main loop...
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████| 15840/15840 [00:06<00:00, 2588.13it/s]
2023-10-18 17:58:21,940 - INFO - main loop finished
2023-10-18 17:58:22,042 - INFO - Broker
Asset amounts
usdc :0 eth :0.0707372737666261654660170258
Markets
uni_market(UniLpMarket)
token0 :usdc token1 :eth fee :0.0500 is 0 base :True
positions
lower_tick upper_tick pending0 pending1 liquidity
0 201318 203819 11.25578570434480907228378585 0.006292688370498359591955467510 387963507047826

2023-10-18 17:58:22,042 - INFO - Account Status
2023-10-18 17:58:22,042 - INFO - net_value usdc eth uni_market_net_value uni_market_base_uncollected uni_market_quote_uncollected uni_market_base_in_position uni_market_quote_in_position uni_market_position_count
2022-08-05 00:00:00 1999.502335313296790636961503 0 0.0707372737666261654660170258 1885.735520583365642398358784 0.002317256849493052428779442252 1.122703402876110291885239120E-8 999.4999999999993698541062311 0.551036957449638596322551911 1
2022-08-05 00:01:00 1999.503159161726183291339706 0 0.0707372737666261654660170258 1885.736344431795035052736987 0.002376082206750144045744816348 4.868986665685556989979950670E-7 999.4999999999993698541062311 0.551036957449638596322551911 1
2022-08-05 00:02:00 1999.505414272171702021080411 0 0.0707372737666261654660170258 1885.738599542240553782477692 0.002376082206750144045744816348 0.000001889068230138732202928621405 999.4999999999993698541062311 0.551036957449638596322551911 1
2022-08-05 00:03:00 2000.010042428917993894827471 0 0.0707372737666261654660170258 1886.186332913802682057146282 0.007769810024119111652365549753 0.000001954612027542642486319156519 1003.389977568857216796791728 0.5486188742034648538134621155 1
2022-08-05 00:04:00 2000.011791998832975502438095 0 0.0707372737666261654660170258 1886.188082483717663664756906 0.008514742480334209166075439656 0.000002578957475953257275966616112 1003.389977568857216796791728 0.5486188742034648538134621155 1
... ... ... ... ... ... ... ... ... ...
2022-08-15 23:55:00 2097.331940403029013849406112 0 0.0707372737666261654660170258 1962.511694511516152630184097 11.25578570434480907228378585 0.006292688370498359591955467510 1939.262489379837990181000271 0 1
2022-08-15 23:56:00 2097.331940403029013849406112 0 0.0707372737666261654660170258 1962.511694511516152630184097 11.25578570434480907228378585 0.006292688370498359591955467510 1939.262489379837990181000271 0 1
2022-08-15 23:57:00 2097.185207452937580868677203 0 0.0707372737666261654660170258 1962.499707685831809325579027 11.25578570434480907228378585 0.006292688370498359591955467510 1939.262489379837990181000271 0 1
2022-08-15 23:58:00 2096.950739708552203283392838 0 0.0707372737666261654660170258 1962.480553678645302295496506 11.25578570434480907228378585 0.006292688370498359591955467510 1939.262489379837990181000271 0 1
2022-08-15 23:59:00 2096.950739708552203283392838 0 0.0707372737666261654660170258 1962.480553678645302295496506 11.25578570434480907228378585 0.006292688370498359591955467510 1939.262489379837990181000271 0 1

[15840 rows x 9 columns]
2023-10-18 17:58:22,049 - INFO - Backtesting finished, execute time 6.25815224647522s

Thanks.

i can change code of demeter-fetch to download 30sec/row data, but dont know is there any way to work with such file with demeter