Back test results are different between cached data and live data about pybroker HOT 3 CLOSED

Hi @none2003,

Thanks for pointing this out! The notebook should include an additional check that 20 bars have been completed:

def rank(ctxs: dict[str, ExecContext]):
    if tuple(ctxs.values())[0].bars <= 20:
        return
    scores = {
        symbol: ctx.indicator('roc_20')[-1]
        for symbol, ctx in ctxs.items()
    }
    sorted_scores = sorted(
        scores.items(), 
        key=lambda score: score[1],
        reverse=True
    )
    threshold = pyb.param('rank_threshold')
    top_scores = sorted_scores[:threshold]
    top_symbols = [score[0] for score in top_scores]
    pyb.param('top_symbols', top_symbols)

def rotate(ctx: ExecContext):
    if ctx.bars <= 20:
        return
    if ctx.long_pos():
        if ctx.symbol not in pyb.param('top_symbols'):
            ctx.sell_all_shares()
    else:
        target_size = pyb.param('target_size')
        ctx.buy_shares = ctx.calc_target_shares(target_size)
        ctx.score = ctx.indicator('roc_20')[-1]

What's happening is that caching does not maintain the order of the rows of the original DataFrame. During the initial 20 bars of the backtest, the 20 day ROC score will be NaN, so the order of the scores will depend on the order that the symbols appear in the DataFrame. That ordering becomes non-deterministic once the cached data is read.

The issue is with the notebook code, not the caching mechanism. I have updated the notebook with the fix:
https://www.pybroker.com/en/latest/notebooks/10.%20Rotational%20Trading.html

Let me know if you need anything else.

from pybroker.

PyBroker does maintain the ordering of the dates, so the time sequence will be correct. However, the order in which the rows appear for different symbols on the same date does not have a guaranteed ordering.

from pybroker.

Hi @none2003,

Thanks for pointing this out! The notebook should include an additional check that 20 bars have been completed:

def rank(ctxs: dict[str, ExecContext]):
    if tuple(ctxs.values())[0].bars <= 20:
        return
    scores = {
        symbol: ctx.indicator('roc_20')[-1]
        for symbol, ctx in ctxs.items()
    }
    sorted_scores = sorted(
        scores.items(), 
        key=lambda score: score[1],
        reverse=True
    )
    threshold = pyb.param('rank_threshold')
    top_scores = sorted_scores[:threshold]
    top_symbols = [score[0] for score in top_scores]
    pyb.param('top_symbols', top_symbols)

def rotate(ctx: ExecContext):
    if ctx.bars <= 20:
        return
    if ctx.long_pos():
        if ctx.symbol not in pyb.param('top_symbols'):
            ctx.sell_all_shares()
    else:
        target_size = pyb.param('target_size')
        ctx.buy_shares = ctx.calc_target_shares(target_size)
        ctx.score = ctx.indicator('roc_20')[-1]

What's happening is that caching does not maintain the order of the rows of the original DataFrame. During the initial 20 bars of the backtest, the 20 day ROC score will be NaN, so the order of the scores will depend on the order that the symbols appear in the DataFrame. That ordering becomes non-deterministic once the cached data is read.

The issue is with the notebook code, not the caching mechanism. I have updated the notebook with the fix: https://www.pybroker.com/en/latest/notebooks/10.%20Rotational%20Trading.html

Let me know if you need anything else.

Thank you for your quick action!

I'm a little confuse about "caching does not maintain the order of the rows of the original DataFrame", so how pybroker make sure the time sequence when get data from cache? Is data get resorted when retrieve from cache when using?

from pybroker.