[Feature request] Improve post-lapse stability analysis about fsrs-optimizer HOT 13 CLOSED

I update the test version for notebook: https://github.com/open-spaced-repetition/fsrs4anki/blob/test/fsrs4anki_optimizer_test.ipynb

You can set the group_key to last_s_bin, last_d_bin and last_r_bin.

from fsrs-optimizer.

Please, add 3 more graphs:

1. new S as a function of previous S
2. new S as a function of last R
3. new S as a function of last D

Done:

from fsrs-optimizer.

Is there no way to speed this code up? It takes more time than any other part of the code related to plotting graphs.

Done in open-spaced-repetition/fsrs4anki@a5ae9f1

from fsrs-optimizer.

Thank you. A few more suggestions, nothing urgent:

Done in: open-spaced-repetition/fsrs4anki@7cb2bb4

from fsrs-optimizer.

@L-M-Sherlock, thank you for making a Google colab notebook! Please, implement my suggestion above as well.

from fsrs-optimizer.

I noticed something, I think Sherlock mentioned a similar phenomenon before.
I replaced the function for calculating PLS with a different one, and while RMSE for "Again" got better, the overall RMSE got worse.

RMSE for "Good" became worse somehow. I have no idea why.
However, I still believe that analyzing how PLS (and eventually S) depends on DSR will be helpful. That would require adding more graphs, which, hopefully, won't bother Sherlock too much.

from fsrs-optimizer.

It's not exactly what I wanted, I wanted 3 charts just like the one you originally made, but thank you.
Also, it would be useful to have some kind of fitness metric for fig5, like RMSE between true and predicted stability or R^2.
Like this: rmse = mean_squared_error(pls_df_group["true_s"], pls_df_group["predicted_s"], sample_weight=pls_df_group["total_count"], squared=False)
EDIT: actuall,y I can just copy-paste the code you provided 3 times. Not the most elegant solution, but eh.

from fsrs-optimizer.

@L-M-Sherlock I have another request. It seems that no matter how I improve the formula for PLS, it somehow makes the overall RMSE worse, which means that we need to analyze the main formula the same way to find out what's wrong. Please, add 3 more graphs:

1. new S as a function of previous S
2. new S as a function of last R
3. new S as a function of last D

And remove grades 2 and 4, leaving only 3. In other words, only reviews when user pressed "Good" should be used for analysis, to simplify things.
Also, for some reason predicted PLS is usually overestimated. Take a look at the following 3 charts (and this is after I modified the formulas):

Notice that the orange curve is usually above blue dots on all 3 charts. I don't know how to fix that.
Maybe S is usually underestimated, and overestimation of PLS somehow offsets underestimation of S, so the optimizer tries that? That's the only idea I have right now.

from fsrs-optimizer.

Is there no way to speed this code up? It takes more time than any other part of the code related to plotting graphs.

ns_dataset["last_state"] = ns_dataset.progress_apply(
    lambda x: my_collection.predict(
        ",".join(x["t_history"].split(",")[:-1]),
        ",".join(x["r_history"].split(",")[:-1]),
    ),
    axis=1,
)

from fsrs-optimizer.

Interestingly, S as function of previous S provides practically a perfect fit (log coordinates):

from fsrs-optimizer.

Thank you. A few more suggestions, nothing urgent:

1. Add RMSE between true S and predicted S (and do that for PLS too), like this:

rmse = mean_squared_error(pls_df_group["true_s"], pls_df_group["predicted_s"], sample_weight=pls_df_group["total_count"], squared=False)
print(f'RMSE={rmse:.2f}')

2. Analyzing new S as a function of previous S is more convenient when both are in log coordinates (same for PLS), so add this:

    if group_key == "last_s_bin":
        ax1.set_ylim(max(ns_df_group["predicted_s"].min(), 0.1), ns_df_group["predicted_s"].max() * 1.1)
        ax1.set_xscale("log")
        ax1.set_yscale("log")

3. Change the bounds. It seems that the estimate of true stability is capped at 365, I would suggest 3650 instead
4. Use this to set y_lim of S and PLS:
   ax1.set_ylim(0, min(pls_df_group["predicted_s"].max(), pls_df_group["true_s"].max()) * 1.1)

from fsrs-optimizer.

By the way, you may be interested in these charts for my collection. Also, this is with a modified formula for PLS that provides a better fit than the previous formula (see open-spaced-repetition/fsrs4anki#461 (comment))

It seems that the main formula provides a good fit overall (though the function for R could be improved, it seems), but the formula for PLS tends to overestimate S

from fsrs-optimizer.

Nice. Once you have enough free time, please test my idea regarding PLS: open-spaced-repetition/fsrs4anki#461 (comment)

from fsrs-optimizer.