Build a good costfunction about synergy-jointpdf HOT 5 CLOSED

Normalization is a problem, there are two possible paths it seems:

1. Use the Kullback-Leibler (NPEET) implementation instead of MSE, this should be the same 'units' as the normalized MI, because MI can be reduced to Kullbar-Leibler
2. Don't try to find one optimum, but find the front where there are no more possibilities where both cost functions are more optimal

from synergy-jointpdf.

Also, I can pass a list-of-lists to NPEET, but it should be [s1, s2, ...] where s1 is the first sample, and is of length n (with n variables in the system).

from synergy-jointpdf.

KL-divergence of NPEET is acting a bit weird so far, and alternative is https://arxiv.org/abs/1702.05222 (sent by Rick).

from synergy-jointpdf.

I think MI is not great for memory, as it ignores that biological systems return to the same state. I think I need to make the following changes:

1. Use the KL-divergence to determine memory
   1. This assumes that dt is in arbitrary time units
   2. This assumes that a natural system always returns to a value at some point
2. Discuss with Jaap Kaandorp what induces limit cycles
3. Brainstorm about the definition of memory: the KL-divergence seems to ignore the possibility off limit cycles
4. Add noise to the ODE itself

from synergy-jointpdf.

The landscape seems hard to traverse for optimizers, genetic algorithms perform relatively well, Nelder-Mead is horrible, as is basinhopping. I will try BFGS.

from synergy-jointpdf.