erzakiev / frictionless Goto Github PK

The tie-adjustment factor $t_{ij}^3$ in the denominator of the formula below for the Friedman's statistic

overflows the int format... Observed when, for example, the number of dropouts (i.e. 0 expression) was tied many times in a big sample of $2700$ cells: $2691$ ties cubed is $19486825371$, which is an order of magnitude bigger than the INT_MAX value which is +2147483647

It was meant to be a troubleshooting feature that would allow for a precise control of the algorithm, but it is clearly unnecessary (and actually annoying) for the end user

Ditto

When the signature size is not fixed at a certain number, we faced a problem of explosive growth of the number of combinations in which a signature can be constructed (the issue is discussed in the study of a similar problem in the case subnetworks).

We tried penalization of the Friedman's S for each sample by dividing it with the number of genes in the solution n exponentiated to some real power alpha [0.1 ... 3]: S'=S/(n^alpha), yet it either produced very big signatures (read half or more of the input gene universe) with the lower values of alpha, or reduced the signatures to the minimum allowed number (3 in our case) in case of higher alphas.

We invite anyone to contribute their ideas on how to properly penalize the statistic so that the signatures don't expand to the totality of the inputted gene space yet don't collapse to the minimum number of genes allowed (i.e. 3).

With the help of Carola Doerr we can avoid redundant calculations, leading to the same basins of attraction over and over again (which is the case for the current vanilla implementation). She suggested using a hybrid taboo/double verification approach, which is in active development and will be posted shortly along with the details of the approach.