Conv-TasNet Cumulative Layer Norm Bug? about dnn-based_source_separation HOT 4 CLOSED

My CumulativeLayerNorm1d is based on official implementation. You can compare it with mine here.
If there is any possibility that I have misunderstood something, let me know in more detail.

from dnn-based_source_separation.

compared to the official implementation, your code writes cum_var=cum_squared_mean - cum_mean**2 while the official code writes cum_var = (cum_pow_sum - 2cum_meancum_sum) / entry_cnt + cum_mean.pow(2).

then cum_squared_mean should = (cum_pow_sum - 2cum_meancum_sum) / entry_cnt
In your code cum_squared_mean=cum_squared_sum/cum_num. as entry_cnt=cum_sum,
cum_squared_sum should = cum_pow_sum-2cum_meancum_sum

however cum_squared_sum is defined as torch.cumsum(step_pow_sum, dim=1) which equals cum_pow_sum in the official implementation, so you're missing 2cum_meancum_sum.

am I missing anything here? or is this omitted for a speed / accuracy tradeoff?

from dnn-based_source_separation.

In official implementation:

In my repo:

I renamed some variables in check_layer_norm.ipynb at #101 (comment) for readability.

from dnn-based_source_separation.

My bad! thank you for clearing it up!

from dnn-based_source_separation.