Working with uniformly initialized Linear layers about feathermap HOT 4 OPEN

If you take a look at the __norm_V method within class FeatherNet

    def __norm_V(self) -> None:
        """Normalize global weight matrix. Currently implemented only for uniform
        intializations"""
        # sigma = M**(-1/4); bound follows from uniform dist.
        bound = sqrt(12) / 2 * (self._size_m ** (-1 / 4))
        torch.nn.init.uniform_(self._V1, -bound, bound)
        torch.nn.init.uniform_(self._V2, -bound, bound)

one can see that, at present, only uniform initializations are supported.

from feathermap.

Yes, also looks like fan-in is only supported as found in __unregister_params

from feathermap.

Thanks! I got it to work by suitably modifying scaler to reflect SIREN's initialisation. Its not perfect, but works.

Seems like the low rank decomposition, even with no compression impacts training dynamics (performance drop). Not surprising, since the task isn't classification & performance is measured by PSNR.

from feathermap.

Yeah, just by setting compression = 1.0 (which still expresses weights as a matrix product, albeit one of the same size as is uncompressed) will cause a drop in performace. This is due to, I think, just an inefficient structuring of weights (nonlinearly coupled).

After experiments and characterization, I've found that the FeatherMap package is really designed to shine at very very deep compressions. Its in this range you can get the largest delta_size/delta_performance

from feathermap.