recurrent state about block-recurrent-transformer-pytorch HOT 5 CLOSED

state_q, state_k, state_v = (self.state_to_q(self.init_state), *self.state_to_kv(self.init_state).chunk(2, dim = -1))

from block-recurrent-transformer-pytorch.

@wj210 only if you don't pass in the previous state, it will use the init_state, much like the initial hiddens of an RNN https://github.com/lucidrains/block-recurrent-transformer-pytorch/blob/main/block_recurrent_transformer_pytorch/block_recurrent_transformer_pytorch.py#L518

from block-recurrent-transformer-pytorch.

@wj210 do let me know if i misunderstood something!

from block-recurrent-transformer-pytorch.

I tried checking the recurrent layers to see if it is trained, for example the projection/mlp layers and it is not. After changing the key and values to orig_state, then it trains. The code shows that the keys and values are always using self.init_state, which is hard coded in, instead of referring to orig_states.

from block-recurrent-transformer-pytorch.

@wj210 🤦‍♂️ my mistake! thank you!

from block-recurrent-transformer-pytorch.