How to measure the embedding/representation difference between layers? about albert_zh HOT 2 CLOSED

may be you can take the first position, which is [CLS] token, it represent the sequence of that layer. so that you can have a hidden states as vector to represent that specific layer.
after you trained the model, or you get a checkpoint from pre-trained model, i think you can measure it, as all the parameters are fixed. it may not relate to a downstream dataset.

from albert_zh.

may be you can take the first position, which is [CLS] token, it represent the sequence of that layer. so that you can have a hidden states as vector to represent that specific layer.
after you trained the model, or you get a checkpoint from pre-trained model, i think you can measure it, as all the parameters are fixed. it may not relate to a downstream dataset.

OK, thanks~

from albert_zh.