Research : Embedding Aware Attention about tabnet HOT 10 CLOSED

Hello @joseluismoreira,

There are two PR on this topic already:

• a very old one from me that was definitely not ideal : #92
• a more recent one and more promising one : #217

I guess the sparsity is not very important since you will only break the sparsity for embeddings from the same feature, so in the end you'll still be looking at only a few original features.

from tabnet.

The model learns to use its attention layer. For each row, the model decides what column should be masked or not.
During inference, the model decides on its own where to put its attention, nothing is hard coded here and every test row gets a different attention mask. You can use the explain method to see where the model has been looking for every row at every step.

from tabnet.

Great issue! Reading the paper and the both original implementation and this one I made the same question about the embeddings from the same columns be used together. Have you benchmarked that already, somehow?

from tabnet.

I guess if we take the mean or the max we will lose the sparcity property... I am not sure, but maybe we can torch.stack the features instead of torch.cat them, and apply the sparse max in this new dimension. I would like to test this idea and maybe we can evaluate together the approachs.

from tabnet.

Thank you, @Optimox . The PR 217 seems what I look for. 👍

from tabnet.

After sparsemax, only 1% feature is nonzeros, is it normal？

from tabnet.

hello @W-void,

The goal of sparsemax activation is to get a sparse mask, with a lot of 0 values. So yes it is expected.

However if you think the masks are too sparse you can play with different parameters:

• switch from sparsemax to entmax (not sure this will have a big impact)
• set lambda_sparse to 0 to reduce the penalization on sparsity
• add more steps and set a larger value of gamma to be sure that the features used by each step are different (in the end the algorithm will use more features)

Let me know if this helped.

from tabnet.

hello @W-void,

The goal of sparsemax activation is to get a sparse mask, with a lot of 0 values. So yes it is expected.

However if you think the masks are too sparse you can play with different parameters:

• switch from sparsemax to entmax (not sure this will have a big impact)
• set lambda_sparse to 0 to reduce the penalization on sparsity
• add more steps and set a larger value of gamma to be sure that the features used by each step are different (in the end the algorithm will use more features)

Let me know if this helped.

@Optimox ，thx！I have set lambda_sparse to 0，It has little improvement. I will switch from sparsemax to entmax as you said.
I'm still confused. If only 2 or 3 features are used, the following FeatTransformer will get little information. I don't think it's a good result.

from tabnet.

I have a question about mask and "explain" method.

I checked on Tabmodel.py that there is an "explain" method of the model. I wonder how different masks are printed for each instance of the test data because the "explain" method does not train the model.

Do we use the mask that has been learned until the last epoch of the training data in each row of the test data?

from tabnet.

@athewsey this has been around for way too long. I have been thinking of a more general way to deal with this and I created a PR here : #443

I'd be happy to have your thoughts on this!

from tabnet.