GPU out of memory with reasonable data about deepctr HOT 4 CLOSED

@LiChen2012

Hi, I'm sorry that I can't test the GPU version codes because it is a holiday now.
Now I don't know how to put some tensors or ops on CPU when using tf.keras,I think there are 2 methods you can try.

1. Modify the line 39 ofdeepctr.models.deepfm model like following:

with tf.device("/cpu:0"):
    deep_emb_list, linear_logit, inputs_list = get_inputs_embedding(feature_dim_dict, embedding_size, l2_reg_embedding, l2_reg_linear, init_std, seed)

2. Reduce embedding_size dimension,for large scale data,usually the embedding_size is small,like [4,8,16,32,64,etc]

from deepctr.

After modification of code, now GPU memory usage decreased. But it seems still not meet my expectations.
Before the fix, sparse feature embedding memory usage was 3 times of what should be.
After the fix, sparse feature embedding memory usage is double of what should be.
I hope you can take some time to fix it. Thanks.

from deepctr.

Another issue:
After modification of code, training is slow. GPU usage jumps between 5% to 95%. I suspected that set tensor to CPU will cause memory copy between GPU and CPU.
Thanks.

from deepctr.

This solution puts the vector itself on the CPU, and the parameters created by the optimizer are still on the GPU, so there is still twice the memory usage.
I'm sorry that I don't have a perfect solution right now, and I think I need help from the community~

from deepctr.