0aqz0 / slr Goto Github PK

isolated & continuous sign language recognition using CNN+LSTM/3D CNN/GCN/Encoder-Decoder

Python 100.00%

sign-language-recognition slr deep-learning 3d-cnn cnn-lstm gcn lstm seq sign-language-translation sign-language-recognition-system

slr's Introduction

SLR

isolated & continuous sign language recognition using CNN+LSTM/3D CNN/GCN/Encoder-Decoder

Requirements

Download and extract CSL Dataset
Download and install PyTorch

Isolated Sign Language Recognition

CNN+LSTM

four layers of Conv2d + one layer of LSTM

Dataset Classes Samples Best Test Acc Best Test Loss

CSL_Isolated 100 25,000 82.08% 0.734426

CSL_Isolated 500 125,000 71.71% 1.332122
ResNet + one layer of LSTM

Dataset Classes Samples Best Test Acc Best Test Loss

CSL_Isolated 100 25,000 93.54% 0.245582

CSL_Isolated 500 125,000 83.17% 0.748759

Dataset	Classes	Samples	Best Test Acc	Best Test Loss
CSL_Isolated	100	25,000	82.08%	0.734426
CSL_Isolated	500	125,000	71.71%	1.332122

Dataset	Classes	Samples	Best Test Acc	Best Test Loss
CSL_Isolated	100	25,000	93.54%	0.245582
CSL_Isolated	500	125,000	83.17%	0.748759

3D CNN

three layers of Conv3d

Dataset Classes Samples Best Test Acc Best Test Loss

CSL_Isolated 100 25,000 58.86% 1.560049

CSL_Isolated 500 125,000 45.07% 2.255563

Dataset	Classes	Samples	Best Test Acc	Best Test Loss
CSL_Isolated	100	25,000	58.86%	1.560049
CSL_Isolated	500	125,000	45.07%	2.255563

3D ResNet

Method	Dataset	Classes	Samples	Best Test Acc	Best Test Loss
ResNet18	CSL_Isolated	100	25,000	93.30%	0.246169
ResNet18	CSL_Isolated	500	125,000	79.42%	0.800490
ResNet34	CSL_Isolated	100	25,000	94.78%	0.207592
ResNet34	CSL_Isolated	500	125,000	81.61%	0.750424
ResNet50	CSL_Isolated	100	25,000	94.36%	0.232631
ResNet50	CSL_Isolated	500	125,000	83.15%	0.803212
ResNet101	CSL_Isolated	100	25,000	95.26%	0.205430
ResNet101	CSL_Isolated	500	125,000	83.18%	0.751727

ResNet (2+1)D

Dataset Classes Samples Best Test Acc Best Test Loss

CSL_Isolated 100 25,000 98.68% 0.043099

CSL_Isolated 500 125,000 94.85% 0.234880

Dataset	Classes	Samples	Best Test Acc	Best Test Loss
CSL_Isolated	100	25,000	98.68%	0.043099
CSL_Isolated	500	125,000	94.85%	0.234880

GCN

Dataset	Classes	Samples	Best Test Acc	Best Test Loss
CSL_Skeleton	100	25,000	79.20%	0.737053
CSL_Skeleton	500	125,000	66.64%	1.165872

Skeleton+LSTM

Dataset	Classes	Samples	Best Test Acc	Best Test Loss
CSL_Skeleton	100	25,000	84.30%	0.488253
CSL_Skeleton	500	125,000	70.62%	1.078730

Continuous Sign Language Recognition

Encoder-Decoder

Encoder is ResNet18+LSTM, and Decoder is LSTM

Dataset	Sentences	Samples	Best Test Wer	Best Test Loss
CSL_Continuous	100	25,000	1.01%	0.034636
CSL_Continuous_Char	100	25,000	1.19%	0.049449

References

slr's People

Contributors

Stargazers

Watchers

slr's Issues

Asking About the code if these code works for RTWK Phoenix2014 or not

Hello guys can these code works for Phoenix2014 continuous sign language data set if we correct the dataset directory or the path
please let met know if they correct

training continuous with ctc loss?

你好，

请问你有试着用ctc loss来训练 continuous SLR 吗？我的模型很简单，就是 pretrained EfficientNet + self-attention(3 layer) + linear + ctc loss. 我是在RWTH-PHOENIX- Weather-2014T 这个数据集上训练的，然后发现loss完全降不下去，维持在5.2左右，而且在解码的时候只预测blank。个人感觉应该不是我的ctc loss使用问题，因为我之前用ctc loss训练machine translation是没有问题的。想知道你有训练过这个数据集和用ctc来训这个模型吗？

acc和loss的一些问题

作者你好，我训练孤立词的时候发现在前几个epoch的时候acc就能到接近100，并且最后的loss值大概是0.0008左右，请问这是什么问题呢

Asking about CSL Dataset specially the continuous sign language part

hello guys can any one tell me about how CSL Dataset data path or directory is prepared i want to try these code in my local sign language dataset but because of the CSL Dataset is private i don't about the data structure please tell me or give me the small data

email: [email protected]

Thank you.