Light

butterfliesss / sdt Goto Github PK

View Code? Open in Web Editor NEW

33.0 1.0 5.0 1.2 MB

Python 98.72% Shell 1.28%

sdt's Introduction

SDT

This repository is the implementation for our paper A Transformer-based Model with Self-distillation for Multimodal Emotion Recognition in Conversations.

Model Architecture

Setup

Check the packages needed or simply run the command:

pip install -r requirements.txt

Download the preprocessed datasets from here, and put them into data/.

Run SDT model

Run the model on IEMOCAP dataset:

bash exec_iemocap.sh

Run the model on MELD dataset:

bash exec_meld.sh

Acknowledgements

Special thanks to the COSMIC and MMGCN for sharing their codes and datasets.

Citation

If you find our work useful for your research, please kindly cite our paper. Thanks!

@article{ma2024sdt,
  author={Ma, Hui and Wang, Jian and Lin, Hongfei and Zhang, Bo and Zhang, Yijia and Xu, Bo},
  journal={IEEE Transactions on Multimedia}, 
  title={A Transformer-Based Model With Self-Distillation for Multimodal Emotion Recognition in Conversations}, 
  year={2024},
  volume={26},
  number={},
  pages={776-788},
  keywords={Emotion recognition;Transformers;Oral communication;Context modeling;Task analysis;Visualization;Logic gates;Multimodal emotion recognition in conversations;intra- and inter-modal interactions;multimodal fusion;modal representation},
  doi={10.1109/TMM.2023.3271019}}

sdt's People

Contributors

Stargazers

Watchers

Forkers

ccnu-jreamy binnchow ritesh-47 ml-edu khanhluong34

sdt's Issues

Questions. multimodal data preprocessing

Hi. Thanks to share good paper and code.

I wondering how to preprocess multimodal(text, audio, video) datas.

You share multimodal feature datasets, so I can implement this code but I wondering how to preprocess datasets(because I want to run my own datasets)

Thank you.

Teacher, I really want to have the data processing part of the code

I really need code for data preprocessing

Because my tutor attaches great importance to the code of the data preprocessing part, but I am not good at it and have never solved this problem. I hope you can provide some help so that I can learn from your code. This is my email: [email protected]
Thank you so much! ! !

Problem of batch_size

Why does it only work if batch_size=1, once it is greater than 1 the following happens：
epoch: 1, train_loss: nan, train_acc: 10.51, train_fscore: 5.0, valid_loss: nan, valid_acc: 6.78, valid_fscore: 0.86, test_loss: nan, test_acc: 8.87, test_fscore: 1.45, time: 1.87 sec
epoch: 2, train_loss: nan, train_acc: 8.92, train_fscore: 1.46, valid_loss: nan, valid_acc: 6.78, valid_fscore: 0.86, test_loss: nan, test_acc: 8.87, test_fscore: 1.45, time: 0.57 sec
epoch: 3, train_loss: nan, train_acc: 8.92, train_fscore: 1.46, valid_loss: nan, valid_acc: 6.78, valid_fscore: 0.86, test_loss: nan, test_acc: 8.87, test_fscore: 1.45, time: 0.6 sec
epoch: 4, train_loss: nan, train_acc: 8.92, train_fscore: 1.46, valid_loss: nan, valid_acc: 6.78, valid_fscore: 0.86, test_loss: nan, test_acc: 8.87, test_fscore: 1.45, time: 0.55 sec
epoch: 5, train_loss: nan, train_acc: 8.92, train_fscore: 1.46, valid_loss: nan, valid_acc: 6.78, valid_fscore: 0.86, test_loss: nan, test_acc: 8.87, test_fscore: 1.45, time: 0.62 sec

About valid and test sets

Hello, in the code, I think you treat the test set as a valid set, saving the best results of the test set each time instead of the valid set, is this approach reasonable? Looking forward to your reply.

Can you provide the code part of data preprocessing

which faces to choose, MELD dataset

hi,
thanks for this work.
in MELD dataset, often, there are many faces per-frame.
how did you select a face in this case for the vision modality?

thanks

Re-upload data

it's a nice job and could you please re-upload the data for this paper including the processed features?thank you.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.