zhouyuxuanyx / hyperformer Goto Github PK

View Code? Open in Web Editor NEW

74.0 4.0 15.0 1.03 MB

This is the official implementation of our paper "Hypergraph Transformer for Skeleton-based Action Recognition."

License: Apache License 2.0

Python 99.13% Shell 0.87%

graph-modeling hypergraph-neural-networks skeleton-based-action-recognition transformer-models

hyperformer's Introduction

Hyperformer

This is the official implementation of our paper Hypergraph Transformer for Skeleton-based Action Recognition.

Excellent Efficiency

Model	Parameters	FLOPs	Training Time/Epoch	Accuracy
ST-TR	12.1M	259.4G	11m48s	89.9
DSTA	4.1M	64.7G	10m36s	91.5
Efficient-GCN-B4 *	2.0M	15.2G*	15m37s*	91.7*
Hyperformer	2.6M	14.8G	6m52s	92.9

Remark: Comparing computation budget under the same setting (1 Tesla V100-32GB, 64 batch size, etc.) on NTU RGB+D 60 Cross-Subject Benchmark. *EfficientNet-B4 requires 4 times higher temporal resolution (288 vs 64 frames) and thus 2 GPUs to fit into memory and reach their reported accuracy.

Attention on Hypergraph

Hyperformer architecture

Preparation

Install torchlight

Run pip install -e torchlight

Download datasets.

There are 3 datasets to download:

NTU RGB+D 60 Skeleton
NTU RGB+D 120 Skeleton
NW-UCLA

NTU RGB+D 60 and 120

Request dataset here: https://rose1.ntu.edu.sg/dataset/actionRecognition
Download the skeleton-only datasets:
1. nturgbd_skeletons_s001_to_s017.zip (NTU RGB+D 60)
2. nturgbd_skeletons_s018_to_s032.zip (NTU RGB+D 120)
3. Extract above files to ./data/nturgbd_raw

NW-UCLA

Download dataset from CTR-GCN
Move all_sqe to ./data/NW-UCLA

Data Processing

Directory Structure

Put downloaded data into the following directory structure:

- data/
  - NW-UCLA/
    - all_sqe
      ... # raw data of NW-UCLA
  - ntu/
  - ntu120/
  - nturgbd_raw/
    - nturgb+d_skeletons/     # from `nturgbd_skeletons_s001_to_s017.zip`
      ...
    - nturgb+d_skeletons120/  # from `nturgbd_skeletons_s018_to_s032.zip`
      ...

Generating Data

Generate NTU RGB+D 60 or NTU RGB+D 120 dataset:

 cd ./data/ntu # or cd ./data/ntu120
 # Get skeleton of each performer
 python get_raw_skes_data.py
 # Remove the bad skeleton 
 python get_raw_denoised_data.py
 # Transform the skeleton to the center of the first frame
 python seq_transformation.py

Evaluation

We provide the pretrained model weights for NTURGB+D 60 and NTURGB+D 120 benchmarks.

To use the pretrained weights for evaluation, please run the following command:

bash evaluate.sh

Training & Testing

Training

bash train.sh

Please check the configuration in the config directory.

Testing

bash evaluate.sh

To ensemble the results of different modalities, run the following command:

bash ensemble.sh

Acknowledgements

This repo is based on 2s-AGCN and CTR-GCN. The data processing is borrowed from SGN and HCN, and the training strategy is based on InfoGCN.

Thanks to the original authors for their work!

Citation

Please cite this work if you find it useful:

 @article{zhou2022hypergraph,
  title={Hypergraph Transformer for Skeleton-based Action Recognition},
  author={Zhou, Yuxuan and Cheng, Zhi-Qi and Li, Chao and Geng, Yifeng and Xie, Xuansong and Keuper, Margret},
  journal={arXiv preprint arXiv:2211.09590},
  year={2022}
}

Contact

For any questions, feel free to contact: [email protected]

hyperformer's People

Contributors

Stargazers

Watchers

Forkers

zhiqic xinxinatg baimaqingxin xidianpqz zangtuo tmukande-debug hucui2022 juraev tianrui-li good-demo egolas zxj12345678 sshuster ugrkilc junmasaki59

hyperformer's Issues

关于NW-UCLA的精度

您好，请问您在实验过程中是否使用CTR-GCN跑了NW-UCLA数据集，是否能达到CTR-GCN提供的精度？
我在实验过程中发现，CTR-GCN在NW-UCLA的进度仅仅达到95.5？请问您是否遇到过同样的问题？
期待您的回复。

Understand of K-hop relative position embedding

I discover in your code about the Multihead Attention Module, there lies a procedure to build improved multi-scale k-hop adjacent matrix just like MS-G3D. However, I can't understand the following code for self.hops variable in your code : https://github.com/ZhouYuxuanYX/Hyperformer/blob/main/model/Hyperformer.py#L283 . Can you simply tell the meaning of these code? Thank you very much!

关于表4的实验：HyperSA的有效性

你好，我尝试验证HyperSA的有效性，也就是复现原论文表格4的内容。
我通过修改此处的attn = a + b + c + d为attn = a、attn = a + b 等去复现表格4的内容，但Full HyperSA并没有比SA带来如表4一样特别大的提升，具体结果如下：

Model	val_acc	remark
Full HyperSA + MS-TC	0.90277	attn = a + b + c + d
SA + MS-TC	0.90083	attn = a
SA + MS-TC + K-Hop RPE	0.90301	attn = a + b
SA + MS-TC + Joint-to-hyperedge attn	0.90101	attn = a + c
SA + MS-TC + Hyperedge-Attentive-Bias	0.90240	attn = a + d

我使用的是3090，其他设置与原代码一致。
我猜测可能是我没有完全复现90.7%的结果或者是我对模型的修改不对？
如果有别的遗漏，请帮我指出一下，谢谢！

Pytorch Version

Hello, what is the version of Pytorch used in your code?

How to train joint_label

Hello,

I am interested in training the joint_label. Based on the discussion in the previous issue, it was mentioned that a parameterized partition matrix could be used. Could you guide me on how to initialize this matrix and then apply the softmax? should the dimensions of this matrix be set to 25x25?

Thank you very much for the help.

可以给出joint label的训练代码吗

您好，在之前的议题中您有详细解释过如何训练joint label，但是很抱歉由于愚笨我没能复现出来。可以劳烦您给出代码吗？

什么年代了还在用np.int

如题

文章收录

您好，请问现在文章收录有结果了吗？

hyperedge

Hello, I would like to ask how you constructed the hyperedge and how to obtain the hyperedge representation

Partition

Hello, I can only see that you use joint_label as partition matrix while you are supposed to use partition strategy to learn the partition matrix. Do I miss anything? If so, could you let me where your learnable partition matrix code is. Thanks!

收录期刊或会议

你好，想问一下这篇工作最后是被哪个期刊或会议接收的吗，看到你的代码仓库上写的是 preprint版的

Hello，I encountered a problem during the implementation process and would like to consult with you

Traceback (most recent call last):
File "/home/hcao/Hyperformer-main/main.py", line 777, in
processor.start()
File "/home/hcao/Hyperformer-main/main.py", line 756, in start
self.eval(epoch=0, save_score=self.arg.save_score, loader_name=['test'], wrong_file=wf, result_file=rf)
File "/home/hcao/Hyperformer-main/main.py", line 573, in eval
output, z = self.model(data, F.one_hot(label, num_classes=self.model.module.num_class))
^^^^^^^^^^^^^^^^^
File "/home/hcao/.conda/envs/Hyperformer/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1614, in getattr
raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'Model' object has no attribute 'module'. Did you mean: 'modules'?

Please Provide Dependencies List

Can you share your requirements.txt, I am having troubles with running the code.

About the learning method for joint_label.

Hello,

our paper has been very informative to me.
In the paper, it mentions using 'soft' partitions with the application of softmax. Does this mean that during training, labels are handled as probabilistic values rather than binary values? Also, how is this matrix initialized during training (zeros or a normal distribution)?"