wingsbrokenangel / semantics-assistedvideocaptioning Goto Github PK

提高tags的数量是否会提高指标？
这篇文章和“Delving Deeper into the Decoder for Video Captioning”都是非常优秀的工作，感谢您分享代码。

Thank you for kindly share your code to the github, Could you tell me how to generate these pkl file file in the Data Section, as to say How to do the preprocess of the two dataset : MSVD and the MSR-VTT ？Thank for your attention!

Hey man, thanks for your excellent work! Could you please tell me what resolution you use for MSR-VTT?

Hey @WingsBrokenAngel nice work thanks for making code publicly for further research.

I have a query regarding implementation, since MSR-VTT have 20 captions for each video. How you have deal with them during?
Did you took random caption for video in each epoch or you have just repeated the features for each caption?
By look the implementation i think you have took all the caption with repeated features. Am I right ?

Hello,

How can I find the msvd equivalent of the checkpoint of tagging/test.py referrred at line 15
( saver.restore(sess, './saves/msrvtt_tag_model_1000_resnext_eco.ckpt'))?

Is the msr-vtt v1.part & v2.part & v3.part corresponds to train/val/test respectively?

I notice that in latest version of 《A Semantics-Assisted Video Captioning Model Trained with Scheduled Sampling》，mAP has a higher value than in a provisional file,can you share the method to get this result。Thanks。

非常感谢您开源代码，我仔细阅读了您的文章并重新复现了代码，收获很多。但是也存在一些疑问：
提取300个单词作为语义分支的训练标签，提取语义特征，该特征被映射到0-1之间。
问题1：语义特征和lstm的视觉输入、前一时刻的word-embedding,前一时刻的隐藏层h(t-1)分别对应相乘，达到语义融合的目的吗？
问题2：文章提到人工选取k个单词，那么选取的标准是什么呢？为什么选择300个，如何找到这300个？（我在您的代码中，去除了schedule sampling仅使用semantic,发现提升很大 ）
问题3：（最困惑的问题）文章提到的sentence-length-related loss function具体体现在代码的哪里呢，我仔细阅读了您的源码，没有找到这部分内容，请您指点，非常感谢！

以下是我在您源码的基础上做的实验（没有找到sentence loss源码，所以无法去除进行对比）：
Base(仅使用LSTM)，Base + schedule sampling,Base + semantic,参数未作改变，50epoch结果如下：

MSRVTT | B-4 | C | M | R | Overall | Time
BASE | 40.3 | 41.3 | 26.1 | 60.1 | 0.896 | 19epo/3900s/10478s
BASE+Samp | 39.1 | 41.8 | 25.9 | 60.1 | 0.915 | 22epo/4500s/10210s
BASE+Semantic | 44.0 | 50.2 | 28.4 | 62.4 | 0.948 | 50epoch/11113.7s

Semantic的作用非常大。
针对上述三个问题尤其问题3，希望得到您的帮助

Thank you for sharing your amazing work.

I need to extract ECO features only. Can you help me how to do that? Specifically, I need to just extract ECO features of videos. Do I have to run all models of ECO in his GitHub repository, or what?

首先非常感谢作者，您的工作和开源代码对我很有帮助。在学习过程中仍然有些疑问想请教您：
1、在Semantics-AssistedVideoCaptioning-master/tagging文件夹下的train_tag_net.py中，命令行参数有
(1)msvd_resnext_eco.npy:msvd数据集的视频特征-->1970x3584
(2)msvd_tag_gt_4_msvd.npy:从msvd选取300个词对msvd做标注，得到的1970x300的真实语义标注
(3)msrvtt_resnext_eco.npy:msrvtt数据集的视频特征-->10000x3584
(4)msrvtt_tag_gt_4_msrvtt.npy：从msrvtt选取300个词对msrvtt做标注，得到的10000x300的真实语义标注
不知道我上面的理解正确吗？
另外，在Data-->MSVD部分，除了提供了msvd_tag_gt_4_msvd.npy，还有一个msrvtt_tag_gt_4_msvd.npy(如下图)，我看它的shape是10000x300，请问这个文件是用msvd的300个词对msrvtt做的真实语义标注吗？
下面还有一句"The previous two files are used to train the tagging network."是想说用这两个文件针对msvd数据集做一个语义检测网络吗？但是我看train_tag_net.py中，是用msvd_tag_gt_4_msvd.npy和msrvtt_tag_gt_4_msrvtt.npy训练了一个统一的语义检测网络啊，这是怎么回事？

2、在提取eco特征时，用到了caffemodel，想知道net = caffe.Net(model_file, model_def_file, caffe.TEST)，model_def_file是您提供的ECO_full_kinetics.caffemodel，那model_file呢，我看网上说是一个deploy.prototxt文件，但您提供的所有文件里并没有这一项，由于我对caffe那一套不了解，不知道具体是怎么回事，还请指教。

3、我注意到，产生resnext特征的文件，generate_res_feat.py，最后产生的是一个1970x32x2048的张量(针对msvd)，并把张量写入一个npy文件，而在您的文章里，是把它们按空间维度(即32所在的维度)进行了平均池化，最终得到1970x2048的特征。
平均池化操作是使用tf.layers.average_pooling3d进行的吗？

4、我看文章中，视频特征是把Ei(第i视频的动态特征)堆叠到Ri(第i视频的静态特征)上去，得到3584维特征。我想知道在您的代码处理过程中，每个视频的3584维特征，具体是eco+resnext(1536+2048)，还是resnext+eco(2048+1536)呢？
5、不好意思，在github上用中文提issues，因为问的东西比较多，还望您能指点一二，谢谢。

Hello dear author:
When I test the method you gave, why does the model still retrain?

I found that train_op is passed to test_step, such that the loss on the test set will be minimized, is this reasonable?

Hi, I tried reproducing your results but the files generated by test.py in the tagging model contain ndarrays with nan values.

I guess, in the tagging network when you mention keep_prob, you actually meant rate.

I'm really interested in your paper, but I haven't found any code about semantic generation for a long time. If it's convenient, can you show me

Thank you very much for your code! It has been very useful to me!
But, when I attempt to load the features in msrvtt_resnext_eco.npy file from the link that you included in the README, ValueError: cannot reshape array of size 13238145 into shape (10000,3584) occurs. Is it the correct file?