sktbrain / sumbt Goto Github PK

Hi, sorry to bother you.
Because the training time of 300 epochs is much long, I am trying a training procedure with learning_rate=1e-4, num_train_epochs=100 and to save memory so that the model could run on a single GPU with 12 GB memory, I set train_batch_size=3 and eval_batch_size=4.
Currently, at the 45th epoch, the joint accuracy on validation set has reached 53.96%. And the model is still on warm up stage. Is it normal? Or there are some problems?



Besides, the dataset I used is MultiWOZ2.1.
Thank you very much.

Hi, very appericate to your contribution and I have two small questions.

1. I saw the num_train_epochs in the shell is 300. How many epochs will the training process long actually?
2. How large is the memory of a single GPU you used? I have tried to train it on a TITAN XP but met OOM and I then set the training batch size as 2.
   Thanks you very much!

Hello,

I found this part a bit weird (line 989) in the evaluation.

https://github.com/SKTBrain/SUMBT/blob/master/code/main-multislot.py#L989

When you compute joint goal accuracy, the score at each turn can only be 0 or 1. But here since that you are using "/" instead of "//" to get a floating number? Can you double check if I am wrong? Many thanks.

During data preprocessing, all values not appeared in ontology are marked as none, this assumption is too strong as the ontology could be incomplete. This is true for the MultiWoz dataset as there are many such values in the dataset. This processing way also changes the test set (>10% data), which provides an unfair comparison to other baselines. Moreover, this assumption results in much higher testing acc as predicting none is much easier. Based on this method, maybe it is better to construct the ontology through traversing the dataset and report the real testing acc

Excuse me. What is the use of "_make_aux_tensors"?

Thanks.

When evaluating the model, I can't see the Exact Joint Accuracy of 0.48806 reported in the paper (both in the output text log and tensorboard log). Is this logged somewhere or do I have to implement it?

Here is the content of eval_results_all.txt, which to my understanding, does not have the exact accuracy score

eval_acc_slot = 0.80290287733078	0.867335855960846	0.7904232144355774	1.0	1.0	1.0	1.0	1.0	0.8619099259376526	0.870455801486969	0.8585187196731567	0.8693705797195435	0.8610960245132446	0.8431904315948486	0.8776451349258423	0.8274552226066589	0.8518719673156738	0.804530680179596	0.7505425810813904	0.8275908827781677	0.8244709968566895	0.8273195624351501	0.7339934706687927	0.8518719673156738	0.7593597173690796	0.9686652421951294	0.9226804375648499	0.9221377968788147	0.9564568400382996	0.799782931804657	0.8491590023040771	0.6471785306930542	0.65233314037323	0.6389039754867554	0.8499728441238403
eval_accuracy = 0.8505464778344914
eval_loss = 28.46753237458331
eval_loss_slot = 0.8817319130627069	1.201549073308302	1.2350054645523776	0.036111686570932194	0.005953068851795991	0.011532692755009056	0.007900203865390863	0.14539033580395558	0.6507461597567916	0.48282753164479253	0.5047258990239433	0.48811625681018606	0.3414967832347172	1.1946264176436954	0.357626472429003	0.5301039830917453	0.5166375836485323	0.40926279729041176	0.7328648737402319	0.7531180265048802	0.706450523843719	1.1251465154754132	1.739766523929245	1.3978182440274964	0.7326630629716331	0.4957690410198754	1.0787263601306851	1.1032195673387335	0.6402477735064305	1.523714961210186	0.7175217050459182	1.2880614138179527	1.9950037940455336	2.0547096399975433	1.3813860977934678
loss = None

Hi, sorry to bother you.

I have a small question that will you conduct an experiment on multiwoz2.1?

Thank you very much!

Hi, Thanks for your great work!
I have reproduced your results on Woz dataset, what's the training command for reproducing results on MultiWoz 2.0?

• Where can I find the pre-processing scripts used to prepare the data
• How can I train the model for MultiWoz 2.3

Why in self.utterance_encoder use attention_mask

and after self.utterance_encoder use the attention_mask again?

Hi, I can't validate on woz devset after I updated recent commits. Here is the log:

Traceback (most recent call last):                                                                                                   
  File "code/main-multislot.py", line 1037, in <module>                                                                
  File "code/main-multislot.py", line 917, in main
    accuracies = eval_all_accs(pred_slot, label_ids, accuracies)
  File "code/main-multislot.py", line 1002, in eval_all_accs
    joint_acc, slot_acc, num_turn, num_data = _eval_acc(pred_slot[:,:,18:25], labels[:,:,18:25])
  File "code/main-multislot.py", line 985, in _eval_acc
    accuracy = (_pred_slot == _labels).view(-1, slot_dim)
RuntimeError: cannot reshape tensor of 0 elements into shape [-1, 0] because the unspecified dimension size -1 can be any value and is ambiguous

Why we need to expand and reshape the tensor to [(slot_dimdsts), bert_seq, hid_size], is that mean every utterance need to mapping into slot_dim times prediction