If possible, you could paste log information here.

from openvqa.

Thanks for you reply @nbgao , did you use the same seed for mcan_small?

We use SEED=1016 for mcan_small in GQA.

from openvqa.

@nbgao The log with first and last epoch

{ BATCH_SIZE        }->64
{ BBOXFEAT_EMB_SIZE }->2048
{ CACHE_PATH        }->./results/cache
{ CKPTS_PATH        }->./ckpts
{ CKPT_EPOCH        }->0
{ CKPT_PATH         }->None
{ CKPT_VERSION      }->4120150
{ DATASET           }->gqa
{ DATA_PATH         }->{'vqa': './data/vqa', 'clevr': './data/clevr', 'gqa': './data/gqa'}
{ DATA_ROOT         }->./data
{ DEVICES           }->[0]
{ DROPOUT_R         }->0.1
{ EVAL_BATCH_SIZE   }->8
{ EVAL_EVERY_EPOCH  }->True
{ FEATS_PATH        }->{'vqa': {'val': './data/vqa/feats/val2014', 'train': './data/vqa/feats/train2014', 'test': './data/vqa/feats/test2015'}, 'clevr': {'val': './data/clevr/feats/val', 'train': './data/clevr/feats/train', 'test': './data/clevr/feats/test'}, 'gqa': {'default-frcn': './data/gqa/feats/gqa-frcn', 'default-grid': './data/gqa/feats/gqa-grid'}}
{ FEAT_SIZE         }->{'vqa': {'BBOX_FEAT_SIZE': (100, 5), 'FRCN_FEAT_SIZE': (100, 2048)}, 'clevr': {'GRID_FEAT_SIZE': (196, 1024)}, 'gqa': {'GRID_FEAT_SIZE': (49, 2048), 'BBOX_FEAT_SIZE': (100, 5), 'FRCN_FEAT_SIZE': (100, 2048)}}
{ FF_SIZE           }->4096
{ FLAT_GLIMPSES     }->1
{ FLAT_MLP_SIZE     }->512
{ FLAT_OUT_SIZE     }->2048
{ GPU               }->7
{ GRAD_ACCU_STEPS   }->4
{ GRAD_NORM_CLIP    }->-1
{ HIDDEN_SIZE       }->1024
{ LAYER             }->6
{ LOG_PATH          }->./results/log
{ LOSS_FUNC         }->ce
{ LOSS_FUNC_NAME_DICT }->{'bce': 'BCEWithLogitsLoss', 'kld': 'KLDivLoss', 'ce': 'CrossEntropyLoss', 'mse': 'MSELoss'}
{ LOSS_FUNC_NONLINEAR }->{'bce': [None, None], 'kld': ['log_softmax', None], 'ce': [None, 'flat'], 'mse': [None, None]}
{ LOSS_REDUCTION    }->sum
{ LR_BASE           }->5e-05
{ LR_DECAY_LIST     }->[8, 10]
{ LR_DECAY_R        }->0.2
{ MAX_EPOCH         }->11
{ MODEL             }->mcan_large
{ MODEL_USE         }->mcan
{ MULTI_HEAD        }->8
{ NUM_WORKERS       }->8
{ N_GPU             }->1
{ OPT               }->Adam
{ OPT_PARAMS        }->{'betas': (0.9, 0.98), 'weight_decay': 0, 'eps': 1e-09, 'amsgrad': False}
{ PIN_MEM           }->True
{ PRED_PATH         }->./results/pred
{ RAW_PATH          }->{'vqa': {'val-anno': './data/vqa/raw/v2_mscoco_val2014_annotations.json', 'val': './data/vqa/raw/v2_OpenEnded_mscoco_val2014_questions.json', 'test': './data/vqa/raw/v2_OpenEnded_mscoco_test2015_questions.json', 'vg': './data/vqa/raw/VG_questions.json', 'train-anno': './data/vqa/raw/v2_mscoco_train2014_annotations.json', 'train': './data/vqa/raw/v2_OpenEnded_mscoco_train2014_questions.json', 'vg-anno': './data/vqa/raw/VG_annotations.json'}, 'clevr': {'val': './data/clevr/raw/questions/CLEVR_val_questions.json', 'train': './data/clevr/raw/questions/CLEVR_train_questions.json', 'test': './data/clevr/raw/questions/CLEVR_test_questions.json'}, 'gqa': {'val': './data/gqa/raw/questions1.2/val_balanced_questions.json', 'val_all': './data/gqa/raw/questions1.2/val_all_questions.json', 'test': './data/gqa/raw/questions1.2/submission_all_questions.json', 'testdev_all': './data/gqa/raw/questions1.2/testdev_all_questions.json', 'val_choices': './data/gqa/raw/eval/val_choices.json', 'train_choices': './data/gqa/raw/eval/train_choices', 'train': './data/gqa/raw/questions1.2/train_balanced_questions.json', 'testdev': './data/gqa/raw/questions1.2/testdev_balanced_questions.json'}}
{ RESULT_PATH       }->./results/result_test
{ RESUME            }->False
{ RUN_MODE          }->train
{ SEED              }->4120150
{ SPLIT             }->{'val': 'testdev', 'train': 'train+val', 'test': 'test'}
{ SPLITS            }->{'vqa': {'val': 'val', 'train': '', 'test': 'test'}, 'clevr': {'val': 'val', 'train': '', 'test': 'test'}, 'gqa': {'val': 'testdev', 'train': 'train+val', 'test': 'test'}}
{ SUB_BATCH_SIZE    }->16
{ TASK_LOSS_CHECK   }->{'vqa': ['bce', 'kld'], 'clevr': ['ce'], 'gqa': ['ce']}
{ TEST_SAVE_PRED    }->False
{ TRAIN_SPLIT       }->train+val
{ USE_AUX_FEAT      }->True
{ USE_BBOX_FEAT     }->True
{ USE_GLOVE         }->True
{ VERBOSE           }->True
{ VERSION           }->default_grid+bb+frcn
{ WARMUP_EPOCH      }->2
{ WORD_EMBED_SIZE   }->300
=====================================
nowTime: 2020-01-15 17:17:00
Epoch: 1, Loss: 1.68896448995679, Lr: 3.3333333333333335e-05
Elapsed time: 12386, Speed(s/batch): 0.7374428591009378

Binary: 56.75%
Open: 34.61%
Accuracy: 44.77%
Distribution: 4.05 (lower is better)
Accuracy / structural type:
  choose: 60.76% (1129 questions)
  compare: 57.05% (589 questions)
  logical: 57.35% (1803 questions)
  query: 34.61% (6805 questions)
  verify: 54.17% (2252 questions)
Accuracy / semantic type:
  attr: 48.98% (5186 questions)
  cat: 40.38% (1149 questions)
  global: 49.68% (157 questions)
  obj: 68.12% (778 questions)
  rel: 38.04% (5308 questions)
Accuracy / steps number:
  1: 60.34% (237 questions)
  2: 40.72% (6395 questions)
  3: 45.99% (4266 questions)
  4: 47.04% (793 questions)
  5: 60.46% (822 questions)
  6: 68.29% (41 questions)
  7: 100.00% (20 questions)
  8: 100.00% (3 questions)
  9: 100.00% (1 questions)
Accuracy / words number:
  3: 23.18% (151 questions)
  4: 41.75% (630 questions)
  5: 35.04% (1290 questions)
  6: 40.98% (2074 questions)
  7: 42.02% (1642 questions)
  8: 46.33% (1185 questions)
  9: 49.65% (1281 questions)
  10: 50.92% (1249 questions)
  11: 46.98% (994 questions)
  12: 52.04% (638 questions)
  13: 49.78% (462 questions)
  14: 47.25% (345 questions)
  15: 51.90% (237 questions)
  16: 51.28% (117 questions)
  17: 45.74% (94 questions)
  18: 53.95% (76 questions)
  19: 58.14% (43 questions)
  20: 56.25% (32 questions)
  21: 42.11% (19 questions)
  22: 50.00% (12 questions)
  23: 25.00% (4 questions)
  24: 100.00% (2 questions)
  25: 100.00% (1 questions)

nowTime: 2020-01-17 03:49:53
Epoch: 11, Loss: 0.27490622613804777, Lr: 2.0000000000000003e-06
Elapsed time: 12362, Speed(s/batch): 0.7360429190743336

Binary: 75.13%
Open: 40.21%
Accuracy: 56.23%
Distribution: 1.93 (lower is better)
Accuracy / structural type:
  choose: 77.06% (1129 questions)
  compare: 62.48% (589 questions)
  logical: 73.60% (1803 questions)
  query: 40.21% (6805 questions)
  verify: 78.69% (2252 questions)
Accuracy / semantic type:
  attr: 64.38% (5186 questions)
  cat: 43.34% (1149 questions)
  global: 53.50% (157 questions)
  obj: 84.58% (778 questions)
  rel: 46.99% (5308 questions)
Accuracy / steps number:
  1: 70.46% (237 questions)
  2: 49.84% (6395 questions)
  3: 59.94% (4266 questions)
  4: 66.20% (793 questions)
  5: 70.56% (822 questions)
  6: 80.49% (41 questions)
  7: 100.00% (20 questions)
  8: 100.00% (3 questions)
  9: 100.00% (1 questions)
Accuracy / words number:
  3: 31.13% (151 questions)
  4: 50.32% (630 questions)
  5: 43.33% (1290 questions)
  6: 52.56% (2074 questions)
  7: 56.70% (1642 questions)
  8: 57.81% (1185 questions)
  9: 62.61% (1281 questions)
  10: 62.21% (1249 questions)
  11: 56.64% (994 questions)
  12: 60.97% (638 questions)
  13: 57.58% (462 questions)
  14: 67.25% (345 questions)
  15: 64.98% (237 questions)
  16: 66.67% (117 questions)
  17: 59.57% (94 questions)
  18: 68.42% (76 questions)
  19: 76.74% (43 questions)
  20: 53.12% (32 questions)
  21: 63.16% (19 questions)
  22: 75.00% (12 questions)
  23: 25.00% (4 questions)
  24: 100.00% (2 questions)
  25: 100.00% (1 questions)```

from openvqa.

@mrfarazi
We set the SEED=1018 in this reported mcan_large model.
The training log file is as follow.

{ BATCH_SIZE }->64
{ BBOXFEAT_EMB_SIZE }->2048
{ CACHE_PATH }->./results/cache
{ CKPTS_PATH }->./ckpts
{ CKPT_EPOCH }->0
{ CKPT_PATH }->None
{ CKPT_VERSION }->4864225
{ DATASET }->gqa
{ DATA_PATH }->{'vqa': '/data-ssd/gaopb/datasets/vqa', 'gqa': '/data-ssd/gaopb/datasets/gqa', 'clevr': '/data-ssd/gaopb/datasets/clevr'}
{ DATA_ROOT }->/data-ssd/gaopb/datasets
{ DEVICES }->[0]
{ DROPOUT_R }->0.1
{ EVAL_BATCH_SIZE }->8
{ EVAL_EVERY_EPOCH }->True
{ FEATS_PATH }->{'vqa': {'train': '/data-ssd/gaopb/datasets/vqa/feats/train2014', 'val': '/data-ssd/gaopb/datasets/vqa/feats/val2014', 'test': '/data-ssd/gaopb/datasets/vqa/feats/test2015'}, 'gqa': {'default-frcn': '/data-ssd/gaopb/datasets/gqa/feats/gqa-frcn', 'default-grid': '/data-ssd/gaopb/datasets/gqa/feats/gqa-grid'}, 'clevr': {'train': '/data-ssd/gaopb/datasets/clevr/feats/train', 'val': '/data-ssd/gaopb/datasets/clevr/feats/val', 'test': '/data-ssd/gaopb/datasets/clevr/feats/test'}}
{ FEAT_SIZE }->{'vqa': {'FRCN_FEAT_SIZE': 2048, 'BBOX_FEAT_SIZE': 5}, 'gqa': {'FRCN_FEAT_SIZE': 2048, 'GRID_FEAT_SIZE': 2048, 'BBOX_FEAT_SIZE': 5}, 'clevr': {'GRID_FEAT_SIZE': 1024}}
{ FF_SIZE }->4096
{ FLAT_GLIMPSES }->1
{ FLAT_MLP_SIZE }->512
{ FLAT_OUT_SIZE }->2048
{ GPU }->1
{ GRAD_ACCU_STEPS }->4
{ GRAD_NORM_CLIP }->-1
{ HIDDEN_SIZE }->1024
{ LAYER }->6
{ LOG_PATH }->./results/log
{ LOSS_FUNC }->ce
{ LOSS_FUNC_NAME_DICT }->{'ce': 'CrossEntropyLoss', 'bce': 'BCEWithLogitsLoss', 'kld': 'KLDivLoss', 'mse': 'MSELoss'}
{ LOSS_FUNC_NONLINEAR }->{'ce': [None, 'flat'], 'bce': [None, None], 'kld': ['log_softmax', None], 'mse': [None, None]}
{ LOSS_REDUCTION }->sum
{ LR_BASE }->5e-05
{ LR_DECAY_LIST }->[8, 10]
{ LR_DECAY_R }->0.2
{ MAX_EPOCH }->11
{ MODEL }->mcan_large
{ MODEL_USE }->mcan
{ MULTI_HEAD }->8
{ NUM_WORKERS }->8
{ N_GPU }->1
{ OPT }->Adam
{ OPT_PARAMS }->{'betas': (0.9, 0.98), 'eps': 1e-09, 'weight_decay': 0, 'amsgrad': False}
{ PIN_MEM }->True
{ PRED_PATH }->./results/pred
{ RAW_PATH }->{'vqa': {'train': '/data-ssd/gaopb/datasets/vqa/raw/v2_OpenEnded_mscoco_train2014_questions.json', 'train-anno': '/data-ssd/gaopb/datasets/vqa/raw/v2_mscoco_train2014_annotations.json', 'val': '/data-ssd/gaopb/datasets/vqa/raw/v2_OpenEnded_mscoco_val2014_questions.json', 'val-anno': '/data-ssd/gaopb/datasets/vqa/raw/v2_mscoco_val2014_annotations.json', 'vg': '/data-ssd/gaopb/datasets/vqa/raw/VG_questions.json', 'vg-anno': '/data-ssd/gaopb/datasets/vqa/raw/VG_annotations.json', 'test': '/data-ssd/gaopb/datasets/vqa/raw/v2_OpenEnded_mscoco_test2015_questions.json'}, 'gqa': {'train': '/data-ssd/gaopb/datasets/gqa/raw/questions1.2/train_balanced_questions.json', 'val': '/data-ssd/gaopb/datasets/gqa/raw/questions1.2/val_balanced_questions.json', 'testdev': '/data-ssd/gaopb/datasets/gqa/raw/questions1.2/testdev_balanced_questions.json', 'test': '/data-ssd/gaopb/datasets/gqa/raw/questions1.2/submission_all_questions.json', 'val_all': '/data-ssd/gaopb/datasets/gqa/raw/questions1.2/val_all_questions.json', 'testdev_all': '/data-ssd/gaopb/datasets/gqa/raw/questions1.2/testdev_all_questions.json', 'train_choices': '/data-ssd/gaopb/datasets/gqa/raw/eval/train_choices', 'val_choices': '/data-ssd/gaopb/datasets/gqa/raw/eval/val_choices.json'}, 'clevr': {'train': '/data-ssd/gaopb/datasets/clevr/raw/questions/CLEVR_train_questions.json', 'val': '/data-ssd/gaopb/datasets/clevr/raw/questions/CLEVR_val_questions.json', 'test': '/data-ssd/gaopb/datasets/clevr/raw/questions/CLEVR_test_questions.json'}}
{ RESULT_PATH }->./results/result_test
{ RESUME }->False
{ RUN_MODE }->train
{ SEED }->1018
{ SPLIT }->{'train': 'train+val', 'val': 'testdev', 'test': 'test'}
{ SPLITS }->{'vqa': {'train': '', 'val': 'val', 'test': 'test'}, 'gqa': {'train': 'train+val', 'val': 'testdev', 'test': 'test'}, 'clevr': {'train': '', 'val': 'val', 'test': 'test'}}
{ SUB_BATCH_SIZE }->16
{ TASK_LOSS_CHECK }->{'vqa': ['bce', 'kld'], 'gqa': ['ce'], 'clevr': ['ce']}
{ TEST_SAVE_PRED }->False
{ TRAIN_SPLIT }->train+val
{ USE_AUX_FEAT }->True
{ USE_BBOX_FEAT }->True
{ USE_GLOVE }->True
{ VERBOSE }->True
{ VERSION }->gqa_mcan_large_train_tv1
{ WARMUP_EPOCH }->2
{ WORD_EMBED_SIZE }->300

nowTime: 2019-10-19 21:16:34
Epoch: 1, Loss: 1.6641010988449478, Lr: 3.3333333333333335e-05
Elapsed time: 21938, Speed(s/batch): 1.306167084564905

Binary: 58.48%
Open: 34.65%
Accuracy: 45.59%
Distribution: 4.15 (lower is better)
Accuracy / structural type:
choose: 60.85% (1129 questions)
compare: 44.48% (589 questions)
logical: 57.90% (1803 questions)
query: 34.65% (6805 questions)
verify: 61.41% (2252 questions)
Accuracy / semantic type:
attr: 50.10% (5186 questions)
cat: 36.99% (1149 questions)
global: 44.59% (157 questions)
obj: 68.51% (778 questions)
rel: 39.71% (5308 questions)
Accuracy / steps number:
1: 55.27% (237 questions)
2: 41.92% (6395 questions)
3: 46.98% (4266 questions)
4: 48.93% (793 questions)
5: 57.79% (822 questions)
6: 75.61% (41 questions)
7: 100.00% (20 questions)
8: 100.00% (3 questions)
9: 100.00% (1 questions)
Accuracy / words number:
3: 27.15% (151 questions)
4: 40.79% (630 questions)
5: 35.12% (1290 questions)
6: 41.71% (2074 questions)
7: 43.18% (1642 questions)
8: 46.50% (1185 questions)
9: 50.82% (1281 questions)
10: 50.36% (1249 questions)
11: 50.40% (994 questions)
12: 49.37% (638 questions)
13: 51.95% (462 questions)
14: 53.62% (345 questions)
15: 56.54% (237 questions)
16: 54.70% (117 questions)
17: 41.49% (94 questions)
18: 50.00% (76 questions)
19: 65.12% (43 questions)
20: 43.75% (32 questions)
21: 42.11% (19 questions)
22: 58.33% (12 questions)
23: 50.00% (4 questions)
24: 100.00% (2 questions)
25: 100.00% (1 questions)

=====================================
nowTime: 2019-10-20 03:23:54
Epoch: 2, Loss: 1.3524777466714077, Lr: 5e-05
Elapsed time: 21912, Speed(s/batch): 1.3046081712712103

Binary: 64.06%
Open: 38.65%
Accuracy: 50.31%
Distribution: 2.05 (lower is better)
Accuracy / structural type:
choose: 67.49% (1129 questions)
compare: 42.44% (589 questions)
logical: 60.73% (1803 questions)
query: 38.65% (6805 questions)
verify: 70.65% (2252 questions)
Accuracy / semantic type:
attr: 54.47% (5186 questions)
cat: 44.39% (1149 questions)
global: 48.41% (157 questions)
obj: 76.74% (778 questions)
rel: 43.71% (5308 questions)
Accuracy / steps number:
1: 66.24% (237 questions)
2: 46.29% (6395 questions)
3: 53.19% (4266 questions)
4: 49.94% (793 questions)
5: 59.61% (822 questions)
6: 78.05% (41 questions)
7: 100.00% (20 questions)
8: 100.00% (3 questions)
9: 100.00% (1 questions)
Accuracy / words number:
3: 33.11% (151 questions)
4: 45.87% (630 questions)
5: 39.30% (1290 questions)
6: 46.72% (2074 questions)
7: 48.48% (1642 questions)
8: 53.50% (1185 questions)
9: 56.83% (1281 questions)
10: 55.24% (1249 questions)
11: 50.40% (994 questions)
12: 53.29% (638 questions)
13: 53.46% (462 questions)
14: 58.26% (345 questions)
15: 62.45% (237 questions)
16: 60.68% (117 questions)
17: 54.26% (94 questions)
18: 46.05% (76 questions)
19: 79.07% (43 questions)
20: 46.88% (32 questions)
21: 52.63% (19 questions)
22: 58.33% (12 questions)
23: 50.00% (4 questions)
24: 100.00% (2 questions)
25: 100.00% (1 questions)

=====================================
nowTime: 2019-10-20 09:30:46
Epoch: 3, Loss: 1.1937491521311219, Lr: 5e-05
Elapsed time: 20965, Speed(s/batch): 1.2482277418097079

Binary: 70.03%
Open: 37.91%
Accuracy: 52.66%
Distribution: 2.33 (lower is better)
Accuracy / structural type:
choose: 68.20% (1129 questions)
compare: 52.80% (589 questions)
logical: 69.61% (1803 questions)
query: 37.91% (6805 questions)
verify: 75.80% (2252 questions)
Accuracy / semantic type:
attr: 59.10% (5186 questions)
cat: 42.65% (1149 questions)
global: 54.14% (157 questions)
obj: 82.39% (778 questions)
rel: 44.12% (5308 questions)
Accuracy / steps number:
1: 66.24% (237 questions)
2: 46.97% (6395 questions)
3: 55.34% (4266 questions)
4: 61.16% (793 questions)
5: 68.37% (822 questions)
6: 73.17% (41 questions)
7: 100.00% (20 questions)
8: 100.00% (3 questions)
9: 100.00% (1 questions)
Accuracy / words number:
3: 31.13% (151 questions)
4: 47.62% (630 questions)
5: 39.69% (1290 questions)
6: 49.32% (2074 questions)
7: 52.92% (1642 questions)
8: 56.12% (1185 questions)
9: 57.38% (1281 questions)
10: 58.69% (1249 questions)
11: 53.02% (994 questions)
12: 58.62% (638 questions)
13: 55.41% (462 questions)
14: 57.68% (345 questions)
15: 60.34% (237 questions)
16: 61.54% (117 questions)
17: 56.38% (94 questions)
18: 57.89% (76 questions)
19: 74.42% (43 questions)
20: 59.38% (32 questions)
21: 47.37% (19 questions)
22: 58.33% (12 questions)
23: 25.00% (4 questions)
24: 100.00% (2 questions)
25: 100.00% (1 questions)

=====================================
nowTime: 2019-10-20 15:21:50
Epoch: 4, Loss: 1.0559891983221752, Lr: 5e-05
Elapsed time: 21643, Speed(s/batch): 1.2885841661108253

Binary: 71.19%
Open: 37.90%
Accuracy: 53.18%
Distribution: 1.75 (lower is better)
Accuracy / structural type:
choose: 70.59% (1129 questions)
compare: 60.78% (589 questions)
logical: 70.83% (1803 questions)
query: 37.90% (6805 questions)
verify: 74.51% (2252 questions)
Accuracy / semantic type:
attr: 59.27% (5186 questions)
cat: 40.73% (1149 questions)
global: 56.05% (157 questions)
obj: 83.03% (778 questions)
rel: 45.46% (5308 questions)
Accuracy / steps number:
1: 66.67% (237 questions)
2: 47.08% (6395 questions)
3: 56.47% (4266 questions)
4: 60.66% (793 questions)
5: 69.71% (822 questions)
6: 82.93% (41 questions)
7: 95.00% (20 questions)
8: 100.00% (3 questions)
9: 100.00% (1 questions)
Accuracy / words number:
3: 35.10% (151 questions)
4: 46.83% (630 questions)
5: 40.78% (1290 questions)
6: 48.70% (2074 questions)
7: 52.92% (1642 questions)
8: 56.03% (1185 questions)
9: 59.64% (1281 questions)
10: 57.65% (1249 questions)
11: 55.53% (994 questions)
12: 59.87% (638 questions)
13: 56.49% (462 questions)
14: 60.00% (345 questions)
15: 65.82% (237 questions)
16: 58.12% (117 questions)
17: 50.00% (94 questions)
18: 56.58% (76 questions)
19: 72.09% (43 questions)
20: 56.25% (32 questions)
21: 57.89% (19 questions)
22: 58.33% (12 questions)
23: 50.00% (4 questions)
24: 100.00% (2 questions)
25: 100.00% (1 questions)

=====================================
nowTime: 2019-10-20 21:24:14
Epoch: 5, Loss: 0.9538466596906512, Lr: 5e-05
Elapsed time: 21822, Speed(s/batch): 1.29927454803296

Binary: 72.30%
Open: 38.91%
Accuracy: 54.24%
Distribution: 2.00 (lower is better)
Accuracy / structural type:
choose: 71.57% (1129 questions)
compare: 59.76% (589 questions)
logical: 71.66% (1803 questions)
query: 38.91% (6805 questions)
verify: 76.47% (2252 questions)
Accuracy / semantic type:
attr: 60.59% (5186 questions)
cat: 44.04% (1149 questions)
global: 51.59% (157 questions)
obj: 83.42% (778 questions)
rel: 46.04% (5308 questions)
Accuracy / steps number:
1: 64.98% (237 questions)
2: 48.30% (6395 questions)
3: 57.34% (4266 questions)
4: 62.55% (793 questions)
5: 70.56% (822 questions)
6: 80.49% (41 questions)
7: 100.00% (20 questions)
8: 100.00% (3 questions)
9: 100.00% (1 questions)
Accuracy / words number:
3: 34.44% (151 questions)
4: 46.19% (630 questions)
5: 40.70% (1290 questions)
6: 52.07% (2074 questions)
7: 52.62% (1642 questions)
8: 56.79% (1185 questions)
9: 60.27% (1281 questions)
10: 60.29% (1249 questions)
11: 55.73% (994 questions)
12: 59.40% (638 questions)
13: 57.58% (462 questions)
14: 59.13% (345 questions)
15: 63.71% (237 questions)
16: 62.39% (117 questions)
17: 60.64% (94 questions)
18: 65.79% (76 questions)
19: 79.07% (43 questions)
20: 62.50% (32 questions)
21: 68.42% (19 questions)
22: 58.33% (12 questions)
23: 25.00% (4 questions)
24: 100.00% (2 questions)
25: 100.00% (1 questions)

=====================================
nowTime: 2019-10-21 03:29:42
Epoch: 6, Loss: 0.8768809864153777, Lr: 5e-05
Elapsed time: 21838, Speed(s/batch): 1.3002370217494323

Binary: 74.09%
Open: 38.41%
Accuracy: 54.79%
Distribution: 2.15 (lower is better)
Accuracy / structural type:
choose: 73.34% (1129 questions)
compare: 64.52% (589 questions)
logical: 72.60% (1803 questions)
query: 38.41% (6805 questions)
verify: 78.15% (2252 questions)
Accuracy / semantic type:
attr: 61.76% (5186 questions)
cat: 43.17% (1149 questions)
global: 58.60% (157 questions)
obj: 83.03% (778 questions)
rel: 46.23% (5308 questions)
Accuracy / steps number:
1: 69.20% (237 questions)
2: 48.58% (6395 questions)
3: 58.02% (4266 questions)
4: 62.42% (793 questions)
5: 72.26% (822 questions)
6: 80.49% (41 questions)
7: 95.00% (20 questions)
8: 100.00% (3 questions)
9: 100.00% (1 questions)
Accuracy / words number:
3: 27.81% (151 questions)
4: 50.48% (630 questions)
5: 40.62% (1290 questions)
6: 51.25% (2074 questions)
7: 53.59% (1642 questions)
8: 56.03% (1185 questions)
9: 60.66% (1281 questions)
10: 62.61% (1249 questions)
11: 55.13% (994 questions)
12: 60.97% (638 questions)
13: 59.09% (462 questions)
14: 64.93% (345 questions)
15: 65.82% (237 questions)
16: 63.25% (117 questions)
17: 59.57% (94 questions)
18: 61.84% (76 questions)
19: 76.74% (43 questions)
20: 59.38% (32 questions)
21: 52.63% (19 questions)
22: 66.67% (12 questions)
23: 25.00% (4 questions)
24: 100.00% (2 questions)
25: 100.00% (1 questions)

=====================================
nowTime: 2019-10-21 09:35:25
Epoch: 7, Loss: 0.8124484499360088, Lr: 5e-05
Elapsed time: 21844, Speed(s/batch): 1.3006059032134938

Binary: 74.33%
Open: 39.71%
Accuracy: 55.60%
Distribution: 2.03 (lower is better)
Accuracy / structural type:
choose: 74.14% (1129 questions)
compare: 62.31% (589 questions)
logical: 72.43% (1803 questions)
query: 39.71% (6805 questions)
verify: 79.09% (2252 questions)
Accuracy / semantic type:
attr: 62.26% (5186 questions)
cat: 45.08% (1149 questions)
global: 56.05% (157 questions)
obj: 83.93% (778 questions)
rel: 47.19% (5308 questions)
Accuracy / steps number:
1: 73.00% (237 questions)
2: 49.73% (6395 questions)
3: 58.79% (4266 questions)
4: 61.92% (793 questions)
5: 71.05% (822 questions)
6: 80.49% (41 questions)
7: 100.00% (20 questions)
8: 100.00% (3 questions)
9: 100.00% (1 questions)
Accuracy / words number:
3: 32.45% (151 questions)
4: 50.63% (630 questions)
5: 41.09% (1290 questions)
6: 52.36% (2074 questions)
7: 55.66% (1642 questions)
8: 56.12% (1185 questions)
9: 61.67% (1281 questions)
10: 63.97% (1249 questions)
11: 57.85% (994 questions)
12: 59.09% (638 questions)
13: 56.71% (462 questions)
14: 64.35% (345 questions)
15: 61.60% (237 questions)
16: 67.52% (117 questions)
17: 55.32% (94 questions)
18: 67.11% (76 questions)
19: 74.42% (43 questions)
20: 59.38% (32 questions)
21: 68.42% (19 questions)
22: 75.00% (12 questions)
23: 25.00% (4 questions)
24: 100.00% (2 questions)
25: 100.00% (1 questions)

=====================================
nowTime: 2019-10-21 15:41:13
Epoch: 8, Loss: 0.760144480078704, Lr: 5e-05
Elapsed time: 22146, Speed(s/batch): 1.3185374425916225

Binary: 74.61%
Open: 39.66%
Accuracy: 55.70%
Distribution: 1.83 (lower is better)
Accuracy / structural type:
choose: 73.69% (1129 questions)
compare: 65.70% (589 questions)
logical: 73.43% (1803 questions)
query: 39.66% (6805 questions)
verify: 78.33% (2252 questions)
Accuracy / semantic type:
attr: 62.86% (5186 questions)
cat: 44.82% (1149 questions)
global: 59.87% (157 questions)
obj: 85.09% (778 questions)
rel: 46.63% (5308 questions)
Accuracy / steps number:
1: 68.35% (237 questions)
2: 49.27% (6395 questions)
3: 59.52% (4266 questions)
4: 62.67% (793 questions)
5: 73.60% (822 questions)
6: 68.29% (41 questions)
7: 100.00% (20 questions)
8: 100.00% (3 questions)
9: 100.00% (1 questions)
Accuracy / words number:
3: 32.45% (151 questions)
4: 48.10% (630 questions)
5: 42.40% (1290 questions)
6: 51.35% (2074 questions)
7: 54.63% (1642 questions)
8: 58.23% (1185 questions)
9: 62.53% (1281 questions)
10: 62.77% (1249 questions)
11: 57.65% (994 questions)
12: 59.72% (638 questions)
13: 59.96% (462 questions)
14: 64.06% (345 questions)
15: 67.93% (237 questions)
16: 61.54% (117 questions)
17: 61.70% (94 questions)
18: 67.11% (76 questions)
19: 76.74% (43 questions)
20: 56.25% (32 questions)
21: 63.16% (19 questions)
22: 66.67% (12 questions)
23: 50.00% (4 questions)
24: 100.00% (2 questions)
25: 100.00% (1 questions)

=====================================
nowTime: 2019-10-21 21:52:02
Epoch: 9, Loss: 0.4970882366093684, Lr: 1e-05
Elapsed time: 22065, Speed(s/batch): 1.3137314969597897

Binary: 77.33%
Open: 41.25%
Accuracy: 57.81%
Distribution: 1.66 (lower is better)
Accuracy / structural type:
choose: 77.41% (1129 questions)
compare: 66.38% (589 questions)
logical: 76.04% (1803 questions)
query: 41.25% (6805 questions)
verify: 81.17% (2252 questions)
Accuracy / semantic type:
attr: 65.43% (5186 questions)
cat: 46.04% (1149 questions)
global: 56.05% (157 questions)
obj: 86.89% (778 questions)
rel: 48.70% (5308 questions)
Accuracy / steps number:
1: 71.73% (237 questions)
2: 51.23% (6395 questions)
3: 61.44% (4266 questions)
4: 66.20% (793 questions)
5: 75.79% (822 questions)
6: 78.05% (41 questions)
7: 100.00% (20 questions)
8: 100.00% (3 questions)
9: 100.00% (1 questions)
Accuracy / words number:
3: 31.79% (151 questions)
4: 49.21% (630 questions)
5: 44.73% (1290 questions)
6: 54.97% (2074 questions)
7: 56.64% (1642 questions)
8: 60.00% (1185 questions)
9: 63.08% (1281 questions)
10: 65.57% (1249 questions)
11: 60.06% (994 questions)
12: 61.91% (638 questions)
13: 60.39% (462 questions)
14: 66.96% (345 questions)
15: 65.40% (237 questions)
16: 68.38% (117 questions)
17: 63.83% (94 questions)
18: 71.05% (76 questions)
19: 81.40% (43 questions)
20: 53.12% (32 questions)
21: 63.16% (19 questions)
22: 66.67% (12 questions)
23: 50.00% (4 questions)
24: 100.00% (2 questions)
25: 100.00% (1 questions)

=====================================
nowTime: 2019-10-22 04:01:29
Epoch: 10, Loss: 0.3919823110563895, Lr: 1e-05
Elapsed time: 21820, Speed(s/batch): 1.2991480892628366

Binary: 77.48%
Open: 40.60%
Accuracy: 57.53%
Distribution: 1.50 (lower is better)
Accuracy / structural type:
choose: 77.77% (1129 questions)
compare: 65.03% (589 questions)
logical: 77.04% (1803 questions)
query: 40.60% (6805 questions)
verify: 80.95% (2252 questions)
Accuracy / semantic type:
attr: 65.16% (5186 questions)
cat: 45.69% (1149 questions)
global: 54.14% (157 questions)
obj: 87.02% (778 questions)
rel: 48.42% (5308 questions)
Accuracy / steps number:
1: 71.73% (237 questions)
2: 50.51% (6395 questions)
3: 61.20% (4266 questions)
4: 66.96% (793 questions)
5: 77.62% (822 questions)
6: 78.05% (41 questions)
7: 100.00% (20 questions)
8: 100.00% (3 questions)
9: 100.00% (1 questions)
Accuracy / words number:
3: 30.46% (151 questions)
4: 50.16% (630 questions)
5: 43.33% (1290 questions)
6: 53.76% (2074 questions)
7: 56.76% (1642 questions)
8: 59.66% (1185 questions)
9: 64.01% (1281 questions)
10: 64.13% (1249 questions)
11: 59.76% (994 questions)
12: 63.01% (638 questions)
13: 60.17% (462 questions)
14: 66.96% (345 questions)
15: 67.09% (237 questions)
16: 67.52% (117 questions)
17: 67.02% (94 questions)
18: 71.05% (76 questions)
19: 76.74% (43 questions)
20: 56.25% (32 questions)
21: 78.95% (19 questions)
22: 75.00% (12 questions)
23: 50.00% (4 questions)
24: 100.00% (2 questions)
25: 100.00% (1 questions)

=====================================
nowTime: 2019-10-22 10:06:53
Epoch: 11, Loss: 0.2924597249457694, Lr: 2.0000000000000003e-06
Elapsed time: 21957, Speed(s/batch): 1.3072959441433467

Binary: 77.43%
Open: 40.87%
Accuracy: 57.65%
Distribution: 1.69 (lower is better)
Accuracy / structural type:
choose: 78.30% (1129 questions)
compare: 67.23% (589 questions)
logical: 75.98% (1803 questions)
query: 40.87% (6805 questions)
verify: 80.82% (2252 questions)
Accuracy / semantic type:
attr: 65.48% (5186 questions)
cat: 45.17% (1149 questions)
global: 57.32% (157 questions)
obj: 87.15% (778 questions)
rel: 48.38% (5308 questions)
Accuracy / steps number:
1: 72.15% (237 questions)
2: 51.07% (6395 questions)
3: 61.28% (4266 questions)
4: 65.45% (793 questions)
5: 76.16% (822 questions)
6: 75.61% (41 questions)
7: 100.00% (20 questions)
8: 100.00% (3 questions)
9: 100.00% (1 questions)
Accuracy / words number:
3: 34.44% (151 questions)
4: 50.63% (630 questions)
5: 43.64% (1290 questions)
6: 54.68% (2074 questions)
7: 56.58% (1642 questions)
8: 59.83% (1185 questions)
9: 62.84% (1281 questions)
10: 65.01% (1249 questions)
11: 59.96% (994 questions)
12: 63.17% (638 questions)
13: 59.31% (462 questions)
14: 66.67% (345 questions)
15: 66.67% (237 questions)
16: 67.52% (117 questions)
17: 63.83% (94 questions)
18: 68.42% (76 questions)
19: 79.07% (43 questions)
20: 53.12% (32 questions)
21: 68.42% (19 questions)
22: 66.67% (12 questions)
23: 25.00% (4 questions)
24: 100.00% (2 questions)
25: 100.00% (1 questions)

from openvqa.

Thanks for you reply @nbgao , did you use the same seed for mcan_small?

from openvqa.