About the training time. about aot-benchmark HOT 5 CLOSED

What were the configs printed at the beginning of the training?

from aot-benchmark.

What were the configs printed at the beginning of the training?

The config is as follows:
Exp default_AOTT:
Use GPU 2 for training VOS.
{
"DATASETS": [
"davis2017"
],
"DATA_DAVIS_REPEAT": 5,
"DATA_DYNAMIC_MERGE_PROB": 0.3,
"DATA_MAX_CROP_STEPS": 10,
"DATA_MAX_SCALE_FACTOR": 1.3,
"DATA_MIN_SCALE_FACTOR": 0.7,
"DATA_RANDOMCROP": [
465,
465
],
"DATA_RANDOMFLIP": 0.5,
"DATA_RANDOM_GAP_DAVIS": 12,
"DATA_RANDOM_GAP_YTB": 3,
"DATA_RANDOM_REVERSE_SEQ": true,
"DATA_SEQ_LEN": 5,
"DATA_SHORT_EDGE_LEN": 480,
"DATA_WORKERS": 8,
"DIR_CKPT": "./results/result/default_AOTT/PRE_YTB_DAV/ckpt",
"DIR_DATA": "./datasets",
"DIR_DAVIS": "./datasets/DAVIS",
"DIR_EMA_CKPT": "./results/result/default_AOTT/PRE_YTB_DAV/ema_ckpt",
"DIR_EVALUATION": "./results/result/default_AOTT/PRE_YTB_DAV/eval",
"DIR_IMG_LOG": "./results/result/default_AOTT/PRE_YTB_DAV/log/img",
"DIR_LOG": "./results/result/default_AOTT/PRE_YTB_DAV/log",
"DIR_RESULT": "./results/result/default_AOTT/PRE_YTB_DAV",
"DIR_ROOT": "./results",
"DIR_STATIC": "./datasets/Static",
"DIR_TB_LOG": "./results/result/default_AOTT/PRE_YTB_DAV/log/tensorboard",
"DIR_YTB": "./datasets/YTB",
"DIST_BACKEND": "nccl",
"DIST_ENABLE": true,
"DIST_START_GPU": 0,
"DIST_URL": "tcp://127.0.0.1:12325",
"EXP_NAME": "default_AOTT",
"MODEL_ALIGN_CORNERS": true,
"MODEL_ATT_HEADS": 8,
"MODEL_DECODER_INTERMEDIATE_LSTT": true,
"MODEL_ENCODER": "mobilenetv2",
"MODEL_ENCODER_DIM": [
24,
32,
96,
1280
],
"MODEL_ENCODER_EMBEDDING_DIM": 256,
"MODEL_ENCODER_PRETRAIN": "./pretrain_models/mobilenet_v2-b0353104.pth",
"MODEL_ENGINE": "aotengine",
"MODEL_EPSILON": 1e-05,
"MODEL_FREEZE_BACKBONE": false,
"MODEL_FREEZE_BN": true,
"MODEL_LSTT_NUM": 1,
"MODEL_MAX_OBJ_NUM": 10,
"MODEL_NAME": "AOTT",
"MODEL_SELF_HEADS": 8,
"MODEL_USE_PREV_PROB": false,
"MODEL_VOS": "aot",
"PRETRAIN": true,
"PRETRAIN_FULL": true,
"PRETRAIN_MODEL": "./results/result/default_AOTT/PRE/ema_ckpt/save_step_100000.pth",
"STAGE_NAME": "PRE_YTB_DAV",
"TEST_CKPT_PATH": null,
"TEST_CKPT_STEP": null,
"TEST_DATASET": "youtubevos",
"TEST_DATASET_FULL_RESOLUTION": false,
"TEST_DATASET_SPLIT": "val",
"TEST_FLIP": false,
"TEST_FRAME_LOG": false,
"TEST_GPU_ID": 0,
"TEST_GPU_NUM": 1,
"TEST_LONG_TERM_MEM_GAP": 9999,
"TEST_MAX_SIZE": 1040.0,
"TEST_MIN_SIZE": null,
"TEST_MULTISCALE": [
1
],
"TEST_WORKERS": 4,
"TRAIN_AUTO_RESUME": true,
"TRAIN_AUX_LOSS_RATIO": 1.0,
"TRAIN_AUX_LOSS_WEIGHT": 1.0,
"TRAIN_BATCH_SIZE": 16,
"TRAIN_CLIP_GRAD_NORM": 5.0,
"TRAIN_DATASET_FULL_RESOLUTION": false,
"TRAIN_EMA_RATIO": 0.1,
"TRAIN_ENABLE_PREV_FRAME": false,
"TRAIN_ENCODER_FREEZE_AT": 2,
"TRAIN_GPUS": 4,
"TRAIN_HARD_MINING_RATIO": 0.5,
"TRAIN_IMG_LOG": true,
"TRAIN_LOG_STEP": 200,
"TRAIN_LONG_TERM_MEM_GAP": 9999,
"TRAIN_LR": 0.0002,
"TRAIN_LR_COSINE_DECAY": false,
"TRAIN_LR_ENCODER_RATIO": 0.1,
"TRAIN_LR_MIN": 2e-05,
"TRAIN_LR_POWER": 0.9,
"TRAIN_LR_RESTART": 1,
"TRAIN_LR_UPDATE_STEP": 1,
"TRAIN_LR_WARM_UP_RATIO": 0.05,
"TRAIN_LSTT_DROPPATH": 0.1,
"TRAIN_LSTT_DROPPATH_LST": false,
"TRAIN_LSTT_DROPPATH_SCALING": false,
"TRAIN_LSTT_EMB_DROPOUT": 0.0,
"TRAIN_LSTT_ID_DROPOUT": 0.0,
"TRAIN_LSTT_LT_DROPOUT": 0.0,
"TRAIN_LSTT_ST_DROPOUT": 0.0,
"TRAIN_MAX_KEEP_CKPT": 8,
"TRAIN_OPT": "adamw",
"TRAIN_RESUME": false,
"TRAIN_RESUME_CKPT": null,
"TRAIN_RESUME_STEP": 0,
"TRAIN_SAVE_STEP": 1000,
"TRAIN_SEQ_TRAINING_FREEZE_PARAMS": [
"patch_wise_id_bank"
],
"TRAIN_SEQ_TRAINING_START_RATIO": 0.5,
"TRAIN_SGD_MOMENTUM": 0.9,
"TRAIN_START_STEP": 0,
"TRAIN_TBLOG": false,
"TRAIN_TBLOG_STEP": 50,
"TRAIN_TOP_K_PERCENT_PIXELS": 0.15,
"TRAIN_TOTAL_STEPS": 100000,
"TRAIN_WEIGHT_DECAY": 0.07,
"TRAIN_WEIGHT_DECAY_EXCLUSIVE": {},
"TRAIN_WEIGHT_DECAY_EXEMPTION": [
"absolute_pos_embed",
"relative_position_bias_table",
"relative_emb_v",
"conv_out"
]
}

The pretraining takes around 0.5s per iteration. However, when I train the "pre_ytb_dav", the dataloader seems to be slow every 18 iteration. Normally ,it takes 1e-3s for data preparation, and it will be slow to 13s every 18 iteration.

from aot-benchmark.

Since the DAVIS dataloarder has 5x60=300 video sequences, an epoch is about 18 iterations.

At the beginning of each epoch, the dataloader will initialize all the data workers and take several seconds.

For PyTorch >= 1.8, you could set persistent_workers=True for the dataloader to avoid re-initialization.

from aot-benchmark.

Thanks a lot! It works for Pytorch1.9.

from aot-benchmark.

For PyTorch >= 1.8, you could set persistent_workers=True for the dataloader to avoid re-initialization.

@z-x-yang Be aware of
pytorch/pytorch#62066

Is this dataloader ready?

from aot-benchmark.