Comments (5)
What were the configs printed at the beginning of the training?
from aot-benchmark.
What were the configs printed at the beginning of the training?
The config is as follows:
Exp default_AOTT:
Use GPU 2 for training VOS.
{
"DATASETS": [
"davis2017"
],
"DATA_DAVIS_REPEAT": 5,
"DATA_DYNAMIC_MERGE_PROB": 0.3,
"DATA_MAX_CROP_STEPS": 10,
"DATA_MAX_SCALE_FACTOR": 1.3,
"DATA_MIN_SCALE_FACTOR": 0.7,
"DATA_RANDOMCROP": [
465,
465
],
"DATA_RANDOMFLIP": 0.5,
"DATA_RANDOM_GAP_DAVIS": 12,
"DATA_RANDOM_GAP_YTB": 3,
"DATA_RANDOM_REVERSE_SEQ": true,
"DATA_SEQ_LEN": 5,
"DATA_SHORT_EDGE_LEN": 480,
"DATA_WORKERS": 8,
"DIR_CKPT": "./results/result/default_AOTT/PRE_YTB_DAV/ckpt",
"DIR_DATA": "./datasets",
"DIR_DAVIS": "./datasets/DAVIS",
"DIR_EMA_CKPT": "./results/result/default_AOTT/PRE_YTB_DAV/ema_ckpt",
"DIR_EVALUATION": "./results/result/default_AOTT/PRE_YTB_DAV/eval",
"DIR_IMG_LOG": "./results/result/default_AOTT/PRE_YTB_DAV/log/img",
"DIR_LOG": "./results/result/default_AOTT/PRE_YTB_DAV/log",
"DIR_RESULT": "./results/result/default_AOTT/PRE_YTB_DAV",
"DIR_ROOT": "./results",
"DIR_STATIC": "./datasets/Static",
"DIR_TB_LOG": "./results/result/default_AOTT/PRE_YTB_DAV/log/tensorboard",
"DIR_YTB": "./datasets/YTB",
"DIST_BACKEND": "nccl",
"DIST_ENABLE": true,
"DIST_START_GPU": 0,
"DIST_URL": "tcp://127.0.0.1:12325",
"EXP_NAME": "default_AOTT",
"MODEL_ALIGN_CORNERS": true,
"MODEL_ATT_HEADS": 8,
"MODEL_DECODER_INTERMEDIATE_LSTT": true,
"MODEL_ENCODER": "mobilenetv2",
"MODEL_ENCODER_DIM": [
24,
32,
96,
1280
],
"MODEL_ENCODER_EMBEDDING_DIM": 256,
"MODEL_ENCODER_PRETRAIN": "./pretrain_models/mobilenet_v2-b0353104.pth",
"MODEL_ENGINE": "aotengine",
"MODEL_EPSILON": 1e-05,
"MODEL_FREEZE_BACKBONE": false,
"MODEL_FREEZE_BN": true,
"MODEL_LSTT_NUM": 1,
"MODEL_MAX_OBJ_NUM": 10,
"MODEL_NAME": "AOTT",
"MODEL_SELF_HEADS": 8,
"MODEL_USE_PREV_PROB": false,
"MODEL_VOS": "aot",
"PRETRAIN": true,
"PRETRAIN_FULL": true,
"PRETRAIN_MODEL": "./results/result/default_AOTT/PRE/ema_ckpt/save_step_100000.pth",
"STAGE_NAME": "PRE_YTB_DAV",
"TEST_CKPT_PATH": null,
"TEST_CKPT_STEP": null,
"TEST_DATASET": "youtubevos",
"TEST_DATASET_FULL_RESOLUTION": false,
"TEST_DATASET_SPLIT": "val",
"TEST_FLIP": false,
"TEST_FRAME_LOG": false,
"TEST_GPU_ID": 0,
"TEST_GPU_NUM": 1,
"TEST_LONG_TERM_MEM_GAP": 9999,
"TEST_MAX_SIZE": 1040.0,
"TEST_MIN_SIZE": null,
"TEST_MULTISCALE": [
1
],
"TEST_WORKERS": 4,
"TRAIN_AUTO_RESUME": true,
"TRAIN_AUX_LOSS_RATIO": 1.0,
"TRAIN_AUX_LOSS_WEIGHT": 1.0,
"TRAIN_BATCH_SIZE": 16,
"TRAIN_CLIP_GRAD_NORM": 5.0,
"TRAIN_DATASET_FULL_RESOLUTION": false,
"TRAIN_EMA_RATIO": 0.1,
"TRAIN_ENABLE_PREV_FRAME": false,
"TRAIN_ENCODER_FREEZE_AT": 2,
"TRAIN_GPUS": 4,
"TRAIN_HARD_MINING_RATIO": 0.5,
"TRAIN_IMG_LOG": true,
"TRAIN_LOG_STEP": 200,
"TRAIN_LONG_TERM_MEM_GAP": 9999,
"TRAIN_LR": 0.0002,
"TRAIN_LR_COSINE_DECAY": false,
"TRAIN_LR_ENCODER_RATIO": 0.1,
"TRAIN_LR_MIN": 2e-05,
"TRAIN_LR_POWER": 0.9,
"TRAIN_LR_RESTART": 1,
"TRAIN_LR_UPDATE_STEP": 1,
"TRAIN_LR_WARM_UP_RATIO": 0.05,
"TRAIN_LSTT_DROPPATH": 0.1,
"TRAIN_LSTT_DROPPATH_LST": false,
"TRAIN_LSTT_DROPPATH_SCALING": false,
"TRAIN_LSTT_EMB_DROPOUT": 0.0,
"TRAIN_LSTT_ID_DROPOUT": 0.0,
"TRAIN_LSTT_LT_DROPOUT": 0.0,
"TRAIN_LSTT_ST_DROPOUT": 0.0,
"TRAIN_MAX_KEEP_CKPT": 8,
"TRAIN_OPT": "adamw",
"TRAIN_RESUME": false,
"TRAIN_RESUME_CKPT": null,
"TRAIN_RESUME_STEP": 0,
"TRAIN_SAVE_STEP": 1000,
"TRAIN_SEQ_TRAINING_FREEZE_PARAMS": [
"patch_wise_id_bank"
],
"TRAIN_SEQ_TRAINING_START_RATIO": 0.5,
"TRAIN_SGD_MOMENTUM": 0.9,
"TRAIN_START_STEP": 0,
"TRAIN_TBLOG": false,
"TRAIN_TBLOG_STEP": 50,
"TRAIN_TOP_K_PERCENT_PIXELS": 0.15,
"TRAIN_TOTAL_STEPS": 100000,
"TRAIN_WEIGHT_DECAY": 0.07,
"TRAIN_WEIGHT_DECAY_EXCLUSIVE": {},
"TRAIN_WEIGHT_DECAY_EXEMPTION": [
"absolute_pos_embed",
"relative_position_bias_table",
"relative_emb_v",
"conv_out"
]
}
The pretraining takes around 0.5s per iteration. However, when I train the "pre_ytb_dav", the dataloader seems to be slow every 18 iteration. Normally ,it takes 1e-3s for data preparation, and it will be slow to 13s every 18 iteration.
from aot-benchmark.
Since the DAVIS dataloarder has 5x60=300 video sequences, an epoch is about 18 iterations.
At the beginning of each epoch, the dataloader will initialize all the data workers and take several seconds.
For PyTorch >= 1.8, you could set persistent_workers=True
for the dataloader to avoid re-initialization.
from aot-benchmark.
Thanks a lot! It works for Pytorch1.9.
from aot-benchmark.
For PyTorch >= 1.8, you could set persistent_workers=True for the dataloader to avoid re-initialization.
@z-x-yang Be aware of
pytorch/pytorch#62066
Is this dataloader ready?
from aot-benchmark.
Related Issues (20)
- [Reproducing quantitative numbers of AOT and DeAOT] HOT 7
- YoutubeVOS quantitative numbers
- Would it help if I had more "ground truth" masks? HOT 2
- Object merging in tracking HOT 3
- Recreating the general examples HOT 2
- Optimizing Model Performance: Exploring ONNX Export and Engine Integration with TensorRT and OpenVino HOT 4
- Work Around for Pytorch-Correlation-extension HOT 5
- Demo.py with more then one GPU HOT 2
- package specified for utils HOT 1
- AUX loss WEIGHT and RATIO HOT 3
- [PAOT] swinv2b
- Evaluator Memory
- F.interpolate upsample gradient with amp HOT 1
- SDPA Attention
- How to train aot? Where is aot's train.py?
- Static image
- INT_MAX `local2global`
- How to reproduce the pre_dav_ytb result?
- To feed truth values into the model during the testing phase? HOT 4
- Does this model convert to onnx to run on edge device? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from aot-benchmark.