jackroos / vl-bert Goto Github PK

View Code? Open in Web Editor NEW

734.0 14.0 110.0 5.54 MB

Code for ICLR 2020 paper "VL-BERT: Pre-training of Generic Visual-Linguistic Representations".

License: MIT License

Python 44.29% C++ 0.91% Cuda 1.29% Shell 0.19% Makefile 0.01% JavaScript 0.84% Jupyter Notebook 52.47%

pre-training representation-learning self-supervised-learning vision-and-language bert pytorch iclr2020 vl-bert

vl-bert's Introduction

VL-BERT

By Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, Jifeng Dai.

This repository is an official implementation of the paper VL-BERT: Pre-training of Generic Visual-Linguistic Representations.

Update on 2020/01/16 Add code of visualization.

Update on 2019/12/20 Our VL-BERT got accepted by ICLR 2020.

Introduction

VL-BERT is a simple yet powerful pre-trainable generic representation for visual-linguistic tasks. It is pre-trained on the massive-scale caption dataset and text-only corpus, and can be fine-tuned for various down-stream visual-linguistic tasks, such as Visual Commonsense Reasoning, Visual Question Answering and Referring Expression Comprehension.

Thanks to PyTorch and its 3rd-party libraries, this codebase also contains following features:

Distributed Training
FP16 Mixed-Precision Training
Various Optimizers and Learning Rate Schedulers
Gradient Accumulation
Monitoring the Training Using TensorboardX

Citing VL-BERT

@inproceedings{
  Su2020VL-BERT:,
  title={VL-BERT: Pre-training of Generic Visual-Linguistic Representations},
  author={Weijie Su and Xizhou Zhu and Yue Cao and Bin Li and Lewei Lu and Furu Wei and Jifeng Dai},
  booktitle={International Conference on Learning Representations},
  year={2020},
  url={https://openreview.net/forum?id=SygXPaEYvH}
}

Prepare

Environment

Ubuntu 16.04, CUDA 9.0, GCC 4.9.4

Python 3.6.x

# We recommend you to use Anaconda/Miniconda to create a conda environment
conda create -n vl-bert python=3.6 pip
conda activate vl-bert

PyTorch 1.0.0 or 1.1.0

conda install pytorch=1.1.0 cudatoolkit=9.0 -c pytorch

Apex (optional, for speed-up and fp16 training)

git clone https://github.com/jackroos/apex
cd ./apex
pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./

Other requirements:

pip install Cython
pip install -r requirements.txt

Compile
```
./scripts/init.sh
```

Data

See PREPARE_DATA.md.

Pre-trained Models

See PREPARE_PRETRAINED_MODELS.md.

Training

Distributed Training on Single-Machine

./scripts/dist_run_single.sh <num_gpus> <task>/train_end2end.py <path_to_cfg> <dir_to_store_checkpoint>

<num_gpus>: number of gpus to use.
<task>: pretrain/vcr/vqa/refcoco.
<path_to_cfg>: config yaml file under ./cfgs/<task>.
<dir_to_store_checkpoint>: root directory to store checkpoints.

Following is a more concrete example:

./scripts/dist_run_single.sh 4 vcr/train_end2end.py ./cfgs/vcr/base_q2a_4x16G_fp32.yaml ./

Distributed Training on Multi-Machine

For example, on 2 machines (A and B), each with 4 GPUs,

run following command on machine A:

./scripts/dist_run_multi.sh 2 0 <ip_addr_of_A> 4 <task>/train_end2end.py <path_to_cfg> <dir_to_store_checkpoint>

run following command on machine B:

./scripts/dist_run_multi.sh 2 1 <ip_addr_of_A> 4 <task>/train_end2end.py <path_to_cfg> <dir_to_store_checkpoint>

Non-Distributed Training

./scripts/nondist_run.sh <task>/train_end2end.py <path_to_cfg> <dir_to_store_checkpoint>

Note:

In yaml files under ./cfgs, we set batch size for GPUs with at least 16G memory, you may need to adapt the batch size and gradient accumulation steps according to your actual case, e.g., if you decrease the batch size, you should also increase the gradient accumulation steps accordingly to keep 'actual' batch size for SGD unchanged.
For efficiency, we recommend you to use distributed training even on single-machine. But for RefCOCO+, you may meet deadlock using distributed training due to unknown reason (it may be related to PyTorch dataloader deadloack), you can simply use non-distributed training to solve this problem.

Evaluation

VCR

Local evaluation on val set:

python vcr/val.py \
  --a-cfg <cfg_of_q2a> --r-cfg <cfg_of_qa2r> \
  --a-ckpt <checkpoint_of_q2a> --r-ckpt <checkpoint_of_qa2r> \
  --gpus <indexes_of_gpus_to_use> \
  --result-path <dir_to_save_result> --result-name <result_file_name>

Note: <indexes_of_gpus_to_use> is gpu indexes, e.g., 0 1 2 3.

Generate prediction results on test set for leaderboard submission:

python vcr/test.py \
  --a-cfg <cfg_of_q2a> --r-cfg <cfg_of_qa2r> \
  --a-ckpt <checkpoint_of_q2a> --r-ckpt <checkpoint_of_qa2r> \
  --gpus <indexes_of_gpus_to_use> \
  --result-path <dir_to_save_result> --result-name <result_file_name>

VQA

Generate prediction results on test set for EvalAI submission:

python vqa/test.py \
  --cfg <cfg_file> \
  --ckpt <checkpoint> \
  --gpus <indexes_of_gpus_to_use> \
  --result-path <dir_to_save_result> --result-name <result_file_name>

RefCOCO+

Local evaluation on val/testA/testB set:

python refcoco/test.py \
  --split <val|testA|testB> \
  --cfg <cfg_file> \
  --ckpt <checkpoint> \
  --gpus <indexes_of_gpus_to_use> \
  --result-path <dir_to_save_result> --result-name <result_file_name>

Visualization

See VISUALIZATION.md.

Acknowledgements

Many thanks to following codes that help us a lot in building this codebase:

vl-bert's People

Contributors

Stargazers

Watchers

Forkers

vl-bert's Issues

Visualizing attention per head and per layer

Dear authors, thank you for your great work and making it available.

I'm currently using vl-bert and I'd like to visualize attention to make sure my model is learning from images and text jointly and not independently. I've tried using the visualization tool available in the notebook under ./viz however it only provides a birds-eye view of attention across all of the model’s layers and heads. I'd be interested in looking at something that looks more like what you show in the Figure 2 of the README with the text and ROI label on each side and focusing on one attention head and one layer at a time. Can this be plotted from some code I haven't found yet in the repo?

Using CUDA 10.1

Hi, my system has CUDA 10.1, so I followed your setup instructions, but using conda install pytorch torchvision cudatoolkit=10.1 -c pytorch. The rest of the installation went fine, but I got the following message when trying to test refcoco+:
ImportError: VL-BERT/refcoco/../common/lib/roi_pooling/C_ROIPooling.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN6caffe26detail36_typeMetaDataInstance_preallocated_7E

I think this could be due to the CUDA version mismatch. Do you know if the compiling scripts have to be run differently for CUDA 10.1?

Google drive access rights for en_corpus

Hi, thank you for making this code available !
I can download the en_corpus .dioc files (bookcorpus, wikipedia) from my PC, but I want them on a server an have to use a bash wget command; when I use the "gdown" library, the command don't work because the permission level seems insufficient
`Maybe you need to change permission over 'Anyone with the link'?``
Can you please change the accessibility level or help me find a way to download the corpus from command line ? Thanks!

VL-BERT(base with pretraining) and VL-BERT large

Hi, Thanks for the great repo.
I wonder that do we need to pretrain model again if I want to apply VL-BERT(base with pretraining) or VL-BERT large to vcr task? Thanks a lot!

Some little errors in preparation of conceptual-captions dataset

When I wanted to extract img feature from conceptual-captions following the instructions, I found some errors during inference with this file and it took me sometime to debug and I'd like to share with you:

Firstly, the calling script should be:

python ./tools/generate_tsv_v2.py --gpu 0,1,2,3,4,5,6,7 --cfg experiments/cfgs/faster_rcnn_end2end_resnet.yml --def models/vg/ResNet-101/faster_rcnn_end2end_final/test.prototxt --net data/faster_rcnn_models/resnet101_faster_rcnn_final.caffemodel --split conceptual_captions_train --data_root {Conceptual_Captions_Root} --out {Conceptual_Captions_Root}/train_frcnn/

A little mistake in --def params

Secondly, in Line46 and Line54, should be:

with open(os.path.join(data_root, 'utils/train.json')) as f:  #Line46
with open(os.path.join(data_root, 'utils/val.json')) as f:  #Line54

Thirdly, in Line67, should be:

zip_image = ziphelper.imread(str("/".join(data_root, im_file)))

Fourthly, in Line142, we need another params:

def generate_tsv(gpu_id, prototxt, weights, image_ids, data_root, outfolder):

And correspondly, in Line170, the same problem:

json.dump(get_detections_from_im(net, im_file, image_id, ziphelper, data_root), f)

Really thank you for your sharing of the code!

subprocess.CalledProcessError: Command xxx returned non-zero exit status 1

hi!
i want vcr training.

but i got this error!

(vl-bert) ailab@ailab:~/vl-bert/VL-BERT$ ./scripts/dist_run_single.sh 2 vcr/train_end2end.py ./cfgs/vcr/base_q2a_4x16G_fp32.yaml ./vcr/saves/q2a/
Traceback (most recent call last):
  File "vcr/train_end2end.py", line 8, in <module>
    from vcr.function.train import train_net
  File "/home/ailab/vl-bert/VL-BERT/vcr/../vcr/function/train.py", line 25, in <module>
    from vcr.data.build import make_dataloader, build_dataset, build_transforms
  File "/home/ailab/vl-bert/VL-BERT/vcr/../vcr/data/build.py", line 5, in <module>
    from .transforms.build import build_transforms
  File "/home/ailab/vl-bert/VL-BERT/vcr/../vcr/data/transforms/__init__.py", line 1, in <module>
    from .transforms import Compose
  File "/home/ailab/vl-bert/VL-BERT/vcr/../vcr/data/transforms/transforms.py", line 6, in <module>
    import torchvision
  File "/home/ailab/anaconda3/envs/vl-bert/lib/python3.6/site-packages/torchvision/__init__.py", line 1, in <module>
    from torchvision import models
  File "/home/ailab/anaconda3/envs/vl-bert/lib/python3.6/site-packages/torchvision/models/__init__.py", line 11, in <module>
    from . import detection
  File "/home/ailab/anaconda3/envs/vl-bert/lib/python3.6/site-packages/torchvision/models/detection/__init__.py", line 1, in <module>
    from .faster_rcnn import *
  File "/home/ailab/anaconda3/envs/vl-bert/lib/python3.6/site-packages/torchvision/models/detection/faster_rcnn.py", line 7, in <module>
    from torchvision.ops import misc as misc_nn_ops
  File "/home/ailab/anaconda3/envs/vl-bert/lib/python3.6/site-packages/torchvision/ops/__init__.py", line 1, in <module>
    from .boxes import nms, box_iou
  File "/home/ailab/anaconda3/envs/vl-bert/lib/python3.6/site-packages/torchvision/ops/boxes.py", line 2, in <module>
    from torchvision import _C
ImportError: /home/ailab/anaconda3/envs/vl-bert/lib/python3.6/site-packages/torchvision/_C.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN3c107Warning4warnENS_14SourceLocationENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE
Traceback (most recent call last):
  File "vcr/train_end2end.py", line 8, in <module>
    from vcr.function.train import train_net
  File "/home/ailab/vl-bert/VL-BERT/vcr/../vcr/function/train.py", line 25, in <module>
    from vcr.data.build import make_dataloader, build_dataset, build_transforms
  File "/home/ailab/vl-bert/VL-BERT/vcr/../vcr/data/build.py", line 5, in <module>
    from .transforms.build import build_transforms
  File "/home/ailab/vl-bert/VL-BERT/vcr/../vcr/data/transforms/__init__.py", line 1, in <module>
    from .transforms import Compose
  File "/home/ailab/vl-bert/VL-BERT/vcr/../vcr/data/transforms/transforms.py", line 6, in <module>
    import torchvision
  File "/home/ailab/anaconda3/envs/vl-bert/lib/python3.6/site-packages/torchvision/__init__.py", line 1, in <module>
    from torchvision import models
  File "/home/ailab/anaconda3/envs/vl-bert/lib/python3.6/site-packages/torchvision/models/__init__.py", line 11, in <module>
    from . import detection
  File "/home/ailab/anaconda3/envs/vl-bert/lib/python3.6/site-packages/torchvision/models/detection/__init__.py", line 1, in <module>
    from .faster_rcnn import *
  File "/home/ailab/anaconda3/envs/vl-bert/lib/python3.6/site-packages/torchvision/models/detection/faster_rcnn.py", line 7, in <module>
    from torchvision.ops import misc as misc_nn_ops
  File "/home/ailab/anaconda3/envs/vl-bert/lib/python3.6/site-packages/torchvision/ops/__init__.py", line 1, in <module>
    from .boxes import nms, box_iou
  File "/home/ailab/anaconda3/envs/vl-bert/lib/python3.6/site-packages/torchvision/ops/boxes.py", line 2, in <module>
    from torchvision import _C
ImportError: /home/ailab/anaconda3/envs/vl-bert/lib/python3.6/site-packages/torchvision/_C.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN3c107Warning4warnENS_14SourceLocationENSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE
Traceback (most recent call last):
  File "./scripts/launch.py", line 200, in <module>
    main()
  File "./scripts/launch.py", line 196, in main
    cmd=process.args)
subprocess.CalledProcessError: Command '['/home/ailab/anaconda3/envs/vl-bert/bin/python', '-u', 'vcr/train_end2end.py', '--cfg', './cfgs/vcr/base_q2a_4x16G_fp32.yaml', '--model-dir', './vcr/saves/q2a/', '--dist']' returned non-zero exit status 1.

how do i do?
thank you!

Can we use VL-BERT as image or text generator?

use VL-Bert as a generator for text or image or both?

Obtaining visual features for the ConceptualCaptions data set

Hi,
I'd like to take a look at the pretraining but I was wondering if there was a way to have access to the train_frcnn.zip files without having to run the caffe model, are they available somewhere?
Best,

Doesn't it offer a fine-tuned VCR model?

hi!

i want fine-tuning for vcr task!
Doesn't it offer a fine-tuned VCR model?

thank you :)!

RuntimeError：CUDA out of memory

when have trained a fine-tuned VCR model nearly approaching a day, the error happened.
Traceback (most recent call last): File "vcr/train_end2end.py", line 59, in <module> main() File "vcr/train_end2end.py", line 53, in main rank, model = train_net(args, config) File "/home/songzijie/project/VLbert/VL-BERT-master/vcr/../vcr/function/train.py", line 337, in train_net gradient_accumulate_steps=config.TRAIN.GRAD_ACCUMULATE_STEPS) File "/home/songzijie/project/VLbert/VL-BERT-master/vcr/../common/trainer.py", line 115, in train outputs, loss = net(*batch) File "/home/songzijie/.conda/envs/vl-bert/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__ result = self.forward(*input, **kwargs) File "/home/songzijie/.conda/envs/vl-bert/lib/python3.6/site-packages/torch/nn/parallel/distributed.py", line 376, in forward output = self.module(*inputs[0], **kwargs[0]) File "/home/songzijie/.conda/envs/vl-bert/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__ result = self.forward(*input, **kwargs) File "/home/songzijie/project/VLbert/VL-BERT-master/vcr/../common/module.py", line 22, in forward return self.train_forward(*inputs, **kwargs) File "/home/songzijie/project/VLbert/VL-BERT-master/vcr/../vcr/modules/resnet_vlbert_for_vcr.py", line 340, in train_forward output_text_and_object_separately=True) File "/home/songzijie/.conda/envs/vl-bert/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__ result = self.forward(*input, **kwargs) File "/home/songzijie/project/VLbert/VL-BERT-master/vcr/../common/nlp/time_distributed.py", line 35, in forward reshaped_outputs = self._module(*reshaped_inputs, **kwargs) File "/home/songzijie/.conda/envs/vl-bert/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__ result = self.forward(*input, **kwargs) File "/home/songzijie/project/VLbert/VL-BERT-master/vcr/../common/visual_linguistic_bert.py", line 140, in forward output_attention_probs=output_attention_probs) File "/home/songzijie/.conda/envs/vl-bert/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__ result = self.forward(*input, **kwargs) File "/home/songzijie/project/VLbert/VL-BERT-master/vcr/../external/pytorch_pretrained_bert/modeling.py", line 410, in forward hidden_states = layer_module(hidden_states, attention_mask, output_attention_probs=output_attention_probs) File "/home/songzijie/.conda/envs/vl-bert/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__ result = self.forward(*input, **kwargs) File "/home/songzijie/project/VLbert/VL-BERT-master/vcr/../external/pytorch_pretrained_bert/modeling.py", line 392, in forward intermediate_output = self.intermediate(attention_output) File "/home/songzijie/.conda/envs/vl-bert/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__ result = self.forward(*input, **kwargs) File "/home/songzijie/project/VLbert/VL-BERT-master/vcr/../external/pytorch_pretrained_bert/modeling.py", line 362, in forward hidden_states = self.dense(hidden_states) File "/home/songzijie/.conda/envs/vl-bert/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__ result = self.forward(*input, **kwargs) File "/home/songzijie/.conda/envs/vl-bert/lib/python3.6/site-packages/torch/nn/modules/linear.py", line 92, in forward return F.linear(input, self.weight, self.bias) File "/home/songzijie/.conda/envs/vl-bert/lib/python3.6/site-packages/torch/nn/functional.py", line 1408, in linear output = input.matmul(weight.t()) RuntimeError: CUDA out of memory. Tried to allocate 24.00 MiB (GPU 1; 11.91 GiB total capacity; 10.43 GiB already allocated; 12.06 MiB free; 348.09 MiB cached)
I tryed training by 2 or 3 GPUs.
And tryed to reduced Batch by changing LOG_FREQUENT from 100 to 2, but no use.
The error still happened in one day train.
I hope can get some help for it.

how much computing resources do you spend to pretrain the model?

Hi, recently I prepare to pretrain a similar model, I am wondering how much computing resources (the number of gpus, the type of gpu and amount of time). Hope you could answer this question to me.

How to use Pre-trained Models?

I have such a data set, each of its records is like this: <text>,<image>,<label>. It is a simple classification task, I have the following questions, I hope you can answer:
1. Can you use this pre-trained model for fine-tuning? So how do you do it?
2. How do I load my dataset into the model?
3. How to get the output of the model?
Thank you for answering my question

How to generate my own vgbua_res101_precomputed files.

Hi,

When I use your code to evaluate my own visual grounding dataset, could you give me some instructions for constructing vgbua_res101_precomputed files?

Thanks

init() missing 2 required positional arguments: 'doc' and 'pos'

Traceback (most recent call last):
File "pretrain/train_end2end.py", line 51, in
main()
File "pretrain/train_end2end.py", line 47, in main
rank, model = train_net(args, config)
File "VL-BERT/pretrain/../pretrain/function/train.py", line 361, in train_net
gradient_accumulate_steps=config.TRAIN.GRAD_ACCUMULATE_STEPS)
File "VL-BERT/pretrain/../common/trainer.py", line 101, in train
for nbatch, batch in enumerate(train_loader):
File "VL-BERT/pretrain/../common/utils/multi_task_dataloader.py", line 35, in next
output_tuple = (*next(self.iters[0]), )
File "anaconda3/envs/vl-bert/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 582, in next
return self._process_next_batch(batch)
File "anaconda3/envs/vl-bert/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 608, in _process_next_batch
raise batch.exc_type(batch.exc_msg)
TypeError: init() missing 2 required positional arguments: 'doc' and 'pos'

GCC version

Hi, I am wondering if there is anyway I can run it with GCC 7.5.0? I don't really know how to downgrade my GCC version. I am getting the ROI_pooling undefined symbol error, so I figure it might be something about gcc version...?

The results on test set about VCR task

Thanks for your great work!
And I wonder that how to get the test result directly, must I generate prediction results on test set for leaderboard submission？

attention_viz

Dear contributor,

Fantastic work on VL task!

Now I have some trouble showing the visualization results between language and ROIs. Could you please hit me your code to generate the result as attention_viz.png showed if possible?

Many thanks.

Could you provide the logfile of pretraining process?

Hi, Thanks for the great repo.

I want to get an understanding of what happened during the pretraining, especially the train and validation loss change.

In other work like lxmert and my own experiment, The MLM loss is highly overfitted. So I wander is that the same thing of VL-BERT? Considering the specia mask operation of VL-BERT.

Is it possible that you can share the logfile of the pretraining process?

i have 'ImportError: /home/ailab/VL-BERT/vcr/../common/lib/roi_pooling/C_ROIPooling.cpython-36m-x86_64-linux-gnu.so: undefined symbol: __cudaPopCallConfiguration' problem!

cuda 9.0
pytorch 1.1.0
ubuntu 16.04

when running the command "./scripts/nondist_run.sh vcr/train_end2end.py ./cfgs/vcr/base_q2a_4x16G_fp32.yaml ./"
i have a problem.
i find this problem caused of mismatch in the cuda version installed on the system and cuda version working in my environment.
but my cuda version installed on the system is 9.0 and cuda version working in my environment is 9.0..
how do i do? help T^T

Traceback (most recent call last):
  File "vcr/train_end2end.py", line 8, in <module>
    from vcr.function.train import train_net
  File "/home/ailab/VL-BERT/vcr/../vcr/function/train.py", line 26, in <module>
    from vcr.modules import *
  File "/home/ailab/VL-BERT/vcr/../vcr/modules/__init__.py", line 1, in <module>
    from .resnet_vlbert_for_vcr import ResNetVLBERT
  File "/home/ailab/VL-BERT/vcr/../vcr/modules/resnet_vlbert_for_vcr.py", line 7, in <module>
    from common.fast_rcnn import FastRCNN
  File "/home/ailab/VL-BERT/vcr/../common/fast_rcnn.py", line 10, in <module>
    from common.lib.roi_pooling.roi_pool import ROIPool
  File "/home/ailab/VL-BERT/vcr/../common/lib/roi_pooling/__init__.py", line 1, in <module>
    from .roi_align import ROIAlign
  File "/home/ailab/VL-BERT/vcr/../common/lib/roi_pooling/roi_align.py", line 8, in <module>
    from . import C_ROIPooling
ImportError: /home/ailab/VL-BERT/vcr/../common/lib/roi_pooling/C_ROIPooling.cpython-36m-x86_64-linux-gnu.so: undefined symbol: __cudaPopCallConfiguration

Can you give a txt file containing all word vectors？

Thank you for your work, I would love to reference your VL-Bert in my network. Is there a VQA dataset word vector file saved in txt format？

数据下载问题

您好，我在学习您提供的参考工程，请问参考工程中的train_gcc-training.tsv以及validation_gcc_1.1.0-Validation.tsv必须从https://ai.google.com/research/ConceptualCaptions/download下载吗，我从国内好像访问不到这个数据，谢谢。

share logs of pretrain

Excellent work!
I used the script and https://github.com/jackroos/VL-BERT/blob/master/cfgs/pretrain/base_e2e_16x16G_fp16.yaml to pretrain，but can't be sure if the loss and accuracy are good enough.
When using my pre-trained model to finetune in VCR task, I cannot get the high score of using the shared pre-trained model.
I will appreciate for your share of pre-trained logs.
Thanks.

torchvision dependency

I follow the document to install VL-BERT, then run scripts
'./scripts/dist_run_single.sh 4 vcr/train_end2end.py ./cfgs/vcr/base_q2a_4x16G_fp32.yaml ./'
Got this error:

fine-tune on VQA

Hi,

Thanks for your code. I want to use VLBERT to fine-tune on VQA dataset. The pretrained weights have 12-layer parameters. But in config file of VQA task, the number of layers is only 4. Can you provide more details about how to load pretrained weights and fine-tune on VQA?

Thank you.

Query regarding Pretrained model

Hi,

In Pre-training VL-BERT section, you've highlighed some tasks on which model was trained.

Pretraining on Conceptual Conceptions
Masked Language Modeling with Visual Clues
Masked RoI Classification with Linguistic Clues

I had some questions:

Is the pretrained model which you've provided was trained on all 3 objectives ? If yes, should it be used for finetuning now ? If not, do I need to perform pretrainining ?
Can you please clarify the order of usage of datasets in all three objectives ? (As in in which dataset was used to satisfy the objective)
In the code, I'm seeing that the pretrained models dict partially updates the weights of the instantiated model. Why so ? Are there some parts of the model that are initialized as mentioned in the paper ?

Pre-Trained Model checkpoints available?

I would like to run the evaluation scrtipt on VCR dataset vcr/val.py, but not sure about the checkpoint files --a-ckpt & --r-ckpt.
Could kindly assist me with this issue.

For the given template:

python vcr/val.py \
  --a-cfg <cfg_of_q2a> --r-cfg <cfg_of_qa2r> \
  --a-ckpt <checkpoint_of_q2a> --r-ckpt <checkpoint_of_qa2r> \
  --gpus <indexes_of_gpus_to_use> \
  --result-path <dir_to_save_result> --result-name <result_file_name>

This is what I am executing:

python3 vcr/val.py \
--a-cfg ./cfgs/vcr/base_q2a_4x16G_fp32.yaml \
--r-cfg ./cfgs/vcr/base_qa2r_4x16G_fp32.yaml \
--gpus 0 1 --result-path ./results/ --result-name eval_vcr \
--a-ckpt ./model/pretrained_model/vl-bert-base-e2e.model

However I am facing the following error:

Traceback (most recent call last):
  File "vcr/val.py", line 218, in <module>
    main()
  File "/home/axe/VirtualEnvs/py3ve/lib/python3.6/site-packages/torch/autograd/grad_mode.py", line 49, in decorate_no_grad
    return func(*args, **kwargs)
  File "vcr/val.py", line 119, in main
    smart_load_model_state_dict(answer_model, a_ckpt['state_dict'])
  File "/home/axe/Projects/VL_BERT/common/utils/load.py", line 16, in smart_load_model_state_dict
    raise ValueError('failed to match key of state dict smartly!')
ValueError: failed to match key of state dict smartly!

This is my ./models directory:

└── pretrained_model
    ├── bert-base-uncased
    │   ├── bert_config.json
    │   ├── pytorch_model.bin
    │   └── vocab.txt
    ├── bert-large-uncased
    │   ├── bert_config.json
    │   ├── pytorch_model.bin
    │   └── vocab.txt
    ├── PREPARE_PRETRAINED_MODELS.md
    ├── resnet101-pt-vgbua-0000.model
    └── vl-bert-base-e2e.model

anyone can help me：error：ImportError: C_ROIPooling.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN6caffe26detail36_typeMetaDataInstance_preallocated_7E

when i run the code， the error occures：

Check failed: error == cudaSuccess (2 vs. 0) out of memory

Dear Author,

When I try to run " generate_tsv_v2_1.py", I meet the problem "Check failed: error == cudaSuccess (2 vs. 0) out of memory".
My GPU is 11G Tesla K40m. I have changed the batch_size to 1, but the problem is still there
Do you know how to solve the problem? Thanks a lot!

The MLMACC without only MLM

Thanks for your great work!
Could you provide the MLMACC, including "val" and "train", on the pretraining stage based on the base-bert model and only conceptual caption dataset?

Additionally, it is easy to be stuck during the pretraining process due to the large I-O. I wonder if you could provide the specific config based on the "slurm setting", like using mc or not, the num_workers per GPU.

The training loss suddenly increases to 'nan' after training 2 epochs. Any methods to avoid this?

THCudaCheck FAIL

I am trying to train VL-BERT for RefCOCO+ (python refcoco/train_end2end.py --cfg cfgs/refcoco/base_detected_regions_4x16G.yaml). However, I am getting the following CUDA-related error.

THCudaCheck FAIL file=/project/ocean/tsriniva/VL-BERT/common/lib/roi_pooling/cuda/ROIAlign_cuda.cu line=297 error=98 : invalid device function
Traceback (most recent call last):
  File "refcoco/train_end2end.py", line 60, in <module>
    main()
  File "refcoco/train_end2end.py", line 54, in main
    rank, model = train_net(args, config)
  File "/project/ocean/tsriniva/VL-BERT/refcoco/../refcoco/function/train.py", line 323, in train_net
    gradient_accumulate_steps=config.TRAIN.GRAD_ACCUMULATE_STEPS)
  File "/project/ocean/tsriniva/VL-BERT/refcoco/../common/trainer.py", line 115, in train
    outputs, loss = net(*batch)
  File "/home/tsriniva/anaconda2/envs/vl-bert/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/project/ocean/tsriniva/VL-BERT/refcoco/../common/module.py", line 22, in forward
    return self.train_forward(*inputs, **kwargs)
  File "/project/ocean/tsriniva/VL-BERT/refcoco/../refcoco/modules/resnet_vlbert_for_refcoco.py", line 96, in train_forward
    segms=None)
  File "/home/tsriniva/anaconda2/envs/vl-bert/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/project/ocean/tsriniva/VL-BERT/refcoco/../common/fast_rcnn.py", line 149, in forward
    roi_align_res = self.roi_align(img_feats['body4'], rois).type(images.dtype)
  File "/home/tsriniva/anaconda2/envs/vl-bert/lib/python3.6/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/project/ocean/tsriniva/VL-BERT/refcoco/../common/lib/roi_pooling/roi_align.py", line 69, in forward
    input.float(), rois.float(), self.output_size, self.spatial_scale, self.sampling_ratio
  File "/project/ocean/tsriniva/VL-BERT/refcoco/../common/lib/roi_pooling/roi_align.py", line 20, in forward
    input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio
RuntimeError: cuda runtime error (98) : invalid device function at /project/ocean/tsriniva/VL-BERT/common/lib/roi_pooling/cuda/ROIAlign_cuda.cu:297
Segmentation fault (core dumped)

Is there any fix for this?

segmentation fault

Thank you for the code. I want to fine tune this model on refcoco dataset. I got a segmentation fault error when I run the non-distributed sh file. Please help.

[Partial Load] non pretrain keys: ['final_mlp.2.weight', 'final_mlp.2.bias']
PROGRESS: 0.00%
Segmentation fault

Working with CUDA 10.1

Hi,
My gcc and Ubuntu versions match exactly those in README.md, but I have CUDA 10.1. I set up the environment exactly as in README.md.
When trying to run the code, I find CUDA 10.1 causing trouble:

if cudatoolkit is not 9.0 -> ImportError: libcudart.so.9.0: cannot open shared object file: No such file or directory
if cudatoolkit is 9.0 -> ImportError:.../VL-BERT/refcoco/../common/lib/roi_pooling/C_ROIPooling.cpython-36m-x86_64-linux-gnu.so: undefined symbol: __cudaRegisterFatBinaryEnd

Is there any fix other than having to install CUDA 9.0?

About adding whole image as a bounding box & the embedding for [END] token

Thank you for your brilliant work.

According to Figure 1 in the paper, for Visual Feature Embedding of image regions, VL-BERT is using features of bounding box proposals. It seems you are concatenating the whole image with other bounding boxes in every dataset.py by assigning add_image_as_a_box=True (e.g. VQA). In my understanding, your code utilizes this first box (whole image) for Visual Feature Embedding of words other than image regions (with [IMG] token). However, it seems your code didn't remove this full image box after that so there's one additional (full image) box at the front of boxes?
I can't see why it make sense to just train a joint new (Token + Visual Feature) Embedding for the final word [END] (e.g. in class VisualLinguisticBert)? It neither used predefined [END] token like [SEP] and [CLS] nor used the full image box as Visual Feature embedding. Did I miss anything here?

Miss the [SEQ] token when truncating the sequence too long

I found a little mistake when loading the data from multimodal dataset, for example the CC. I'm not sure whether it will make a big difference.
Ref to you src paper, the pretrain input is in the format of [CLS] + Caption Tokens + [SEQ] + ROIs + [END] and I also noticed that when the input sequence is over long, the code will truncate the sequence to max_len, e.g. 64, at here. My question is, when the code do text truncation, like

text = text[:text_len_keep]
mlm_labels = mlm_labels[:text_len_keep]

The text has already added [CLS] & [SEQ] in the beginning and end, so this truncation op will definitely Move Out the [SEQ] Token. I don't know whether this small mistake matters. But to keep input format, it's better to truncate the input caption tokens first, then add [CLS] & [SEQ], then convert to ids..
Thank you for your reply in advance

where can I find the category outputs for bottom-up features?

Hello. Thanks for sharing your code
I downloaded trainval2014_resnet101_faster_rcnn_genome file from the google drive link. I could only find image_id, image_w, image_h, num_boxes, boxes, features inside it. But I can't find the predicted category for each box, in which you use in your masked ROI classification. May you tell me where are these located?

_pickle.UnpicklingError: invalid load key, '-'.

Thanks for your great code.
I try to train on the vcr task to see result.
when i did
python vcr/val.py \ --a-cfg ./cfgs/vcr/base_q2a_4x16G_fp32.yaml --r-cfg ./cfgs/vcr/base_qa2r_4x16G_fp32.yaml \ --a-ckpt ./output/base_q2a_4x16G_fp32.yaml --r-ckpt ./output/base_qa2r_4x16G_fp32.yaml \ --gpus 0 1 \ --result-path ./results/ --result-name eval_vcr,
the mistake happened.
As follows:
warnings.warn('miss keys: {}'.format(miss_keys)) Warnings: Unexpected keys: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.transform.LayerNorm.gamma', 'cls.predictions.transform.LayerNorm.beta', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias']. Traceback (most recent call last): File "vcr/val.py", line 214, in <module> main() File "/home/songzijie/.conda/envs/vlbert/lib/python3.6/site-packages/torch/autograd/grad_mode.py", line 43, in decorate_no_grad return func(*args, **kwargs) File "vcr/val.py", line 114, in main a_ckpt = torch.load(args.a_ckpt, map_location=lambda storage, loc: storage) File "/home/songzijie/.conda/envs/vlbert/lib/python3.6/site-packages/torch/serialization.py", line 387, in load return _load(f, map_location, pickle_module, **pickle_load_args) File "/home/songzijie/.conda/envs/vlbert/lib/python3.6/site-packages/torch/serialization.py", line 564, in _load magic_number = pickle_module.load(f, **pickle_load_args) _pickle.UnpicklingError: invalid load key, '-'.
I hope can get some help to solve the problem.
Thanks a lot.

VL-BERT with other dataset

Hi, Thank you for sharing nice work.
It seems really good at vision and Language multi-grounding.
I wanna check the performance with other tasks.
What should i follow, if i wanna pre-train the model with another dataset?
Thank you.

subprocess.CalledProcessError: Command xxx returned non-zero exit status 1.

When I try training on VCR dataset with the comand ./scripts/dist_run_single.sh 1 vcr/train_end2end.py ./cfgs/vcr/base_q2a_4x16G_fp32.yaml ./, I got an error like this:

Traceback (most recent call last):
  File "vcr/train_end2end.py", line 59, in <module>
    main()
  File "vcr/train_end2end.py", line 53, in main
    rank, model = train_net(args, config)
  File "/gruntdata/guimin.gm/vlbert/vcr/../vcr/function/train.py", line 87, in train_net
    group_name='mtorch')
  File "/home/guimin.gm/miniconda3/envs/pt/lib/python3.6/site-packages/torch/distributed/distributed_c10d.py", line 406, in init_process_group
    store, rank, world_size = next(rendezvous(url))
  File "/home/guimin.gm/miniconda3/envs/pt/lib/python3.6/site-packages/torch/distributed/rendezvous.py", line 95, in _tcp_rendezvous_handler
    store = TCPStore(result.hostname, result.port, world_size, start_daemon)
RuntimeError: Address already in use
Traceback (most recent call last):
  File "./scripts/launch.py", line 200, in <module>
    main()
  File "./scripts/launch.py", line 196, in main
    cmd=process.args)
subprocess.CalledProcessError: Command '['/home/guimin.gm/miniconda3/envs/pt/bin/python', '-u', 'vcr/train_end2end.py', '--cfg', './cfgs/vcr/base_q2a_4x16G_fp32.yaml', '--model-dir', './', '--dist']' returned non-zero exit status 1.

I haven`t found the solution, is there anybody can help me? Thanks a lot.

Hyperparameters mismatch

Hi @jackroos and thanks for the great repo!

I was looking at the cfgs file for VQA and noticed different hyperparameters than in the appendix of the paper.
For instance, 5 epochs instead of 20, 500 warmup steps instead of 2000, smaller learning rate, ...
Should we follow -- in this and other tasks -- the values in the repository or the ones in the paper?
Also, are inputs not truncated to a maximum length during fine-tuning?

Thanks!

Is the roialign part forgotten compiled ?

It seems that the setup.py only compiles roipooling but not roialign.

apex compiler error

I follow the document to install apex, then run scripts:
pip install -v --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./
and then got the error as follow:

/home/qzhang/anaconda3/envs/vl-bert/lib/python3.6/site-packages/pip/_internal/commands/install.py:244: UserWarning: Disabling all use of wheels due to the use of --build-option / --global-option / --install-option.
  cmdoptions.check_install_build_global(options)
Non-user install because site-packages writeable
Created temporary directory: /tmp/pip-ephem-wheel-cache-3gl_tl_j
Created temporary directory: /tmp/pip-req-tracker-ctu9auww
Initialized build tracking at /tmp/pip-req-tracker-ctu9auww
Created build tracker: /tmp/pip-req-tracker-ctu9auww
Entered build tracker: /tmp/pip-req-tracker-ctu9auww
Created temporary directory: /tmp/pip-install-y028bhtk
Looking in indexes: https://mirrors.aliyun.com/pypi/simple
Processing /home/qzhang/projects/image.clef.project/VL-BERT/apex
  Created temporary directory: /tmp/pip-req-build-p9qpie0o
  Added file:///home/qzhang/projects/image.clef.project/VL-BERT/apex to build tracker '/tmp/pip-req-tracker-ctu9auww'
    Running setup.py (path:/tmp/pip-req-build-p9qpie0o/setup.py) egg_info for package from file:///home/qzhang/projects/image.clef.project/VL-BERT/apex
    Running command python setup.py egg_info
    torch.__version__  =  1.1.0
    running egg_info
    creating /tmp/pip-req-build-p9qpie0o/pip-egg-info/apex.egg-info
    writing /tmp/pip-req-build-p9qpie0o/pip-egg-info/apex.egg-info/PKG-INFO
    writing dependency_links to /tmp/pip-req-build-p9qpie0o/pip-egg-info/apex.egg-info/dependency_links.txt
    writing top-level names to /tmp/pip-req-build-p9qpie0o/pip-egg-info/apex.egg-info/top_level.txt
    writing manifest file '/tmp/pip-req-build-p9qpie0o/pip-egg-info/apex.egg-info/SOURCES.txt'
    reading manifest file '/tmp/pip-req-build-p9qpie0o/pip-egg-info/apex.egg-info/SOURCES.txt'
    writing manifest file '/tmp/pip-req-build-p9qpie0o/pip-egg-info/apex.egg-info/SOURCES.txt'
  Source in /tmp/pip-req-build-p9qpie0o has version 0.1, which satisfies requirement apex==0.1 from file:///home/qzhang/projects/image.clef.project/VL-BERT/apex
  Removed apex==0.1 from file:///home/qzhang/projects/image.clef.project/VL-BERT/apex from build tracker '/tmp/pip-req-tracker-ctu9auww'
Skipping wheel build for apex, due to binaries being disabled for it.
Installing collected packages: apex
  Created temporary directory: /tmp/pip-record-htjucn0b
    Running command /home/qzhang/anaconda3/envs/vl-bert/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-p9qpie0o/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-p9qpie0o/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' --cpp_ext --cuda_ext install --record /tmp/pip-record-htjucn0b/install-record.txt --single-version-externally-managed --compile --install-headers /home/qzhang/anaconda3/envs/vl-bert/include/python3.6m/apex
    torch.__version__  =  1.1.0

    Compiling cuda extensions with
    nvcc: NVIDIA (R) Cuda compiler driver
    Copyright (c) 2005-2019 NVIDIA Corporation
    Built on Wed_Oct_23_19:24:38_PDT_2019
    Cuda compilation tools, release 10.2, V10.2.89
    from /usr/local/cuda/bin

    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-req-build-p9qpie0o/setup.py", line 64, in <module>
        check_cuda_torch_binary_vs_bare_metal(torch.utils.cpp_extension.CUDA_HOME)
      File "/tmp/pip-req-build-p9qpie0o/setup.py", line 54, in check_cuda_torch_binary_vs_bare_metal
        "https://github.com/NVIDIA/apex/pull/323#discussion_r287021798.  "
    RuntimeError: Cuda extensions are being compiled with a version of Cuda that does not match the version used to compile Pytorch binaries.  Pytorch binaries were compiled with Cuda 9.0.176.
    In some cases, a minor-version mismatch will not cause later errors:  https://github.com/NVIDIA/apex/pull/323#discussion_r287021798.  You can try commenting out this check (at your own risk).
    Running setup.py install for apex ... error
Cleaning up...
  Removing source in /tmp/pip-req-build-p9qpie0o
Removed build tracker: '/tmp/pip-req-tracker-ctu9auww'
ERROR: Command errored out with exit status 1: /home/qzhang/anaconda3/envs/vl-bert/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-p9qpie0o/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-p9qpie0o/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' --cpp_ext --cuda_ext install --record /tmp/pip-record-htjucn0b/install-record.txt --single-version-externally-managed --compile --install-headers /home/qzhang/anaconda3/envs/vl-bert/include/python3.6m/apex Check the logs for full command output.
Exception information:
Traceback (most recent call last):
  File "/home/qzhang/anaconda3/envs/vl-bert/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 186, in _main
    status = self.run(options, args)
  File "/home/qzhang/anaconda3/envs/vl-bert/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 404, in run
    use_user_site=options.use_user_site,
  File "/home/qzhang/anaconda3/envs/vl-bert/lib/python3.6/site-packages/pip/_internal/req/__init__.py", line 71, in install_given_reqs
    **kwargs
  File "/home/qzhang/anaconda3/envs/vl-bert/lib/python3.6/site-packages/pip/_internal/req/req_install.py", line 829, in install
    scheme=scheme,
  File "/home/qzhang/anaconda3/envs/vl-bert/lib/python3.6/site-packages/pip/_internal/operations/install/legacy.py", line 72, in install
    cwd=install_req.unpacked_source_directory,
  File "/home/qzhang/anaconda3/envs/vl-bert/lib/python3.6/site-packages/pip/_internal/utils/subprocess.py", line 275, in runner
    spinner=spinner,
  File "/home/qzhang/anaconda3/envs/vl-bert/lib/python3.6/site-packages/pip/_internal/utils/subprocess.py", line 242, in call_subprocess
    raise InstallationError(exc_msg)
pip._internal.exceptions.InstallationError: Command errored out with exit status 1: /home/qzhang/anaconda3/envs/vl-bert/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-p9qpie0o/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-p9qpie0o/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' --cpp_ext --cuda_ext install --record /tmp/pip-record-htjucn0b/install-record.txt --single-version-externally-managed --compile --install-headers /home/qzhang/anaconda3/envs/vl-bert/include/python3.6m/apex Check the logs for full command output.
1 location(s) to search for versions of pip:
* https://mirrors.aliyun.com/pypi/simple/pip/
Fetching project page and analyzing links: https://mirrors.aliyun.com/pypi/simple/pip/
Getting page https://mirrors.aliyun.com/pypi/simple/pip/
Found index url https://mirrors.aliyun.com/pypi/simple
Starting new HTTPS connection (1): mirrors.aliyun.com:443
https://mirrors.aliyun.com:443 "GET /pypi/simple/pip/ HTTP/1.1" 200 13652
  Found link https://mirrors.aliyun.com/pypi/packages/18/ad/c0fe6cdfe1643a19ef027c7168572dac6283b80a384ddf21b75b921877da/pip-0.2.1.tar.gz#sha256=83522005c1266cc2de97e65072ff7554ac0f30ad369c3b02ff3a764b962048da (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 0.2.1
  Found link https://mirrors.aliyun.com/pypi/packages/3d/9d/1e313763bdfb6a48977b65829c6ce2a43eaae29ea2f907c8bbef024a7219/pip-0.2.tar.gz#sha256=88bb8d029e1bf4acd0e04d300104b7440086f94cc1ce1c5c3c31e3293aee1f81 (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 0.2
  Found link https://mirrors.aliyun.com/pypi/packages/0a/bb/d087c9a1415f8726e683791c0b2943c53f2b76e69f527f2e2b2e9f9e7b5c/pip-0.3.1.tar.gz#sha256=34ce534f17065c78f980702928e988a6b6b2d8a9851aae5f1571a1feb9bb58d8 (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 0.3.1
  Found link https://mirrors.aliyun.com/pypi/packages/17/05/f66144ef69b436d07f8eeeb28b7f77137f80de4bf60349ec6f0f9509e801/pip-0.3.tar.gz#sha256=183c72455cb7f8860ac1376f8c4f14d7f545aeab8ee7c22cd4caf79f35a2ed47 (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 0.3
  Found link https://mirrors.aliyun.com/pypi/packages/cf/c3/153571aaac6cf999f4bb09c019b1ff379b7b599ea833813a41c784eec995/pip-0.4.tar.gz#sha256=28fc67558874f71fddda7168f73595f1650523dce3bc5bf189713ecdfc1e456e (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 0.4
  Found link https://mirrors.aliyun.com/pypi/packages/9a/aa/f536b6d14fe03343367da2ff44eee28f340ae650cd017ca088b6be13084a/pip-0.5.1.tar.gz#sha256=e27650538c41fe1007a41abd4cfd0f905b822622cbe1f8e7e09d1215af207694 (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 0.5.1
  Found link https://mirrors.aliyun.com/pypi/packages/8d/c7/f05c87812fa5d9562ecbc5f4f1fc1570444f53c81c834a7f662af406e3c1/pip-0.5.tar.gz#sha256=328d8412782f22568508a0d0c78a49c9920a82e44c8dfca49954fe525c152b2a (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 0.5
  Found link https://mirrors.aliyun.com/pypi/packages/91/cd/105f4d3c75d0ae18e12623acc96f42168aaba408dd6e43c4505aa21f8e37/pip-0.6.1.tar.gz#sha256=efe47e84ffeb0ea4804f9858b8a94bebd07f5452f907ebed36d03aed06a9f9ec (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 0.6.1
  Found link https://mirrors.aliyun.com/pypi/packages/1c/c7/c0e1a9413c37828faf290f29a85a4d6034c145cc04bf1622ba8beb662ad8/pip-0.6.2.tar.gz#sha256=1c1a504d7e70d2c24246f95bd16e3d5fcec740fd144df69a407bf65a2ee67586 (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 0.6.2
  Found link https://mirrors.aliyun.com/pypi/packages/3f/af/c4b9d49fb0f286996b28dbc0955c3ad359794697eb98e0e69863908070b0/pip-0.6.3.tar.gz#sha256=1a6df71eb29b98cba11bde6d6a0d8c6dd8b0518e74ceb71fb31ea4fbb42fd313 (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 0.6.3
  Found link https://mirrors.aliyun.com/pypi/packages/db/e6/fdf7be8a17b032c533d3f91e91e2c63dd81d3627cbe4113248a00c2d39d8/pip-0.6.tar.gz#sha256=4cf47db6815b2f435d1f44e1f35ff04823043f6161f7df9aec71a123b0c47f0d (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 0.6
  Found link https://mirrors.aliyun.com/pypi/packages/a5/63/11303863c2f5e9d9a15d89fcf7513a4b60987007d418862e0fb65c09fff7/pip-0.7.1.tar.gz#sha256=f54f05aa17edd0036de433c44892c8fedb1fd2871c97829838feb995818d24c3 (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 0.7.1
  Found link https://mirrors.aliyun.com/pypi/packages/cd/a9/1debaa96bbc1005c1c8ad3b79fec58c198d35121546ea2e858ce0894268a/pip-0.7.2.tar.gz#sha256=98df2eb779358412bbbae75980171ae85deebc846d87e244d086520b1212da09 (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 0.7.2
  Found link https://mirrors.aliyun.com/pypi/packages/ec/7a/6fe91ff0079ad0437830957c459d52f3923e516f5b453218f2a93d09a427/pip-0.7.tar.gz#sha256=ceaea0b9e494d893c8a191895301b79c1db33e41f14d3ad93e3d28a8b4e9bf27 (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 0.7
  Found link https://mirrors.aliyun.com/pypi/packages/5c/79/5e8381cc3078bae92166f2ba96de8355e8c181926505ba8882f7b099a500/pip-0.8.1.tar.gz#sha256=7176a87f35675f6468341212f3b959bb51d23ea66eb1c3692bf746c45c716fa2 (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 0.8.1
  Found link https://mirrors.aliyun.com/pypi/packages/17/3e/0a98ab032991518741e7e712a719633e6ae160f51b3d3e855194530fd308/pip-0.8.2.tar.gz#sha256=f80a3549c048bc3bbcb47844826e9c7c6fcd87e77b92bef0d9e66d1b397c4962 (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 0.8.2
  Found link https://mirrors.aliyun.com/pypi/packages/f7/9a/943fc6d879ed7220bac2e7e53096bfe78abec88d77f2f516400e0129679e/pip-0.8.3.tar.gz#sha256=1be2e18edd38aa75b5e4ef38a99ec33ba9247177cfcb4a6d2d2b3e73430e3001 (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 0.8.3
  Found link https://mirrors.aliyun.com/pypi/packages/74/54/f785c327fb3d163560a879b36edae5c78ee07806be282c9d4807f6be7dd1/pip-0.8.tar.gz#sha256=9017e4484a212dd4e1a43dd9f039dd7fc8338d4eea1c339d5ae1c80726de5b0f (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 0.8
  Found link https://mirrors.aliyun.com/pypi/packages/10/d9/f584e6107ef98ad7eaaaa5d0f756bfee12561fa6a4712ffdb7209e0e1fd4/pip-1.0.1.tar.gz#sha256=37d2f18213d3845d2038dd3686bc71fc12bb41ad66c945a8b0dfec2879f3497b (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 1.0.1
  Found link https://mirrors.aliyun.com/pypi/packages/16/90/5e6f80364d8a656f60681dfb7330298edef292d43e1499bcb3a4c71ff0b9/pip-1.0.2.tar.gz#sha256=a6ed9b36aac2f121c01a2c9e0307a9e4d9438d100a407db701ac65479a3335d2 (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 1.0.2
  Found link https://mirrors.aliyun.com/pypi/packages/24/33/6eb675fb6db7b71d69d6928b33dea61b8bf5cfe1e5649be70ec84ce2fc09/pip-1.0.tar.gz#sha256=34ba07e2d14ba86d5088ba896ac80bed845a9b276ab8acb279b8d99bc77fec8e (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 1.0
  Found link https://mirrors.aliyun.com/pypi/packages/25/57/0d42cf5307d79913a082c5c4397d46f3793bc35e1138a694136d6e31be99/pip-1.1.tar.gz#sha256=993804bb947d18508acee02141281c77d27677f8c14eaa64d6287a1c53ef01c8 (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 1.1
  Found link https://mirrors.aliyun.com/pypi/packages/c3/a2/a63244da32afd9ce9a8ca1bd86e71610039adea8b8314046ebe5047527a6/pip-1.2.1.tar.gz#sha256=12a9302acfca62cdc7bc5d83386cac3e0581db61ac39acdb3a4e766a16b88eb1 (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 1.2.1
  Found link https://mirrors.aliyun.com/pypi/packages/ba/c3/4e1f892f41aaa217fe0d1f827fa05928783349c69f3cc06fdd68e112678a/pip-1.2.tar.gz#sha256=2b168f1987403f1dc6996a1f22a6f6637b751b7ab6ff27e78380b8d6e70aa314 (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 1.2
  Found link https://mirrors.aliyun.com/pypi/packages/5b/ce/f5b98104f1c10d868936c25f7c597f492d4371aa9ad5fb61a94954ee7208/pip-1.3.1.tar.gz#sha256=145eaa5d1ea1b062663da1f3a97780d7edea4c63c68a37c463b1deedf7bb4957 (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 1.3.1
  Found link https://mirrors.aliyun.com/pypi/packages/00/45/69d4f2602b80550bfb26cfd2f62c2f05b3b5c7352705d3766cd1e5b27648/pip-1.3.tar.gz#sha256=d6a13c5be316cb21a0243047c7f163f47e88973ebccff8d32e63ca1bf4d9321c (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 1.3
  Found link https://mirrors.aliyun.com/pypi/packages/3f/f8/da390e0df72fb61d176b25a4b95262e3dcc14bda0ad25ac64d56db38b667/pip-1.4.1.tar.gz#sha256=4e7a06554711a624c35d0c646f63674b7f6bfc7f80221bf1eb1f631bd890d04e (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 1.4.1
  Found link https://mirrors.aliyun.com/pypi/packages/5f/d0/3b3958f6a58783bae44158b2c4c7827ae89abaecdd4bed12cff402620b9a/pip-1.4.tar.gz#sha256=1fd43cbf07d95ddcecbb795c97a1674b3ddb711bb4a67661284a5aa765aa1b97 (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 1.4
  Found link https://mirrors.aliyun.com/pypi/packages/44/5d/1dca53b5de6d287e7eb99bd174bb022eb6cb0d6ca6e19ca6b16655dde8c2/pip-1.5.1-py2.py3-none-any.whl#sha256=00960db3b0b8724dd37fe37cfb9c72ecb8f59fab9db7d17c5c1e89a1adab49ce (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 1.5.1
  Found link https://mirrors.aliyun.com/pypi/packages/21/3f/d86a600c9b2f41a75caacf768a24130f343def97652de2345da15ef7911f/pip-1.5.1.tar.gz#sha256=e60e936fbc101d56668c6134c1f2b5b40fcbec8b4fc4ca7fc34842b6b4c5c130 (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 1.5.1
  Found link https://mirrors.aliyun.com/pypi/packages/3d/1f/227d77d5e9ed2df5162de4ba3616799a351eccb1ecd668ae824dd26153a1/pip-1.5.2-py2.py3-none-any.whl#sha256=6903909ccdcdbc3297b74118590e71344d6d262827acd1f5c0e2fcfce9807499 (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 1.5.2
  Found link https://mirrors.aliyun.com/pypi/packages/ed/94/391a003107f6ec997c314199d03bff1c105af758ee490e3255353574487b/pip-1.5.2.tar.gz#sha256=2a8a3e08e652d3a40edbb39264bf01f8ff3c32520a79113357cca1f30533f738 (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 1.5.2
  Found link https://mirrors.aliyun.com/pypi/packages/df/e9/bdb53d44fad1465b43edaf6bc7dd3027ed5af81405cc97603fdff0721ebb/pip-1.5.3-py2.py3-none-any.whl#sha256=f0037aed3ce6cf96b9e9117d42e967a74bea9ebe19088a2fdea5de93d5762fee (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 1.5.3
  Found link https://mirrors.aliyun.com/pypi/packages/55/de/671a48ad313c808623041fc475f7c8f7610401d9f573f06b40eeb84e74e3/pip-1.5.3.tar.gz#sha256=dc53b4d28b88556a37cd73052b6d1d08cc644c6724e37c4d38a2e3c03c5440b2 (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 1.5.3
  Found link https://mirrors.aliyun.com/pypi/packages/a9/9a/9aa19fe00de4c025562e5fb3796ff8520165a7dd1a5662c6ec9816e1ae99/pip-1.5.4-py2.py3-none-any.whl#sha256=fb7282556a42e84464f2e963a859ac4012d8134ba6218b70c1d82d145fcfa82f (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 1.5.4
  Found link https://mirrors.aliyun.com/pypi/packages/78/d8/6e58a7130d457edadb753a0ea5708e411c100c7e94e72ad4802feeef735c/pip-1.5.4.tar.gz#sha256=70208a250bb4afdbbdd74c3ac35d4ab9ba1eb6852d02567a6a87f2f5104e30b9 (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 1.5.4
  Found link https://mirrors.aliyun.com/pypi/packages/ce/c2/10d996b9c51b126a9f0bb9e14a9edcdd5c88888323c0685bb9b392b6c47c/pip-1.5.5-py2.py3-none-any.whl#sha256=fe7a5808190067b2598d85def9b83db46e5d64a00848ad843e107c36e1db4ae6 (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 1.5.5
  Found link https://mirrors.aliyun.com/pypi/packages/88/01/a442fde40bd9aaf837612536f16ab751fac628807fd718690795b8ade77d/pip-1.5.5.tar.gz#sha256=4b7f5124364ae9b5ba833dcd8813a84c1c06fba1d7c8543323c7af4b33188eca (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 1.5.5
  Found link https://mirrors.aliyun.com/pypi/packages/3f/08/7347ca4021e7fe0f1ab8f93cbc7d2a7a7350012300ad0e0227d55625e2b8/pip-1.5.6-py2.py3-none-any.whl#sha256=fbc1351ffedf09ca7560428758845a88d648b9730b63ce9e5df53a7c89f039a4 (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 1.5.6
  Found link https://mirrors.aliyun.com/pypi/packages/45/db/4fb9a456b4ec4d3b701456ef562b9d72d76b6358e0c1463d17db18c5b772/pip-1.5.6.tar.gz#sha256=b1a4ae66baf21b7eb05a5e4f37c50c2706fa28ea1f8780ce8efe14dcd9f1726c (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 1.5.6
  Found link https://mirrors.aliyun.com/pypi/packages/4f/7d/e53bc80667378125a9e07d4929a61b0bd7128a1129dbe6f07bb3228652a3/pip-1.5.tar.gz#sha256=25f81d1a0e55d3b1709818dd57fdfb954b028f229f09bd69cb0bc80a8e03e048 (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 1.5
  Found link https://mirrors.aliyun.com/pypi/packages/62/a1/0d452b6901b0157a0134fd27ba89bf95a857fbda64ba52e1ca2cf61d8412/pip-10.0.0-py2.py3-none-any.whl#sha256=86a60a96d85e329962a9e6f6af612cbc11106293dbc83f119802b5bee9874cf3 (from https://mirrors.aliyun.com/pypi/simple/pip/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*), version: 10.0.0
  Found link https://mirrors.aliyun.com/pypi/packages/e0/69/983a8e47d3dfb51e1463c1e962b2ccd1d74ec4e236e232625e353d830ed2/pip-10.0.0.tar.gz#sha256=f05a3eeea64bce94e85cc6671d679473d66288a4d37c3fcf983584954096b34f (from https://mirrors.aliyun.com/pypi/simple/pip/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*), version: 10.0.0
  Found link https://mirrors.aliyun.com/pypi/packages/4b/5a/8544ae02a5bd28464e03af045e8aabde20a7b02db1911a9159328e1eb25a/pip-10.0.0b1-py2.py3-none-any.whl#sha256=dbd5d24cd461be23429625085a36cc8732cbcac4d2aaf673031f80f6ac07d844 (from https://mirrors.aliyun.com/pypi/simple/pip/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*), version: 10.0.0b1
  Found link https://mirrors.aliyun.com/pypi/packages/aa/6d/ffbb86abf18b750fb26f27eda7c7732df2aacaa669c420d2eb2ad6df3458/pip-10.0.0b1.tar.gz#sha256=8d6e63d8b99752e4b53f272b66f9cd7b59e2b288e9a863a61c48d167203a2656 (from https://mirrors.aliyun.com/pypi/simple/pip/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*), version: 10.0.0b1
  Found link https://mirrors.aliyun.com/pypi/packages/97/72/1d514201e7d7fc7fff5aac3de9c7b892cd72fb4bf23fd983630df96f7412/pip-10.0.0b2-py2.py3-none-any.whl#sha256=79f55588912f1b2b4f86f96f11e329bb01b25a484e2204f245128b927b1038a7 (from https://mirrors.aliyun.com/pypi/simple/pip/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*), version: 10.0.0b2
  Found link https://mirrors.aliyun.com/pypi/packages/32/67/572f642e6e42c580d3154964cfbab7d9322c23b0f417c6c01fdd206a2777/pip-10.0.0b2.tar.gz#sha256=ad6adec2150ce4aed8f6134d9b77d928fc848dbcb887fb1a455988cf99da5cae (from https://mirrors.aliyun.com/pypi/simple/pip/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*), version: 10.0.0b2
  Found link https://mirrors.aliyun.com/pypi/packages/0f/74/ecd13431bcc456ed390b44c8a6e917c1820365cbebcb6a8974d1cd045ab4/pip-10.0.1-py2.py3-none-any.whl#sha256=717cdffb2833be8409433a93746744b59505f42146e8d37de6c62b430e25d6d7 (from https://mirrors.aliyun.com/pypi/simple/pip/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*), version: 10.0.1
  Found link https://mirrors.aliyun.com/pypi/packages/ae/e8/2340d46ecadb1692a1e455f13f75e596d4eab3d11a57446f08259dee8f02/pip-10.0.1.tar.gz#sha256=f2bd08e0cd1b06e10218feaf6fef299f473ba706582eb3bd9d52203fdbd7ee68 (from https://mirrors.aliyun.com/pypi/simple/pip/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*), version: 10.0.1
  Found link https://mirrors.aliyun.com/pypi/packages/5f/25/e52d3f31441505a5f3af41213346e5b6c221c9e086a166f3703d2ddaf940/pip-18.0-py2.py3-none-any.whl#sha256=070e4bf493c7c2c9f6a08dd797dd3c066d64074c38e9e8a0fb4e6541f266d96c (from https://mirrors.aliyun.com/pypi/simple/pip/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*), version: 18.0
  Found link https://mirrors.aliyun.com/pypi/packages/69/81/52b68d0a4de760a2f1979b0931ba7889202f302072cc7a0d614211bc7579/pip-18.0.tar.gz#sha256=a0e11645ee37c90b40c46d607070c4fd583e2cd46231b1c06e389c5e814eed76 (from https://mirrors.aliyun.com/pypi/simple/pip/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*), version: 18.0
  Found link https://mirrors.aliyun.com/pypi/packages/c2/d7/90f34cb0d83a6c5631cf71dfe64cc1054598c843a92b400e55675cc2ac37/pip-18.1-py2.py3-none-any.whl#sha256=7909d0a0932e88ea53a7014dfd14522ffef91a464daaaf5c573343852ef98550 (from https://mirrors.aliyun.com/pypi/simple/pip/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*), version: 18.1
  Found link https://mirrors.aliyun.com/pypi/packages/45/ae/8a0ad77defb7cc903f09e551d88b443304a9bd6e6f124e75c0fbbf6de8f7/pip-18.1.tar.gz#sha256=c0a292bd977ef590379a3f05d7b7f65135487b67470f6281289a94e015650ea1 (from https://mirrors.aliyun.com/pypi/simple/pip/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*), version: 18.1
  Found link https://mirrors.aliyun.com/pypi/packages/60/64/73b729587b6b0d13e690a7c3acd2231ee561e8dd28a58ae1b0409a5a2b20/pip-19.0-py2.py3-none-any.whl#sha256=249ab0de4c1cef3dba4cf3f8cca722a07fc447b1692acd9f84e19c646db04c9a (from https://mirrors.aliyun.com/pypi/simple/pip/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*), version: 19.0
  Found link https://mirrors.aliyun.com/pypi/packages/46/dc/7fd5df840efb3e56c8b4f768793a237ec4ee59891959d6a215d63f727023/pip-19.0.1-py2.py3-none-any.whl#sha256=aae79c7afe895fb986ec751564f24d97df1331bb99cdfec6f70dada2f40c0044 (from https://mirrors.aliyun.com/pypi/simple/pip/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*), version: 19.0.1
  Found link https://mirrors.aliyun.com/pypi/packages/c8/89/ad7f27938e59db1f0f55ce214087460f65048626e2226531ba6cb6da15f0/pip-19.0.1.tar.gz#sha256=e81ddd35e361b630e94abeda4a1eddd36d47a90e71eb00f38f46b57f787cd1a5 (from https://mirrors.aliyun.com/pypi/simple/pip/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*), version: 19.0.1
  Found link https://mirrors.aliyun.com/pypi/packages/d7/41/34dd96bd33958e52cb4da2f1bf0818e396514fd4f4725a79199564cd0c20/pip-19.0.2-py2.py3-none-any.whl#sha256=6a59f1083a63851aeef60c7d68b119b46af11d9d803ddc1cf927b58edcd0b312 (from https://mirrors.aliyun.com/pypi/simple/pip/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*), version: 19.0.2
  Found link https://mirrors.aliyun.com/pypi/packages/4c/4d/88bc9413da11702cbbace3ccc51350ae099bb351febae8acc85fec34f9af/pip-19.0.2.tar.gz#sha256=f851133f8b58283fa50d8c78675eb88d4ff4cde29b6c41205cd938b06338e0e5 (from https://mirrors.aliyun.com/pypi/simple/pip/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*), version: 19.0.2
  Found link https://mirrors.aliyun.com/pypi/packages/d8/f3/413bab4ff08e1fc4828dfc59996d721917df8e8583ea85385d51125dceff/pip-19.0.3-py2.py3-none-any.whl#sha256=bd812612bbd8ba84159d9ddc0266b7fbce712fc9bc98c82dee5750546ec8ec64 (from https://mirrors.aliyun.com/pypi/simple/pip/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*), version: 19.0.3
  Found link https://mirrors.aliyun.com/pypi/packages/36/fa/51ca4d57392e2f69397cd6e5af23da2a8d37884a605f9e3f2d3bfdc48397/pip-19.0.3.tar.gz#sha256=6e6f197a1abfb45118dbb878b5c859a0edbdd33fd250100bc015b67fded4b9f2 (from https://mirrors.aliyun.com/pypi/simple/pip/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*), version: 19.0.3
  Found link https://mirrors.aliyun.com/pypi/packages/11/31/c483614095176ddfa06ac99c2af4171375053b270842c7865ca0b4438dc1/pip-19.0.tar.gz#sha256=c82bf8bc00c5732f0dd49ac1dea79b6242a1bd42a5012e308ed4f04369b17e54 (from https://mirrors.aliyun.com/pypi/simple/pip/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*), version: 19.0
  Found link https://mirrors.aliyun.com/pypi/packages/f9/fb/863012b13912709c13cf5cfdbfb304fa6c727659d6290438e1a88df9d848/pip-19.1-py2.py3-none-any.whl#sha256=8f59b6cf84584d7962d79fd1be7a8ec0eb198aa52ea864896551736b3614eee9 (from https://mirrors.aliyun.com/pypi/simple/pip/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*), version: 19.1
  Found link https://mirrors.aliyun.com/pypi/packages/5c/e0/be401c003291b56efc55aeba6a80ab790d3d4cece2778288d65323009420/pip-19.1.1-py2.py3-none-any.whl#sha256=993134f0475471b91452ca029d4390dc8f298ac63a712814f101cd1b6db46676 (from https://mirrors.aliyun.com/pypi/simple/pip/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*), version: 19.1.1
  Found link https://mirrors.aliyun.com/pypi/packages/93/ab/f86b61bef7ab14909bd7ec3cd2178feb0a1c86d451bc9bccd5a1aedcde5f/pip-19.1.1.tar.gz#sha256=44d3d7d3d30a1eb65c7e5ff1173cdf8f7467850605ac7cc3707b6064bddd0958 (from https://mirrors.aliyun.com/pypi/simple/pip/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*), version: 19.1.1
  Found link https://mirrors.aliyun.com/pypi/packages/51/5f/802a04274843f634469ef299fcd273de4438386deb7b8681dd059f0ee3b7/pip-19.1.tar.gz#sha256=d9137cb543d8a4d73140a3282f6d777b2e786bb6abb8add3ac5b6539c82cd624 (from https://mirrors.aliyun.com/pypi/simple/pip/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*), version: 19.1
  Found link https://mirrors.aliyun.com/pypi/packages/3a/6f/35de4f49ae5c7fdb2b64097ab195020fb48faa8ad3a85386ece6953c11b1/pip-19.2-py2.py3-none-any.whl#sha256=468c67b0b1120cd0329dc72972cf0651310783a922e7609f3102bd5fb4acbf17 (from https://mirrors.aliyun.com/pypi/simple/pip/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*), version: 19.2
  Found link https://mirrors.aliyun.com/pypi/packages/62/ca/94d32a6516ed197a491d17d46595ce58a83cbb2fca280414e57cd86b84dc/pip-19.2.1-py2.py3-none-any.whl#sha256=80d7452630a67c1e7763b5f0a515690f2c1e9ad06dda48e0ae85b7fdf2f59d97 (from https://mirrors.aliyun.com/pypi/simple/pip/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*), version: 19.2.1
  Found link https://mirrors.aliyun.com/pypi/packages/8b/8a/1b2aadd922db1afe6bc107b03de41d6d37a28a5923383e60695fba24ae81/pip-19.2.1.tar.gz#sha256=258d702483dd749400aec59c23d638a5b2249ae28a0f478b6cab12ad45681a80 (from https://mirrors.aliyun.com/pypi/simple/pip/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*), version: 19.2.1
  Found link https://mirrors.aliyun.com/pypi/packages/8d/07/f7d7ced2f97ca3098c16565efbe6b15fafcba53e8d9bdb431e09140514b0/pip-19.2.2-py2.py3-none-any.whl#sha256=4b956bd8b7b481fc5fa222637ff6d0823a327e5118178f1ec47618a480e61997 (from https://mirrors.aliyun.com/pypi/simple/pip/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*), version: 19.2.2
  Found link https://mirrors.aliyun.com/pypi/packages/aa/1a/62fb0b95b1572c76dbc3cc31124a8b6866cbe9139eb7659ac7349457cf7c/pip-19.2.2.tar.gz#sha256=e05103825871e210d50a44c7e448587b0ed99dd775d3ef586304c58f40224a53 (from https://mirrors.aliyun.com/pypi/simple/pip/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*), version: 19.2.2
  Found link https://mirrors.aliyun.com/pypi/packages/30/db/9e38760b32e3e7f40cce46dd5fb107b8c73840df38f0046d8e6514e675a1/pip-19.2.3-py2.py3-none-any.whl#sha256=340a0ba40fdeb16413914c0fcd8e0b4ebb0bf39a900ec80e11c05d836c05103f (from https://mirrors.aliyun.com/pypi/simple/pip/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*), version: 19.2.3
  Found link https://mirrors.aliyun.com/pypi/packages/00/9e/4c83a0950d8bdec0b4ca72afd2f9cea92d08eb7c1a768363f2ea458d08b4/pip-19.2.3.tar.gz#sha256=e7a31f147974362e6c82d84b91c7f2bdf57e4d3163d3d454e6c3e71944d67135 (from https://mirrors.aliyun.com/pypi/simple/pip/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*), version: 19.2.3
  Found link https://mirrors.aliyun.com/pypi/packages/41/13/b6e68eae78405af6e4e9a93319ae5bb371057786f1590b157341f7542d7d/pip-19.2.tar.gz#sha256=aa6fdd80d13caac75d92b5eced06778712859b1606ba92d62389c11be12b2dad (from https://mirrors.aliyun.com/pypi/simple/pip/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*), version: 19.2
  Found link https://mirrors.aliyun.com/pypi/packages/4a/08/6ca123073af4ebc4c5488a5bc8a010ac57aa39ce4d3c8a931ad504de4185/pip-19.3-py2.py3-none-any.whl#sha256=e100a7eccf085f0720b4478d3bb838e1c179b1e128ec01c0403f84e86e0e2dfb (from https://mirrors.aliyun.com/pypi/simple/pip/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*), version: 19.3
  Found link https://mirrors.aliyun.com/pypi/packages/00/b6/9cfa56b4081ad13874b0c6f96af8ce16cfbc1cb06bedf8e9164ce5551ec1/pip-19.3.1-py2.py3-none-any.whl#sha256=6917c65fc3769ecdc61405d3dfd97afdedd75808d200b2838d7d961cebc0c2c7 (from https://mirrors.aliyun.com/pypi/simple/pip/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*), version: 19.3.1
  Found link https://mirrors.aliyun.com/pypi/packages/ce/ea/9b445176a65ae4ba22dce1d93e4b5fe182f953df71a145f557cffaffc1bf/pip-19.3.1.tar.gz#sha256=21207d76c1031e517668898a6b46a9fb1501c7a4710ef5dfd6a40ad9e6757ea7 (from https://mirrors.aliyun.com/pypi/simple/pip/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*), version: 19.3.1
  Found link https://mirrors.aliyun.com/pypi/packages/af/7a/5dd1e6efc894613c432ce86f1011fcc3bbd8ac07dfeae6393b7b97f1de8b/pip-19.3.tar.gz#sha256=324d234b8f6124846b4e390df255cacbe09ce22791c3b714aa1ea6e44a4f2861 (from https://mirrors.aliyun.com/pypi/simple/pip/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*), version: 19.3
  Found link https://mirrors.aliyun.com/pypi/packages/60/65/16487a7c4e0f95bb3fc89c2e377be331fd496b7a9b08fd3077de7f3ae2cf/pip-20.0-py2.py3-none-any.whl#sha256=eea07b449d969dbc8c062c157852cf8ed2ad1b8b5ac965a6b819e62929e41703 (from https://mirrors.aliyun.com/pypi/simple/pip/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*), version: 20.0
  Found link https://mirrors.aliyun.com/pypi/packages/57/36/67f809c135c17ec9b8276466cc57f35b98c240f55c780689ea29fa32f512/pip-20.0.1-py2.py3-none-any.whl#sha256=b7110a319790ae17e8105ecd6fe07dbcc098a280c6d27b6dd7a20174927c24d7 (from https://mirrors.aliyun.com/pypi/simple/pip/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*), version: 20.0.1
  Found link https://mirrors.aliyun.com/pypi/packages/28/af/2c76c8aa46ccdf7578b83d97a11a2d1858794d4be4a1610ade0d30182e8b/pip-20.0.1.tar.gz#sha256=3cebbac2a1502e09265f94e5717408339de846b3c0f0ed086d7b817df9cab822 (from https://mirrors.aliyun.com/pypi/simple/pip/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*), version: 20.0.1
  Found link https://mirrors.aliyun.com/pypi/packages/54/0c/d01aa759fdc501a58f431eb594a17495f15b88da142ce14b5845662c13f3/pip-20.0.2-py2.py3-none-any.whl#sha256=4ae14a42d8adba3205ebeb38aa68cfc0b6c346e1ae2e699a0b3bad4da19cef5c (from https://mirrors.aliyun.com/pypi/simple/pip/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*), version: 20.0.2
  Found link https://mirrors.aliyun.com/pypi/packages/8e/76/66066b7bc71817238924c7e4b448abdb17eb0c92d645769c223f9ace478f/pip-20.0.2.tar.gz#sha256=7db0c8ea4c7ea51c8049640e8e6e7fde949de672bfa4949920675563a5a6967f (from https://mirrors.aliyun.com/pypi/simple/pip/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*), version: 20.0.2
  Found link https://mirrors.aliyun.com/pypi/packages/8c/5c/c18d58ab5c1a702bf670e0bd6a77cd4645e4aeca021c6118ef850895cc96/pip-20.0.tar.gz#sha256=5128e9a9401f1d16c1d15b2ed766a79d7813db1538428d0b0ce74838249e3a41 (from https://mirrors.aliyun.com/pypi/simple/pip/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*), version: 20.0
  Found link https://mirrors.aliyun.com/pypi/packages/54/2e/df11ea7e23e7e761d484ed3740285a34e38548cf2bad2bed3dd5768ec8b9/pip-20.1-py2.py3-none-any.whl#sha256=4fdc7fd2db7636777d28d2e1432e2876e30c2b790d461f135716577f73104369 (from https://mirrors.aliyun.com/pypi/simple/pip/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*), version: 20.1
  Found link https://mirrors.aliyun.com/pypi/packages/43/84/23ed6a1796480a6f1a2d38f2802901d078266bda38388954d01d3f2e821d/pip-20.1.1-py2.py3-none-any.whl#sha256=b27c4dedae8c41aa59108f2fa38bf78e0890e590545bc8ece7cdceb4ba60f6e4 (from https://mirrors.aliyun.com/pypi/simple/pip/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*), version: 20.1.1
  Found link https://mirrors.aliyun.com/pypi/packages/08/25/f204a6138dade2f6757b4ae99bc3994aac28a5602c97ddb2a35e0e22fbc4/pip-20.1.1.tar.gz#sha256=27f8dc29387dd83249e06e681ce087e6061826582198a425085e0bf4c1cf3a55 (from https://mirrors.aliyun.com/pypi/simple/pip/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*), version: 20.1.1
  Found link https://mirrors.aliyun.com/pypi/packages/d1/05/059c78cd5d740d2299266ffa15514dad6692d4694df571bf168e2cdd98fb/pip-20.1.tar.gz#sha256=572c0f25eca7c87217b21f6945b7192744103b18f4e4b16b8a83b227a811e192 (from https://mirrors.aliyun.com/pypi/simple/pip/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*), version: 20.1
  Found link https://mirrors.aliyun.com/pypi/packages/ec/05/82d3fababbf462d876883ebc36f030f4fa057a563a80f5a26ee63679d9ea/pip-20.1b1-py2.py3-none-any.whl#sha256=4cf0348b683937da883ccaae8c8bcfc9b4c7ba4c48b38cc2d89cd7b8d0b220d9 (from https://mirrors.aliyun.com/pypi/simple/pip/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*), version: 20.1b1
  Found link https://mirrors.aliyun.com/pypi/packages/cd/81/c1184456fe506bd50992571c9f8581907976ce71502e36741f033e2da1f1/pip-20.1b1.tar.gz#sha256=699880a47f6d306f4f9a87ca151ef33d41d2223b81ff343b786d38c297923a19 (from https://mirrors.aliyun.com/pypi/simple/pip/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*), version: 20.1b1
  Found link https://mirrors.aliyun.com/pypi/packages/fe/3b/0fc5e63eb277d5a50a95ce5c896f742ef243be27382303a4a44dd0197e29/pip-20.2b1-py2.py3-none-any.whl#sha256=b4e230e2b8ece18c5a19b818f3c20a8d4eeac8172962779fd9898d7c4ceb1636 (from https://mirrors.aliyun.com/pypi/simple/pip/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*), version: 20.2b1
  Found link https://mirrors.aliyun.com/pypi/packages/77/3e/6a1fd8e08a06e3e0f54182c7c937bba3f4e9cf1b26f54946d3915021ea2e/pip-20.2b1.tar.gz#sha256=dbf65ecb1c30d35d72f5fda052fcd2f1ea9aca8eaf03d930846d990f51d3f6f6 (from https://mirrors.aliyun.com/pypi/simple/pip/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*), version: 20.2b1
  Found link https://mirrors.aliyun.com/pypi/packages/dc/7c/21191b5944b917b66e4e4e06d74f668d814b6e8a3ff7acd874479b6f6b3d/pip-6.0-py2.py3-none-any.whl#sha256=5ec6732505bd8be49fe1f8ad557b88253ffb085736396df4d6bea753fc2a8f2c (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 6.0
  Found link https://mirrors.aliyun.com/pypi/packages/e9/7a/cdbc1a12ed52410d557e48d4646f4543e9e991ff32d2374dc6db849aa617/pip-6.0.1-py2.py3-none-any.whl#sha256=322aea7d1f7b9ee68ad87ac4704cad5df97f77e70668c0bd18f964c5daa78173 (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 6.0.1
  Found link https://mirrors.aliyun.com/pypi/packages/4d/c3/8675b90cd89b9b222062f4f6c7e9d48b0387f5b35cbf747a74403a883e56/pip-6.0.1.tar.gz#sha256=fa2f7c68da4a405d673aa38542f9df009d60026db4f532429ac9cbfbda1f959d (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 6.0.1
  Found link https://mirrors.aliyun.com/pypi/packages/71/3c/b5a521e5e99cfff091e282231591f21193fd80de079ec5fb8ed9c6614044/pip-6.0.2-py2.py3-none-any.whl#sha256=7d17b0f267f7c9cd17cd2924bbbe2b4a3d407322c0e09084ca3f1295c1fed50d (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 6.0.2
  Found link https://mirrors.aliyun.com/pypi/packages/4c/5a/f9e8e3de0153282c7cb54a9b991af225536ac914bac858ca664cf883bb3e/pip-6.0.2.tar.gz#sha256=6fa90667706a679e3dc75b27a51fddafa64401c45e96f8ae6c20978183290077 (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 6.0.2
  Found link https://mirrors.aliyun.com/pypi/packages/73/cb/3eebf42003791df29219a3dfa1874572aa16114b44c9b1b0ac66bf96e8c0/pip-6.0.3-py2.py3-none-any.whl#sha256=b72655b6ac6aef1c86dd07f51e8ace8d7aabd6a1c4ff88db87155276fa32a073 (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 6.0.3
  Found link https://mirrors.aliyun.com/pypi/packages/ce/63/8d99ae60d11ae1a65f5d4fc39a529a598bd3b8e067132210cb0c4d9e9f74/pip-6.0.3.tar.gz#sha256=b091a35f5fa0faffac0b27b97e1e1e93ffe63b463c2ea8dbde0c1fb987933614 (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 6.0.3
  Found link https://mirrors.aliyun.com/pypi/packages/c5/0e/c974206726542bc495fc7443dd97834a6d14c2f0cba183fcfcd01075225a/pip-6.0.4-py2.py3-none-any.whl#sha256=8dfd95de29a7a3bb1e7d368cc83d566938eb210b04d553ebfe5e3a422f4aec65 (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 6.0.4
  Found link https://mirrors.aliyun.com/pypi/packages/02/a1/c90f19910ee153d7a0efca7216758121118d7e93084276541383fe9ca82e/pip-6.0.4.tar.gz#sha256=1dbbff9c369e510c7468ab68ba52c003f68f83c99c2f8259acd51099e8799f1e (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 6.0.4
  Found link https://mirrors.aliyun.com/pypi/packages/e9/1b/c6a375a337fb576784cdea3700f6c3eaf1420f0a01458e6e034cc178a84a/pip-6.0.5-py2.py3-none-any.whl#sha256=b2c20e3a2a43b2bbb1d19ad98be27eccc7b0f0ece016da602ccaa757a862b0e2 (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 6.0.5
  Found link https://mirrors.aliyun.com/pypi/packages/19/f2/58628768f618c8c9fea878e0fb97730c0b8a838d3ab3f325768bf12dac94/pip-6.0.5.tar.gz#sha256=3bf42d28be9085ab2e9aecfd69a6da2d31563fe833304bf71a620a30c38ab8a2 (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 6.0.5
  Found link https://mirrors.aliyun.com/pypi/packages/64/fc/4a49ccb18f55a0ceeb76e8d554bd4563217117492997825d194ed0017cc1/pip-6.0.6-py2.py3-none-any.whl#sha256=fb04f8afe1ba57626783f0c8e2f3d46bbaebaa446fcf124f434e968a2fee595e (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 6.0.6
  Found link https://mirrors.aliyun.com/pypi/packages/f6/ce/d9e4e178b66c766c117f62ddf4fece019ef9d50127a8926d2f60300d615e/pip-6.0.6.tar.gz#sha256=3a14091299dcdb9bab9e9004ae67ac401f2b1b14a7c98de074ca74fdddf4bfa0 (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 6.0.6
  Found link https://mirrors.aliyun.com/pypi/packages/7a/8e/2bbd4fcf3ee06ee90ded5f39ec12f53165dfdb9ef25a981717ad38a16670/pip-6.0.7-py2.py3-none-any.whl#sha256=93a326304c7db749896bcef822bbbac1ab29dad5651c6d732e245975239890e6 (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 6.0.7
  Found link https://mirrors.aliyun.com/pypi/packages/52/85/b160ebdaa84378df6bb0176d4eed9f57edca662446174eead7a9e2e566d6/pip-6.0.7.tar.gz#sha256=35a5a43ac6b7af83ed47ea5731a365f43d350a3a7267e039e5f06b61d42ab3c2 (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 6.0.7
  Found link https://mirrors.aliyun.com/pypi/packages/63/65/55b71647adec1ad595bf0e5d76d028506dfc002df30c256f022ff7a660a5/pip-6.0.8-py2.py3-none-any.whl#sha256=3c22b0a8ff92727bd737a82f72700790591f177541df08c07bc1f90d6b72ac19 (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 6.0.8
  Found link https://mirrors.aliyun.com/pypi/packages/ef/8a/e3a980bc0a7f791d72c1302f65763ed300f2e14c907ac033e01b44c79e5e/pip-6.0.8.tar.gz#sha256=0d58487a1b7f5be2e5e965c11afbea1dc44ecec8069de03491a4d0d6c85f4551 (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 6.0.8
  Found link https://mirrors.aliyun.com/pypi/packages/38/fd/065c66a88398f240e344fdf496b9707f92d75f88eedc3d10ff847b28a657/pip-6.0.tar.gz#sha256=6103897f1bb68d3f933edd60f3e3830c4ea6b8abf7a4b500db148921b11f6c9b (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 6.0
  Found link https://mirrors.aliyun.com/pypi/packages/24/fb/8a56a46243514681e569bbafd8146fa383476c4b7c725c8598c452366f31/pip-6.1.0-py2.py3-none-any.whl#sha256=435a018f6d29e34d4f901bf4e6860d8a5fa1816b68d62008c18ca062a306db31 (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 6.1.0
  Found link https://mirrors.aliyun.com/pypi/packages/6c/84/432eb60bbcb414b9cdfcb135d5f4925e253c74e7d6916ada79990d6cc1a0/pip-6.1.0.tar.gz#sha256=89f120e2ab3d25ab70c36eb28ad4f280fc9ba71736e74d3055f609c1f9173768 (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 6.1.0
  Found link https://mirrors.aliyun.com/pypi/packages/67/f0/ba0fb41dbdbfc4aa3e0c16b40269aca6b9e3d59cacdb646218aa2e9b1d2c/pip-6.1.1-py2.py3-none-any.whl#sha256=a67e54aa0f26b6d62ccec5cc6735eff205dd0fed075f56ac3d3111e91e4467fc (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 6.1.1
  Found link https://mirrors.aliyun.com/pypi/packages/bf/85/871c126b50b8ee0b9819e8a63b614aedd264577e73478caedcd447e8f28c/pip-6.1.1.tar.gz#sha256=89f3b626d225e08e7f20d85044afa40f612eb3284484169813dc2d0631f2a556 (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 6.1.1
  Found link https://mirrors.aliyun.com/pypi/packages/5a/9b/56d3c18d0784d5f2bbd446ea2dc7ffa7476c35e3dc223741d20cfee3b185/pip-7.0.0-py2.py3-none-any.whl#sha256=309c48399c7d68501a10ef206abd6e5c541fedbf84b95435d9063bd454b39df7 (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 7.0.0
  Found link https://mirrors.aliyun.com/pypi/packages/c6/16/6475b142927ca5d03e3b7968efa5b0edd103e4684ecfde181a25f6fa2505/pip-7.0.0.tar.gz#sha256=7b46bfc1b95494731de306a688e2a7bc056d7fa7ad27e026908fb2ae67fed23d (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 7.0.0
  Found link https://mirrors.aliyun.com/pypi/packages/5a/10/bb7a32c335bceba636aa673a4c977effa1e73a79f88856459486d8d670cf/pip-7.0.1-py2.py3-none-any.whl#sha256=d26b8573ba1ac1ec99a9bdbdffee2ff2b06c7790815211d0eb4dc1462a089705 (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 7.0.1
  Found link https://mirrors.aliyun.com/pypi/packages/4a/83/9ae4362a80739657e0c8bb628ea3fa0214a9aba7c8590dacc301ea293f73/pip-7.0.1.tar.gz#sha256=cfec177552fdd0b2d12b72651c8e874f955b4c62c1c2c9f2588cbdc1c0d0d416 (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 7.0.1
  Found link https://mirrors.aliyun.com/pypi/packages/64/7f/7107800ae0919a80afbf1ecba21b90890431c3ee79d700adac3c79cb6497/pip-7.0.2-py2.py3-none-any.whl#sha256=83c869c5ab7113866e2d69641ec470d47f0faae68ca4550a289a4d3db515ad65 (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 7.0.2
  Found link https://mirrors.aliyun.com/pypi/packages/75/b1/66532c273bca0133e42c3b4540a1609289f16e3046f1830f18c60794d661/pip-7.0.2.tar.gz#sha256=ba28fa60b573a9444e7b78ccb3b0f261d1f66f46d20403f9dce37b18a6aed405 (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 7.0.2
  Found link https://mirrors.aliyun.com/pypi/packages/96/76/33a598ae42dd0554207d83c7acc60e3b166dbde723cbf282f1f73b7a127c/pip-7.0.3-py2.py3-none-any.whl#sha256=7b1cb03e827d58d2d05e68ea96a9e27487ed4b0afcd951ac6e40847ce94f0738 (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 7.0.3
  Found link https://mirrors.aliyun.com/pypi/packages/35/59/5b23115758ba0f2fc465c459611865173ef006202ba83f662d1f58ed2fb8/pip-7.0.3.tar.gz#sha256=b4c598825a6f6dc2cac65968feb28e6be6c1f7f1408493c60a07eaa731a0affd (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 7.0.3
  Found link https://mirrors.aliyun.com/pypi/packages/f7/c0/9f8dac88326609b4b12b304e8382f64f7d5af7735a00d2fac36cf135fc30/pip-7.1.0-py2.py3-none-any.whl#sha256=80c29f899d3a00a448d65f8158544d22935baec7159af8da1a4fa1490ced481d (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 7.1.0
  Found link https://mirrors.aliyun.com/pypi/packages/7e/71/3c6ece07a9a885650aa6607b0ebfdf6fc9a3ef8691c44b5e724e4eee7bf2/pip-7.1.0.tar.gz#sha256=d5275ba3221182a5dd1b6bcfbfc5ec277fb399dd23226d6fa018048f7e0f10f2 (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 7.1.0
  Found link https://mirrors.aliyun.com/pypi/packages/1c/56/094d563c508917081bccff365e4f621ba33073c1c13aca9267a43cfcaf13/pip-7.1.1-py2.py3-none-any.whl#sha256=ce13000878d34c1178af76cb8cf269e232c00508c78ed46c165dd5b0881615f4 (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 7.1.1
  Found link https://mirrors.aliyun.com/pypi/packages/3b/bb/b3f2a95494fd3f01d3b3ae530e7c0e910dc25e88e30787b0a5e10cbc0640/pip-7.1.1.tar.gz#sha256=b22fe3c93a13fc7c04f145a42fd2ad50a9e3e1b8a7eed2e2b1c66e540a0951da (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 7.1.1
  Found link https://mirrors.aliyun.com/pypi/packages/b2/d0/cd115fe345dd6f07ec1c780020a7dfe74966fceeb171e0f20d1d4905b0b7/pip-7.1.2-py2.py3-none-any.whl#sha256=b9d3983b5cce04f842175e30169d2f869ef12c3546fd274083a65eada4e9708c (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 7.1.2
  Found link https://mirrors.aliyun.com/pypi/packages/d0/92/1e8406c15d9372084a5bf79d96da3a0acc4e7fcf0b80020a4820897d2a5c/pip-7.1.2.tar.gz#sha256=ca047986f0528cfa975a14fb9f7f106271d4e0c3fe1ddced6c1db2e7ae57a477 (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 7.1.2
  Found link https://mirrors.aliyun.com/pypi/packages/00/ae/bddef02881ee09c6a01a0d6541aa6c75a226a4e68b041be93142befa0cd6/pip-8.0.0-py2.py3-none-any.whl#sha256=262ed1823eb7fbe3f18a9bedb4800e59c4ab9a6682aff8c37b5ee83ea840910b (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 8.0.0
  Found link https://mirrors.aliyun.com/pypi/packages/e3/2d/03c014d11e66628abf2fda5ca00f779cbe7b5292c5cd13d42a95b94aa9b8/pip-8.0.0.tar.gz#sha256=90112b296152f270cb8dddcd19b7b87488d9e002e8cf622e14c4da9c2f6319b1 (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 8.0.0
  Found link https://mirrors.aliyun.com/pypi/packages/45/9c/6f9a24917c860873e2ce7bd95b8f79897524353df51d5d920cd6b6c1ec33/pip-8.0.1-py2.py3-none-any.whl#sha256=dedaac846bc74e38a3253671f51a056331ffca1da70e3f48d8128f2aa0635bba (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 8.0.1
  Found link https://mirrors.aliyun.com/pypi/packages/ea/66/a3d6187bd307159fedf8575c0d9ee2294d13b1cdd11673ca812e6a2dda8f/pip-8.0.1.tar.gz#sha256=477c50b3e538a7ac0fa611fb8b877b04b33fb70d325b12a81b9dbf3eb1158a4d (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 8.0.1
  Found link https://mirrors.aliyun.com/pypi/packages/e7/a0/bd35f5f978a5e925953ce02fa0f078a232f0f10fcbe543d8cfc043f74fda/pip-8.0.2-py2.py3-none-any.whl#sha256=249a6f3194be8c2e8cb4d4be3f6fd16a9f1e3336218caffa8e7419e3816f9988 (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 8.0.2
  Found link https://mirrors.aliyun.com/pypi/packages/ce/15/ee1f9a84365423e9ef03d0f9ed0eba2fb00ac1fffdd33e7b52aea914d0f8/pip-8.0.2.tar.gz#sha256=46f4bd0d8dfd51125a554568d646fe4200a3c2c6c36b9f2d06d2212148439521 (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 8.0.2
  Found link https://mirrors.aliyun.com/pypi/packages/ae/d4/2b127310f5364610b74c28e2e6a40bc19e2d3c9a9a4e012d3e333e767c99/pip-8.0.3-py2.py3-none-any.whl#sha256=b0335bc837f9edb5aad03bd43d0973b084a1cbe616f8188dc23ba13234dbd552 (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 8.0.3
  Found link https://mirrors.aliyun.com/pypi/packages/22/f3/14bc87a4f6b5ec70b682765978a6f3105bf05b6781fa97e04d30138bd264/pip-8.0.3.tar.gz#sha256=30f98b66f3fe1069c529a491597d34a1c224a68640c82caf2ade5f88aa1405e8 (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 8.0.3
  Found link https://mirrors.aliyun.com/pypi/packages/1e/c7/78440b3fb882ed001e6e12d8770bd45e73d6eced4e57f7c072b829ce8a3d/pip-8.1.0-py2.py3-none-any.whl#sha256=a542b99e08002ead83200198e19a3983270357e1cb4fe704247990b5b35471dc (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 8.1.0
  Found link https://mirrors.aliyun.com/pypi/packages/3c/72/6981d5adf880adecb066a1a1a4c312a17f8d787a3b85446967964ac66d55/pip-8.1.0.tar.gz#sha256=d8faa75dd7d0737b16d50cd0a56dc91a631c79ecfd8d38b80f6ee929ec82043e (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 8.1.0
  Found link https://mirrors.aliyun.com/pypi/packages/31/6a/0f19a7edef6c8e5065f4346137cc2a08e22e141942d66af2e1e72d851462/pip-8.1.1-py2.py3-none-any.whl#sha256=44b9c342782ab905c042c207d995aa069edc02621ddbdc2b9f25954a0fdac25c (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 8.1.1
  Found link https://mirrors.aliyun.com/pypi/packages/41/27/9a8d24e1b55bd8c85e4d022da2922cb206f183e2d18fee4e320c9547e751/pip-8.1.1.tar.gz#sha256=3e78d3066aaeb633d185a57afdccf700aa2e660436b4af618bcb6ff0fa511798 (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 8.1.1
  Found link https://mirrors.aliyun.com/pypi/packages/9c/32/004ce0852e0a127f07f358b715015763273799bd798956fa930814b60f39/pip-8.1.2-py2.py3-none-any.whl#sha256=6464dd9809fb34fc8df2bf49553bb11dac4c13d2ffa7a4f8038ad86a4ccb92a1 (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 8.1.2
  Found link https://mirrors.aliyun.com/pypi/packages/e7/a8/7556133689add8d1a54c0b14aeff0acb03c64707ce100ecd53934da1aa13/pip-8.1.2.tar.gz#sha256=4d24b03ffa67638a3fa931c09fd9e0273ffa904e95ebebe7d4b1a54c93d7b732 (from https://mirrors.aliyun.com/pypi/simple/pip/), version: 8.1.2
  Found link https://mirrors.aliyun.com/pypi/packages/3f/ef/935d9296acc4f48d1791ee56a73781271dce9712b059b475d3f5fa78487b/pip-9.0.0-py2.py3-none-any.whl#sha256=c856ac18ca01e7127456f831926dc67cc7d3ab663f4c13b1ec156e36db4de574 (from https://mirrors.aliyun.com/pypi/simple/pip/) (requires-python:>=2.6,!=3.0.*,!=3.1.*,!=3.2.*), version: 9.0.0
  Found link https://mirrors.aliyun.com/pypi/packages/5e/53/eaef47e5e2f75677c9de0737acc84b659b78a71c4086f424f55346a341b5/pip-9.0.0.tar.gz#sha256=f62fb70e7e000e46fce12aaeca752e5281a5446977fe5a75ab4189a43b3f8793 (from https://mirrors.aliyun.com/pypi/simple/pip/) (requires-python:>=2.6,!=3.0.*,!=3.1.*,!=3.2.*), version: 9.0.0
  Found link https://mirrors.aliyun.com/pypi/packages/b6/ac/7015eb97dc749283ffdec1c3a88ddb8ae03b8fad0f0e611408f196358da3/pip-9.0.1-py2.py3-none-any.whl#sha256=690b762c0a8460c303c089d5d0be034fb15a5ea2b75bdf565f40421f542fefb0 (from https://mirrors.aliyun.com/pypi/simple/pip/) (requires-python:>=2.6,!=3.0.*,!=3.1.*,!=3.2.*), version: 9.0.1
  Found link https://mirrors.aliyun.com/pypi/packages/11/b6/abcb525026a4be042b486df43905d6893fb04f05aac21c32c638e939e447/pip-9.0.1.tar.gz#sha256=09f243e1a7b461f654c26a725fa373211bb7ff17a9300058b205c61658ca940d (from https://mirrors.aliyun.com/pypi/simple/pip/) (requires-python:>=2.6,!=3.0.*,!=3.1.*,!=3.2.*), version: 9.0.1
  Found link https://mirrors.aliyun.com/pypi/packages/e7/f9/e801dcea22886cd513f6bd2e8f7e581bd6f67bb8e8f1cd8e7b92d8539280/pip-9.0.2-py2.py3-none-any.whl#sha256=b135491ddb061f39719b8472d8abb59c613816a2b86069c332db74d1cd208ab2 (from https://mirrors.aliyun.com/pypi/simple/pip/) (requires-python:>=2.6,!=3.0.*,!=3.1.*,!=3.2.*), version: 9.0.2
  Found link https://mirrors.aliyun.com/pypi/packages/e5/8f/3fc66461992dc9e9fcf5e005687d5f676729172dda640df2fd8b597a6da7/pip-9.0.2.tar.gz#sha256=88110a224e9d30e5d76592a0b2130ef10e7e67a6426e8617bb918fffbfe91fe5 (from https://mirrors.aliyun.com/pypi/simple/pip/) (requires-python:>=2.6,!=3.0.*,!=3.1.*,!=3.2.*), version: 9.0.2
  Found link https://mirrors.aliyun.com/pypi/packages/ac/95/a05b56bb975efa78d3557efa36acaf9cf5d2fd0ee0062060493687432e03/pip-9.0.3-py2.py3-none-any.whl#sha256=c3ede34530e0e0b2381e7363aded78e0c33291654937e7373032fda04e8803e5 (from https://mirrors.aliyun.com/pypi/simple/pip/) (requires-python:>=2.6,!=3.0.*,!=3.1.*,!=3.2.*), version: 9.0.3
  Found link https://mirrors.aliyun.com/pypi/packages/c4/44/e6b8056b6c8f2bfd1445cc9990f478930d8e3459e9dbf5b8e2d2922d64d3/pip-9.0.3.tar.gz#sha256=7bf48f9a693be1d58f49f7af7e0ae9fe29fd671cde8a55e6edca3581c4ef5796 (from https://mirrors.aliyun.com/pypi/simple/pip/) (requires-python:>=2.6,!=3.0.*,!=3.1.*,!=3.2.*), version: 9.0.3
Given no hashes to check 141 links for project 'pip': discarding no candidates

undefined symbol: _ZN6caffe26detail37_typeMetaDataInstance_preallocated_32E

when running the command "./scripts/dist_run_single.sh 4 vcr/train_end2end.py ./cfgs/vcr/base_q2a_4x16G_fp32.yaml ./"

I got this error

ImportError: /data/workspace/VL-BERT/vcr/../common/lib/roi_pooling/C_ROIPooling.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN6caffe26detail37_typeMetaDataInstance_preallocated_32E
Traceback (most recent call last):
File "./scripts/launch.py", line 200, in
main()
File "./scripts/launch.py", line 196, in main
cmd=process.args)
subprocess.CalledProcessError: Command '['/root/anaconda3/envs/vl-bert/bin/python', '-u', 'vcr/train_end2end.py', '--cfg', './cfgs/vcr/base_q2a_4x16G_fp32.yaml', '--model-dir', './', '--dist']' returned non-zero exit status 1.
(vl-bert) root@6cbe3808c2b8:/data/workspace/VL-BERT# Traceback (most recent call last):
File "vcr/train_end2end.py", line 8, in
from vcr.function.train import train_net
File "/data/workspace/VL-BERT/vcr/../vcr/function/train.py", line 26, in
from vcr.modules import *
File "/data/workspace/VL-BERT/vcr/../vcr/modules/init.py", line 1, in
from .resnet_vlbert_for_vcr import ResNetVLBERT
File "/data/workspace/VL-BERT/vcr/../vcr/modules/resnet_vlbert_for_vcr.py", line 7, in
from common.fast_rcnn import FastRCNN
File "/data/workspace/VL-BERT/vcr/../common/fast_rcnn.py", line 10, in
from common.lib.roi_pooling.roi_pool import ROIPool
File "/data/workspace/VL-BERT/vcr/../common/lib/roi_pooling/init.py", line 1, in
from .roi_align import ROIAlign
File "/data/workspace/VL-BERT/vcr/../common/lib/roi_pooling/roi_align.py", line 8, in
from . import C_ROIPooling
ImportError: /data/workspace/VL-BERT/vcr/../common/lib/roi_pooling/C_ROIPooling.cpython-36m-x86_64-linux-gnu.so: undefined symbol: _ZN6caffe26detail37_typeMetaDataInstance_preallocated_32E
Traceback (most recent call last):
File "vcr/train_end2end.py", line 8, in
from vcr.function.train import train_net
File "/data/workspace/VL-BERT/vcr/../vcr/function/train.py", line 26, in
from vcr.modules import *
File "/data/workspace/VL-BERT/vcr/../vcr/modules/init.py", line 1, in
from .resnet_vlbert_for_vcr import ResNetVLBERT
File "/data/workspace/VL-BERT/vcr/../vcr/modules/resnet_vlbert_for_vcr.py", line 7, in
from common.fast_rcnn import FastRCNN
File "/data/workspace/VL-BERT/vcr/../common/fast_rcnn.py", line 10, in
from common.lib.roi_pooling.roi_pool import ROIPool
File "/data/workspace/VL-BERT/vcr/../common/lib/roi_pooling/init.py", line 1, in
from .roi_align import ROIAlign
File "/data/workspace/VL-BERT/vcr/../common/lib/roi_pooling/roi_align.py", line 8, in
from . import C_ROIPooling

I think it may be caused by ROI_pooling.xx.so

But I've compiled C_ROIPooling.cpython-36m-x86_64-linux-gnu.so with "./scripts/init.sh"

cuda version

安装说明也没装成功，真的太郁闷了

Which config file should I use when doing pre-training and fine-tuning on each task to reproduce the paper results?

Hi. I have noticed that there are several config files in cfg/pretrain, cfg/vqa and cfg/refcoco (like there are 3 base-model configs base_e2e_16x16G_fp16.yaml, base_prec_4x16G_fp32.yaml, base_prec_withouttextonly_4x16G_fp32.yaml existing in cfg/pretrain) Can you provide more details about the differences of these configs? If I want to reproduce the paper results, which configs among them should I use? Thank you!

some question about the codes in ./common/faster_rcnn.py

Hi, there. I read some of your codes and have some questions about class FastRCNN in file './common/faster_rcnn.py'. As we can see, it's forward method has args like (self, images, boxes, box_mask, im_info, classes=None, segms=None, mvrc_ops=None, mask_visual_embed=None), but isn't it Faster-RCNN's work to get the RoIs, so why is there parameters like 'boxes' and so on?

pretrained model for conceptual-captions

In .\VL-BERT-master\data\conceptual-captions\ReadMe.txt there is a request in step 8(3): "Download pretrained model (https://www.dropbox.com/s/wqada4qiv1dz9dk/resnet101_faster_rcnn_final.caffemodel?dl=1)", which is not available now.
Could you please provide another solution?
Thx!

Thanks!

conceptual caption数据集330万张图片大约有多少G？

如题

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.