uber-research / deeppruner Goto Github PK

I would like to ask the author, if the size of the image used for network training is 960x540, then when I want to use this network test, the resolution of the photos taken by my camera is very large, such as 4608x3456. When I take this picture to test, the parallax map effect is very bad. When I resize the original image to 960x540, the parallax map effect is very good OK, but I don't know how to restore the parallax value of the original image. If the high-resolution image（4608x3456） is cut（960x540） and then put into the prediction, the prediction effect of each small resolution image cut out is not good.Or how to use your network to test higher resolution images without enough similar data sets to train? Hope that the author can guide, thank you very much!

Nice work！I have been reading your works recently, and there is a question that makes me very confused.
How to understand the propagation，aggregating disparity of adjacent pixels using convolution？
THX

Hi do you have any benchmarks for the Deep Pruner on Nvidia Jetson's?

Hi! Thanks so much for your code! When I saw the code, I found the weight_init in SubModule you defined was not used. Do I understand correctly? Is there any problem if the init method you defined is not used?

Hi
I checked runtime using code like this:

checked_runtime = 0
for i in range(100):
runtime = runtime_fe()
if i != 0:
checked_runtime += runtime
print('feature_extraction module runtime: %.5f' %(checked_runtime/99))

def runtime_fe():
net = feature_extraction()
net.eval()
test_in = Variable(torch.randn(1,3,544,960))
if torch.cuda.is_available():
net = nn.DataParallel(net)
net.cuda()
test_in.cuda()
with torch.no_grad():
start_time = time.time()
result = net(test_in)
torch.cuda.synchronize()
end_time = time.time()
return end_time-start_time

and I get 40 ms but correct value is 54 ms in your paper.
can you tell me how to get inference runtime?

@ShivamDuggal4
args.cuda = True,submission_kitti.py runs correctly,when I set --no-cuda,args.cuda=False,such error appeared:

Traceback (most recent call last):
File "submission_kitti.py", line 83, in
model.load_state_dict(state_dict['state_dict'], strict=True)
File "/home/xxm/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 845, in load_state_dict
self.class.name, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for DeepPruner:

Great work!
However, I encounter a very confusing problem. As reported in the paper <DeepPruner: Learning Efficient Stereo Matching via Differentiable PatchMatch>, the inference time of DeepPruner-Best model on SceneFlow test set is 182ms using Titan Xp and EPE=0.86. But when I run the provided model, I find the inference time is about 43ms and EPE=1.037. Big gap!
So, I doubt that the released code of DeepPruner-Best model is somehow simplified so that the inference speed is faster and the EPE performance is worse.
Is it the case? And how can I reproduce the inference time and EPE performance in the paper? Thanks!

my test code of time costs is as following:

def test(imgL, imgR):
    model.eval()
    with torch.no_grad():
        imgL = Variable(torch.FloatTensor(imgL))
        imgR = Variable(torch.FloatTensor(imgR))
        imgL, imgR = imgL.cuda(), imgR.cuda()

        for i in range(10):
            start_time = time.time()
            refined_disparity = model(imgL, imgR)
            end_time = time.time()
            print("time costs:{}".format(end_time - start_time))
    return refined_disparity

and the outputs is :

my config is as :

config = {
    "max_disp": 192,
    "cost_aggregator_scale": 4, # for DeepPruner-fast change this to 8.
    "mode": "evaluation", # for evaluation/ submission, change this to evaluation.
    "feature_extractor_ca_level_outplanes": 32,
    "feature_extractor_refinement_level_outplanes": 32, # for DeepPruner-fast change this to 64.
    "feature_extractor_refinement_level_1_outplanes": 32,
    "patch_match_args": {
        "sample_count": 12,
        "iteration_count": 2,
        "propagation_filter_size": 3
    },
    "post_CRP_sample_count": 7,
    "post_CRP_sampler_type": "uniform", #change to patch_match for Sceneflow model. 
    "hourglass_inplanes": 16
}

and the results is even accurate, why the time differs so greatly? And the first time runs faster 10 times then the later?

Environments:
GTX1080, python 3.5, Ubuntu16.04

Hi,

I am getting the following error. Can you let me know how to solve this? This happens only for the DeepPruner-fast*.tar models.

File "/root/anaconda3/envs/mccnn_env/lib/python2.7/site-packages/torch/nn/modules/module.py", line 769, in load_state_dict
self.class.name, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for DataParallel:
Missing key(s) in state_dict: "module.feature_extraction.branch1.1.0.weight", "module.feature_extraction.branch1.1.1.running_var", "module.feature_extraction.branch1.1.1.bias", "module.feature_extraction.branch1.1.1.weight", "module.feature_extraction.branch1.1.1.running_mean".
Unexpected key(s) in state_dict: "module.refinement_net1.conv1.0.0.weight", "module.refinement_net1.conv1.0.1.weight", "module.refinement_net1.conv1.0.1.bias", "module.refinement_net1.conv1.0.1.running_mean", "module.refinement_net1.conv1.0.1.running_var", "module.refinement_net1.conv1.0.1.num_batches_tracked", "module.refinement_net1.conv1.1.0.weight", "module.refinement_net1.conv1.1.1.weight", "module.refinement_net1.conv1.1.1.bias", "module.refinement_net1.conv1.1.1.running_mean", "module.refinement_net1.conv1.1.1.running_var", "module.refinement_net1.conv1.1.1.num_batches_tracked", "module.refinement_net1.conv1.2.0.weight", "module.refinement_net1.conv1.2.1.weight", "module.refinement_net1.conv1.2.1.bias", "module.refinement_net1.conv1.2.1.running_mean", "module.refinement_net1.conv1.2.1.running_var", "module.refinement_net1.conv1.2.1.num_batches_tracked", "module.refinement_net1.conv1.3.0.weight", "module.refinement_net1.conv1.3.1.weight", "module.refinement_net1.conv1.3.1.bias", "module.refinement_net1.conv1.3.1.running_mean", "module.refinement_net1.conv1.3.1.running_var", "module.refinement_net1.conv1.3.1.num_batches_tracked", "module.refinement_net1.conv1.4.0.weight", "module.refinement_net1.conv1.4.1.weight", "module.refinement_net1.conv1.4.1.bias", "module.refinement_net1.conv1.4.1.running_mean", "module.refinement_net1.conv1.4.1.running_var", "module.refinement_net1.conv1.4.1.num_batches_tracked", "module.refinement_net1.conv1.5.0.weight", "module.refinement_net1.conv1.5.1.weight", "module.refinement_net1.conv1.5.1.bias", "module.refinement_net1.conv1.5.1.running_mean", "module.refinement_net1.conv1.5.1.running_var", "module.refinement_net1.conv1.5.1.num_batches_tracked", "module.refinement_net1.classif1.weight".
size mismatch for module.refinement_net.conv1.0.0.weight: copying a param with shape torch.Size([32, 74, 3, 3]) from checkpoint, the shape in current model is torch.Size([32, 42, 3, 3]).
size mismatch for module.feature_extraction.lastconv.0.0.weight: copying a param with shape torch.Size([128, 352, 3, 3]) from checkpoint, the shape in current model is torch.Size([128, 320, 3, 3]).

When I run python submission_kitti.py, I get an error:
File "DeepPruner/deeppruner/submission_kitti.py", line 123, in main
disparity = disparity[top_pad:, :-left_pad].data.cpu().numpy()
TypeError: tuple indices must be integers or slices, not tuple

Hi, @ShivamDuggal4 ,
In your paper and README.md, you mentioned that your fast model consumed about 800MB Cuda memory. However, when I tested your model on my machine, I observed a memory consumption of 1300MB. Was there anything I missed?
My test environment: PyTorch 1.1; torchvision 0.4.0; python 3.6; RTX 2080Ti; CUDA 10

I set args.cuda=False .But RuntimeError: Device index must not be negative
This error occurred in deeppruner.py
device = left_input.get_device()
if stage is "pre":
min_disparity = torch.zeros((left_input.size()[0], 1, left_input.size()[2], left_input.size()[3]),
device=device)
max_disparity = torch.zeros((left_input.size()[0], 1, left_input.size()[2], left_input.size()[3]),
device=device) + self.max_disp

Hello, I am trying to find some quickly code about patchmatch, then I have read your code about Differentiable patchmatch. the key about first step is to extractor features by conv3d. So I wonder if I wanna know is this differentiable patchmatch is just fit to Stereo Matching or it can be treated as a torch implemention version of patchmatch?

HI @ShivamDuggal4 ,

I am implementing deep pruner from this repository for stereo disparity estimation. However, in data loading step, I am having errors related to reading pfm file. Even though the directory sturcture is as per the scene_collector.py file, I am not able to get past reading the data in following line in train_sceneflow.py.

Line 208: for batch_idx, (imgL_crop, imgR_crop, disp_crop_L) in enumerate(TrainImgLoader):

Any help would be greatly appreciated.

Regards,
Nakul

hello，the model weights list is corrupt，please check and update，ths！

Model Weights
DeepPruner was first trained on Sceneflow dataset and then finetuned on KITTI (Combined 394 images of KITTI-2012 and KITTI-2015) dataset.

DeepPruner-fast (KITTI)
DeepPruner-best (KITTI)
DeepPruner-fast (Sceneflow)
DeepPruner-best (Sceneflow)

Hello and thanks for the source code!

I have been using DeepPruner with the KITTI dataset to reconstruct point clouds using the disparity images from DeepPruner together with the KITTI camera parameters. The figure below illustrates the consecutive accumulation of reconstructed stereo point clouds with DeepPruner over the KITTI ground truth trajectory.

This point cloud has been cropped to remove the ground plane and data above a certain limit. The objects in these point clouds belong to the following real life objects:

As a reference, these are the accumulated point clouds for LiDAR data.

My question is regarding the accuracy of the point clouds, since there seems to be around 1m total variance in between the estimated depth of an object, from the first until the last time it is seen, and around a 1m error in spacial accuracy, given the mean position for stereo depths with respect to the LiDAR depths. Is this bias and variance reasonable for the DeepPruner performance, or might I have introduced some other source of error in the process? Thanks in advance!

Hi,

It seems that in the requirements it should be written Pytorch>=0.4.1 and not 0.4.0 because some of the functions you are using doesn't exist in 0.4.0.
Thank you

Hello,

I am using this code, it is a nice job!

I find out that the F.grid_sample function takes lots of GPU memories because it creates a copy of tensor in every iteration.

If I replace F.grid_sample in line 135 135 by torch.gather like the 1 dimension case in line 111 111,

Does it still create many copies of tensors?

Or do you have any better options to solve this question? Because in my understanding, these copies are not neccessary, they are only for updating the offsets.

Thanks.

Hi authors and communities,

Do anyone convert successfully DeepPruner / DeepPruner Fast Pytorch to ONNX model ? Could you guys help me to convert DeepPruner Pytoch to ONNX model ?

I tried to convert to ONNX, however, the uniform operator is not supported with all version of ONNX opset.
How I did : in submission_kitti.py , at function test()

def test(imgL, imgR):
    model.eval()
    with torch.no_grad():
        imgL = Variable(torch.FloatTensor(imgL))
        imgR = Variable(torch.FloatTensor(imgR))

        if args.cuda:
            imgL, imgR = imgL.cuda(), imgR.cuda()


        onnx_file = f"deepPruner_fast.onnx"
        torch.onnx.export(model,
                        args=(imgL,imgR),
                        f=onnx_file,
                        opset_version=13)
        import onnx
        from onnxsim import simplify
        model_load = onnx.load(onnx_file)
        model_simp, check = simplify(model_load)
        onnx.save(model_simp, onnx_file)
        import sys
        sys.exit(0)

        refined_disparity = model(imgL, imgR)
        return refined_disparity

Many thanks in advance.

I want to give two pictures, one is the left and the other is right, to predict the depth, so how should I run your code? Should I run finetune_kitti.py? I don't understand your Readme.txt, is finetune_kitti.py to train and test a model? I have downloaded the pre-trained model and I just want to use it not to train it, should I run another .py docunment? Really hope you can help me.

Hello, I read your code, there is a bit of confusion about normalized_disparity_samples in patch_match.py, why would you use two variables: disparity_samples and normalized_disparity_samples to do the PatchMatch?
It feels just to use disparity_samples for propagation and evaluation, and it can achieve the same goal and saves calculations.

disparity.max()
tensor(2277291.5000, device='cuda:0')

disparity.min()
tensor(0., device='cuda:0')

I find it difficult to construct an environment where I can run DeepPruner. PyTorch 0.4.0 and Torchvision 0.2.0 are so old that I fail to install them with conda or pip. Do you have any scripts for setting up a proper environment for DeepPruner? Preferably as some kind of container (Docker or Singularity) but even a simple script that installs the necessary libraries in some versions that are supported by DeepPruner would be a huge help.

I tried running DeepPruner using the latest PyTorch/Torchvision (1.3.1 and 0.4.2 respectively) and it doesn't work, so clearly there is some cutoff version when it stopped working. I get warnings about stuff being changed since 0.4.0 so it seems like we really need 0.4.0 to run DeepPruner (see below).

/opt/conda/lib/python2.7/site-packages/torch/nn/functional.py:2404: UserWarning: nn.functional.upsample is deprecated. Use nn.functional.interpolate instead. warnings.warn("nn.functional.upsample is deprecated. Use nn.functional.interpolate instead.") /opt/conda/lib/python2.7/site-packages/torch/nn/functional.py:2494: UserWarning: Default upsampling behavior when mode=bilinear is changed to align_corners=False since 0.4.0. Please specify align_corners=True if the old behavior is desired. See the documentation of nn.Upsample for details.

I am trying to train your model on colab with only "KITTI best" pretrained weights and with only KITTI 2015. after solving some of the errors. I get stuck on this error:
I am passing !python finetune_kitti.py --loadmodel /content/DeepPruner/deeppruner/DeepPruner-best-kitti.tar --train_datapath_2015 /content/DeepPruner/kitti_2015/training --val_datapath_2015 /content/DeepPruner/kitti_2015/testing

finetune_kitti.py: loading model...
finetune_kitti.py: Number of model parameters: 7390142
finetune_kitti.py: learning rate = 0.00010
Traceback (most recent call last):
File "finetune_kitti.py", line 218, in
main()
File "finetune_kitti.py", line 198, in main
for batch_idx, (imgL_crop, imgR_crop, disp_crop_L) in enumerate(TrainImgLoader):
File "/usr/local/lib/python2.7/dist-packages/torch/utils/data/dataloader.py", line 336, in next
return self._process_next_batch(batch)
File "/usr/local/lib/python2.7/dist-packages/torch/utils/data/dataloader.py", line 357, in _process_next_batch
raise batch.exc_type(batch.exc_msg)
IndexError: Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/torch/utils/data/dataloader.py", line 106, in _worker_loop
samples = collate_fn([dataset[i] for i in batch_indices])
File "/content/DeepPruner/deeppruner/dataloader/kitti_loader.py", line 53, in getitem
left_disp = self.left_disp[index]
IndexError: list index out of range

Hello, I want to reproduce your model, but I found that the quality is not as good as the model you provided. In your paper, you used batchsize = 16, but you used batchsize = 1 in your code. so which is the batchsize used by your model?
I use the model you provided to predict my test set. The overall performance is very good and complete. the disp is as follow:

However, if I use the model I trained to predict, I will lose some details (such as fingers). the disp is as follow:

What do you think may caused this difference?

Hi,

I need all the parameters for the layers in deeppruner. I tried extracting this information from the text and figure 1 but some information is missing and hence I am not able to figure out the parameters.
Can you please share this information with me?

regards,
Diksha

in torch 2.0.0

disparity = torch.cat(disparity, dim=0)

need to be added to the submission_kitti.py after

disparity = test(imgL, imgR)

Hi @ShivamDuggal4

I am trying to load a pretrained model to do evaluation with the code. The commandline I am using is as follows:

python submission_kitti.py --loadmodel ./dataloader/sceneflow_loader.py --save_dir ./ --datapath ./test_directory

However, I am getting the following error:

Traceback (most recent call last): File "submission_kitti.py", line 84, in <module> state_dict = torch.load(args.loadmodel) File "/home/benjamin/anaconda3/lib/python3.6/site-packages/torch/serialization.py", line 386, in load return _load(f, map_location, pickle_module, **pickle_load_args) File "/home/benjamin/anaconda3/lib/python3.6/site-packages/torch/serialization.py", line 563, in _load magic_number = pickle_module.load(f, **pickle_load_args) _pickle.UnpicklingError: invalid load key, '#'.

Is sceneflow_loader.py the right file to try to load? I also tried sceneflow_collector.py and a couple others in the dataloader folder, but they all gave the same error.

Do I need to download the pretrained model from somewhere?

Hello!
I tried downloading your pre-trained models from the links provided but it seems that the tarballs are broken and do not unpack.

tar -xvf DeepPruner-fast-kitti.tar 
tar: This does not look like a tar archive
tar: Skipping to next header
tar: Exiting with failure status due to previous errors

This issue is the same for all 4 models.
I have re-downloaded and verified on different systems and the problem persists.
I would like to try your models and would be grateful for help

I can't decompress the DeepPruner-best-kitti.tar document etc.

Nice job!
About reviewing your code about patch_match, I have some confusions about the definition and meaning of variables : filter_size=3, temperature=7 defined in function 'Evaluate' in patch_match.py.

Can you explain more about this two variables?
THX

Hi, I am running into an error on the 2nd to final line of the submission_kitti.py file, namely, the following line:

"skimage.io.imsave(os.path.join(args.save_dir, image_L_path.split(('/')[-1]), (disparity_comb * 256).astype('uint16')))"

The error I am getting is:

Traceback (most recent call last):
File "submission_kitti.py", line 173, in
main()
File "submission_kitti.py", line 160, in main
skimage.io.imsave(os.path.join(args.save_dir, (disparity * 256).astype('uint16')))
File "/home/benjamin/anaconda3/lib/python3.6/posixpath.py", line 92, in join
genericpath._check_arg_types('join', a, *p)
File "/home/benjamin/anaconda3/lib/python3.6/genericpath.py", line 149, in _check_arg_types
(funcname, s.class.name)) from None
TypeError: join() argument must be str or bytes, not 'ndarray'

Do you have any idea why this might be?
When I print out args.save_dir, I get ./ and when I print out image_L_path.split(('/')[-1]), (disparity_comb * 256).astype('uint16')), I get ['.', 'unity_chair', 'rgb_chair_left', '0002.jpg'] []

My code is below as follows:

`from future import print_function
import argparse
import os
import random
import torch
import torch.nn as nn
import torch.nn.parallel
import torch.backends.cudnn as cudnn
import torch.optim as optim
import torch.utils.data
from torch.autograd import Variable
import torch.nn.functional as F
import skimage.io
import numpy as np
import logging
from dataloader import kitti_submission_collector as ls
from dataloader import preprocess
from PIL import Image
from models.deeppruner import DeepPruner
from models.config import config as config_args
from setup_logging import setup_logging
import cv2

parser = argparse.ArgumentParser(description='DeepPruner')
parser.add_argument('--datapath', default='/',
help='datapath')
parser.add_argument('--loadmodel', default=None,
help='load model')
parser.add_argument('--save_dir', default='./',
help='save directory')
parser.add_argument('--logging_filename', default='./submission_kitti.log',
help='filename for logs')
parser.add_argument('--no-cuda', action='store_true', default=False,
help='enables CUDA training')
parser.add_argument('--seed', type=int, default=1, metavar='S',
help='random seed (default: 1)')

args = parser.parse_args()
torch.backends.cudnn.benchmark = True
args.cuda = not args.no_cuda and torch.cuda.is_available()

args.cost_aggregator_scale = config_args.cost_aggregator_scale
args.downsample_scale = args.cost_aggregator_scale * 8.0

setup_logging(args.logging_filename)

if args.cuda:
torch.manual_seed(args.seed)
np.random.seed(args.seed)
random.seed(args.seed)
torch.cuda.manual_seed(args.seed)
torch.backends.cudnn.deterministic = True

#test_left_img, test_right_img = ls.datacollector(args.datapath)

model = DeepPruner()
model = nn.DataParallel(model)

if args.cuda:
model.cuda()

logging.info('Number of model parameters: {}'.format(sum([p.data.nelement() for p in model.parameters()])))

if args.loadmodel is not None:
logging.info("loading model...")
state_dict = torch.load(args.loadmodel)
model.load_state_dict(state_dict['state_dict'], strict=True)

def test(imgL, imgR):
model.eval()
with torch.no_grad():
imgL = Variable(torch.FloatTensor(imgL))
imgR = Variable(torch.FloatTensor(imgR))

    if args.cuda:
        imgL, imgR = imgL.cuda(), imgR.cuda()

    refined_disparity = model(imgL, imgR)
    return refined_disparity

def main():

name_save_idx = 0
#for left_image_path, right_image_path in zip(test_left_img, test_right_img):
for i in range(2,3):
    name_save_idx += 1
    left_img_path = './unity_chair/rgb_chair_left/'
    right_img_path = './unity_chair/rgb_chair_right/'
    image_L_path = './unity_chair/rgb_chair_left/' + '{:04d}'.format(i) + '.jpg' #CHECK TO CHANGE B4 RUNNING
    print('i:', i)
    image_R_path = './unity_chair/rgb_chair_right/' + '{:04d}'.format(i) + '.jpg' #CHECK TO CHANGE B4 RUNNING
    #imgL = cv2.imread(image_L_path,0)
    #imgR = cv2.imread(image_R_path,0)
    #imgL = cv2.imread(image_L_path)
    #imgR = cv2.imread(image_R_path)

    imgL = np.asarray(Image.open(image_L_path))
    imgR = np.asarray(Image.open(image_R_path))
    print('imgL 0',imgL)
    print('imgR 0',imgR)
    print('POINT 0')
    #print('IMGL SHAPE:',imgL.shape)
    #iprint('IMGR SHAPE:',imgR.shape)
    processed = preprocess.get_transform()
    print('POINT 1')
    imgL = processed(imgL).numpy()
    imgR = processed(imgR).numpy()
    print('IMG 1',imgL)
    print('IMGR 1',imgR)
    imgL = np.reshape(imgL, [1, 3, imgL.shape[1], imgL.shape[2]])
    imgR = np.reshape(imgR, [1, 3, imgR.shape[1], imgR.shape[2]])
    print('IMGL 2',imgL)
    print('IMGR 2',imgR)

    w = imgL.shape[3]
    h = imgL.shape[2]
    dw = int(args.downsample_scale - (w%args.downsample_scale + (w%args.downsample_scale==0)*args.downsample_scale))
    dh = int(args.downsample_scale - (h%args.downsample_scale + (h%args.downsample_scale==0)*args.downsample_scale))

    top_pad = dh
    left_pad = dw
    imgL = np.lib.pad(imgL, ((0, 0), (0, 0), (top_pad, 0), (0, left_pad)), mode='constant', constant_values=0)
    imgR = np.lib.pad(imgR, ((0, 0), (0, 0), (top_pad, 0), (0, left_pad)), mode='constant', constant_values=0)

    disparity = test(imgL, imgR)
    print('POINT 0')
    print('DISPARITY:',disparity)
    print('DISPARITY TYPE:',type(disparity))
    #print('DISPARITY SHAPE:',disparity.shape)
    #disparity = list(disparity)
    #disparity = np.array(disparity)
    #print(disparity)
    print('TOP PAD:',top_pad)
    print('LEFT_PAD:',left_pad)
    disparity = disparity[0, top_pad:, :-left_pad].data.cpu().numpy()  
    print('POINT 1')
    print('NEED TO FIX ISSUE BELOW')

    skimage.io.imsave(os.path.join(args.save_dir, image_L_path.split(('/')[-1]), (disparity_comb * 256).astype('uint16')))
    print('POINT 3')
    #skimage.io.imsave(os.path.join(args.save_dir, (disparity * 256).astype('uint16')))
    filename = 'unity_chair_estimated_disparity_pair_dp_' + '{:04d}'.format(name_save_idx) + '.png'
    #disparity = np.uint16(disparity)
    print('POINT 4')
    #img = Image.fromarray(disparity_comb, 'RGB')
    #img.save(filename)
    ('POINT 5')
    #cv2.imwrite(filename,disparity_comb)

    logging.info("Disparity for {} generated at: {}".format(left_image_path, os.path.join(args.save_dir, left_image_path.split('/')[-1])))

if name == 'main':
main()`

Hello:
When I run submission_kitti.py I get RuntimeError: cuda runtime error (11): invalid argument at /pytorch/aten/src/THC/THCGeneral.cpp:663, what should I do, thanks

Traceback (most recent call last):
File "submission_kitti.py", line 83, in
model.load_state_dict(state_dict['state_dict'], strict=True)
File "/home/xxm/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 845, in load_state_dict
self.class.name, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for DeepPruner:

Hello, do you have a pre-training model?

Hi，it‘s a really nice work. But I still have a question about the training process on KITTI2015. In your paper,you mentioned that "We reserved out 40 images from the total 394 images for validation", and then you said "For submission to the KITTI test benchmark, we
re-trained the model on all the 394 training images for 1040 epochs." So,what's the meaning of the first train with validation? And how did you choose the model without the validation set? I hope you can answer my questions, which is very important for my work. Thank you again for your excellent work！

Is there something wrong in line 183, 256, 265 of file DeepPruner/deeppruner/models/patch_match.py

1. line 183: modify if type is "horizontal": to if propagation_type is "horizontal":
2. line 256: modify normalized_disparity_samples = self.propagation(normalized_disparity_samples, device, 1) to normalized_disparity_samples = self.propagation(normalized_disparity_samples, device, 'horizontal')
3. line 265: modify normalized_disparity_samples = self.propagation(normalized_disparity_samples, device, 0) to normalized_disparity_samples = self.propagation(normalized_disparity_samples, device, 'vertical')

Hi, thanks for your sharing and work. I watched your paper&code and some questions really confuse me. I will appreciate if you could answer these questions.

I tried to only use the Differential-PatchMatch of your network to predict the depth, but I didn't get any useful disparity information. It seemed that the random part disturbed the training process.
How do you think of the random part of the network? Does it really works under the forward-backward method?

Thanks.

Hello, I'm very interested in your code，how can I test my own dataset (640 * 480)?
I made an error while testing my dataset, as follows:
File "submission_kitti.py", line 185, in
main()
File "submission_kitti.py", line 165, in main
[-1]), (disparity * 256).astype('uint16'))
File "/home/dkm/local/anaconda3/envs/deep/lib/python3.7/site-packages/skimage/io/_io.py", line 139, in imsave
if is_low_contrast(arr):
File "/home/dkm/local/anaconda3/envs/deep/lib/python3.7/site-packages/skimage/exposure/exposure.py", line 504, in is_low_contrast
limits = np.percentile(image, [lower_percentile, upper_percentile])
File "/home/dkm/local/anaconda3/envs/deep/lib/python3.7/site-packages/numpy/lib/function_base.py", line 3540, in percentile
a, q, axis, out, overwrite_input, interpolation, keepdims)
File "/home/dkm/local/anaconda3/envs/deep/lib/python3.7/site-packages/numpy/lib/function_base.py", line 3652, in _quantile_unchecked
interpolation=interpolation)
File "/home/dkm/local/anaconda3/envs/deep/lib/python3.7/site-packages/numpy/lib/function_base.py", line 3250, in _ureduce
r = func(a, **kwargs)
File "/home/dkm/local/anaconda3/envs/deep/lib/python3.7/site-packages/numpy/lib/function_base.py", line 3767, in _quantile_ureduce_func
x1 = take(ap, indices_below, axis=axis) * weights_below
File "/home/dkm/local/anaconda3/envs/deep/lib/python3.7/site-packages/numpy/core/fromnumeric.py", line 181, in take
return _wrapfunc(a, 'take', indices, axis=axis, out=out, mode=mode)
File "/home/dkm/local/anaconda3/envs/deep/lib/python3.7/site-packages/numpy/core/fromnumeric.py", line 51, in _wrapfunc
return getattr(obj, method)(*args, **kwds)
IndexError: cannot do a non-empty take from an empty axes.

How do you do? Excuse me for using poor English to consult you.

1. For the evaluation of scene flow data set, EPE (also known as MAE) is generally used as the evaluation standard, and the evaluation function can also be implemented in the code for evaluation.

2. For the kitti2012 data set, the evaluation criteria are > 2px, > 3px, > 4PX, > 5px of Noc and Occ (all), and the evaluation of error rate and error pixel number of mean error. Do these evaluations need to implement their functions in their own code? Or do you need to submit it to the official website of Kitti to generate the evaluation results?

3. For the Kitti 2015 data set, the evaluation criteria include the error rate evaluation of all (Occ) and Noc's D1-bg, D1-fg and D1-all. Do you need to implement the evaluation function in the code, or must you submit the evaluation results to the Kitti official website?

4. For kitti12, the evaluation standard can be implemented by its own code; but for kitti2015, it can't be implemented by itself. How to implement D1-bg, D1-fg and D1-all?

5. Besides, do the experimental data of Kitti 12 and 15 have to come from Kitti's official website?

For the above problems, it's still confusing at present. Kitti website seems to say that it can't be used for debugging programs. Everyone can only submit it once within the specified time and can't apply for multiple accounts.

So for these evaluation criteria, I hope you can spare time in your busy schedule, thank you very much!!! ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~Xiaobai thanks first!!!!!!!

What is the default maximum disparity range used in this model?

I cannot find any such parameter in the code. When I ran PSMNet, there was a parameter max_disp to set the maximum disparity value. I am looking for one such parameter in deeppruner.