Hello,

Many thanks for your great work! It's very impressive. Please forgive my ignorance since I am still new to this research.

In your conclusion you mention adopting a photometric reprojection loss. I assume similar to Goddard et al. and Zhou et al.

Is my assumption correct?

I was wondering if you have already begun working on this? What kind of timeline are you expecting?

If you have not yet begun this, I am also considering adding it to your pipeline as the loss function in place of supervised learning.

My Python/C++ skills are pretty good, but my CNN skills still need a lot of work. Would it be OK to reach out to someone on your team for collaboration on this? Is that even a possibility?

Thanks,
Sameh

I thought the original idea of ASPP was to have parallel convolutions with varying dilation rates and then concatenate them all together. But you do ASPP in a sequential manner with skip connections. Do you have specific reasoning for this? Judging from the ablation experiments, it seems like the ASPP did not have much of an effect.

Hi,

Thanks a lot for releasing this code, and for taking the time to release the pytorch version too!

I tested your pre-trained model on kitti, as the README file suggests, and indeed got the expected results (rmse = 2.798).
However - training gets NANs after about 6.5K iterations. (Before that, the loss got to ~1.0):

[epoch][s/s_per_e/gs]: [1][1608/5790/7398], lr: 0.000097927423, loss: nan

1. Could you perhaps verify the reason for getting NANs?
2. What should be the final loss at the final epoch?

Thanks again for the great work!
Best,
Z.

Minor comments:

1. I'm using your recently-release pytorch version

"fa5face5eb76b16119613aebf9dc9965d6070e4d ; "Replacing bts_parameters with args"

2. The only difference from the setup you provide, is that i disable "multiprocessing_distributed", since it provides this error:

OMP: Error #13: Assertion failure at z_Linux_util.cpp(2361).
OMP: Hint Please submit a bug report with this message, compile and run commands used, and machine configuration info including native compiler and operating system versions. Faster response will be obtained by including all program sources. For information on submitting this issue, please see http://www.intel.com/software/products/support/.

Hello,
I've been doing some testing with your eigen model and I ran into an issue that I hope you could help me with.

I have two images, of different field of view, from the same scene... I've also calculated their respective focal length (pixels).
(90 Degrees FOV, focal length = 620.5 pixels)

(120 Degrees FOV, focal length = 385.25 pixels)

I only made the following modification to the configurations...
1095_90FOV_CROP.jpg None 620.5
1095_120FOV_CROP.jpg None 385.25

I sampled the resulting depth map and found the following depths...

Shouldn't these two depths be the same?

Additionally, I changed the focal lengths to some arbitrary number (ie 100) and the depth values remained unchanged.

Is the focal length not used in testing? What am I missing?

Thanks in Advance!

Hi,

I want to train bts (pytorch version) on high resolution images (around (960, 2500)). How do I modify the network, if needed? Is it possible considering the pretrained encoder is used?

Right now, I'm getting the error in the model:

Traceback (most recent call last):
  File "bts_main.py", line 611, in <module>
    main()
  File "bts_main.py", line 607, in main
    main_worker(args.gpu, ngpus_per_node, args)
  File "bts_main.py", line 451, in main_worker
    lpg8x8, lpg4x4, lpg2x2, reduc1x1, depth_est = model(image, focal)
  File "/nfs/interns/kharshit/miniconda3/envs/pylatest/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/nfs/interns/kharshit/miniconda3/envs/pylatest/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 150, in forward
    return self.module(*inputs[0], **kwargs[0])
  File "/nfs/interns/kharshit/miniconda3/envs/pylatest/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/nfs/interns/kharshit/DepthEstimation/bts/pytorch/bts.py", line 322, in forward
    return self.decoder(skip_feat, focal)
  File "/nfs/interns/kharshit/miniconda3/envs/pylatest/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/nfs/interns/kharshit/DepthEstimation/bts/pytorch/bts.py", line 201, in forward
    concat5 = torch.cat([upconv5, skip3], dim=1)
RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 1. Got 125 and 124 in dimension 3 at /opt/conda/conda-bld/pytorch_1565272271120/work/aten/src/THC/generic/THCTensorMath.cu:71

Thanks!

Hello,
Thanks for your work!But I have some problems when I download the bts_nyu.zip, use your script download_from_gdrive.py。the error is that:

python utils/download_from_gdrive.py 1ipme-fkV4pIx87sOs31R9CD_Qg-85__h models/bts_nyu.zip

Traceback (most recent call last):
File "utils/download_from_gdrive.py", line 45, in
download_file_from_google_drive(file_id, destination)
File "utils/download_from_gdrive.py", line 26, in download_file_from_google_drive
response = session.get(URL, params = { 'id' : id }, stream = True)
File "C:\ProgramData\Anaconda2\lib\site-packages\requests\sessions.py", line 546, in get
return self.request('GET', url, **kwargs)
File "C:\ProgramData\Anaconda2\lib\site-packages\requests\sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "C:\ProgramData\Anaconda2\lib\site-packages\requests\sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "C:\ProgramData\Anaconda2\lib\site-packages\requests\adapters.py", line 510, in send
raise ProxyError(e, request=request)
requests.exceptions.ProxyError: HTTPSConnectionPool(host='docs.google.com', port=443): Max retries exceeded with url: /uc?export=download&id=1AysroWpfISmm-yRFGBgFTrLy6FjQwvwP (Caused by ProxyError('Cannot connect to proxy.', error('Tunnel connection failed: 407 Proxy Authentication Required ( Forefront TMG \xe9\x9c\x80\xe8\xa6\x81\xe6\x8e\x88\xe6\x9d\x83\xe6\x89\x8d\xe8\x83\xbd\xe5\xae\x8c\xe6\x88\x90\xe8\xaf\xb7\xe6\xb1\x82\xe3\x80\x82\xe6\x8b\x92\xe7\xbb\x9d\xe8\xae\xbf\xe9\x97\xae Web \xe4\xbb\xa3\xe7\x90\x86\xe7\xad\x9b\xe9\x80\x89\xe5\x99\xa8\xe3\x80\x82 )',)))

I am looking forward to your reply！

Hello,
I follow instructions in the readme, but I got segmentation fault. My python version is 2.7

segmentation fault (core dumped)  python bts_main.py arguments_train_nyu.txt

Hi @cogaplex-bts

I followed ur Tensorflow instruction. The results are shown below:
GT files reading done
45 GT files missing
Computing errors
d1, d2, d3, AbsRel, SqRel, RMSE, RMSElog, SILog, log10
0.348, 0.740, 0.986, 0.339, 1.792, 5.804, 0.354, 33.288, 0.134
Done.
I used densenet161 as encoder and the trained checkpoint you gave testing on KITTI. but the results are not so good.

Hi,
Thanks for your work. I had some problem with adapting my own dataset. I generated depth images from Carla simulator. The max value of carla depth images is 1km (so the images have values from 3 to 1000) but i have meaningful information until 70 meters (which is the max depth that i fixed).
I noticed that in the other datasets (kitti and nyu) the color of meaningless information is black instead in the images of my dataset it's white. So i changed the value to 0 in order to obtain something similar to the other dataset.
I tryu to show what I mean:

This is an image taken from Nyu dataset

After the division by 1000, the result mask is

This is an image taken from my dataset

If I apply the same code applied above I obtain this:

I think that it's not correct... So after i have changed the max value (1000) with 0 I obtain this:

is it correct?

Nevertheless, the network doesn't learn well... what should I do?
And what about the parameter focal? What is it?
Thanks

Line 95, file bts_dataloader.py: It's not working for tf 1.5. Is there any solution?

Dear author,

Thanks for the wonderful works, here I would like to ask 2 questions about the proposed LPG layer:

1. Figure 4 in the paper, the size of the feature map is H/k before Planar Guidance, and under my understanding, the Planar Guidance operation is corresponded to Eq(1), which just fuse four planes into a 1-channel maps without upsample the size of the input. But the output size of Planar Guidance in Figure 4 becomes to H, so I'm wondering if there's a bilinear operation in Planar Guidance?

2. From the picture I highlighted, it says there's a "scaling factor" after the sigmoid layer. For my understanding, it means that n4 will multiply this factor before concatenation, but from the code it seems it didn't do that way, so I'm wondering if there is something I misunderstood.

Thank you

I trained bts (pytorch) on NYU for few epochs, but during testing, I'm getting the following error while loading the model:

Traceback (most recent call last):
  File "bts_test.py", line 221, in <module>
    test(args)
  File "bts_test.py", line 94, in test
    model.load_state_dict(checkpoint['model'])
  File "/nfs/interns/kharshit/miniconda3/envs/pylatest/lib/python3.7/site-packages/torch/nn/modules/module.py", line 845, in load_state_dict
    self.__class__.__name__, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for DataParallel:
	Missing key(s) in state_dict: "module.encoder.base_model.denseblock3.denselayer25.norm1.weight", "module.encoder.base_model.denseblock3.denselayer25.norm1.bias", "module.encoder.base_model.denseblock3.denselayer25.norm1.running_mean", "module.encoder.base_model.denseblock3.denselayer25.norm1.running_var", "module.encoder.base_model.denseblock3.denselayer25.conv1.weight", "module.encoder.base_model.denseblock3.denselayer25.norm2.weight", "module.encoder.base_model.denseblock3.denselayer25.norm2.bias",
...
size mismatch for module.encoder.base_model.conv0.weight: copying a param with shape torch.Size([64, 3, 7, 7]) from checkpoint, the shape in current model is torch.Size([96, 3, 7, 7]).
	size mismatch for module.encoder.base_model.norm0.weight: copying a param with shape torch.Size([64]) from checkpoint, the shape in current model is torch.Size([96]).
	size mismatch for module.encoder.base_model.norm0.bias: copying a param with shape torch.Size([64]) from checkpoint, the shape in current model is torch.Size([96]).
...

The only thing I changed was to remove --multiprocessing_distributed from the arguments_train_nyu.txt file and added --gpu 0 and run script using CUDA_VISIBLE_DEVICES=0 to train using single GPU.

in bts.py

concat4_daspp = tf.concat([iconv4, daspp_3, daspp_6, daspp_12, daspp_18, daspp_24], 3) 
daspp_feat = conv(concat4_daspp, num_filters / 2, 3, 1)  
...  
upconv3 = upconv(daspp_feat, num_filters, 3, 2)  # H/4

Here input of upconv3 is daspp_feat which is after convolution. In paper in figure 2, input of upconv3 is before convolution which is concat4_daspp. Am I misunderstanding something ?

The bts_test.py in pytorch version of BTS only allows testing/inference on the KITTI or NYU V2 dataset. How do I perform inference on a single image?

hi, thanks for your work!But I have some problems when I download the zip, use your script download_from_gdrive.py。the error is that:

python download_from_gdrive.py 1rn7xBF5eSISFKL2bIa8o3d8dNnsrlWfJ models/densenet161_imagenet.zip
Traceback (most recent call last):
File "C:\Users\catalina\AppData\Local\Programs\Python\Python37\lib\site-packages\urllib3\connection.py", line 160, in _new_conn
(self._dns_host, self.port), self.timeout, **extra_kw)
File "C:\Users\catalina\AppData\Local\Programs\Python\Python37\lib\site-packages\urllib3\util\connection.py", line 80, in create_connection
raise err
File "C:\Users\catalina\AppData\Local\Programs\Python\Python37\lib\site-packages\urllib3\util\connection.py", line 70, in create_connection
sock.connect(sa)
TimeoutError: [WinError 10060] ▒▒▒▒▒▒▒ӷ▒▒▒һ▒▒ʱ▒▒▒û▒▒▒▒ȷ▒𸴻▒▒▒▒ӵ▒▒▒▒▒û▒з▒Ӧ▒▒▒▒▒ӳ▒▒▒ʧ▒ܡ▒

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Users\catalina\AppData\Local\Programs\Python\Python37\lib\site-packages\urllib3\connectionpool.py", line 603, in urlopen
chunked=chunked)
File "C:\Users\catalina\AppData\Local\Programs\Python\Python37\lib\site-packages\urllib3\connectionpool.py", line 344, in _make_request
self._validate_conn(conn)
File "C:\Users\catalina\AppData\Local\Programs\Python\Python37\lib\site-packages\urllib3\connectionpool.py", line 843, in _validate_conn
conn.connect()
File "C:\Users\catalina\AppData\Local\Programs\Python\Python37\lib\site-packages\urllib3\connection.py", line 316, in connect
conn = self._new_conn()
File "C:\Users\catalina\AppData\Local\Programs\Python\Python37\lib\site-packages\urllib3\connection.py", line 169, in _new_conn
self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x0000020B95FCFEC8>: Failed to establish a new connection: [WinError 10060] ▒▒▒▒▒▒▒ӷ▒▒▒һ▒▒ʱ▒▒▒û▒▒▒▒ȷ▒𸴻▒▒▒▒ӵ▒▒▒▒▒û▒з▒Ӧ▒▒▒▒▒ӳ▒▒▒ʧ▒ܡ▒

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Users\catalina\AppData\Local\Programs\Python\Python37\lib\site-packages\requests\adapters.py", line 449, in send
timeout=timeout
File "C:\Users\catalina\AppData\Local\Programs\Python\Python37\lib\site-packages\urllib3\connectionpool.py", line 641, in urlopen
_stacktrace=sys.exc_info()[2])
File "C:\Users\catalina\AppData\Local\Programs\Python\Python37\lib\site-packages\urllib3\util\retry.py", line 399, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='docs.google.com', port=443): Max retries exceeded with url: /uc?export=download&id=1rn7xBF5eSISFKL2bIa8o3d8dNnsrlWfJ (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x0000020B95FCFEC8>: Failed to establish a new connection: [WinError 10060] ▒▒▒▒▒▒▒ӷ▒▒▒һ▒▒ʱ▒▒▒û▒▒▒▒ȷ▒𸴻▒▒▒▒ӵ▒▒▒▒▒û▒з▒Ӧ▒▒▒▒▒ӳ▒▒▒ʧ▒ܡ▒'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "download_from_gdrive.py", line 43, in
download_file_from_google_drive(file_id, destination)
File "download_from_gdrive.py", line 24, in download_file_from_google_drive
response = session.get(URL, params = { 'id' : id }, stream = True)
File "C:\Users\catalina\AppData\Local\Programs\Python\Python37\lib\site-packages\requests\sessions.py", line 546, in get
return self.request('GET', url, **kwargs)
File "C:\Users\catalina\AppData\Local\Programs\Python\Python37\lib\site-packages\requests\sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "C:\Users\catalina\AppData\Local\Programs\Python\Python37\lib\site-packages\requests\sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "C:\Users\catalina\AppData\Local\Programs\Python\Python37\lib\site-packages\requests\adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='docs.google.com', port=443): Max retries exceeded with url: /uc?export=download&id=1rn7xBF5eSISFKL2bIa8o3d8dNnsrlWfJ (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x0000020B95FCFEC8>: Failed to establish a new connection: [WinError 10060] ▒▒▒▒▒▒▒ӷ▒▒▒һ▒▒ʱ▒▒▒û▒▒▒▒ȷ▒𸴻▒▒▒▒ӵ▒▒▒▒▒û▒з▒Ӧ▒▒▒▒▒ӳ▒▒▒ʧ▒ܡ▒'))

Hi,
is it possible to use the custom layer in pytorch (with a new cmake) or implement it from scratch?
Thanks

I think the readme is linking to the wrong paper.

Thank you for your interesting work and excellent reproduction code.

1. Is the code fit for multiple GPU training?
2. Did you experience any degradation in results?
3. Should I change something in the arguments file or the command: 'python bts_main.py arguments_train_deci.txt'?

• I am specifically referring to the NYU Pytorch training

Thanks a lot

Hi @cogaplex-bts

thanks for ur effort. I am reading ur tensorflow pipeline. I got one question about LPG. As ur paper mentioned, n4 will go through singmoid and n1, n2, n3 will pass into tanh then L2_nor. Then concat them together. But in bts class, I found that
plane_eq_8x8 = self.reduction_1x1(daspp_feat, num_filters / 2)
plane_normal_8x8 = tf.nn.l2_normalize(plane_eq_8x8[:, :, :, 0:3], axis=3)
plane_dist_8x8 = plane_eq_8x8[:, :, :, 3]
plane_eq_8x8 = tf.concat([plane_normal_8x8, tf.expand_dims(plane_dist_8x8, 3)], 3)
depth_8x8 = lpg.local_planar_guidance(plane_eq_8x8, upratio=8, focal=self.focal)
depth_8x8_scaled = tf.expand_dims(depth_8x8, 3) / self.max_depth
depth_8x8_scaled_ds = self.downsample_nn(depth_8x8_scaled, 4)
plane_normal_8x8 did not pass into sigmoid.
same as plane_eq_8x8
So May I ask why?

Thanks,
Derek

Thanks for this nice work!
I notice that there is a light-weight backbone densenet121 in your code, could you please share the pretrained model? Or just tell me where to download this pretrained model or checkpoint?
Regards!

Hello,

Thanks for ur sharing this code. I am currently trying the tensorflow version. I have downloaded models under tensorflow folder. Then I run python bts_test.py arguments_test_eigen.txt.
error shows that:
Traceback (most recent call last):
File "bts_test.py", line 66, in
for key, val in vars(import(args.model_name)).items():
File "./models/bts_eigen_v2/bts_eigen_v2.py", line 25, in
from resnet_v1 import *
ImportError: No module named resnet_v1

Hello I am not able to find an explanation for class reduction_1x1(nn.Sequential): function in pytorch bts.py file.
Thanks

Thanks for great work.
I am wondering, How to test on any image or video
thanks in advance
Regards

I am interested in your work. I wonder if it is possible to eliminate the c++ and cuda part to make the code more simple and readable. If you use TensorFlow it backpropagates the gradient automatically. It also helps to use your code in future work and make some changes easily. Here is my suggestion for the elimination of cuda part in BTS v1. I am not sure about my transformation because some parts are not described in the paper. Please consider it: https://github.com/saeid-h/bts-fully-tf

Use GPU:5 for training
Use GPU:2 for training
Use GPU:3 for training
Use GPU:4 for training
Use GPU:1 for training
Fixing first conv layer
Total number of parameters:47000688
Total number of learning parameters:46766640

The code stuck in this step.What's wrong?

Thank you for releasing this code.

As other issues have asked, I know this can be trained on nyu and kitti however when applying on a different scene, can this algorithm still perform well?

Issue #15 is having this problem as well where they are applying your code on other scenes and are getting bad depth prediction...

Is it as easy as providing the correct focal length in "eigen_train_files_with_gt.txt" and code will provide good depth estimation for other scenes?

Why do you normalize your image coordinates the way you do in the local planar guidance layer?

u = self.u.repeat(plane_eq.size(0), plane_eq.size(2) * int(self.upratio), plane_eq.size(3)).cuda()
u = (u - (self.upratio - 1) * 0.5) / self.upratio
        
v = self.v.repeat(plane_eq.size(0), plane_eq.size(2), plane_eq.size(3) * int(self.upratio)).cuda()
v = (v - (self.upratio - 1) * 0.5) / self.upratio

I would have thought that normalizing meant to normalize them to be between 0 and 1.
Instead, you normalize it so that it is contrained between (1-r)/2r and (r-1)/2r where r is the upsample ratio. Why is that?

Hello

I used anaconda environment with tensorflow=1.14.0 (by conda install tensorflow-gpu=1.14.0), cuda10.0, and python 3.6.

The libcompute_depth.so can be compiled following the steps in readme. But when I try to load this library, e.g.,

compute_depth_grad_module = tf.load_op_library('custom_layer/build/libcompute_depth.so')

It reports the following error

lib_handle = py_tf.TF_LoadLibrary(library_filename)
tensorflow.python.framework.errors_impl.NotFoundError: custom_layer/build/libcompute_depth.so: undefined symbol: _ZN10tensorflow12OpDefBuilder4AttrESs

Do you have any suggestions for this problem? Thanks

Update: Solved! never mind. Don't know images started getting saved after trying another time.

I'm using pytorch version of bts on NYU V2. Though the testing is shown to be complete, there are no saved images.

This will save results to ./result_bts_nyu_v2_pytorch_densenet161.

now testing 654 files with ../models/bts_nyu_v2_pytorch_densenet161/model
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 654/654 [01:05<00:00, 10.00it/s]
Elapesed time: 65.43908762931824
Done.
Saving result pngs..
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 654/654 [00:15<00:00, 41.09it/s]

I can't find the the folder named result_bts_nyu_v2_pytorch_densenet161 containing predicted images, I even tried creating the empty folder before testing.

Hello, I test bts cpu version, not gpu, use bts_squence.py, nyu dataset settting, when input a image, I got some nan, is it normal?
model download with:python utils/download_from_gdrive.py 1goRL8aZw8bwZ8cZmne_cJTBnBOT6ii0S models/bts_nyu_v2.zip

with the following command and arguments:
python bts_sequence.py --model_name=bts_nyu --encoder=densenet161_bts --dataset=nyu --image_path=../grpcstar/test_images/ --out_path=./out/ --checkpoint_path=./models/bts_nyu/model

input image shape:(480, 640), have 523 nan .

use desktop logitech HD1080p camera

result like following:

test on nyu dataset:

when running python bts_test.py arguments_test_eigen.txt, I encounter the following error:

tensorflow.python.framework.errors_impl.InvalidArgumentError: slice index 2 of dimension 0 out of bounds.

If anyone had the same issue let me know what was the rectification required.

TIA

I have tried pip install tensorflow-gpu==1.13.2 with python 3.6 and 2.7 but I get gpu_kernel_helper.h not found error, see below:

fatal error: tensorflow/core/util/gpu_kernel_helper.h: No such file or directory
 #include "tensorflow/core/util/gpu_kernel_helper.h"
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
CMake Error at compute_depth_generated_compute_depth.cu.o.cmake:215 (message):
  Error generating
  /<path>/bts-master/custom_layer/build/CMakeFiles/compute_depth.dir//./compute_depth_generated_compute_depth.cu.o

In your reply to another issue you said that pip install tensorflow-gpu==1.13.2 works but I get above error... I looked into "tensorflow/include/tensorflow/core/util/" and I could not find gpu_kernel_helper.h or gpu_device_functions.h

I have also tried pip install tensorflow-gpu==1.14.0 with python 3.6 and I also get segmentation fault

In your code, the a scale-invariant error is "self.silog_loss = tf.sqrt(tf.reduce_mean(d ** 2) - 0.85 * (tf.reduce_mean(d) ** 2)) * 10.0", It is different from the Eq.(4) of the paper because there is no "tf.sqrt"

hi~, your work is very impressive. I wonder if you have shown the normal vector and does it make sense?

Greetings,

I have made PyTorch implementation of this paper, and during training phase, some of the predictions have this weird noise, i was wondering if you guys experienced something similar to this?

This noise isnt present in all predictions, some of them seem normal

I have this error:
error: tensorflow/include/tensorflow/core/util/cuda_kernel_helper.h: No such file or directory

I am running:
ubuntu: 19.04
gcc: 7.4
Cuda: 10.0
TensorFlow: 1.13.2 installed via pip

CMake .. output:
-- The C compiler identification is GNU 7.4.0
-- The CXX compiler identification is GNU 7.4.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
/home/hema/.local/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:526: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/hema/.local/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:527: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/hema/.local/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:528: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/hema/.local/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:529: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/hema/.local/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:530: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/hema/.local/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:535: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
/home/hema/.local/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:526: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/hema/.local/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:527: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/hema/.local/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:528: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/hema/.local/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:529: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/hema/.local/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:530: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/hema/.local/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:535: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
/home/hema/.local/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:526: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/hema/.local/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:527: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/hema/.local/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:528: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/hema/.local/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:529: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/hema/.local/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:530: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/hema/.local/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:535: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
/home/hema/.local/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:526: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/hema/.local/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:527: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/hema/.local/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:528: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/hema/.local/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:529: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/hema/.local/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:530: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/hema/.local/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:535: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
TF_INC: /home/hema/.local/lib/python3.7/site-packages/tensorflow/include
TF_LIB: /home/hema/.local/lib/python3.7/site-packages/tensorflow
TF_COMPILE_FLAGS: -I/home/hema/.local/lib/python3.7/site-packages/tensorflow/include -D_GLIBCXX_USE_CXX11_ABI=1
TF_LINK_FLAGS: -L/home/hema/.local/lib/python3.7/site-packages/tensorflow -ltensorflow_framework
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Found CUDA: /usr/local/cuda (found version "10.0")
/home/hema/.local/lib/python3.7/site-packages/tensorflow/include/external/nsync/public/
CMAKE_CXX_FLAGS: -std=c++11 -fPIC -I/home/hema/.local/lib/python3.7/site-packages/tensorflow/include -D_GLIBCXX_USE_CXX11_ABI=1 -DGOOGLE_CUDA
-- Configuring done
-- Generating done
-- Build files have been written to: /media/hema/New Volume/Programming/ComputerVision/Depth/bts/custom_layer/build

make -j output:
/media/hema/New Volume/Programming/ComputerVision/Depth/bts/custom_layer/compute_depth.cu:23:10: fatal error: tensorflow/include/tensorflow/core/util/cuda_kernel_helper.h: No such file or directory
#include "tensorflow/include/tensorflow/core/util/cuda_kernel_helper.h" //HEMA
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
CMake Error at compute_depth_generated_compute_depth.cu.o.cmake:215 (message):
Error generating /media/hema/New
Volume/Programming/ComputerVision/Depth/bts/custom_layer/build/CMakeFiles/compute_depth.dir//./compute_depth_generated_compute_depth.cu.o

make[2]: *** [CMakeFiles/compute_depth.dir/build.make:65: CMakeFiles/compute_depth.dir/compute_depth_generated_compute_depth.cu.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:72: CMakeFiles/compute_depth.dir/all] Error 2
make: *** [Makefile:84: all] Error 2

Hi, the normalized coordinate of pixel i, donated as [u, v], means [(x-cx)/fx, (y-cy)/fy]. But in your code, it is implemented as

float v = ((float)(row % num_threads_row) - (float)(num_threads_row - 1.0f) / 2.0f) / fo;
float u = ((float)(col % num_threads_col) - (float)(num_threads_col - 1.0f) / 2.0f) / fo;

So I was confused about the implementation in the code.

Hi, thanks for your work.
I want to get absolute depth value in real word, expressed in meters. How to do that ?

https://github.com/cogaplex-bts/bts/blob/118a9daf110b87c986ff8d57a4542c88c624521b/pytorch/bts_main.py#L310

cnt not defined
cnt-->var_cnt

Hello and thanks for your great work. I'm trying to understand your paper and code. In the encoder class (pytorch version of the bts.py file) you have used the following code:

 if params.encoder == 'densenet121_bts':
            self.base_model = models.densenet121(pretrained=True).features
            self.feat_names = ['relu0', 'pool0', 'transition1', 'transition2', 'norm5']
            self.feat_out_channels = [64, 64, 128, 256, 1024]

and in the same class (forward method) you have used the code:

def forward(self, x):
        features = [x]
        skip_feat = [x]
        for k, v in self.base_model._modules.items():
            if 'fc' in k or 'avgpool' in k:
                continue
            feature = v(features[-1])
            features.append(feature)
            if any(x in k for x in self.feat_names):
                skip_feat.append(feature)
        
        return skip_feat

It seems that you didn't use all the densenet161 network layers for dense feature extraction (DFE). Am I right? And why you ignored the fc and avgpool layers?

According to my understanding, n1, n2, n3 in the following formula represent the unit normal vector of the local plane of k x k size, and n4 represents the distance. And the purpose of this formula is to calculate the depth of each pixel of the local area of ​​k x k size.

d = n4 / (n1*u + n2*v + n3)

I am curious about how this formula is obtained.

Hello,
I followed these commands, but 'bathroom_0039', 'living_room_0019', 'living_room_0018', 'bedroom_0076a', 'living_room_0020' are missing.

$ cd ~/workspace/dataset/nyu_depth_v2
$ mkdir raw && cd raw
$ aria2c -x 16 -i ../../bts/utils/nyudepthv2_archives_to_download.txt
$ parallel unzip ::: *.zip

Hi,
It seems that the codes you released have some difference with the network in paper. For example, in Figture 2, only the 1/2, 1/4 1/8 size of origin input shape have been used, but in code https://github.com/cogaplex-bts/bts/blob/master/bts.py#L241 1/16 features have been used too. Did you do some other experiments to compare the network in code and paper? Thank you.

Hi, thanks for your great work!

I also use both raw velodyne scans and official ground truth on KITTI in my work, so I am referring to your paper.
In the paper, the results in other methods have slightly different values presented in their paper.
(i.e., DORN a1 0.932 a2 0.984 a3 0.994 rel 0.072 sq rel 0.307 rmse 2.727 rmse log 0.120 in their paper,
and a1 0.936 a2 0.986 a3 0.995 rel 0.081 sq rel 0.337 rmse 2.930 rmse log 0.121 in your paper as well as results of LRC. )

Could you explain how did you get the quantitative results of other work in your paper?

Hi BTS Team,

Could you help me convert the Eigen model to ONNX format?

Many Thanks!

Hi @cogaplex-bts , thank you very much for your great work.
Unfortunately, I don't seem to be able to download the bts_eigen_v2_pytorch_densenet161.zip files with the link you give in your pytorch readme.
(https://cogaplex-bts.s3.ap-northeast-2.amazonaws.com/bts_eigen_v2_pytorch_densenet161.zip)

Is this link outdated and is the data otherwise available?

Thank you,
Dane

Dear Authur,

Thank you for sharing your codes.

I have a question about equation 5:

Do you meant h= beta x logy - beta x log y* ?
The way you described in the paper, betas are going to cancel out each other and have no effect!

Please fix me if I am wrong.
Thanks,
Ali

I have tried using your LPG layers for unsupervised depth estimation. And it didn't seem to improve results very much. From 0.122 to 0.120 abs rel error. Granted, I am using resnet18 as DFE which has output stride of 32, so there are two extra lpg layers: lpg_32x32 and lpg_16x16, but I still would have expected a more significant improvement, given the ablation experiments in the paper. I am wondering if I did something wrong, because visualizing the outputs of the LPG layers, they have significant gridding artificats which I do not see in the LPG outputs in the paper.

The picture is showing disp0, disp1, disp2, disp3, which correspond to final depth, lpg_2x2, lpg_4x4, lpg_8x8.

You can clearly see the patches created by the LPG layers. 8x8 is the most obvious because it has the biggest patches. This could mean the (u, v) do not change enough when re-creating the kxk patch.

I was wondering if the author ever saw this artifacts during his training of bts?