Giter VIP home page Giter VIP logo

voxelnet's Introduction

Introduction

This is an unofficial inplementation of VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection in TensorFlow. A large part of this project is based on the work here. Thanks to @jeasinema. This work is a modified version with bugs fixed and better experimental settings to chase the results reported in the paper (still ongoing).

Dependencies

  • python3.5+
  • TensorFlow (tested on 1.4.1)
  • opencv
  • shapely
  • numba
  • easydict

Installation

  1. Clone this repository.
  2. Compile the Cython module
$ python3 setup.py build_ext --inplace
  1. Compile the evaluation code
$ cd kitti_eval
$ g++ -o evaluate_object_3d_offline evaluate_object_3d_offline.cpp
  1. grant the execution permission to evaluation script
$ cd kitti_eval
$ chmod +x launch_test.sh

Data Preparation

  1. Download the 3D KITTI detection dataset from here. Data to download include:

    • Velodyne point clouds (29 GB): input data to VoxelNet
    • Training labels of object data set (5 MB): input label to VoxelNet
    • Camera calibration matrices of object data set (16 MB): for visualization of predictions
    • Left color images of object data set (12 GB): for visualization of predictions
  2. In this project, we use the cropped point cloud data for training and validation. Point clouds outside the image coordinates are removed. Update the directories in data/crop.py and run data/crop.py to generate cropped data. Note that cropped point cloud data will overwrite raw point cloud data.

  3. Split the training set into training and validation set according to the protocol here. And rearrange the folders to have the following structure:

└── DATA_DIR
       ├── training   <-- training data
       |   ├── image_2
       |   ├── label_2
       |   └── velodyne
       └── validation  <--- evaluation data
       |   ├── image_2
       |   ├── label_2
       |   └── velodyne
  1. Update the dataset directory in config.py and kitti_eval/launch_test.sh

Train

  1. Specify the GPUs to use in config.py
  2. run train.py with desired hyper-parameters to start training:
$ python3 train.py --alpha 1 --beta 10

Note that the hyper-parameter settings introduced in the paper are not able to produce high quality results. So, a different setting is specified here.

Training on two Nvidia 1080 Ti GPUs takes around 3 days (160 epochs as reported in the paper). During training, training statistics are recorded in log/default, which can be monitored by tensorboard. And models are saved in save_model/default. Intermediate validation results will be dumped into the folder predictions/XXX/data with XXX as the epoch number. And metrics will be calculated and saved in predictions/XXX/log. If the --vis flag is set to be True, visualizations of intermediate results will be dumped in the folder predictions/XXX/vis.

  1. When the training is done, executing parse_log.py will generate the learning curve.
$ python3 parse_log.py predictions
  1. There is a pre-trained model for car in save_model/pre_trained_car.

Evaluate

  1. run test.py -n default to produce final predictions on the validation set after training is done. Change -n flag to pre_trained_car will start testing for the pre-trained model (only car model provided for now).
$ python3 test.py

results will be dumped into predictions/data. Set the --vis flag to True if dumping visualizations and they will be saved into predictions/vis.

  1. run the following command to measure quantitative performances of predictions:
$ ./kitti_eval/evaluate_object_3d_offline [DATA_DIR]/validation/label_2 ./predictions

Performances

The current implementation and training scheme are able to produce results in the tables below.

Bird's eye view detection performance: AP on KITTI validation set
Car Easy Moderate Hard
Reported 89.60 84.81 78.57
Reproduced 85.41 83.16 77.10
3D detection performance: AP on KITTI validation set
Car Easy Moderate Hard
Reported 81.97 65.46 62.85
Reproduced 53.43 48.78 48.06

TODO

  • improve the performances
  • reproduce results for Pedestrian and Cyclist
  • fix the deadlock problem in multi-thread processing in training
  • fix the infinite loop problem in test.py
  • replace averaged calibration matrices with correct ones

voxelnet's People

Contributors

qianguih avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

voxelnet's Issues

visualize voxel grid

Hi,

very good work :-)

I have a question about the visualization of the grid on the LIDAR point clouds that is mentioned in the paper.

In the part of voxel partitioning, do you have in your code implemented already the visualization of this grid on the input raw point cloud? (figure 2 of https://arxiv.org/pdf/1711.06396.pdf)

thanks in advance

precision very low

i set the RPN_SCORE_THRESH = 0.90,and RPN_NMS_THRESH = 0.001
but the results have very much boxes that not include anything
does anyone know how to solve it

Multiprocessing error

The following error keeps appearing over and over again when I run the evaluation code on Windows 10 x64 with python test.py -n pre_trained_car:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\Attila\Anaconda2\envs\python364\lib\multiprocessing\spawn.py", line 105, in spawn_main
    exitcode = _main(fd)
  File "C:\Users\Attila\Anaconda2\envs\python364\lib\multiprocessing\spawn.py", line 114, in _main
    prepare(preparation_data)
  File "C:\Users\Attila\Anaconda2\envs\python364\lib\multiprocessing\spawn.py", line 225, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\Attila\Anaconda2\envs\python364\lib\multiprocessing\spawn.py", line 277, in _fixup_main_from_path
    run_name="__mp_main__")
  File "C:\Users\Attila\Anaconda2\envs\python364\lib\runpy.py", line 263, in run_path
    pkg_name=pkg_name, script_name=fname)
  File "C:\Users\Attila\Anaconda2\envs\python364\lib\runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "C:\Users\Attila\Anaconda2\envs\python364\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\Attila\Projects\intelligent-vehicles\voxelnet\voxelnet\test.py", line 10, in <module>
    from model import RPN3D
  File "C:\Users\Attila\Projects\intelligent-vehicles\voxelnet\voxelnet\model\__init__.py", line 12, in <module>
    from model.model import *
  File "C:\Users\Attila\Projects\intelligent-vehicles\voxelnet\voxelnet\model\model.py", line 12, in <module>
    from utils import *
  File "C:\Users\Attila\Projects\intelligent-vehicles\voxelnet\voxelnet\utils\__init__.py", line 12, in <module>
    from utils.kitti_loader import *
  File "C:\Users\Attila\Projects\intelligent-vehicles\voxelnet\voxelnet\utils\kitti_loader.py", line 44, in <module>
    TRAIN_POOL = multiprocessing.Pool(4)
  File "C:\Users\Attila\Anaconda2\envs\python364\lib\multiprocessing\context.py", line 119, in Pool
    context=self.get_context())
  File "C:\Users\Attila\Anaconda2\envs\python364\lib\multiprocessing\pool.py", line 174, in __init__
    self._repopulate_pool()
  File "C:\Users\Attila\Anaconda2\envs\python364\lib\multiprocessing\pool.py", line 239, in _repopulate_pool
    w.start()
  File "C:\Users\Attila\Anaconda2\envs\python364\lib\multiprocessing\process.py", line 105, in start
    self._popen = self._Popen(self)
  File "C:\Users\Attila\Anaconda2\envs\python364\lib\multiprocessing\context.py", line 322, in _Popen
    return Popen(process_obj)
  File "C:\Users\Attila\Anaconda2\envs\python364\lib\multiprocessing\popen_spawn_win32.py", line 33, in __init__
    prep_data = spawn.get_preparation_data(process_obj._name)
  File "C:\Users\Attila\Anaconda2\envs\python364\lib\multiprocessing\spawn.py", line 143, in get_preparation_data
    _check_not_importing_main()
  File "C:\Users\Attila\Anaconda2\envs\python364\lib\multiprocessing\spawn.py", line 136, in _check_not_importing_main
    is not going to be frozen to produce an executable.''')
RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.


About inference time

Hi, thanks for sharing your code, it's really wonderful.

It would be great to know what's the inference time of the code, the paper claimed a 225ms inference time where the voxel input feature computation takes 5ms, feature learning net takes 20ms, convolutional middle layers take 170ms and I'm a little curious about that. Can you please share that? thanks!

关于vfe

嗨,你好,我发现你的这个版本在group_pointcloud.py里去掉了vfe2之后的全连接层,直接进行了reduce_max。因此在vfe里,最后的操作也有一个reduce_max,然后在进行tile和concat。
那进行tile和concat之后,再进行reduce_max,得到的结果和vfe2里做的reduce_max的结果不就一样的么?那这样不就相当于做了重复操作?
另外,为何要去掉那个全连接层呢?是对性能有影响么?

Boost Library Import

For some reason, when i compile the evaluation code, all the import lines aren't read as they are written in angle brackets <import xx>. but when i change them to double quotations "import xx", it works. i can't just change all the import lines of the Boost library files as they are all pointing to each other, so this is an impossible process. how did you manage the Boost library files? is there a specific way by which i should organize those files?

Problem with pre-trained model

There seems to be an issue with the pre-trained model included (save_model/pre_trained_car).
When I run test.py, the visulizations seem to give highly inaccurate results. Furthermore, the ground truth boxes also seem to be flipped along the y-axis in the top-down lidar visualizations.
Is there something wrong with the pre-trained model or coordinate conversions?
Below are two example results.

000001_bv
000001_front

000002_bv
000002_front

test.py writes wrong boxes to the prediction/*.txt files

I run the test.py and the visualizations seems to be good, however they are flipped as in #18. I assume that blue boxes are the ones with the score lower than threshold and that violet boxes are correct predictions. However, the number of lines in prediction/*.txt files is equal to the number of blue boxes. Additionally, evaluation script gives a following performance, which proves that wrong boxes are being checked.

start to eval 3D bounding boxes... 
save ./predictions/plot/car_detection_3d.txt
car_detection_3d AP: 0.003730 0.003682 0.003682

Could anybody comment on this, please?

Bug in corner_to_center_box3d function in utils.py

Follwing Line 270 in utils.py

x, y, z = np.sum(roi, axis=0) / 8

should be replaced with the follwoing

x = np.sum(roi[:, 0], axis=0)/ 8
y = np.sum(roi[0:4, 1], axis=0)/ 4
z = np.sum(roi[:, 2], axis=0)/ 8

This makes sure that the x,y, z is the location of the bottom center of the bounding box.

关于运行时间和卷积层

请问作者你的运行环境是什么?运行时间显示的是多少呢?
我的运行环境是一块1080ti显卡,显示的运行时间是0.6-0.8秒左右一张

还有就是我看您使用的卷积操作全部是在一个convMD函数中,但是我将他们全部替换成tf.layer.conv2d和tf.layer.conv3d了,而且我将2D卷积替换成了标准的VGG16了,请问这样对网络会有影响吗?

test.txt in train val split protocol

Hi friends,
I am trying to split the training set into training and validation set according to the protocol. But I find the test.txt in it. But now we only have training and validation set from the KITTI official website. So what the test.txt stands for? Maybe it has some connection with the process of KITTI submission? A test dataset will give us with the test.txt if we want to submit our results on KITTI?

Thank you!

question about tf.scatter_nd

hello, in your implementation, you use
self.outputs = tf.scatter_nd(
self.coordinate, voxelwise, [self.batch_size, 10, cfg.INPUT_HEIGHT, cfg.INPUT_WIDTH, 128])

I want to ask if I use pytorch, which function should I use to do this operation.

Thank you very much!

Memory usage

In my case, it allocated ~9GB GPU(1080Ti) memory when batch_size = 1, how could you train 2 batch on one GPU without out of memory?

format of pre-trained model output?

The pre-trained car model outputs an array of size-9 for each detected object.
What do these refer to?
How to use them to get lidar-coordinates for detections?

Out of Memory

Hi, many thanks for your source, very helpful.
But when I run the train.py, there is out of memory issue as below:
ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[1,12,402,354,128].......
The memory = 1 x 12 x 402 x 354 x 128 = 172GB, super big. I wonder does it make sense.

Would you please give some hints how it works? Very appreciate for your help.

IndexError: list index out of range

First thank you for your source.
When I was running python3 train.py --alpha 1 --beta 10. I met this problem. I am quite confused now. Can you help me with it?
Thanks!
list index out of range

(I only have 1 1080Ti GPU so I changed single-batch size from 2 to 1.)

Question about name_node and reuse_variable

Hi, thanks for your implementation of voxelnet. Since this is my first time to train with multi-gpu, I am confused with the way you use scope for multi-gpu training.

  1. What's the goal of using name node when creating model.

  2. From my understanding of get_variable_scope().reuse_variables, it is used when we want to reuse the variable under the same scope. However, since variables are created under different name scopes, how can variables be reused under different scopes? Does it mean the name node is not being used when getting variable scope?

Tensorflow Error: Loaded runtime CuDNN library: 7103 (compatibility version 7100) but source was compiled with 7005 (compatibility version 7000)

When I run train.py, got the Error:

2019-01-18 21:14:06.030010: E tensorflow/stream_executor/cuda/cuda_dnn.cc:378] Loaded runtime CuDNN library: 7103 (compatibility version 7100) but source was compiled with 7005 (compatibility version 7000).  If using a binary install, upgrade your CuDNN library to match.  If building from sources, make sure the library loaded at runtime matches a compatible version specified during compile configuration.
2019-01-18 21:14:06.030486: F tensorflow/core/kernels/conv_ops_3d.cc:395] Check failed: stream->parent()->GetConvolveAlgorithms( conv_parameters.ShouldIncludeWinogradNonfusedAlgo<T>(), &algorithms) 

Then I install CuDNN7.1.3 and CuDNN7.0.5 but the Error still exists. I'm not sure how the error occures.
I use conda environment, and tensorflow-gpu=1.4.1

How to train VLP-16 points cloud

Hi,
I tried to train velodyne-16 points cloud on VoxelNet. I modified preprocess.py parameters, but I test the trained model which the training makes, the results is not good. Are there any suggestions that can optimize the results. Thx!

Best Regards,
Xuhui

Coordinate mismatch

It seems there is a mismatch between the coordinate systems in data_aug.py. The voxel_dict is generated in lidar coordinates while the label is generated in the camera coordinates.

There is a problem in the 3D box when evaluating

When viewing the 3D box through the tensorboard, there are a few extra lines that appear to be very strange, and only a few images appear on this issue, with two image attached below. Have you encountered the same situation? How to solve this problem? thank you very much!

image

image

a question about angle_in_limit

the function angle_in_limit in utils.py is:

def angle_in_limit(angle):
    # To limit the angle in -pi/2 - pi/2
    limit_degree = 5
    while angle >= np.pi / 2:
        angle -= np.pi
    while angle < -np.pi / 2:
        angle += np.pi
    if abs(angle + np.pi / 2) < limit_degree / 180 * np.pi:
        angle = np.pi / 2
    return angle

However, what does the last condition mean? when -95 < angle < -85(in degree), angle = 90?

how to achieve the performance in .img/learning_curve.jpg?

I wonder to know if the performances in ./img/learning_curve.jpg is the final result by fine-tone with the pre-trained model. I've done a lot of work to achieve the same result(such as freezing the front layer parameters, adjusting hyper parameter .etc), but there's still a big gap. And I'd appreciate it if you could give me any hints.
THX!!!

error: ‘make_array’ is not a member of ‘boost::serialization’

When compiling the evaluation code, an error occurred:

In file included from /usr/include/boost/numeric/ublas/vector.hpp:21:0,
from /usr/include/boost/numeric/ublas/matrix.hpp:18,
from evaluate_object_3d_offline.cpp:12:
/usr/include/boost/numeric/ublas/storage.hpp: In member function ‘void boost::numeric::ublas::unbounded_array<T, ALLOC>::serialize(Archive&, unsigned int)’:
/usr/include/boost/numeric/ublas/storage.hpp:299:18: error: ‘make_array’ is not a member of ‘boost::serialization’
ar & serialization::make_array(data_, s);
^
/usr/include/boost/numeric/ublas/storage.hpp: In member function ‘void boost::numeric::ublas::bounded_array<T, N, ALLOC>::serialize(Archive&, unsigned int)’:
/usr/include/boost/numeric/ublas/storage.hpp:494:18: error: ‘make_array’ is not a member of ‘boost::serialization’
ar & serialization::make_array(data_, s);
^
In file included from evaluate_object_3d_offline.cpp:12:0:
/usr/include/boost/numeric/ublas/matrix.hpp: In member function ‘void boost::numeric::ublas::c_matrix<T, M, N>::serialize(Archive&, unsigned int)’:
/usr/include/boost/numeric/ublas/matrix.hpp:5977:18: error: ‘make_array’ is not a member of ‘boost::serialization’
ar & serialization::make_array(data_, N)

Offline Evaluation Code

Hello there,

Thanks for the great code you are supplying!
I have a question regarding the offline evaluation code.
I'm trying to verify that it works by using a singe ground truth file twice, once as gt and once as detection as input for the evaluation program. I changed gt file so it adheres to the kitti detection standard (basically just adding a score as 16th column).
Therefore I should get 100% AP correct? The actual results from my tests differ, depending on the file used I sometimes get somewhere between 0 and 36% AP.

Am I using the eval tool wrong?
Does it need a minimum number of detections (that's why it fails in this simple test case?)?
Do I need to modify the evaluation code somehow so it works with the standard kitti detection result format?

Thanks again!

关于精度

有人使用作者给的预训练权重微调之后能得到作者在代码里给出来的精度吗?我的精度不管是怎样都很低

About Split the training set?

In Data Preparation,
"Split the training set into training and validation set according to the protocol here. And rearrange the folders to have the following structure"

I don't know exactly what this means.

  • I ran crop.py. So, Files in the velodyne folder have been removed prefix (000). Why? Is not overwrite ?
  • Do you mean to manually relocate the img / point file in the training / validation folder in the order of the txt file you attached?

python3 test.py crashes on pretrained model? Kindly help

After executing python3 test.py crashes with erroe below

Traceback (most recent call last):
File "/home/administrator/Documents.orig/Kumar_FileSystem/3D_Object_Detection/voxelnet-master/test.py", line 69, in
tags, results, front_images, bird_views, heatmaps = model.predict_step(sess, batch, summary=False, vis=True)
File "/home/administrator/Documents.orig/Kumar_FileSystem/3D_Object_Detection/voxelnet-master/model/model.py", line 285, in predict_step
probs, deltas = session.run(output_feed, input_feed)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 900, in run
run_metadata_ptr)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1135, in _run
feed_dict_tensor, options, run_metadata)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1316, in do_run
run_metadata)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1335, in do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.FailedPreconditionError: Attempting to use uninitialized value MiddleAndRPN
/conv21/kernel
[[Node: MiddleAndRPN
/conv21/kernel/read = IdentityT=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"]]
[[Node: concat_101/_449 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1999_concat_101", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

Caused by op 'MiddleAndRPN_/conv21/kernel/read', defined at:
File "/home/administrator/Documents.orig/Kumar_FileSystem/3D_Object_Detection/voxelnet-master/test.py", line 58, in
avail_gpus=cfg.GPU_AVAILABLE.split(',')
File "/home/administrator/Documents.orig/Kumar_FileSystem/3D_Object_Detection/voxelnet-master/model/model.py", line 70, in init
input=feature.outputs, alpha=self.alpha, beta=self.beta, training=self.is_train)
File "/home/administrator/Documents.orig/Kumar_FileSystem/3D_Object_Detection/voxelnet-master/model/rpn.py", line 97, in init
temp_conv, training=self.training, activation=False, bn=False, name='conv21')
File "/home/administrator/Documents.orig/Kumar_FileSystem/3D_Object_Detection/voxelnet-master/model/rpn.py", line 142, in ConvMD
temp_conv = tf.layers.conv2d(pad, Cout, k, strides=s, padding="valid", reuse=tf.AUTO_REUSE, name=scope)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/layers/convolutional.py", line 621, in conv2d
return layer.apply(inputs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/layers/base.py", line 828, in apply
return self.call(inputs, *args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/layers/base.py", line 699, in call
self.build(input_shapes)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/layers/convolutional.py", line 144, in build
dtype=self.dtype)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/layers/base.py", line 546, in add_variable
partitioner=partitioner)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/checkpointable.py", line 436, in _add_variable_with_custom_getter
**kwargs_for_getter)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/variable_scope.py", line 1317, in get_variable
constraint=constraint)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/variable_scope.py", line 1079, in get_variable
constraint=constraint)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/variable_scope.py", line 425, in get_variable
constraint=constraint)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/variable_scope.py", line 394, in _true_getter
use_resource=use_resource, constraint=constraint)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/variable_scope.py", line 786, in _get_single_variable
use_resource=use_resource)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/variable_scope.py", line 2220, in variable
use_resource=use_resource)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/variable_scope.py", line 2210, in
previous_getter = lambda **kwargs: default_variable_creator(None, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/variable_scope.py", line 2193, in default_variable_creator
constraint=constraint)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/variables.py", line 235, in init
constraint=constraint)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/variables.py", line 397, in _init_from_args
self._snapshot = array_ops.identity(self._variable, name="read")
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/array_ops.py", line 142, in identity
return gen_array_ops.identity(input, name=name)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gen_array_ops.py", line 3187, in identity
"Identity", input=input, name=name)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 3392, in create_op
op_def=op_def)
File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 1718, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

FailedPreconditionError (see above for traceback): Attempting to use uninitialized value MiddleAndRPN_/conv21/kernel
[[Node: MiddleAndRPN_/conv21/kernel/read = IdentityT=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"]]
[[Node: concat_101/_449 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1999_concat_101", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

Process finished with exit code 1

网络耗时

你好,我对整个网络的耗时性进行了一个测试,我在model.py中添加了如下代码:

run_options = tf.RunOptions(trace_level=tf.RunOptions.FULL_TRACE)
 run_metadata = tf.RunMetadata()

        
probs, deltas = session.run(output_feed, input_feed,run_metadata=run_metadata,options=run_options)

trace = timeline.Timeline(step_stats=run_metadata.step_stats)
trace_file = open('/predictions/'+str(time.time()) + '-train-timeline.ctf.json','w')
trace_file.write(trace.generate_chrome_trace_format(show_memory=False))
trace_file.close()

测试的时间应该是准确的,发现整个网络做一次预测,是比论文里提到的更加耗时的,整体耗时超过400ms(我是在GTX1080,单个GPU),其中卷积中间层耗时大概不到100ms(比论文还低),但是有一个操作非常耗时,将近300ms,是一个tile操作,在point_cloud.py中:

def apply(self, inputs, mask, training):
   # [K, T, 7] tensordot [7, units] = [K, T, units]
   pointwise = self.batch_norm.apply(self.dense.apply(inputs), training)

   #n [K, 1, units]
   aggregated = tf.reduce_max(pointwise, axis=1, keep_dims=True)

   # [K, T, units]
   repeated = tf.tile(aggregated, [1, cfg.VOXEL_POINT_COUNT, 1])

   # [K, T, 2 * units]
   concatenated = tf.concat([pointwise, repeated], axis=2)

   mask = tf.tile(mask, [1, 1, 2 * self.units])##这个tile操作非常耗时
   concatenated = tf.multiply(concatenated, tf.cast(mask, tf.float32))

   return concatenated

我在将这个tile操作注释掉之后,速度就提升了。
所以,我比较奇怪。
1、这个tile操作,实际操作的数据量并不大,为什么会这么耗时呢?
2、去掉这个tile操作后,不知道会不会影响训练效果,(= = 我还在训练中),这个地方是否有改进的空间呢?

How did you define K, the limited number of voxels in the paper?

K is the max voxel number in the paper and the author said "it is possible to only store a limited number of voxels (K) and ignore points coming from voxels with few points". But I cannot find where you define K in the code. For example, # [ΣK, 35/45, 7] self.feature = tf.placeholder(tf.float32, [None, cfg.VOXEL_POINT_COUNT, 7], name='feature') where K can be any shape in class FeatureNet. Could you please explain why? Thanks!

missing activation=False for p_map layer?

( see issue tsinghua-rll/VoxelNet-tensorflow#33 )

i'm having trouble understanding why the p_map layer doesn't have 'activation=False':

it's fed into a sigmoid, so if the activations are ReLU'd to be non-negative, it seems the objectness outputs will be limited to [.5,1]. maybe even if that's a bit wrong-ish, it doesn't much matter somehow? I.e. it messes up the gradient for the objectness somewhat, and compresses the range of confidences, but maybe that doesn't have a huge impact in the end?

it seems that, with the fixes in this repo, you're close to reproducing the original-paper-quality results, so i guess that means the impact can't be too large. BTW, thanks for sharing your efforts and fixes!

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.