cardwing / codes-for-lane-detection Goto Github PK

Hello,I got a strange bug when run you code to train a model. in this line 'net = lanenet_merge_model.LaneNet()',it throw a error 'TypeError: init() missing 1 required positional argument: 'phase'',and I check your code carefully,but not found any questions.my python version is 3.6,tensorflow version is 1.12. can you help me solve this problem?
thank you very much.

Hi, cardwing. Recently,I am also studying lane detection and see your good job about it. And I have several questions to ask you for some help. Firstly, I don't know how you annonate the lane in the picture and use what kinds of tools. Secondly, I downloaded the CULane dataset, I can not understand the meaning of the .txt file , can you tell the meaning of it's data in the .txt?

Hi,

I started training from scratch, during training its printing accuracy and mean_accuracy as shown below in snapshot

What does it mean between accuracy and mean accuracy.?
From term mean i understand like its mean of accuracy of each and every epoch...

Kindly give some interpretation towards the mentioned terms.

The repo says that this is an implementation of the "Spatial as Deep" paper: https://arxiv.org/abs/1712.06080, but it seems to implement Lanenet, which is this paper: https://arxiv.org/pdf/1807.01726.pdf
Moreover, this seems to be an unofficial fork of the Lanenet implementation available here: https://github.com/MaybeShewill-CV/lanenet-lane-detection

So, my question is, what exactly is the purpose of this fork, what model does it aim to implement?

感谢大佬分享，请问一下，如果在culane上训练的话需要选取ROI删除引擎盖区域吗，不这样的话换自己拍摄的视频来测试训练好的模型是否会效果不佳

could you tell me Which annotation tool do you use?,I want label my data. thinks.

@cardwing hello, I am coming again. I have retrained your pre-train-model in CULane data. In the training process, I think the train is normal. Like accuracy is 70% to 80 %, accuracy-back is 90% to 98%. The times of train epoch is 90000. They are normal.
But when I test them, I also get the not good result. I evaluate the results. Get the information below:

Evaluating the results...
tp: 6783 FP: 96950 fn: 98103
finished process file
precision: 0.065389
recall: 0.0646702
F-measure: 0.0650276

The result in real pictures are like below:

Is there anything wrong in test code? I retrain it, the details are normal. But the test results were not good.

按照readme准备了数据集，使用2块GPU 训练模型时报错如下：
我的 tensorflow = 1.12.0 它提示API已经弃用。是这个原因儿找不到对的文件吗？

W0121 15:26:00.527571 23387 tf_logging.py:125] From tools/train_lanenet.py:249: start_queue_runners (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.
Instructions for updating:
To construct input pipelines, use the tf.data module.
2019-01-21 15:26:02.450372: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at whole_file_read_ops.cc:114 : Not found: laneseg_label_w16/driver_23_30frame/05160929_0487.MP4/00015.png; No such file or directory
2019-01-21 15:26:02.450372: W tensorflow/core/framework/op_kernel.cc:1273] OP_REQUIRES failed at whole_file_read_ops.cc:114 : Not found: driver_23_30frame/05160929_0487.MP4/00015.jpg; No such file or directory
E0121 15:26:02.450782 23387 tf_logging.py:105] Exception in QueueRunner: laneseg_label_w16/driver_23_30frame/05160929_0487.MP4/00015.png; No such file or directory
[[{{node data_augmentation/ReadFile_1}} = ReadFile_device="/job:localhost/replica:0/task:0/device:CPU:0"]]
[[{{node data_augmentation/DecodeJpeg/_53}} = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_17_data_augmentation/DecodeJpeg", tensor_type=DT_UINT8, _device="/job:localhost/replica:0/task:0/device:GPU:0"]]

How to judge existence ground truth is '0' or '1', I refer https://github.com/XingangPan/seg_label_generate this repo but I can not understand code without any comments. So could you kindly let me know the ways to generate existence ground truth?

"test_lanenet(args.image_path, args.weights_path, args.use_gpu, image_name, args.batch_size)"
NameError: name 'image_name' is not defined.
Can you please fix this..?

你好，我看到tf实现的模型中，lane detection的输出是卷积结果直接resize后输出，而softmax被放入了lane existence prediction branch，如下：

           ret['prob_output'] = tf.image.resize_images(conv_output, [CFG.TRAIN.IMG_HEIGHT, 
           CFG.TRAIN.IMG_WIDTH])

            ### add lane existence prediction branch ###

            # spatial softmax #
            features = conv_output  # N x H x W x C
            softmax = tf.nn.softmax(features)

            avg_pool = self.avgpooling(softmax, kernel_size=2, stride=2)

这个实现与论文中Fig 5给出的结构有些不同，请问这个改动有什么作用呢？

Hi
While training the model (train_lanenet.py), I could see for each and every training epoch you are doing validation epochs 1209 in number ( epoch_val in range(int(9675 / 8.0)) ) .

So what is logic behind this logic? can we do only 1-val epoch per 1-train epoch?

HI,

Wanted to do inference and plot the detected lanes onto image.
You mentioned need that to follow SCNN to get curve lines from probability maps.

In your inference code you are not storing the probabilities maps, so can I kindly know how to save those probability (in what format) and then how to pass it through MATLAB code for generation of lines from probability map.

kindly provide your inputs, Look forward.

There are several tricks to boost the performance of SCNN-Tensorflow:

• Decrease the coefficient of the lane existence prediction loss
• Decrease the coefficient of the background pixels in the cross-entropy loss
• First pre-train the VGG-16 backbone on CULane. Then add the message passing module to VGG-16 and train the whole model jointly
• Introduce auxiliary tasks like drivable area detection and lane point regression
• Use a better optimizer, like SGD + Nesterov Momentum
• Balance the proportion of different driving categories like normal, shadow and curve, in the training process
• Hard example mining
• Add weight decay

感谢分享代码！在训练过程中观察发现在跑验证步的时候，精度似乎没有随着训练精度的提升而提升。怀疑是验证时用的网络并不是训练的网络。仔细看了一下代码，感觉是train_lanenet.py中192-192行，两次forward生成两个网络（而不是共用一个），所以验证的时候一直用的初始化权重而不是训练的权重？不知大家在训练的过程中是否遇到类似现象？

hello , i try to run the test,but it still have problem,and i don't where and how to solve.All the need file i have prepared.
when i use this CUDA_VISIBLE_DEVICES="0" python tools/test_lanenet.py --
weights_path /home/pch/test/SCNN/culane_lanenet_vgg_2018-12-01-14-38-37.ckpt-10000 --image_path /home/pch/test/SCNN/test_img.txt

get the problem 'Segmentation fault (core dumped)' and i don't know how to solve it.

@cardwing Hi, If I want to evaluate SCNN on TuSimple Lane Detection Challenge dataset, how can I generate the json file needed for evaluation?（pred_file.json）

There is some useless line of code that raise error
from lanenet_model import lanenet_discriminative_loss
at lanenet_merge_model.py

Thank you for code. If I understand correctly code you realized SCNN from https://arxiv.org/pdf/1712.06080.pdf in lane-detection-model folder. Do you have pretrained model for tensorflow? Can you share that?

hello, i test the results use the prerain-model you offered, but i find the result is very worse .In some clear lane mark picture also not good,i want to know why.And is this model your last testing model? And 是否存在坐标的转换问题，我获得的测试结果，利用SCNN的tools生成结果图的。下面是我其中一张。

@cardwing Hello，i find an interesting thing. In my own computer(1080ti 11G), I try to change different tensorflow such as 1.4, 1.7... and I also re-install cudu9, Nvidia driver and cudnn. I all get the not good results. But on another computer in my laboratory, the result is ok.

Hi

Im utilizing SCNN (Original) code for getting coordinates using matlab scripts.
in getLane.m files it computing the cordinates based on probmaps
the snapshot of functionality is shown below,

in blue marked line, what does 580 & 288 number stand for ?
In your TensorFlow SCNN should we need to mention same numbers?

Kindly provide your inputs

Hi.

I try to get the model to run according to README. Apparently there is a problem with the checkpoint file. Is the file broken or am I doing something wrong?

python test_lanenet.py --weights_path model_culane-71-3/culane_lanenet_vgg_2018-12-01-14-38-37.ckpt-10000.index --image_path images.txt
...
INFO:tensorflow:Restoring parameters from model_culane-71-3/culane_lanenet_vgg_2018-12-01-14-38-37.ckpt-10000.index I1215 22:58:33.538790 7892 tf_logging.py:82] Restoring parameters from model_culane-71-3/culane_lanenet_vgg_2018-12-01-14-38-37.ckpt-10000.index Traceback (most recent call last): File "/home/crr/miniconda3/envs/tensorflow/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1327, in _do_call return fn(*args) File "/home/crr/miniconda3/envs/tensorflow/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1306, in _run_fn status, run_metadata) File "/home/crr/miniconda3/envs/tensorflow/lib/python3.5/contextlib.py", line 66, in __exit__ next(self.gen) File "/home/crr/miniconda3/envs/tensorflow/lib/python3.5/site-packages/tensorflow/python/framework/errors_impl.py", line 466, in raise_exception_on_not_ok_status pywrap_tensorflow.TF_GetCode(status)) tensorflow.python.framework.errors_impl.NotFoundError: Tensor name "lanenet_loss/inference/encode/dense_1/kernel" not found in checkpoint files model_culane-71-3/culane_lanenet_vgg_2018-12-01-14-38-37.ckpt-10000.index [[Node: save/RestoreV2_84 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_84/tensor_names, save/RestoreV2_84/shape_and_slices)]]

Hi,

You mentioned that during testing phase, to compute the accuracy and F2 related measure need to use original SCNN implementation.

In your training process, there were IOU computation and accuracy, can we utilize the same thing for testing purpose just like you used during the validation phase??.

These is the one im doing duirbg testing phase

and also

And also

I computed, IoU_1, IoU_2, IoU_3, IoU_4 as per training code....

Look forward for your valuable response.

Thanks,
Anil.

Hello! I am trying your repo on Tusimple Dataset. I have met a problem which looks like as following:

It seems that all the output becomes all zero. I wonder if you have met this problem and how can I solve this problem! I just use your list and adjust the input size to 256 x 512. My tf verson is 1.10.0. Thank you very much!

Hi, cardwing.
As I know, you have also studied VPGNet. I would like to ask you a quetion. After reading his paper, we can know, the author use spline to annotate the lane first, ang the convert it to grid cell annotation. However, how to annotate other marks like zebra line, stop line direct arrow and so on. waiting for your answer.

And I print the value: print (feature_list_new):

Could you help to find out the problem? Thx~

Is it ok to use run.sh in windows when doing evaluation? just to follow the original steps?

Hello, thanks for your brilliant work.

I am relatively new to this and I am trying to test culane pretrained model real-time over webcam. However, at first I need to freeze the model. What output nodes should I keep for the frozen model to work?

Optional: any suggestions for real-time inference?

Thanks for your time

In vgg_encoder.py, it seems that one message passing procedure (say top down) share one conv module.

But in the paper SCNN, it seems that conv modules are different in one message passing procedure. (According to Equation (1) in Page 3)

BTW, in Xingang Pan's souce code, it seems in one message passing procedure, 'conv:clone' is done. I'm not quite sure whether this clone procedure is a deep clone in lua torch. Maybe you could give me a definite answer. Much appreciated!

Happy Chinese New year! I read the code of test_inference function, and found the code I paste here is an average pooling process since the kernel is initialized by 1.0/81 and is not trainable. I want to ask whether my opinion is true? If it is true, why not use average pooling layer instead of it.

kernel = tf.get_variable('kernel', [9, 9, 1, 1], initializer=tf.constant_initializer(1.0 / 81),
                            trainable=False)

with tf.variable_scope("convs_smooth"):
    prob_smooth = tf.nn.conv2d(tf.cast(tf.expand_dims(binary_seg_ret[:, :, :, 0], axis=3), tf.float32),
                                kernel, [1, 1, 1, 1], 'SAME')
    prob_list.append(prob_smooth)

for cnt in range(1, binary_seg_ret.get_shape().as_list()[3]):
    with tf.variable_scope("convs_smooth", reuse=True):
        prob_smooth = tf.nn.conv2d(
            tf.cast(tf.expand_dims(binary_seg_ret[:, :, :, cnt], axis=3), tf.float32), kernel, [1, 1, 1, 1],
            'SAME')
        prob_list.append(prob_smooth)
processed_prob = tf.stack(prob_list, axis=4)
processed_prob = tf.squeeze(processed_prob, axis=3)
binary_seg_ret = processed_prob

Hello @cardwing

I am currently trying to train the lanenet model with my own dataset but i missing some files.
M current status:

• I trained successfully the network with the 6 images which are in traning data example directory
  Now i need to train it with my own dataset (I generated images from a simulation program)
  In order to train the network i need additionaly the gt_images_instance and the gt_image_binary of each image.

For that i can use the generate_tusimple_dataset.py file in the tools directory and my files need to be structured as in the tuSimple directory, but my problem is i can not run the file because i dont have the label_data.json . How can i generate the json file with all all the attributes

• "lanes"
• "h_samples"
• "raw_file"
  and, am i doing something wrong or is there perhaps a better way to train the network ?

I thank you in advance for any answer.

I just follow your test steps and get the probability maps from the model. U mentioned 'To get the final performance, you need to follow SCNN to get curve lines from probability maps as well as calculate precision, recall and F1-measure'. The problem is that I run your code in windows but the original SCNN testing is running in Linux. Does is it run succesfully in Windows? Do u get the test result in Windows?

I edit the default batch size in test_lanenet.py and global_config then get a problem like this:
...
Traceback (most recent call last):
File "/home/aquarius/anaconda3/envs/tensorflow_gpu/lib/python3.5/site-packages/tensorflow/python/framework/common_shapes.py", line 654, in _call_cpp_shape_fn_impl
input_tensors_as_shapes, status)
File "/home/aquarius/anaconda3/envs/tensorflow_gpu/lib/python3.5/contextlib.py", line 66, in exit
next(self.gen)
File "/home/aquarius/anaconda3/envs/tensorflow_gpu/lib/python3.5/site-packages/tensorflow/python/framework/errors_impl.py", line 466, in raise_exception_on_not_ok_status
pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: Index out of range using input dim 3; input has only 3 dims for 'lanenet_loss/inference/encode/strided_slice_37' (op: 'StridedSlice') with input shapes: [36,100,128], [4], [4], [4] and with computed input tensors: input[3] = <1 1 1 1>.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "tools/test_lanenet.py", line 138, in
test_lanenet(args.image_path, args.weights_path, args.use_gpu, img_name, args.batch_size)
File "tools/test_lanenet.py", line 68, in test_lanenet
binary_seg_ret, instance_seg_ret = net.inference(input_tensor=input_tensor, name='lanenet_loss')
File "/home/aquarius/Codes-for-Lane-Detection/SCNN-Tensorflow/lane-detection-model/lanenet_model/lanenet_merge_model.py", line 114, in inference
inference_ret = self._build_model(input_tensor=input_tensor, name='inference')
File "/home/aquarius/Codes-for-Lane-Detection/SCNN-Tensorflow/lane-detection-model/lanenet_model/lanenet_merge_model.py", line 49, in _build_model
name='encode')
File "/home/aquarius/Codes-for-Lane-Detection/SCNN-Tensorflow/lane-detection-model/encoder_decoder_model/vgg_encoder.py", line 240, in encode
feature_list_old.append(tf.expand_dims(processed_feature[:, :, cnt, :], axis=2))
File "/home/aquarius/anaconda3/envs/tensorflow_gpu/lib/python3.5/site-packages/tensorflow/python/ops/array_ops.py", line 509, in _SliceHelper
name=name)
File "/home/aquarius/anaconda3/envs/tensorflow_gpu/lib/python3.5/site-packages/tensorflow/python/ops/array_ops.py", line 677, in strided_slice
shrink_axis_mask=shrink_axis_mask)
File "/home/aquarius/anaconda3/envs/tensorflow_gpu/lib/python3.5/site-packages/tensorflow/python/ops/gen_array_ops.py", line 3744, in strided_slice
shrink_axis_mask=shrink_axis_mask, name=name)
File "/home/aquarius/anaconda3/envs/tensorflow_gpu/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op
op_def=op_def)
File "/home/aquarius/anaconda3/envs/tensorflow_gpu/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 2632, in create_op
set_shapes_for_outputs(ret)
File "/home/aquarius/anaconda3/envs/tensorflow_gpu/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1911, in set_shapes_for_outputs
shapes = shape_func(op)
File "/home/aquarius/anaconda3/envs/tensorflow_gpu/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1861, in call_with_requiring
return call_cpp_shape_fn(op, require_shape_fn=True)
File "/home/aquarius/anaconda3/envs/tensorflow_gpu/lib/python3.5/site-packages/tensorflow/python/framework/common_shapes.py", line 595, in call_cpp_shape_fn
require_shape_fn)
File "/home/aquarius/anaconda3/envs/tensorflow_gpu/lib/python3.5/site-packages/tensorflow/python/framework/common_shapes.py", line 659, in _call_cpp_shape_fn_impl
raise ValueError(err.message)
ValueError: Index out of range using input dim 3; input has only 3 dims for 'lanenet_loss/inference/encode/strided_slice_37' (op: 'StridedSlice') with input shapes: [36,100,128], [4], [4], [4] and with computed input tensors: input[3] = <1 1 1 1>.

If I use batch size = 8, I encounter this problem:
2018-12-06 16:02:34.287785: E tensorflow/stream_executor/cuda/cuda_dnn.cc:371] could not create cudnn handle: CUDNN_STATUS_INTERNAL_ERROR
2018-12-06 16:02:34.287807: E tensorflow/stream_executor/cuda/cuda_dnn.cc:338] could not destroy cudnn handle: CUDNN_STATUS_BAD_PARAM
2018-12-06 16:02:34.287813: F tensorflow/core/kernels/conv_ops.cc:672] Check failed: stream->parent()->GetConvolveAlgorithms( conv_parameters.ShouldIncludeWinogradNonfusedAlgo(), &algorithms)
Aborted (core dumped)

I am using Ubuntu 16.04 with conda python 3.5 to work on your code.

1.Whether the tusimple dataset can only evaluate instance segmentation，can`t evaluate Semantic segmentation lane result ? Because the official script is evaluated on an example of each instance lane .

Hello,happy new year! 如果有的话能分享一下轻量化模型（ENet-SAD）的代码吗，感谢^~^

An error seems trace back to line 70 in tools/test_lanenet.py: binary_seg_ret, instance_seg_ret = net.inference(input_tensor=input_tensor, name='lanenet_loss')

My command is python3 tools/test_lanenet.py --weights_path ~/test-model-68-8/culane_lanenet_vgg_2018-11-27-16-03-42.ckpt-76000 --image_path demo_file/test_img.txt --batch_size 1 --use_gpu 0

Here is the link of Traceback >> log.txt
`

Now I have a project that requires lane detection accuracy of 80% in night， If I use SCNN or LaneNet to train the data set in night alone, can I reach 80 percent?

Thank you for your work, I use the latest code, and have not made any changes, but the loss and accuracy of the training has been 0, some details are not noticed or caused by the TensorFlow version? I am using 1.10.0. Thank you!

Just like the label images in CULane, how to decide the lane's Id in the TuSimple dataset? Cause there will be 0~5 lanes in an image.
And can you share your code for generating the training label images？

I met a problem when run the main.m to do evaluation in the original scnn. I modified all the path in the main.m code. But It seems like exist.txt can't get read.

您好！您能提供生成文件夹laneseg_label_w16下，实例分割标签png文件的python脚本吗？我想调整一下线宽，另外程序为什么没有在sess.run()的时候feed input 和 label数据呢？还有就是在dataset类中，我该如何使用numpy函数出来tensor呢？希望 您能解答，谢谢！

awesome work, do you mind producing a demo on a random dashcam video