terminated,exit value:139 (Segmentation fault (core dumped)) about surfacenet HOT 7 OPEN

I my opinion, it could because of 2 reasons:

1. out of memory issue: please check #4 (comment)
2. inconsistent cuda/cudnn version: can you post the result of command python -c "import lasagne" after you source activate <conda_env_cuda8.0_cudnn5.1>, or please post the entire error log (from the very beginning). Then let's see what we can do.

from surfacenet.

@mjiUST Thanks

1. I dont think its a memory issue like #4. To avoid out of memory issue, I have ajusted these settings: gpuarray.preallocate=0.8 ,cnmem=0.8 and __GPUMemoryGB = 11. And both GPU memory and ram are not fully used.
2. python -c "import lasagne"
   Using cuDNN version 5005 on context None
   Mapped name None to device cuda: GeForce GTX 1080 Ti (0000:02:00.0)
   Is this alright? I followed your advice in #3.
   I use a safe way to export path. The cuda/cudnn path will not export to $PATH and $LD_LIBRARY_PATH in .bashrc. They only export to $PATH and $LD_LIBRARY_PATH in activate.d's script. In SurfaceNet's script is cuda 8.0+cudnn5.0.

Sadly, there is no error log, just the python interpreter stoped.(<terminated, exit value: 139>main.py [debug] [/home/cd/miniconda2/envs/SurfaceNet/bin/python2.7])
I think it means a memory access problem. I track it in debug mode, It stops on earlyRejection.py line 31

patches_embedding[:,:] = patch2embedding_fn(patch_allBlack)[0]# don't use np.repeat (out of memory)
I think this is will do a lot of memery access and use a lot of memory.

It may cause by a array problem, patch2embedding_fn is a suspicions array.
I think patch2embedding_fn is a theano.compile.function_module.Function object.
So I doubt my theano(0.9) have some problems. If set the value optimizer = fast_run in .theanorc, then a lot of error arise(C source file output in the error log). Can not run in C mode?
I am new to theano. Could you give me some advice?

from surfacenet.

@cdb0y511

kindly note that the output of command python -c "import lasagne" on my machine looks like this

Using cuDNN version 5110 on context None
Mapped name None to device cuda: TITAN Xp (0000:01:00.0)

From your printout "...Using cuDNN version 5005 on context None...", we infer that the cudnn version (5.1) is not correctly linked to your theano. Please make sure the cudnn in the /usr/local/cuda-8.0 folder is deleted.

If your machine have the following configuration:

cuda 8.0+ cudnn5.1 and cuda 9.0+cudnn7.1.1

I recommend you locate and delete the zombie cudnn (5.0) before continue.

You can checkout the output of the commands python -c "import lasagne" AND echo $CUDA_ROOT before & after source activate <conda_env> to make sure that the conda_env's activate.d works.

from surfacenet.

@mjiUST Thanks
I notice it,and replace the cudnn 5.05 with 5.1. But the Segmentation fault (core dumped) still comes.
I use "gdb python core file" to trace this error. I think it is related with libcuda.so.1. It may be a cuda installtion problem or compatible problem. This drives me crazzzy!
gdb python corefile outputs like this:

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `python main.py'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007f59b9efca20 in ?? () from /usr/lib/x86_64-linux-gnu/libcuda.so.1
[Current thread is 1 (Thread 0x7f59d37e7700 (LWP 4287))]
(gdb) where
#0 0x00007f59b9efca20 in ?? () from /usr/lib/x86_64-linux-gnu/libcuda.so.1
#1 0x00007f59b9efcb50 in ?? () from /usr/lib/x86_64-linux-gnu/libcuda.so.1
#2 0x00007f59b9e0d174 in ?? () from /usr/lib/x86_64-linux-gnu/libcuda.so.1
#3 0x00007f59b9f57d0d in cuEventRecord ()
from /usr/lib/x86_64-linux-gnu/libcuda.so.1
#4 0x00007f5997af198b in ?? () from /usr/local/cuda-8.0/lib64/libcudnn.so.5
#5 0x00007f5997b21eac in ?? () from /usr/local/cuda-8.0/lib64/libcudnn.so.5
#6 0x00007f59975ce934 in ?? () from /usr/local/cuda-8.0/lib64/libcudnn.so.5
#7 0x00007f5997660dc2 in ?? () from /usr/local/cuda-8.0/lib64/libcudnn.so.5
#8 0x00007f59976617ee in ?? () from /usr/local/cuda-8.0/lib64/libcudnn.so.5
#9 0x00007f59975d2af6 in cudnnConvolutionForward ()
from /usr/local/cuda-8.0/lib64/libcudnn.so.5
#10 0x00007f59731cb186 in (anonymous namespace)::__struct_compiled_op_aeac713786253a5100563478542a8b48::conv_fwd_node_aeac713786253a5100563478542a8b48_0 (
_handle=0x55aed67b81f0, output=, beta=0, alpha=1,
desc=0x55af029f2b10, om=, kerns=0x7f59b0131650,
input=0x7f596d6da0d0, this=0x55aefee695b0)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/tmphSDDVa/mod.cpp:493
#11 (anonymous namespace)::__struct_compiled_op_aeac713786253a5100563478542a8b48::run (this=0x55aefee695b0)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/tmphSDDVa/mod.cpp:1011
---Type to continue, or q to quit---
#12 __struct_compiled_op_aeac713786253a5100563478542a8b48_executor (self=0x55aefee695b0)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/tmphSDDVa/mod.cpp:1132
#13 0x00007f59c13d3798 in c_call (self=0x7f59aa2a52c0, node_idx=276, verbose=0)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:533
#14 0x00007f59c13d41d2 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=337, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:699
#15 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=338, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#16 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=339, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#17 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=341, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#18 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=342, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#19 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=343, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#20 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=344, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#21 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=361, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#22 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=362, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#23 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=363, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#24 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=365, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#25 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=366, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#26 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=370, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#27 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=371, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#28 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=377, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#29 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=401, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#30 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=402, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#31 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=403, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#32 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=405, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#33 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=406, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#34 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=407, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#35 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=408, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#36 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=425, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
---Type to continue, or q to quit---
#37 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=426, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#38 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=427, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#39 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=429, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#40 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=430, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#41 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=434, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#42 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=435, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#43 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=441, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#44 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=465, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#45 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=466, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#46 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=467, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#47 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=469, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#48 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=470, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#49 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=471, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#50 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=472, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#51 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=489, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#52 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=490, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#53 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=491, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#54 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=493, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#55 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=494, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#56 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=495, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#57 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=496, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#58 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=513, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#59 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=514, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#60 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=515, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#61 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=517, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
---Type to continue, or q to quit---
#62 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=518, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#63 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=522, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#64 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=523, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#65 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=529, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#66 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=553, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#67 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=554, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#68 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=555, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#69 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=557, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#70 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=558, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#71 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=559, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#72 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=560, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#73 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=577, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#74 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=578, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#75 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=579, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#76 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=581, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#77 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=582, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#78 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=583, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#79 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=584, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#80 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=601, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#81 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=602, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#82 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=603, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#83 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=605, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#84 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=606, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#85 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=610, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#86 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=616, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
---Type to continue, or q to quit---
#87 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=618, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#88 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=641, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#89 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=642, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#90 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=643, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#91 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=645, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#92 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=646, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#93 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=647, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#94 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=648, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#95 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=665, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#96 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=666, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#97 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=667, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#98 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=669, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#99 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=670, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#100 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=671, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#101 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=672, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#102 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=689, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#103 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=690, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#104 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=691, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#105 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=693, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#106 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=694, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#107 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=698, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#108 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=705, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#109 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=706, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#110 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=713, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#111 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=718, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
---Type to continue, or q to quit---
#112 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=719, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#113 0x00007f59c13d4137 in lazy_rec_eval (self=0x7f59aa2a52c0, var_idx=720, one=0x55aed49aa168, zero=0x55aed49aa180)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:690
#114 0x00007f59c13d4a58 in CLazyLinker_call (_self=0x7f59aa2a52c0, args=0x7f59d37a6050, kwds=0x0)
at /home/cd/.theano/compiledir_Linux-4.13--generic-x86_64-with-debian-stretch-sid-x86_64-2.7.14-64/lazylinker_ext/mod.cpp:873
#115 0x00007f59d325e773 in PyObject_Call () from /home/cd/miniconda2/envs/SurfaceNet/bin/../lib/libpython2.7.so.1.0
#116 0x00007f59d32f253b in PyEval_EvalFrameEx () from /home/cd/miniconda2/envs/SurfaceNet/bin/../lib/libpython2.7.so.1.0
#117 0x00007f59d32fa4e9 in PyEval_EvalCodeEx () from /home/cd/miniconda2/envs/SurfaceNet/bin/../lib/libpython2.7.so.1.0
#118 0x00007f59d3282fda in function_call () from /home/cd/miniconda2/envs/SurfaceNet/bin/../lib/libpython2.7.so.1.0
#119 0x00007f59d325e773 in PyObject_Call () from /home/cd/miniconda2/envs/SurfaceNet/bin/../lib/libpython2.7.so.1.0
#120 0x00007f59d326d50d in instancemethod_call () from /home/cd/miniconda2/envs/SurfaceNet/bin/../lib/libpython2.7.so.1.0
#121 0x00007f59d325e773 in PyObject_Call () from /home/cd/miniconda2/envs/SurfaceNet/bin/../lib/libpython2.7.so.1.0
#122 0x00007f59d32b7574 in slot_tp_call () from /home/cd/miniconda2/envs/SurfaceNet/bin/../lib/libpython2.7.so.1.0
#123 0x00007f59d325e773 in PyObject_Call () from /home/cd/miniconda2/envs/SurfaceNet/bin/../lib/libpython2.7.so.1.0
#124 0x00007f59d32f253b in PyEval_EvalFrameEx () from /home/cd/miniconda2/envs/SurfaceNet/bin/../lib/libpython2.7.so.1.0
#125 0x00007f59d32fa4e9 in PyEval_EvalCodeEx () from /home/cd/miniconda2/envs/SurfaceNet/bin/../lib/libpython2.7.so.1.0
#126 0x00007f59d32f7482 in PyEval_EvalFrameEx () from /home/cd/miniconda2/envs/SurfaceNet/bin/../lib/libpython2.7.so.1.0
#127 0x00007f59d32f8dac in PyEval_EvalFrameEx () from /home/cd/miniconda2/envs/SurfaceNet/bin/../lib/libpython2.7.so.1.0
#128 0x00007f59d32fa4e9 in PyEval_EvalCodeEx () from /home/cd/miniconda2/envs/SurfaceNet/bin/../lib/libpython2.7.so.1.0
#129 0x00007f59d32fa70a in PyEval_EvalCode () from /home/cd/miniconda2/envs/SurfaceNet/bin/../lib/libpython2.7.so.1.0
#130 0x00007f59d331393d in run_mod () from /home/cd/miniconda2/envs/SurfaceNet/bin/../lib/libpython2.7.so.1.0
#131 0x00007f59d3314ab8 in PyRun_FileExFlags () from /home/cd/miniconda2/envs/SurfaceNet/bin/../lib/libpython2.7.so.1.0
#132 0x00007f59d3315cd8 in PyRun_SimpleFileExFlags () from /home/cd/miniconda2/envs/SurfaceNet/bin/../lib/libpython2.7.so.1.0
#133 0x00007f59d3327d3c in Py_Main () from /home/cd/miniconda2/envs/SurfaceNet/bin/../lib/libpython2.7.so.1.0
#134 0x00007f59d253f830 in __libc_start_main (main=0x55aed3a7b850

from surfacenet.

@cdb0y511
Can you try

1. delete the folder ~/.theano and source into the conda_env
2. check python -c "import lasagne" AND
3. make sure echo $CUDA_ROOT only output /usr/local/cuda-8.0 and WITHOUT /usr/local/cuda except that the symbolic link cuda is mapping to cuda-8.0. You can also post the output of these commands.
4. run python -m pdb main.py, followed by c button and an enter button

Let's see what we can do.

from surfacenet.

@mjiUST Thanks
I solved it by following steps:
1.delete all libcuda* in /usr/lib/x86_64-linux-gnu/
like libcuda.so.1, libcuda.so.
They come wih cuda drivers. Since they come from a newer driver(390.30). And Theano(0.9) use an old backend, it definitely causes a Segmetaion fault!
2.uninstall cuda-driver completely
3.purge cuda-8-0
4.reinstall cuda-8-0 from deb, and the driver 384.111 come with it.
Maybe this info can help someone.

Your programme has just finished(middbury Dino). It takes about 38mins(fast_run). The nets is loaded from 2D_2_3D-19-0.918_0.951.model.
Does it mean reconstruction need 38mins? Why @Rubikplayer took 2 hours?(My cpu is i7-4700k,CPU differs?)
Is the 2D_2_3D-19-0.918_0.951.model pre-trained from DTU database?(5days on TITIAN XP?)
And sometimes even 11GB GPU memory is free, cnnm cant be larger than 0.8, otherwise OUT OF MEMORY ERROR arise. it may be a Theano bug.

from surfacenet.

@cdb0y511
Thanks a lot for your feedback. The provided solution must be helpful for others.

Does it mean reconstruction need 38mins? Why @Rubikplayer took 2 hours?(My cpu is i7-4700k,CPU differs?)

Yes, your running time is similar to mine.

Is the 2D_2_3D-19-0.918_0.951.model pre-trained from DTU database?(5days on TITIAN XP?)

Yes, the training and validation split are from the DTU dataset. And it generalize well on the Dino model.

from surfacenet.