mjiust / surfacenet Goto Github PK

I'm running this code in my Amazon AWS server
Configuration of my system
`==============NVSMI LOG==============

Timestamp : Tue Apr 24 17:43:40 2018
Driver Version : 375.66

Attached GPUs : 1
GPU 0000:00:1E.0
Product Name : Tesla K80
Product Brand : Tesla
CUDA versionnvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2016 NVIDIA Corporation
Built on Tue_Jan_10_13:22:03_CST_2017
Cuda compilation tools, release 8.0, V8.0.61
`

So when I run python main.py
I got this error

attached as log file
log.txt

Hi thanks for previous feedback in another thread. After I setup up Cuda8.0/CuDNN 5.1 and theano 0.9, I can run some part of main.py. But there's still some error when executing patch2embedding() function in the early rejection stage.

More specifically:

Traceback (most recent call last):
  File "./main.py", line 27, in <module>
    save_npz_file_path = main_reconstruct.reconstruction(datasetFolder, _model, imgNamePattern, poseNamePattern, outputFolder, N_viewPairs4inference, resol, BB, viewList)
  File "/home/ICT2000/tli/Workspace/SurfaceNet/main_reconstruct.py", line 77, in reconstruction
    cubeCenter_hw = np.stack([img_h_cubesCenter, img_w_cubesCenter], axis=0))    # (N_cubes, N_views, D_embedding), (N_cubes, N_views)
  File "./utils/earlyRejection.py", line 31, in patch2embedding
    patches_embedding[:,:] = patch2embedding_fn(patch_allBlack)[0] # don't use np.repeat (out of memory)
  File "/home/ICT2000/tli/.conda/envs/SurfaceNet/lib/python2.7/site-packages/theano/compile/function_module.py", line 898, in __call__
    storage_map=getattr(self.fn, 'storage_map', None))
  File "/home/ICT2000/tli/.conda/envs/SurfaceNet/lib/python2.7/site-packages/theano/gof/link.py", line 325, in raise_with_op
    reraise(exc_type, exc_value, exc_trace)
  File "/home/ICT2000/tli/.conda/envs/SurfaceNet/lib/python2.7/site-packages/theano/compile/function_module.py", line 884, in __call__
    self.fn() if output_subset is None else\
RuntimeError: error doing operation: CUDNN_STATUS_INTERNAL_ERROR
Apply node that caused the error: GpuDnnConv{algo='small', inplace=False}(GpuContiguous.0, GpuContiguous.0, GpuAllocEmpty{dtype='float32', context_name=None}.0, GpuDnnConvDesc{border_mode=(1, 1), subsample=(1, 1), conv_mode='cross', precision='float32'}.0, Cast{float32}.0, Cast{float32}.0)
Toposort index: 276
Inputs types: [GpuArrayType<None>(float32, (False, False, False, False)), GpuArrayType<None>(float32, (False, False, False, False)), GpuArrayType<None>(float32, (False, False, False, False)), <theano.gof.type.CDataType object at 0x7fbd6848bc90>, Scalar(float32), Scalar(float32)]
Inputs shapes: [(1, 3, 64, 64), (64, 3, 3, 3), (1, 64, 64, 64), 'No shapes', (), ()]
Inputs strides: [(49152, 16384, 256, 4), (108, 36, 12, 4), (1048576, 16384, 256, 4), 'No strides', (), ()]
Inputs values: ['not shown', 'not shown', 'not shown', <capsule object NULL at 0x7fbb43bd10c0>, 1.0, 0.0]
Inputs type_num: [11, 11, 11, '', 11, 11]
Outputs clients: [[HostFromGpu(gpuarray)(GpuDnnConv{algo='small', inplace=False}.0)]]

Detail error log can be seen here:
err_log.txt

I have tried:

• Delete Theano cache theano-cache purge or rm -rf ./.theano
• Adjust CNMeM: (https://devtalk.nvidia.com/default/topic/950158/cnmem-limitations-when-using-cudnn/)
• Remove nvidia cache and reboot: (https://stackoverflow.com/questions/45810356/runtimeerror-cudnn-status-internal-error)

None has worked so far.

Have you seen this type of error before? Or did I set my computer correctly?
I observed you have a params.py to specify all parameters. Some has mentioned this error can result from lack of memory (link), and it seems your code did something for batch processing.

Info of my setting:

• Ubuntu 16.04
• CUDA 8.0 / CuDNN 5.1
• GPU: Nvidia 1080 Ti (11GB memory) --- I also tried on another machine with Titan X, not working
• theano 0.9

My ~/.theanorc:

[global] 
floatX=float32 
device=cuda0
optimizer=None
 
allow_gc=True 
#gpuarray.preallocate=0.95
gcc.cxxflags=-Wno-narrowing
exception_verbosity=high

[lib]
cnmem=0.75

[nvcc]
nvcc.fastmath=True 

[cuda] 
root=/usr/local/cuda-8.0

If you have any suggestions, please let me know! Thanks for your help and support!

Update:

After I tried to remove other versions of CuDNN: (https://groups.google.com/forum/#!topic/theano-users/w4M3Xy0ec60), the error changes to the following.

Traceback (most recent call last):
  File "./main.py", line 27, in <module>
    save_npz_file_path = main_reconstruct.reconstruction(datasetFolder, _model, imgNamePattern, poseNamePattern, outputFolder, N_viewPairs4inference, resol, BB, viewList)
  File "/home/ICT2000/tli/Workspace/SurfaceNet/main_reconstruct.py", line 77, in reconstruction
    cubeCenter_hw = np.stack([img_h_cubesCenter, img_w_cubesCenter], axis=0))    # (N_cubes, N_views, D_embedding), (N_cubes, N_views)
  File "./utils/earlyRejection.py", line 48, in patch2embedding
    _patches_embedding_inScope[_batch] = patch2embedding_fn(_patches_preprocessed[_batch])     # (N_batch, 3/1, patchSize, patchSize) --> (N_batch, D_embedding). similarityNet: patch --> embedding
  File "/home/ICT2000/tli/.conda/envs/SurfaceNet/lib/python2.7/site-packages/theano/compile/function_module.py", line 898, in __call__
    storage_map=getattr(self.fn, 'storage_map', None))
  File "/home/ICT2000/tli/.conda/envs/SurfaceNet/lib/python2.7/site-packages/theano/gof/link.py", line 325, in raise_with_op
    reraise(exc_type, exc_value, exc_trace)
  File "/home/ICT2000/tli/.conda/envs/SurfaceNet/lib/python2.7/site-packages/theano/compile/function_module.py", line 884, in __call__
    self.fn() if output_subset is None else\
  File "pygpu/gpuarray.pyx", line 676, in pygpu.gpuarray.pygpu_empty
  File "pygpu/gpuarray.pyx", line 290, in pygpu.gpuarray.array_empty
pygpu.gpuarray.GpuArrayException: cuMemAlloc: CUDA_ERROR_OUT_OF_MEMORY: out of memory
Apply node that caused the error: GpuDnnConv{algo='small', inplace=False}(GpuContiguous.0, GpuContiguous.0, GpuAllocEmpty{dtype='float32', context_name=None}.0, GpuDnnConvDesc{border_mode=(1, 1), subsample=(1, 1), conv_mode='cross', precision='float32'}.0, Cast{float32}.0, Cast{float32}.0)
Toposort index: 276
Inputs types: [GpuArrayType<None>(float32, (False, False, False, False)), GpuArrayType<None>(float32, (False, False, False, False)), GpuArrayType<None>(float32, (False, False, False, False)), <theano.gof.type.CDataType object at 0x7f703a019c90>, Scalar(float32), Scalar(float32)]
Inputs shapes: [(1100, 3, 64, 64), (64, 3, 3, 3), (1100, 64, 64, 64), 'No shapes', (), ()]
Inputs strides: [(49152, 16384, 256, 4), (108, 36, 12, 4), (1048576, 16384, 256, 4), 'No strides', (), ()]
Inputs values: ['not shown', 'not shown', 'not shown', <capsule object NULL at 0x7f6e133930c0>, 1.0, 0.0]
Inputs type_num: [11, 11, 11, '', 11, 11]
Outputs clients: [[HostFromGpu(gpuarray)(GpuDnnConv{algo='small', inplace=False}.0)]]

After I follow this #4, the enviroment problem seems to be gone.
But evertime python main.py occour Segmentation fault (core dumped), the python interpreter stops on main_reconstruc.py line 73
patches_embedding, inScope_cubes_vs_views = earlyRejection.patch2embedding(
images_list, img_h_cubesCorner, img_w_cubesCorner, patch2embedding_fn, patches_mean_bgr,
N_cubes, N_views, D_embedding, patchSize = params.__imgPatch_hw_size,
batchSize = params.__batchSize_similNet_patch2embedding,
cubeCenter_hw = np.stack([img_h_cubesCenter, img_w_cubesCenter], axis=0)) # (N_cubes, N_views, D_embedding), (N_cubes, N_views).

output like this
start reconstruction ...
loaded img ./inputs/Middlebury/dinoSparseRing/dinoSR0007.png
loaded img ./inputs/Middlebury/dinoSparseRing/dinoSR0008.png
loaded img ./inputs/Middlebury/dinoSparseRing/dinoSR0009.png
loaded img ./inputs/Middlebury/dinoSparseRing/dinoSR0010.png
loaded img ./inputs/Middlebury/dinoSparseRing/dinoSR0011.png
loaded img ./inputs/Middlebury/dinoSparseRing/dinoSR0012.png
xyz bounding box of the reconstructed scene: [-0.061897 0.010897], [-0.018874 0.068227], [-0.057845 0.015495]
loaded similarityNet model: ./inputs/SurfaceNet_models/epoch33_acc_tr0.707_val0.791.model
loaded SurfaceNet model: ./inputs/SurfaceNet_models/2D_2_3D-19-0.918_0.951.model
Segmentation fault (core dumped)

It seems like a stack or memery problem. Can you help me? Any advice?
My OS is
Ubuntu 1604 x64
cuda 8.0+ cudnn5.1 and cuda 9.0+cudnn7.1.1
1080Ti

.theanorc
[global]
floatX=float32
device=cuda
optimizer=None

allow_gc=True
gpuarray.preallocate=0.75
gcc.cxxflags=-Wno-narrowing
exception_verbosity=high

[lib]
cnmem=0.75

[nvcc]
nvcc.fastmath=True

[cuda]
root=/usr/local/cuda-8.0

@mjiUST @Rubikplayer

Thanks for the great released code! I am wondering what the detailed information recorded in the pos file is. Is that an essential matrix or a homography matrix? And how can we generate it by ourselves? Eager for your reply.

I installed lasagne 0.1 and theano 0.9 at first, but it doesn't work
ImportError: Could not import Theano. Please make sure you install a recent enough version of Theano.
Then I installed lasagne 0.2.dev1 and theano 1 instead but theano.sandbox.cuda.dnn is discarded in theano 1(like #3 )
import similarityNet File "./nets/similarityNet.py", line 6, in <module> if lasagne.utils.theano.sandbox.cuda.dnn_available(): # when cuDNN available AttributeError: 'module' object has no attribute 'cuda'
@mjiUST could U help me？ thanks

I just installed your code and ran it succesfully. Here are some comments:

• You should mention that your install script requires us to allow the miniconda install to append to our .bashrc - I didn't allow this and it took me a bit to figure out why the rest of the script wasn't working.
• you are sourcing .bashrc without the ~/ in

  SurfaceNet/installEnv.sh

  Line 16 in 2e93c47

  . .bashrc # to enable the conda command

  
  which just always fails
• the else clause is commented here:
  

  SurfaceNet/installEnv.sh

  Line 28 in 2e93c47

  echo "The '$conda_env_name' conda env exists." # but the env will be visible even thoug the installation was terminated. In this case you should delete this conda env first.else # if the env does not exist.

aside from those, I got everything working pretty smoothly

Could you update your source file layer.py?
Because theano.sandbox.cuda.dnn is discarded in theano 1(>theano 0.9).
from theano.sandbox.cuda.dnn import gpu_contiguous, GpuDnnConvDesc, gpu_alloc_empty, GpuDnnConv3dGradW wont work, and if lasagne.utils.theano.sandbox.cuda.dnn_available() in similarityNet.py.
Could you use theano.gpuarray.dnn instead?
I cant replace gpu_contiguous, GpuDnnConvDesc, gpu_alloc_empty, GpuDnnConv3dGradW with classes of theano.gpuarray.dnn by myself.
And I cant backwards to theano 0.9 either, due to the new vision of cudnn does not support old theano and pygpu.
plz help me,thanks