Error running IAM demo about returnn HOT 6 CLOSED

The output contains

'Not able to select available GPU from 1 cards (all CUDA-capable devices are busy or unavailable

Are you sure, that there was a free GPU when running it?
I think you need to set device=cpu in your .theanorc

from returnn.

I was able to run the GPU demo given at (http://deeplearning.net/software/theano/tutorial/using_gpu.html). The gpu was detected in that case and the demo program ran without any issues, outputting "Used the gpu". The GPU had no other process running at the time when I tried to run the demo or when I tried to run the go.sh file.

I had set device=cuda in my .theanorc file while running the above demo program.

from returnn.

Please set device=cpu in the .theanorc and try again.
Even though it will be set to cpu, RETURNN will acquire the GPU in a background process.

from returnn.

I get a new error upon setting device=cpu in .theanorc. I suppose I need to ensure cudnn is being detected by theano ?

RETURNN starting up, version 20160714.042013--git-5a40490-dirty, pid 36431, cwd /home/kartik/new_returnn/demos/mdlstm/IAM
RETURNN command line options: ['config_fwd_kd']
Theano: 0.9.0 ( in /home/kartik/.local/lib/python2.7/site-packages/theano)
WARNING (theano.sandbox.cuda): The cuda backend is deprecated and will be removed in the next release (v0.10). Please switch to the gpuarray backend. You can get more information about how to switch at this URL:
https://github.com/Theano/Theano/wiki/Converting-to-the-new-gpu-back-end%28gpuarray%29

Using gpu device 0: GeForce GTX 1080 Ti (CNMeM is disabled, cuDNN not available)
Device gpuX proc starting up, pid 36557
Device gpuX proc: THEANO_FLAGS = 'compiledir_format=compiledir_%(platform)s-%(processor)s-%(python_version)s-%(python_bitwidth)s--dev-gpuZ,device=gpu,force_device=True'
Device train-network: Used data keys: ['classes', 'data', 'sizes']
^[[F^[[ADevice gpuX proc exception: ('The following error happened while compiling the node', CuDNNConvHWBCOp{border_mode='valid'}(GpuContiguous.0, GpuContiguous.0, GpuContiguous.0), '\n', "We can't determine the cudnn version as it is not available", "Can not compile with cuDNN. We got this error:\nnvcc warning : The 'compute_20', 'sm_20', and 'sm_21' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning).\n/tmp/try_flags_mfyHIk.c:5:19: fatal error: cudnn.h: No such file or directory\ncompilation terminated.\n")
Unhandled exception <type 'exceptions.Exception'> in thread <_MainThread(MainThread, started 140098489444096)>, proc 36557.

Thread current, main, <_MainThread(MainThread, started 140098489444096)>:
(Excluded thread.)

That were all threads.
EXCEPTION
Traceback (most recent call last):
File "/home/kartik/new_returnn/Device.py", line 982, in process
line: self.process_inner(device, config, self.update_specs, asyncTask)
locals:
self = <Device.Device object at 0x7f6ac2f621d0>
self.process_inner = <bound method Device.process_inner of <Device.Device object at 0x7f6ac2f621d0>>
device = 'gpuX'
config = <Config.Config instance at 0x7f6ac2f676c8>
self.update_specs = {'layers': [], 'block_size': 0, 'update_params': {}, 'update_rule': 'global'}
asyncTask = <TaskSystem.AsyncTask instance at 0x7f6ad0c58f80>
File "/home/kartik/new_returnn/Device.py", line 1035, in process_inner
line: self.initialize(config, update_specs=update_specs)
locals:
self = <Device.Device object at 0x7f6ac2f621d0>
self.initialize = <bound method Device.initialize of <Device.Device object at 0x7f6ac2f621d0>>
config = <Config.Config instance at 0x7f6ac2f676c8>
update_specs = {'layers': [], 'block_size': 0, 'update_params': {}, 'update_rule': 'global'}
File "/home/kartik/new_returnn/Device.py", line 808, in initialize
line: self.extractor = theano.function(inputs = [],
outputs = source if len(source) == 1 else [T.concatenate(source, axis=-1)],
givens = givens,
on_unused_input=config.value('theano_on_unused_input', 'ignore'),
name = "extractor")
locals:
self = <Device.Device object at 0x7f6ac2f621d0>
self.extractor = !AttributeError: 'Device' object has no attribute 'extractor'
theano = <module 'theano' from '/home/kartik/.local/lib/python2.7/site-packages/theano/init.pyc'>
theano.function = <function function at 0x7f6b32b738c0>
inputs =
outputs =
source = [Elemwise{switch,no_inplace}.0]
len =
T = <module 'theano.tensor' from '/home/kartik/.local/lib/python2.7/site-packages/theano/tensor/init.pyc'>
T.concatenate = <function concatenate at 0x7f6b29823848>
axis =
givens = [(y_classes, y_classes), (x, y_data), (y_sizes, y_sizes), (j_classes, j_classes), (i, j_data), (j_sizes, j_sizes), (tags, tags_var)], len = 7
on_unused_input =
config = <Config.Config instance at 0x7f6ac2f676c8>
config.value = <bound method Config.value of <Config.Config instance at 0x7f6ac2f676c8>>
name =
File "/home/kartik/.local/lib/python2.7/site-packages/theano/compile/function.py", line 326, in function
line: fn = pfunc(params=inputs,
outputs=outputs,
mode=mode,
updates=updates,
givens=givens,
no_default_updates=no_default_updates,
accept_inplace=accept_inplace, name=name,
rebuild_strict=rebuild_strict,
allow_input_downcast=allow_input_downcast,
on_unused_input=on_unused_input,
profile=profile,
output_keys=output_keys)
locals:
fn =
pfunc = <function pfunc at 0x7f6b32bd3758>
params =
inputs = []
outputs = [Elemwise{switch,no_inplace}.0]
mode = None
updates = []
givens = [(y_classes, y_classes), (x, y_data), (y_sizes, y_sizes), (j_classes, j_classes), (i, j_data), (j_sizes, j_sizes), (tags, tags_var)], len = 7
no_default_updates = False
accept_inplace = False
name = 'extractor', len = 9
rebuild_strict = True
allow_input_downcast = None
on_unused_input = 'ignore', len = 6
profile = None
output_keys = None
File "/home/kartik/.local/lib/python2.7/site-packages/theano/compile/pfunc.py", line 486, in pfunc
line: return orig_function(inputs, cloned_outputs, mode,
accept_inplace=accept_inplace, name=name,
profile=profile, on_unused_input=on_unused_input,
output_keys=output_keys)
locals:
orig_function = <function orig_function at 0x7f6b32bbb1b8>
inputs = [In(y_classes), In(y_data), In(y_sizes), In(j_classes), In(j_data), In(j_sizes), In(tags_var), In(W_conv0), In(b_conv0), In(W1_mdlstm0), In(W2_mdlstm0), In(W3_mdlstm0), In(W4_mdlstm0), In(U1_mdlstm0), In(U2_mdlstm0), In(U3_mdlstm0), In(U4_mdlstm0), In(V1_mdlstm0), In(V2_mdlstm0), In(V3_mdlstm0), ..., len = 99
cloned_outputs = [Elemwise{switch,no_inplace}.0]
mode = None
accept_inplace = False
name = 'extractor', len = 9
profile = None
on_unused_input = 'ignore', len = 6
output_keys = None
File "/home/kartik/.local/lib/python2.7/site-packages/theano/compile/function_module.py", line 1795, in orig_function
line: fn = Maker(inputs,
outputs,
mode,
accept_inplace=accept_inplace,
profile=profile,
on_unused_input=on_unused_input,
output_keys=output_keys).create(
defaults)
locals:
fn =
Maker = <class 'theano.compile.function_module.FunctionMaker'>
inputs = [In(y_classes), In(y_data), In(y_sizes), In(j_classes), In(j_data), In(j_sizes), In(tags_var), In(W_conv0), In(b_conv0), In(W1_mdlstm0), In(W2_mdlstm0), In(W3_mdlstm0), In(W4_mdlstm0), In(U1_mdlstm0), In(U2_mdlstm0), In(U3_mdlstm0), In(U4_mdlstm0), In(V1_mdlstm0), In(V2_mdlstm0), In(V3_mdlstm0), ..., len = 99
outputs = [Out(Elemwise{switch,no_inplace}.0,False)]
mode = <theano.compile.mode.Mode object at 0x7f6b236fd690>
accept_inplace = False
profile = None
on_unused_input = 'ignore', len = 6
output_keys = None
create =
defaults = [<array([[0]], dtype=int32)>, <CudaNdarray([[[ 0.]]])>, <array([[0]], dtype=int32)>, <array([[0]], dtype=int8)>, <array([[0]], dtype=int8)>, <array([[0]], dtype=int8)>, <array([], shape=(0, 0), dtype=int8)>, <CudaNdarray([[[[-0.21161591 0.08376592 -0.04271842]
[ 0.19575438 0.19206183 -0.1560..., len = 99
File "/home/kartik/.local/lib/python2.7/site-packages/theano/compile/function_module.py", line 1661, in create
line: _fn, _i, _o = self.linker.make_thunk(
input_storage=input_storage_lists, storage_map=storage_map)
locals:
_fn =
_i =
_o =
self = <theano.compile.function_module.FunctionMaker object at 0x7f6ab498ca10>
self.linker = <theano.gof.vm.VM_Linker object at 0x7f6ab498c450>
self.linker.make_thunk = <bound method VM_Linker.make_thunk of <theano.gof.vm.VM_Linker object at 0x7f6ab498c450>>
input_storage = [<array([[0]], dtype=int32)>, <CudaNdarray([[[ 0.]]])>, <array([[0]], dtype=int32)>, <array([[0]], dtype=int8)>, <array([[0]], dtype=int8)>, <array([[0]], dtype=int8)>, <array([], shape=(0, 0), dtype=int8)>, <CudaNdarray([[[[-0.21161591 0.08376592 -0.04271842]
[ 0.19575438 0.19206183 -0.1560..., len = 99
input_storage_lists = [[array([[0]], dtype=int32)], [CudaNdarray([[[ 0.]]])], [array([[0]], dtype=int32)], [array([[0]], dtype=int8)], [array([[0]], dtype=int8)], [array([[0]], dtype=int8)], [array([], shape=(0, 0), dtype=int8)], [CudaNdarray([[[[-0.21161591 0.08376592 -0.04271842]
[ 0.19575438 0.19206183 -0.1560..., len = 99, [0]: {[0]: {len = 1, _[0]: {len = 1}}}
storage_map = None
File "/home/kartik/.local/lib/python2.7/site-packages/theano/gof/link.py", line 699, in make_thunk
line: return self.make_all(input_storage=input_storage,
output_storage=output_storage,
storage_map=storage_map)[:3]
locals:
self = <theano.gof.vm.VM_Linker object at 0x7f6ab498c450>
self.make_all = <bound method VM_Linker.make_all of <theano.gof.vm.VM_Linker object at 0x7f6ab498c450>>
input_storage = [[array([[0]], dtype=int32)], [CudaNdarray([[[ 0.]]])], [array([[0]], dtype=int32)], [array([[0]], dtype=int8)], [array([[0]], dtype=int8)], [array([[0]], dtype=int8)], [array([], shape=(0, 0), dtype=int8)], [CudaNdarray([[[[-0.21161591 0.08376592 -0.04271842]
[ 0.19575438 0.19206183 -0.1560..., len = 99, [0]: {[0]: {len = 1, [0]: {len = 1}}}
output_storage = None
storage_map = None
File "/home/kartik/.local/lib/python2.7/site-packages/theano/gof/vm.py", line 1047, in make_all
line: thunks.append(node.op.make_thunk(node,
storage_map,
compute_map,
no_recycling,
impl=impl))
locals:
thunks = [<function rval at 0x7f6ab5efed70>, <function rval at 0x7f6ab8306050>, <function rval at 0x7f6ab5d4f1b8>, <function rval at 0x7f6ab5d4f320>, <function rval at 0x7f6ab5d4fb90>, <function rval at 0x7f6ab5d4fc08>, <function rval at 0x7f6ab5d4f398>, <function rval at 0x7f6ab82a4758>, <function rval a..., len = 183
thunks.append = <built-in method append of list object at 0x7f6aaf520998>
node = CuDNNConvHWBCOp{border_mode='valid'}(GpuContiguous.0, GpuContiguous.0, GpuContiguous.0)
node.op = <cuda_implementation.CuDNNConvHWBCOp.CuDNNConvHWBCOp object at 0x7f6ac30273d0>
node.op.make_thunk = <bound method CuDNNConvHWBCOp.make_thunk of <cuda_implementation.CuDNNConvHWBCOp.CuDNNConvHWBCOp object at 0x7f6ac30273d0>>
storage_map = {b_conv3: [CudaNdarray([ 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0..., len = 733
compute_map = {ScalarFromTensor.0: [False], V1_mdlstm2: [True], b_conv3: [True], Subtensor{::, int64}.0: [False], W1_mdlstm3: [True], Subtensor{::, int64}.0: [False], W2_mdlstm3: [True], MultiDirectionalTwoDLSTMOp.0: [False], y_sizes: [True], Elemwise{int_div,no_inplace}.0: [False], MultiDirectionalTwoDLSTMOp...., len = 733
no_recycling = set([HostFromGpu.0]), len = 1
impl = None
File "/home/kartik/.local/lib/python2.7/site-packages/theano/gof/op.py", line 935, in make_thunk
line: return self.make_c_thunk(node, storage_map, compute_map,
no_recycling)
locals:
self = <cuda_implementation.CuDNNConvHWBCOp.CuDNNConvHWBCOp object at 0x7f6ac30273d0>
self.make_c_thunk = <bound method CuDNNConvHWBCOp.make_c_thunk of <cuda_implementation.CuDNNConvHWBCOp.CuDNNConvHWBCOp object at 0x7f6ac30273d0>>
node = CuDNNConvHWBCOp{border_mode='valid'}(GpuContiguous.0, GpuContiguous.0, GpuContiguous.0)
storage_map = {b_conv3: [CudaNdarray([ 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0..., len = 733
compute_map = {ScalarFromTensor.0: [False], V1_mdlstm2: [True], b_conv3: [True], Subtensor{::, int64}.0: [False], W1_mdlstm3: [True], Subtensor{::, int64}.0: [False], W2_mdlstm3: [True], MultiDirectionalTwoDLSTMOp.0: [False], y_sizes: [True], Elemwise{int_div,no_inplace}.0: [False], MultiDirectionalTwoDLSTMOp...., len = 733
no_recycling = set([HostFromGpu.0]), len = 1
File "/home/kartik/.local/lib/python2.7/site-packages/theano/gof/op.py", line 839, in make_c_thunk
line: outputs = cl.make_thunk(input_storage=node_input_storage,
output_storage=node_output_storage)
locals:
outputs =
cl = <theano.gof.cc.CLinker object at 0x7f6aac699810>
cl.make_thunk = <bound method CLinker.make_thunk of <theano.gof.cc.CLinker object at 0x7f6aac699810>>
input_storage =
node_input_storage = [[None], [None], [None]]
output_storage =
node_output_storage = [[None]]
File "/home/kartik/.local/lib/python2.7/site-packages/theano/gof/cc.py", line 1190, in make_thunk
line: cthunk, in_storage, out_storage, error_storage = self.compile(
input_storage, output_storage, storage_map,
keep_lock=keep_lock)
locals:
cthunk =
in_storage =
out_storage =
error_storage =
self = <theano.gof.cc.CLinker object at 0x7f6aac699810>
self.compile = <bound method CLinker.compile of <theano.gof.cc.CLinker object at 0x7f6aac699810>>
input_storage = [[None], [None], [None]]
output_storage = [[None]]
storage_map = None
keep_lock = False
File "/home/kartik/.local/lib/python2.7/site-packages/theano/gof/cc.py", line 1131, in compile
line: thunk = self.cthunk_factory(error_storage,
input_storage,
output_storage,
storage_map,
keep_lock=keep_lock)
locals:
thunk =
self = <theano.gof.cc.CLinker object at 0x7f6aac699810>
self.cthunk_factory = <bound method CLinker.cthunk_factory of <theano.gof.cc.CLinker object at 0x7f6aac699810>>
error_storage = [None, None, None]
input_storage = ([None], [None], [None])
output_storage = ([None],)
storage_map = None
keep_lock = False
File "/home/kartik/.local/lib/python2.7/site-packages/theano/gof/cc.py", line 1586, in cthunk_factory
line: module = get_module_cache().module_from_key(
key=key, lnk=self, keep_lock=keep_lock)
locals:
module =
get_module_cache = <function get_module_cache at 0x7f6b33b6b140>
module_from_key =
key = (((3, 3), (3,), (3,), (3,), (3,)), ('CLinker.cmodule_key', ('-DCUDA_NDARRAY_CUH=c72d035fdf91890f3b36710688069b2e', '-DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION', '-O3', '-Wno-unused-label', '-Wno-unused-variable', '-Wno-write-strings', '-arch=sm_61', '-fno-math-errno'), ('cublas', 'cuda_ndarray',...
lnk =
self = <theano.gof.cc.CLinker object at 0x7f6aac699810>
keep_lock = False
File "/home/kartik/.local/lib/python2.7/site-packages/theano/gof/cmodule.py", line 1122, in module_from_key
line: src_code = lnk.get_src_code()
locals:
src_code =
lnk = <theano.gof.cc.CLinker object at 0x7f6aac699810>
lnk.get_src_code = <bound method CLinker.get_src_code of <theano.gof.cc.CLinker object at 0x7f6aac699810>>
File "/home/kartik/.local/lib/python2.7/site-packages/theano/gof/cc.py", line 1462, in get_src_code
line: mod = self.get_dynamic_module()
locals:
mod =
self = <theano.gof.cc.CLinker object at 0x7f6aac699810>
self.get_dynamic_module = <bound method CLinker.get_dynamic_module of <theano.gof.cc.CLinker object at 0x7f6aac699810>>
File "/home/kartik/.local/lib/python2.7/site-packages/theano/gof/cc.py", line 1506, in get_dynamic_module
line: self.code_gen()
locals:
self = <theano.gof.cc.CLinker object at 0x7f6aac699810>
self.code_gen = <bound method CLinker.code_gen of <theano.gof.cc.CLinker object at 0x7f6aac699810>>
File "/home/kartik/.local/lib/python2.7/site-packages/theano/gof/cc.py", line 826, in code_gen
line: behavior = op.c_code(node, name, isyms, osyms, sub)
locals:
behavior =
op = <cuda_implementation.CuDNNConvHWBCOp.CuDNNConvHWBCOp object at 0x7f6ac30273d0>
op.c_code = <bound method CuDNNConvHWBCOp.c_code of <cuda_implementation.CuDNNConvHWBCOp.CuDNNConvHWBCOp object at 0x7f6ac30273d0>>
node = CuDNNConvHWBCOp{border_mode='valid'}(<CudaNdarrayType(float32, 4D)>, <CudaNdarrayType(float32, 4D)>, <CudaNdarrayType(float32, vector)>)
name = 'node<<<<HASH_PLACEHOLDER>>>>_0', len = 31
isyms = ['V3', 'V5', 'V7']
osyms = ['V1']
sub = {'fail': '{\n __failure = 9;\n if (!PyErr_Occurred()) {\n PyErr_SetString(PyExc_RuntimeError,\n "Unexpected error in an Op's C code. "\n "No Python exception was set.");\n }\n goto __label_9;}', 'id': 9, 'failure_var': '_f...
File "/home/kartik/new_returnn/cuda_implementation/CuDNNConvHWBCOp.py", line 235, in c_code
line: cudnn_version = theano.sandbox.cuda.dnn.version()[0]/1000
locals:
cudnn_version =
theano = <module 'theano' from '/home/kartik/.local/lib/python2.7/site-packages/theano/init.pyc'>
theano.sandbox = <module 'theano.sandbox' from '/home/kartik/.local/lib/python2.7/site-packages/theano/sandbox/init.pyc'>
theano.sandbox.cuda = <module 'theano.sandbox.cuda' from '/home/kartik/.local/lib/python2.7/site-packages/theano/sandbox/cuda/init.pyc'>
theano.sandbox.cuda.dnn = <module 'theano.sandbox.cuda.dnn' from '/home/kartik/.local/lib/python2.7/site-packages/theano/sandbox/cuda/dnn.pyc'>
theano.sandbox.cuda.dnn.version = <function dnn_version at 0x7f6b2387b0c8>
File "/home/kartik/.local/lib/python2.7/site-packages/theano/sandbox/cuda/init.py", line 424, in dnn_version
line: raise Exception(
"We can't determine the cudnn version as it is not available",
dnn_available.msg)
locals:
Exception = <type 'exceptions.Exception'>
dnn_available = <function dnn_available at 0x7f6b27eb5e60>
dnn_available.msg = "Can not compile with cuDNN. We got this error:\nnvcc warning : The 'compute_20', 'sm_20', and 'sm_21' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning).\n/tmp/try_flags_mfyHIk.c:5:19: fatal error: cudnn.h: No such file or d..., len = 327
Exception: ('The following error happened while compiling the node', CuDNNConvHWBCOp{border_mode='valid'}(GpuContiguous.0, GpuContiguous.0, GpuContiguous.0), '\n', "We can't determine the cudnn version as it is not available", "Can not compile with cuDNN. We got this error:\nnvcc warning : The 'compute_20', 'sm_20', and 'sm_21' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning).\n/tmp/try_flags_mfyHIk.c:5:19: fatal error: cudnn.h: No such file or directory\ncompilation terminated.\n")
Device proc gpuX (gpuZ) died: ProcConnectionDied('recv_bytes EOFError: ',)
Theano flags: compiledir_format=compiledir%(platform)s-%(processor)s-%(python_version)s-%(python_bitwidth)s--dev-gpuZ,device=gpu,force_device=True
EXCEPTION
Traceback (most recent call last):
File "/home/kartik/new_returnn/Device.py", line 332, in startProc
line: self._startProc(*args, **kwargs)
locals:
self = <Device.Device object at 0x7f0dde7dc250>
self._startProc = <bound method Device._startProc of <Device.Device object at 0x7f0dde7dc250>>
args = ('gpuZ',)
kwargs = {}
File "/home/kartik/new_returnn/Device.py", line 386, in _startProc
line: interrupt_main()
locals:
interrupt_main = <function interrupt_main at 0x7f0ddf429398>
File "/home/kartik/new_returnn/Util.py", line 637, in interrupt_main
line: sys.exit(1) # And exit the thread.
locals:
sys = <module 'sys' (built-in)>
sys.exit =
SystemExit: 1
KeyboardInterrupt
EXCEPTION
Traceback (most recent call last):
File "../../../rnn.py", line 532, in main
line: init(commandLineOptions=argv[1:])
locals:
init = <function init at 0x7f0dde7d07d0>
commandLineOptions =
argv = ['../../../rnn.py', 'config_fwd_kd'], _[0]: {len = 15}
File "../../../rnn.py", line 345, in init
line: devices = initDevices()
locals:
devices =
initDevices = <function initDevices at 0x7f0dde7d0410>
File "../../../rnn.py", line 158, in initDevices
line: time.sleep(0.25)
locals:
time = <module 'time' (built-in)>
time.sleep =
KeyboardInterrupt
Quitting

from returnn.

it says

We can't determine the cudnn version as it is not available

Did you properly set up cudnn?
Please have a look at http://deeplearning.net/software/theano/library/sandbox/cuda/dnn.html

from returnn.

Thank you very much for your help. It is functioning now.

from returnn.