Error while runnin IAM demo about returnn HOT 10 CLOSED

You must use float32, not float64. You can set that via THEANO_FLAGS, or theanorc file.

from returnn.

Thank you so much for the quick reply albertz. I have changed the float to 32 and the error is gone but after calculating gradient i get another error regarding to cudnn i guess.I am using cudnn 5 which i am not sure if it is compatible with the RETURNN.here is the error i get:
...........................................................................................................................................................

('converting IAM_lines to', 'features/raw/demo.h5')
features/raw/demo.h5
(0, '/', 3)
RETURNN starting up, version 20180405.225130--git-538ed96-dirty, date/time 2018-04-09-13-42-29 (UTC+0200), pid 18637, cwd /home/arman/returnn/demos/mdlstm/IAM, Python /usr/bin/python
RETURNN command line options: ['config_demo']
faulthandler import error. No module named faulthandler
Theano: 0.9.0 ( in /usr/local/lib/python2.7/dist-packages/theano)
pynvml not available, memory information missing
WARNING (theano.sandbox.cuda): The cuda backend is deprecated and will be removed in the next release (v0.10). Please switch to the gpuarray backend. You can get more information about how to switch at this URL:
https://github.com/Theano/Theano/wiki/Converting-to-the-new-gpu-back-end%28gpuarray%29

Using gpu device 0: GeForce GTX 960M (CNMeM is disabled, cuDNN Mixed dnn version. The header is from one version, but we link with a different version (5110, 6021))
Device gpuX proc starting up, pid 18663
Device gpuX proc: THEANO_FLAGS = 'compiledir_format=compiledir_%(platform)s-%(processor)s-%(python_version)s-%(python_bitwidth)s--dev-gpuZ,device=gpu,force_device=True'
faulthandler import error. No module named faulthandler
Device train-network: Used data keys: ['classes', 'data', 'sizes']
using adam with nag and momentum schedule
Device gpuX proc exception: ('The following error happened while compiling the node', CuDNNConvHWBCOp{border_mode='valid'}(GpuContiguous.0, GpuContiguous.0, GpuContiguous.0), '\n', "We can't determine the cudnn version as it is not available", 'Mixed dnn version. The header is from one version, but we link with a different version (5110, 6021)')
Unhandled exception <type 'exceptions.Exception'> in thread <_MainThread(MainThread, started 140595116275456)>, proc 18663.

Thread current, main, <_MainThread(MainThread, started 140595116275456)>:
(Excluded thread.)

That were all threads.
EXCEPTION
Traceback (most recent call last):
File "/home/arman/returnn/Device.py", line 1027, in process
line: self.process_inner(device, config, self.update_specs, asyncTask)
locals:
self = <Device.Device object at 0x7fdeb0064c90>
self.process_inner = <bound method Device.process_inner of <Device.Device object at 0x7fdeb0064c90>>
device = 'gpuX'
config = <Config.Config instance at 0x7fdeb007b488>
self.update_specs = {'layers': [], 'block_size': 0, 'update_params': {}, 'update_rule': 'global'}
asyncTask = <TaskSystem.AsyncTask instance at 0x7fded5adb680>
File "/home/arman/returnn/Device.py", line 1080, in process_inner
line: self.initialize(config, update_specs=update_specs)
locals:
self = <Device.Device object at 0x7fdeb0064c90>
self.initialize = <bound method Device.initialize of <Device.Device object at 0x7fdeb0064c90>>
config = <Config.Config instance at 0x7fdeb007b488>
update_specs = {'layers': [], 'block_size': 0, 'update_params': {}, 'update_rule': 'global'}
File "/home/arman/returnn/Device.py", line 592, in initialize
line: self.trainer = theano.function(inputs=[self.block_start, self.block_end],
outputs=outputs,
givens=train_givens,
updates=self.updater.getUpdateList(),
on_unused_input=config.value('theano_on_unused_input', 'ignore'),
no_default_updates=exclude,
name="train_and_updater")
locals:
self = <Device.Device object at 0x7fdeb0064c90>
self.trainer = !AttributeError: 'Device' object has no attribute 'trainer'
theano = <module 'theano' from '/usr/local/lib/python2.7/dist-packages/theano/init.pyc'>
theano.function = <function function at 0x7fdec2091410>
inputs =
self.block_start = <TensorType(int64, scalar)>
self.block_end = <TensorType(int64, scalar)>
outputs = [Sum{acc_dtype=float64}.0, Sum{axis=[0], acc_dtype=float64}.0]
givens =
train_givens = [(y_classes, Subtensor{::, int64:int64:}.0), (x, Subtensor{::, int64:int64:}.0), (y_sizes, Subtensor{::, int64:int64:}.0), (j_classes, Subtensor{::, int64:int64:}.0), (i, Subtensor{::, int64:int64:}.0), (j_sizes, Subtensor{::, int64:int64:}.0), (TensorConstant{0}, epoch_var), (tags, tags_var)], len = 8
updates =
self.updater = <Updater.Updater instance at 0x7fde5846c4d0>
self.updater.getUpdateList = <bound method Updater.getUpdateList of <Updater.Updater instance at 0x7fde5846c4d0>>
on_unused_input =
config = <Config.Config instance at 0x7fdeb007b488>
config.value = <bound method Config.value of <Config.Config instance at 0x7fdeb007b488>>
no_default_updates =
exclude = []
name =
File "/usr/local/lib/python2.7/dist-packages/theano/compile/function.py", line 326, in function
line: fn = pfunc(params=inputs,
outputs=outputs,
mode=mode,
updates=updates,
givens=givens,
no_default_updates=no_default_updates,
accept_inplace=accept_inplace, name=name,
rebuild_strict=rebuild_strict,
allow_input_downcast=allow_input_downcast,
on_unused_input=on_unused_input,
profile=profile,
output_keys=output_keys)
locals:
fn =
pfunc = <function pfunc at 0x7fdec20f12a8>
params =
inputs = [<TensorType(int64, scalar)>, <TensorType(int64, scalar)>]
outputs = [Sum{acc_dtype=float64}.0, Sum{axis=[0], acc_dtype=float64}.0]
mode = None
updates = [(counter, Elemwise{add,no_inplace}.0), (prev_epoch, TensorConstant{0}), (updater_epoch, Elemwise{switch,no_inplace}.0), (momemtum_cache, Elemwise{mul,no_inplace}.0), (nadam_m_b_conv4, Elemwise{add,no_inplace}.0), (nadam_v_b_conv4, Elemwise{add,no_inplace}.0), (momemtum_cache, Elemwise{mul,no_inp..., len = 372
givens = [(y_classes, Subtensor{::, int64:int64:}.0), (x, Subtensor{::, int64:int64:}.0), (y_sizes, Subtensor{::, int64:int64:}.0), (j_classes, Subtensor{::, int64:int64:}.0), (i, Subtensor{::, int64:int64:}.0), (j_sizes, Subtensor{::, int64:int64:}.0), (TensorConstant{0}, epoch_var), (tags, tags_var)], len = 8
no_default_updates = []
accept_inplace = False
name = 'train_and_updater', len = 17
rebuild_strict = True
allow_input_downcast = None
on_unused_input = 'ignore', len = 6
profile = None
output_keys = None
File "/usr/local/lib/python2.7/dist-packages/theano/compile/pfunc.py", line 486, in pfunc
line: return orig_function(inputs, cloned_outputs, mode,
accept_inplace=accept_inplace, name=name,
profile=profile, on_unused_input=on_unused_input,
output_keys=output_keys)
locals:
orig_function = <function orig_function at 0x7fdec20dbde8>
inputs = [In(<TensorType(int64, scalar)>), In(<TensorType(int64, scalar)>), In(y_classes), In(y_data), In(y_sizes), In(j_classes), In(j_data), In(j_sizes), In(epoch_var), In(tags_var), In(b_output -> GpuFromHost.0), In(<CudaNdarrayType(float32, vector)> -> GPU_mrg_uniform{CudaNdarrayType(float32, vector),..., len = 389
cloned_outputs = [Sum{acc_dtype=float64}.0, Sum{axis=[0], acc_dtype=float64}.0]
mode = None
accept_inplace = False
name = 'train_and_updater', len = 17
profile = None
on_unused_input = 'ignore', len = 6
output_keys = None
File "/usr/local/lib/python2.7/dist-packages/theano/compile/function_module.py", line 1795, in orig_function
line: fn = Maker(inputs,
outputs,
mode,
accept_inplace=accept_inplace,
profile=profile,
on_unused_input=on_unused_input,
output_keys=output_keys).create(
defaults)
locals:
fn =
Maker = <class 'theano.compile.function_module.FunctionMaker'>
inputs = [In(<TensorType(int64, scalar)>), In(<TensorType(int64, scalar)>), In(y_classes), In(y_data), In(y_sizes), In(j_classes), In(j_data), In(j_sizes), In(epoch_var), In(tags_var), In(b_output -> GpuFromHost.0), In(<CudaNdarrayType(float32, vector)> -> GPU_mrg_uniform{CudaNdarrayType(float32, vector),..., len = 389
outputs = [Out(Sum{acc_dtype=float64}.0,False), Out(Sum{axis=[0], acc_dtype=float64}.0,False)]
mode = <theano.compile.mode.Mode object at 0x7fdeb3c38890>
accept_inplace = False
profile = None
on_unused_input = 'ignore', len = 6
output_keys = None
create =
defaults = [None, None, <array([[0]], dtype=int32)>, <CudaNdarray([[[0.]]])>, <array([[0]], dtype=int32)>, <array([[0]], dtype=int8)>, <array([[0]], dtype=int8)>, <array([[0]], dtype=int8)>, <array(0, dtype=int32)>, <array([], shape=(0, 0), dtype=int8)>, <CudaNdarray([-4.3820267 -4.3820267 -4.3820267 -4.382..., len = 389
File "/usr/local/lib/python2.7/dist-packages/theano/compile/function_module.py", line 1661, in create
line: _fn, _i, _o = self.linker.make_thunk(
input_storage=input_storage_lists, storage_map=storage_map)
locals:
_fn =
_i =
o =
self = <theano.compile.function_module.FunctionMaker object at 0x7fde5a259210>
self.linker = <theano.gof.vm.VM_Linker object at 0x7fde334cd410>
self.linker.make_thunk = <bound method VM_Linker.make_thunk of <theano.gof.vm.VM_Linker object at 0x7fde334cd410>>
input_storage = [None, None, <array([[0]], dtype=int32)>, <CudaNdarray([[[0.]]])>, <array([[0]], dtype=int32)>, <array([[0]], dtype=int8)>, <array([[0]], dtype=int8)>, <array([[0]], dtype=int8)>, <array(0, dtype=int32)>, <array([], shape=(0, 0), dtype=int8)>, <CudaNdarray([-4.3820267 -4.3820267 -4.3820267 -4.382..., len = 389
input_storage_lists = [[None], [None], [array([[0]], dtype=int32)], [CudaNdarray([[[0.]]])], [array([[0]], dtype=int32)], [array([[0]], dtype=int8)], [array([[0]], dtype=int8)], [array([[0]], dtype=int8)], [array(0, dtype=int32)], [array([], shape=(0, 0), dtype=int8)], [CudaNdarray([-4.3820267 -4.3820267 -4.3820267 -4..., len = 389
storage_map = None
File "/usr/local/lib/python2.7/dist-packages/theano/gof/link.py", line 699, in make_thunk
line: return self.make_all(input_storage=input_storage,
output_storage=output_storage,
storage_map=storage_map)[:3]
locals:
self = <theano.gof.vm.VM_Linker object at 0x7fde334cd410>
self.make_all = <bound method VM_Linker.make_all of <theano.gof.vm.VM_Linker object at 0x7fde334cd410>>
input_storage = [[None], [None], [array([[0]], dtype=int32)], [CudaNdarray([[[0.]]])], [array([[0]], dtype=int32)], [array([[0]], dtype=int8)], [array([[0]], dtype=int8)], [array([[0]], dtype=int8)], [array(0, dtype=int32)], [array([], shape=(0, 0), dtype=int8)], [CudaNdarray([-4.3820267 -4.3820267 -4.3820267 -4..., len = 389
output_storage = None
storage_map = None
File "/usr/local/lib/python2.7/dist-packages/theano/gof/vm.py", line 1047, in make_all
line: thunks.append(node.op.make_thunk(node,
storage_map,
compute_map,
no_recycling,
impl=impl))
locals:
thunks = [<function rval at 0x7fde39dafd70>, <function rval at 0x7fde39daf410>, <function rval at 0x7fde39daf9b0>, <function rval at 0x7fde39daf5f0>, <function rval at 0x7fde39daf848>, <function rval at 0x7fde39daff50>, <function rval at 0x7fde39daf578>, <function rval at 0x7fde39dafb90>, <function rval a..., len = 1031
thunks.append = <built-in method append of list object at 0x7fde3782b830>
node = CuDNNConvHWBCOp{border_mode='valid'}(GpuContiguous.0, GpuContiguous.0, GpuContiguous.0)
node.op = <cuda_implementation.CuDNNConvHWBCOp.CuDNNConvHWBCOp object at 0x7fdeb012c890>
node.op.make_thunk = <bound method CuDNNConvHWBCOp.make_thunk of <cuda_implementation.CuDNNConvHWBCOp.CuDNNConvHWBCOp object at 0x7fdeb012c890>>
storage_map = {MultiDirectionalTwoDLSTMOpGrad{inplace=True}.11: [None], GpuDimShuffle{x,x}.0: [None], GpuElemwise{Composite{((i0 * Switch(i1, i2, i3)) + Switch(i4, i5, (i6 * i7)))}}[(0, 3)].0: [None], if{shape}.3: [None], Elemwise{Composite{Switch(GE(Composite{Switch(LT(i0, i1), i2, i0)}(Composite{Switch(i0, (..., len = 3394
compute_map = {MultiDirectionalTwoDLSTMOpGrad{inplace=True}.11: [False], Elemwise{Composite{Switch(i0, i1, maximum(minimum((i2 + (i3 - i4)), i5), i6))}}.0: [False], GpuElemwise{Composite{((i0 * Switch(i1, i2, i3)) + Switch(i4, i5, (i6 * i7)))}}[(0, 3)].0: [False], GpuDimShuffle{x,x,x,0}.0: [False], GpuFromHost..., len = 3394
no_recycling = set([GpuDimShuffle{x,x}.0, GpuElemwise{Composite{((i0 * Switch(i1, sqr(i2), i3)) + (i4 * sqr(i2)))},no_inplace}.0, GpuElemwise{Composite{((i0 * Switch(i1, sqr(i2), i3)) + (i4 * sqr(i2)))}}[(0, 3)].0, GpuFromHost.0, GpuElemwise{mul,no_inplace}.0, GpuDimShuffle{x,x}.0, GpuElemwise{Composite{(i0 + S..., len = 530
impl = None
File "/usr/local/lib/python2.7/dist-packages/theano/gof/op.py", line 935, in make_thunk
line: return self.make_c_thunk(node, storage_map, compute_map,
no_recycling)
locals:
self = <cuda_implementation.CuDNNConvHWBCOp.CuDNNConvHWBCOp object at 0x7fdeb012c890>
self.make_c_thunk = <bound method CuDNNConvHWBCOp.make_c_thunk of <cuda_implementation.CuDNNConvHWBCOp.CuDNNConvHWBCOp object at 0x7fdeb012c890>>
node = CuDNNConvHWBCOp{border_mode='valid'}(GpuContiguous.0, GpuContiguous.0, GpuContiguous.0)
storage_map = {MultiDirectionalTwoDLSTMOpGrad{inplace=True}.11: [None], GpuDimShuffle{x,x}.0: [None], GpuElemwise{Composite{((i0 * Switch(i1, i2, i3)) + Switch(i4, i5, (i6 * i7)))}}[(0, 3)].0: [None], if{shape}.3: [None], Elemwise{Composite{Switch(GE(Composite{Switch(LT(i0, i1), i2, i0)}(Composite{Switch(i0, (..., len = 3394
compute_map = {MultiDirectionalTwoDLSTMOpGrad{inplace=True}.11: [False], Elemwise{Composite{Switch(i0, i1, maximum(minimum((i2 + (i3 - i4)), i5), i6))}}.0: [False], GpuElemwise{Composite{((i0 * Switch(i1, i2, i3)) + Switch(i4, i5, (i6 * i7)))}}[(0, 3)].0: [False], GpuDimShuffle{x,x,x,0}.0: [False], GpuFromHost..., len = 3394
no_recycling = set([GpuDimShuffle{x,x}.0, GpuElemwise{Composite{((i0 * Switch(i1, sqr(i2), i3)) + (i4 * sqr(i2)))},no_inplace}.0, GpuElemwise{Composite{((i0 * Switch(i1, sqr(i2), i3)) + (i4 * sqr(i2)))}}[(0, 3)].0, GpuFromHost.0, GpuElemwise{mul,no_inplace}.0, GpuDimShuffle{x,x}.0, GpuElemwise{Composite{(i0 + S..., len = 530
File "/usr/local/lib/python2.7/dist-packages/theano/gof/op.py", line 839, in make_c_thunk
line: outputs = cl.make_thunk(input_storage=node_input_storage,
output_storage=node_output_storage)
locals:
outputs =
cl = <theano.gof.cc.CLinker object at 0x7fde266ec0d0>
cl.make_thunk = <bound method CLinker.make_thunk of <theano.gof.cc.CLinker object at 0x7fde266ec0d0>>
input_storage =
node_input_storage = [[None], [None], [None]]
output_storage =
node_output_storage = [[None]]
File "/usr/local/lib/python2.7/dist-packages/theano/gof/cc.py", line 1190, in make_thunk
line: cthunk, in_storage, out_storage, error_storage = self.compile(
input_storage, output_storage, storage_map,
keep_lock=keep_lock)
locals:
cthunk =
in_storage =
out_storage =
error_storage =
self = <theano.gof.cc.CLinker object at 0x7fde266ec0d0>
self.compile = <bound method CLinker.compile of <theano.gof.cc.CLinker object at 0x7fde266ec0d0>>
input_storage = [[None], [None], [None]]
output_storage = [[None]]
storage_map = None
keep_lock = False
File "/usr/local/lib/python2.7/dist-packages/theano/gof/cc.py", line 1131, in compile
line: thunk = self.cthunk_factory(error_storage,
input_storage,
output_storage,
storage_map,
keep_lock=keep_lock)
locals:
thunk =
self = <theano.gof.cc.CLinker object at 0x7fde266ec0d0>
self.cthunk_factory = <bound method CLinker.cthunk_factory of <theano.gof.cc.CLinker object at 0x7fde266ec0d0>>
error_storage = [None, None, None]
input_storage = ([None], [None], [None])
output_storage = ([None],)
storage_map = None
keep_lock = False
File "/usr/local/lib/python2.7/dist-packages/theano/gof/cc.py", line 1586, in cthunk_factory
line: module = get_module_cache().module_from_key(
key=key, lnk=self, keep_lock=keep_lock)
locals:
module =
get_module_cache = <function get_module_cache at 0x7fded553d0c8>
module_from_key =
key = (((3, 3), (3,), (3,), (3,), (3,)), ('CLinker.cmodule_key', ('-DCUDA_NDARRAY_CUH=c72d035fdf91890f3b36710688069b2e', '-DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION', '-O3', '-Wno-unused-label', '-Wno-unused-variable', '-Wno-write-strings', '-arch=sm_50', '-fno-math-errno'), ('cublas', 'cuda_ndarray',...
lnk =
self = <theano.gof.cc.CLinker object at 0x7fde266ec0d0>
keep_lock = False
File "/usr/local/lib/python2.7/dist-packages/theano/gof/cmodule.py", line 1122, in module_from_key
line: src_code = lnk.get_src_code()
locals:
src_code =
lnk = <theano.gof.cc.CLinker object at 0x7fde266ec0d0>
lnk.get_src_code = <bound method CLinker.get_src_code of <theano.gof.cc.CLinker object at 0x7fde266ec0d0>>
File "/usr/local/lib/python2.7/dist-packages/theano/gof/cc.py", line 1462, in get_src_code
line: mod = self.get_dynamic_module()
locals:
mod =
self = <theano.gof.cc.CLinker object at 0x7fde266ec0d0>
self.get_dynamic_module = <bound method CLinker.get_dynamic_module of <theano.gof.cc.CLinker object at 0x7fde266ec0d0>>
File "/usr/local/lib/python2.7/dist-packages/theano/gof/cc.py", line 1506, in get_dynamic_module
line: self.code_gen()
locals:
self = <theano.gof.cc.CLinker object at 0x7fde266ec0d0>
self.code_gen = <bound method CLinker.code_gen of <theano.gof.cc.CLinker object at 0x7fde266ec0d0>>
File "/usr/local/lib/python2.7/dist-packages/theano/gof/cc.py", line 826, in code_gen
line: behavior = op.c_code(node, name, isyms, osyms, sub)
locals:
behavior =
op = <cuda_implementation.CuDNNConvHWBCOp.CuDNNConvHWBCOp object at 0x7fdeb012c890>
op.c_code = <bound method CuDNNConvHWBCOp.c_code of <cuda_implementation.CuDNNConvHWBCOp.CuDNNConvHWBCOp object at 0x7fdeb012c890>>
node = CuDNNConvHWBCOp{border_mode='valid'}(<CudaNdarrayType(float32, 4D)>, <CudaNdarrayType(float32, 4D)>, <CudaNdarrayType(float32, vector)>)
name = 'node<<<<HASH_PLACEHOLDER>>>>_0', len = 31
isyms = ['V3', 'V5', 'V7']
osyms = ['V1']
sub = {'fail': '{\n __failure = 9;\n if (!PyErr_Occurred()) {\n PyErr_SetString(PyExc_RuntimeError,\n "Unexpected error in an Op's C code. "\n "No Python exception was set.");\n }\n goto __label_9;}', 'id': 9, 'failure_var': '_f...
File "/home/arman/returnn/cuda_implementation/CuDNNConvHWBCOp.py", line 241, in c_code
line: cudnn_version = theano.sandbox.cuda.dnn.version()[0]/1000
locals:
cudnn_version =
theano = <module 'theano' from '/usr/local/lib/python2.7/dist-packages/theano/init.pyc'>
theano.sandbox = <module 'theano.sandbox' from '/usr/local/lib/python2.7/dist-packages/theano/sandbox/init.pyc'>
theano.sandbox.cuda = <module 'theano.sandbox.cuda' from '/usr/local/lib/python2.7/dist-packages/theano/sandbox/cuda/init.pyc'>
theano.sandbox.cuda.dnn = <module 'theano.sandbox.cuda.dnn' from '/usr/local/lib/python2.7/dist-packages/theano/sandbox/cuda/dnn.pyc'>
theano.sandbox.cuda.dnn.version = <function dnn_version at 0x7fdeb3e61938>
File "/usr/local/lib/python2.7/dist-packages/theano/sandbox/cuda/init.py", line 424, in dnn_version
line: raise Exception(
"We can't determine the cudnn version as it is not available",
dnn_available.msg)
locals:
Exception = <type 'exceptions.Exception'>
dnn_available = <function dnn_available at 0x7fdeb7f01b18>
dnn_available.msg = 'Mixed dnn version. The header is from one version, but we link with a different version (5110, 6021)', len = 100
Exception: ('The following error happened while compiling the node', CuDNNConvHWBCOp{border_mode='valid'}(GpuContiguous.0, GpuContiguous.0, GpuContiguous.0), '\n', "We can't determine the cudnn version as it is not available", 'Mixed dnn version. The header is from one version, but we link with a different version (5110, 6021)')
Device proc gpuX (gpuZ) died: ProcConnectionDied('recv_bytes EOFError: ',)
Theano flags: compiledir_format=compiledir%(platform)s-%(processor)s-%(python_version)s-%(python_bitwidth)s--dev-gpuZ,device=gpu,force_device=True
EXCEPTION
Traceback (most recent call last):
File "/home/arman/returnn/Device.py", line 347, in startProc
line: self._startProc(*args, **kwargs)
locals:
self = <Device.Device object at 0x7fba1e207b10>
self._startProc = <bound method Device._startProc of <Device.Device object at 0x7fba1e207b10>>
args = ('gpuZ',)
kwargs = {}
File "/home/arman/returnn/Device.py", line 401, in _startProc
line: interrupt_main()
locals:
interrupt_main = <function interrupt_main at 0x7fba1ee795f0>
File "/home/arman/returnn/Util.py", line 665, in interrupt_main
line: sys.exit(1) # And exit the thread.
locals:
sys = <module 'sys' (built-in)>
sys.exit =
SystemExit: 1
KeyboardInterrupt
EXCEPTION
Traceback (most recent call last):
File "../../../rnn.py", line 546, in main
line: init(commandLineOptions=argv[1:])
locals:
init = <function init at 0x7fba1e1fda28>
commandLineOptions =
argv = ['../../../rnn.py', 'config_demo'], _[0]: {len = 15}
File "../../../rnn.py", line 343, in init
line: devices = initDevices()
locals:
devices =
initDevices = <function initDevices at 0x7fba1e1fd668>
File "../../../rnn.py", line 154, in initDevices
line: time.sleep(0.25)
locals:
time = <module 'time' (built-in)>
time.sleep =
KeyboardInterrupt
Quitting

from returnn.

my cudnn version is 5110

from returnn.

'Mixed dnn version. The header is from one version, but we link with a different version (5110, 6021)'

^ that's your problem. reinstall the headers and library of one of the cudnn versions and this should be resolved.

from returnn.

Ok thank you. I will try to reinstall and see the result.

from returnn.

Ok i just wanted to give an update. seems like reinstalling cudnn solved the problem and training begings.

Thanks

from returnn.

Guys I have a question. Since im using my laptop for training i wanted to know if there is anyways to stop training and continue later. I am training the whole IAM dataset now i wanna know if i can resume training but i can not find any models saved in models folder.

from returnn.

Yes you can. It stores the model. You can configure that. See the documentation.

from returnn.

Hi albertz

Thanks for the reply. I have read the config_real and it is supposed to save models in model folder, Although it is saving the mdlstm_real in log folder which carries the information of training but still Model folder is empty. While i was training the demo of IAM i could see the saved models in Model folder.Thats why i am confused. I have also checked out your website for documentation but did not find any answers. Can you give me a reference which i can read and get a clue?

Thank you

from returnn.

See here, and the code.
Related options are model and save_interval.
You should also see related information when and where a model is saved in the log output.

from returnn.