Comments (10)
pvoigtlaender
commented on July 17, 2024
the problem is that the file was written for python2.
I converted it to python3 using
2to3 -w create_iam_dataset.py
and pushed the new file. So just pull and try again.
Please confirm if it's now working, so we can close the issue
from returnn.
pedro-dreyer
commented on July 17, 2024
A new error occurs:
deep@deep-System-Product-Name ~/Documents/Deep Learning Libraries/returnn/demos/mdlstm/IAM $ ./go.sh
converting IAM_lines to features/raw/demo.h5
features/raw/demo.h5
0 / 3
CRNN starting up, version 20171010.211508--git-e19cbd3-dirty, pid 25112, cwd /home/deep/Documents/Deep Learning Libraries/returnn/demos/mdlstm/IAM
CRNN command line options: ['config_demo']
Theano: 0.9.0.dev-c697eeab84... (<site-package> in /home/deep/anaconda3/lib/python3.6/site-packages/theano)
pynvml not available, memory information missing
WARNING (theano.sandbox.cuda): The cuda backend is deprecated and will be removed in the next release (v0.10). Please switch to the gpuarray backend. You can get more information about how to switch at this URL:
https://github.com/Theano/Theano/wiki/Converting-to-the-new-gpu-back-end%28gpuarray%29
Using gpu device 0: TITAN Xp (CNMeM is enabled with initial size: 95.0% of memory, cuDNN 5110)
Device gpuX proc starting up, pid 25140
Device gpuX proc: THEANO_FLAGS = 'compiledir_format=compiledir_%(platform)s-%(processor)s-%(python_version)s-%(python_bitwidth)s--dev-gpuZ,device=gpu,force_device=True'
Device gpuX proc exception: ('Shape must be integers', MakeVector{dtype='float32'}.0, 'float32')
Unhandled exception <class 'TypeError'> in thread <_MainThread(MainThread, started 140714657150720)>, proc 25140.
Thread current, main, <_MainThread(MainThread, started 140714657150720)>:
(Excluded thread.)
That were all threads.
EXCEPTION
Traceback (most recent call last):
File "/home/deep/Documents/Deep Learning Libraries/returnn/Device.py", line 977, in process
line: self.process_inner(device, config, self.update_specs, asyncTask)
locals:
self = <local> <Device.Device object at 0x7ffaaa1738d0>
self.process_inner = <local> <bound method Device.process_inner of <Device.Device object at 0x7ffaaa1738d0>>
device = <local> 'gpuX'
config = <local> <Config.Config object at 0x7ffa696b6518>
self.update_specs = <local> {'update_rule': 'global', 'update_params': {}, 'layers': [], 'block_size': 0}
asyncTask = <local> <TaskSystem.AsyncTask object at 0x7ffa9514bf28>
File "/home/deep/Documents/Deep Learning Libraries/returnn/Device.py", line 1030, in process_inner
line: self.initialize(config, update_specs=update_specs)
locals:
self = <local> <Device.Device object at 0x7ffaaa1738d0>
self.initialize = <local> <bound method Device.initialize of <Device.Device object at 0x7ffaaa1738d0>>
config = <local> <Config.Config object at 0x7ffa696b6518>
update_specs = <local> {'update_rule': 'global', 'update_params': {}, 'layers': [], 'block_size': 0}
File "/home/deep/Documents/Deep Learning Libraries/returnn/Device.py", line 439, in initialize
line: self.trainnet = LayerNetwork.from_config_topology(config, train_flag=True, eval_flag=False)
locals:
self = <local> <Device.Device object at 0x7ffaaa1738d0>
self.trainnet = <local> !AttributeError: 'Device' object has no attribute 'trainnet'
LayerNetwork = <global> <class 'Network.LayerNetwork'>
LayerNetwork.from_config_topology = <global> <bound method LayerNetwork.from_config_topology of <class 'Network.LayerNetwork'>>
config = <local> <Config.Config object at 0x7ffa696b6518>
train_flag = <not found>
eval_flag = <local> False
File "/home/deep/Documents/Deep Learning Libraries/returnn/Network.py", line 119, in from_config_topology
line: return cls.from_json_and_config(json_content, config, mask=mask, **kwargs)
locals:
cls = <local> <class 'Network.LayerNetwork'>
cls.from_json_and_config = <local> <bound method LayerNetwork.from_json_and_config of <class 'Network.LayerNetwork'>>
json_content = <local> {'classes_source': {'class': 'source', 'data_key': 'sizes', 'from': [], 'n_out': 2, 'dtype': 'int32'}, '1Dto2D': {'class': '1Dto2D', 'from': ['data', 'classes_source']}, 'conv0': {'class': 'conv2', 'n_features': 15, 'filter': [3, 3], 'pool_size': [2, 2], 'from': ['1Dto2D']}, 'mdlstm0': {'class': ..., len = 13
config = <local> <Config.Config object at 0x7ffa696b6518>
mask = <local> None
kwargs = <local> {'train_flag': True, 'eval_flag': False}
File "/home/deep/Documents/Deep Learning Libraries/returnn/Network.py", line 204, in from_json_and_config
line: network = cls.from_json(json_content, **dict_joined(kwargs, cls.init_args_from_config(config)))
locals:
network = <not found>
cls = <local> <class 'Network.LayerNetwork'>
cls.from_json = <local> <bound method LayerNetwork.from_json of <class 'Network.LayerNetwork'>>
json_content = <local> {'classes_source': {'class': 'source', 'data_key': 'sizes', 'from': [], 'n_out': 2, 'dtype': 'int32'}, '1Dto2D': {'class': '1Dto2D', 'from': ['data', 'classes_source']}, 'conv0': {'class': 'conv2', 'n_features': 15, 'filter': [3, 3], 'pool_size': [2, 2], 'from': ['1Dto2D']}, 'mdlstm0': {'class': ..., len = 13
dict_joined = <global> <function dict_joined at 0x7ffa7407de18>
kwargs = <local> {'mask': None, 'train_flag': True, 'eval_flag': False}
cls.init_args_from_config = <local> <bound method LayerNetwork.init_args_from_config of <class 'Network.LayerNetwork'>>
config = <local> <Config.Config object at 0x7ffa696b6518>
File "/home/deep/Documents/Deep Learning Libraries/returnn/Network.py", line 422, in from_json
line: traverse(json_content, layer_name, trg, index)
locals:
traverse = <local> <function LayerNetwork.from_json.<locals>.traverse at 0x7ffa6912bd08>
json_content = <local> {'classes_source': {'class': 'source', 'data_key': 'sizes', 'from': [], 'n_out': 2, 'dtype': 'int32'}, '1Dto2D': {'class': '1Dto2D', 'from': ['data', 'classes_source']}, 'conv0': {'class': 'conv2', 'n_features': 15, 'filter': [3, 3], 'pool_size': [2, 2], 'from': ['1Dto2D']}, 'mdlstm0': {'class': ..., len = 13
layer_name = <local> 'output', len = 6
trg = <local> 'classes', len = 7
index = <local> j_classes
File "/home/deep/Documents/Deep Learning Libraries/returnn/Network.py", line 350, in traverse
line: index = traverse(content, prev, target, index)
locals:
index = <local> j_classes
traverse = <local> <function LayerNetwork.from_json.<locals>.traverse at 0x7ffa6912bd08>
content = <local> {'classes_source': {'class': 'source', 'data_key': 'sizes', 'from': [], 'n_out': 2, 'dtype': 'int32'}, '1Dto2D': {'class': '1Dto2D', 'from': ['data', 'classes_source']}, 'conv0': {'class': 'conv2', 'n_features': 15, 'filter': [3, 3], 'pool_size': [2, 2], 'from': ['1Dto2D']}, 'mdlstm0': {'class': ..., len = 13
prev = <local> 'mdlstm4', len = 7
target = <local> 'classes', len = 7
File "/home/deep/Documents/Deep Learning Libraries/returnn/Network.py", line 350, in traverse
line: index = traverse(content, prev, target, index)
locals:
index = <local> j_classes
traverse = <local> <function LayerNetwork.from_json.<locals>.traverse at 0x7ffa6912bd08>
content = <local> {'classes_source': {'class': 'source', 'data_key': 'sizes', 'from': [], 'n_out': 2, 'dtype': 'int32'}, '1Dto2D': {'class': '1Dto2D', 'from': ['data', 'classes_source']}, 'conv0': {'class': 'conv2', 'n_features': 15, 'filter': [3, 3], 'pool_size': [2, 2], 'from': ['1Dto2D']}, 'mdlstm0': {'class': ..., len = 13
prev = <local> 'conv4'
target = <local> 'classes', len = 7
File "/home/deep/Documents/Deep Learning Libraries/returnn/Network.py", line 350, in traverse
line: index = traverse(content, prev, target, index)
locals:
index = <local> j_classes
traverse = <local> <function LayerNetwork.from_json.<locals>.traverse at 0x7ffa6912bd08>
content = <local> {'classes_source': {'class': 'source', 'data_key': 'sizes', 'from': [], 'n_out': 2, 'dtype': 'int32'}, '1Dto2D': {'class': '1Dto2D', 'from': ['data', 'classes_source']}, 'conv0': {'class': 'conv2', 'n_features': 15, 'filter': [3, 3], 'pool_size': [2, 2], 'from': ['1Dto2D']}, 'mdlstm0': {'class': ..., len = 13
prev = <local> 'mdlstm3', len = 7
target = <local> 'classes', len = 7
File "/home/deep/Documents/Deep Learning Libraries/returnn/Network.py", line 350, in traverse
line: index = traverse(content, prev, target, index)
locals:
index = <local> j_classes
traverse = <local> <function LayerNetwork.from_json.<locals>.traverse at 0x7ffa6912bd08>
content = <local> {'classes_source': {'class': 'source', 'data_key': 'sizes', 'from': [], 'n_out': 2, 'dtype': 'int32'}, '1Dto2D': {'class': '1Dto2D', 'from': ['data', 'classes_source']}, 'conv0': {'class': 'conv2', 'n_features': 15, 'filter': [3, 3], 'pool_size': [2, 2], 'from': ['1Dto2D']}, 'mdlstm0': {'class': ..., len = 13
prev = <local> 'conv3'
target = <local> 'classes', len = 7
File "/home/deep/Documents/Deep Learning Libraries/returnn/Network.py", line 350, in traverse
line: index = traverse(content, prev, target, index)
locals:
index = <local> j_classes
traverse = <local> <function LayerNetwork.from_json.<locals>.traverse at 0x7ffa6912bd08>
content = <local> {'classes_source': {'class': 'source', 'data_key': 'sizes', 'from': [], 'n_out': 2, 'dtype': 'int32'}, '1Dto2D': {'class': '1Dto2D', 'from': ['data', 'classes_source']}, 'conv0': {'class': 'conv2', 'n_features': 15, 'filter': [3, 3], 'pool_size': [2, 2], 'from': ['1Dto2D']}, 'mdlstm0': {'class': ..., len = 13
prev = <local> 'mdlstm2', len = 7
target = <local> 'classes', len = 7
File "/home/deep/Documents/Deep Learning Libraries/returnn/Network.py", line 350, in traverse
line: index = traverse(content, prev, target, index)
locals:
index = <local> j_classes
traverse = <local> <function LayerNetwork.from_json.<locals>.traverse at 0x7ffa6912bd08>
content = <local> {'classes_source': {'class': 'source', 'data_key': 'sizes', 'from': [], 'n_out': 2, 'dtype': 'int32'}, '1Dto2D': {'class': '1Dto2D', 'from': ['data', 'classes_source']}, 'conv0': {'class': 'conv2', 'n_features': 15, 'filter': [3, 3], 'pool_size': [2, 2], 'from': ['1Dto2D']}, 'mdlstm0': {'class': ..., len = 13
prev = <local> 'conv2'
target = <local> 'classes', len = 7
File "/home/deep/Documents/Deep Learning Libraries/returnn/Network.py", line 350, in traverse
line: index = traverse(content, prev, target, index)
locals:
index = <local> j_classes
traverse = <local> <function LayerNetwork.from_json.<locals>.traverse at 0x7ffa6912bd08>
content = <local> {'classes_source': {'class': 'source', 'data_key': 'sizes', 'from': [], 'n_out': 2, 'dtype': 'int32'}, '1Dto2D': {'class': '1Dto2D', 'from': ['data', 'classes_source']}, 'conv0': {'class': 'conv2', 'n_features': 15, 'filter': [3, 3], 'pool_size': [2, 2], 'from': ['1Dto2D']}, 'mdlstm0': {'class': ..., len = 13
prev = <local> 'mdlstm1', len = 7
target = <local> 'classes', len = 7
File "/home/deep/Documents/Deep Learning Libraries/returnn/Network.py", line 350, in traverse
line: index = traverse(content, prev, target, index)
locals:
index = <local> j_classes
traverse = <local> <function LayerNetwork.from_json.<locals>.traverse at 0x7ffa6912bd08>
content = <local> {'classes_source': {'class': 'source', 'data_key': 'sizes', 'from': [], 'n_out': 2, 'dtype': 'int32'}, '1Dto2D': {'class': '1Dto2D', 'from': ['data', 'classes_source']}, 'conv0': {'class': 'conv2', 'n_features': 15, 'filter': [3, 3], 'pool_size': [2, 2], 'from': ['1Dto2D']}, 'mdlstm0': {'class': ..., len = 13
prev = <local> 'conv1'
target = <local> 'classes', len = 7
File "/home/deep/Documents/Deep Learning Libraries/returnn/Network.py", line 350, in traverse
line: index = traverse(content, prev, target, index)
locals:
index = <local> j_classes
traverse = <local> <function LayerNetwork.from_json.<locals>.traverse at 0x7ffa6912bd08>
content = <local> {'classes_source': {'class': 'source', 'data_key': 'sizes', 'from': [], 'n_out': 2, 'dtype': 'int32'}, '1Dto2D': {'class': '1Dto2D', 'from': ['data', 'classes_source']}, 'conv0': {'class': 'conv2', 'n_features': 15, 'filter': [3, 3], 'pool_size': [2, 2], 'from': ['1Dto2D']}, 'mdlstm0': {'class': ..., len = 13
prev = <local> 'mdlstm0', len = 7
target = <local> 'classes', len = 7
File "/home/deep/Documents/Deep Learning Libraries/returnn/Network.py", line 350, in traverse
line: index = traverse(content, prev, target, index)
locals:
index = <local> j_classes
traverse = <local> <function LayerNetwork.from_json.<locals>.traverse at 0x7ffa6912bd08>
content = <local> {'classes_source': {'class': 'source', 'data_key': 'sizes', 'from': [], 'n_out': 2, 'dtype': 'int32'}, '1Dto2D': {'class': '1Dto2D', 'from': ['data', 'classes_source']}, 'conv0': {'class': 'conv2', 'n_features': 15, 'filter': [3, 3], 'pool_size': [2, 2], 'from': ['1Dto2D']}, 'mdlstm0': {'class': ..., len = 13
prev = <local> 'conv0'
target = <local> 'classes', len = 7
File "/home/deep/Documents/Deep Learning Libraries/returnn/Network.py", line 350, in traverse
line: index = traverse(content, prev, target, index)
locals:
index = <local> j_classes
traverse = <local> <function LayerNetwork.from_json.<locals>.traverse at 0x7ffa6912bd08>
content = <local> {'classes_source': {'class': 'source', 'data_key': 'sizes', 'from': [], 'n_out': 2, 'dtype': 'int32'}, '1Dto2D': {'class': '1Dto2D', 'from': ['data', 'classes_source']}, 'conv0': {'class': 'conv2', 'n_features': 15, 'filter': [3, 3], 'pool_size': [2, 2], 'from': ['1Dto2D']}, 'mdlstm0': {'class': ..., len = 13
prev = <local> '1Dto2D', len = 6
target = <local> 'classes', len = 7
File "/home/deep/Documents/Deep Learning Libraries/returnn/Network.py", line 408, in traverse
line: return network.add_layer(layer_class(**params)).index
locals:
network = <local> <Network.LayerNetwork object at 0x7ffa6911dfd0>
network.add_layer = <local> <bound method LayerNetwork.add_layer of <Network.LayerNetwork object at 0x7ffa6911dfd0>>
layer_class = <local> <class 'NetworkTwoDLayer.OneDToTwoDLayer'>
params = <local> {'sources': [<<class 'NetworkBaseLayer.SourceLayer'> class:source name:data>, <<class 'NetworkBaseLayer.SourceLayer'> class:source name:classes_source>], 'dropout': 0.0, 'name': '1Dto2D', 'train_flag': True, 'eval_flag': False, 'network': <Network.LayerNetwork object at 0x7ffa6911dfd0>, 'mask': N..., len = 9
index = <local> j_sizes
File "/home/deep/Documents/Deep Learning Libraries/returnn/NetworkTwoDLayer.py", line 51, in __init__
line: sizes = sizes.reshape((2, sizes.size / 2)).dimshuffle(1, 0)
locals:
sizes = <local> Elemwise{Cast{float32}}.0
sizes.reshape = <local> <bound method _tensor_py_operators.reshape of Elemwise{Cast{float32}}.0>
sizes.size = <local> Prod{axis=None, dtype='int64', acc_dtype='int64'}.0
dimshuffle = <not found>
File "/home/deep/anaconda3/lib/python3.6/site-packages/theano/tensor/var.py", line 321, in reshape
line: return theano.tensor.basic.reshape(self, shape, ndim=ndim)
locals:
theano = <global> <module 'theano' from '/home/deep/anaconda3/lib/python3.6/site-packages/theano/__init__.py'>
theano.tensor = <global> <module 'theano.tensor' from '/home/deep/anaconda3/lib/python3.6/site-packages/theano/tensor/__init__.py'>
theano.tensor.basic = <global> <module 'theano.tensor.basic' from '/home/deep/anaconda3/lib/python3.6/site-packages/theano/tensor/basic.py'>
theano.tensor.basic.reshape = <global> <function reshape at 0x7ffa882c2a60>
self = <local> Elemwise{Cast{float32}}.0
shape = <local> (2, Elemwise{true_div,no_inplace}.0)
ndim = <local> None
File "/home/deep/anaconda3/lib/python3.6/site-packages/theano/tensor/basic.py", line 4910, in reshape
line: rval = op(x, newshape)
locals:
rval = <not found>
op = <local> <theano.tensor.basic.Reshape object at 0x7ffa68f8d358>
x = <local> Elemwise{Cast{float32}}.0
newshape = <local> MakeVector{dtype='float32'}.0
File "/home/deep/anaconda3/lib/python3.6/site-packages/theano/gof/op.py", line 615, in __call__
line: node = self.make_node(*inputs, **kwargs)
locals:
node = <not found>
self = <local> <theano.tensor.basic.Reshape object at 0x7ffa68f8d358>
self.make_node = <local> <bound method Reshape.make_node of <theano.tensor.basic.Reshape object at 0x7ffa68f8d358>>
inputs = <local> (Elemwise{Cast{float32}}.0, MakeVector{dtype='float32'}.0)
kwargs = <local> {}
File "/home/deep/anaconda3/lib/python3.6/site-packages/theano/tensor/basic.py", line 4748, in make_node
line: raise TypeError("Shape must be integers", shp, shp.dtype)
locals:
TypeError = <builtin> <class 'TypeError'>
shp = <local> MakeVector{dtype='float32'}.0
shp.dtype = <local> 'float32', len = 7
TypeError: ('Shape must be integers', MakeVector{dtype='float32'}.0, 'float32')
Device proc gpuX (gpuZ) died: ProcConnectionDied('recv_bytes EOFError: ',)
Theano flags: compiledir_format=compiledir_%(platform)s-%(processor)s-%(python_version)s-%(python_bitwidth)s--dev-gpuZ,device=gpu,force_device=True
EXCEPTION
Traceback (most recent call last):
File "/home/deep/Documents/Deep Learning Libraries/returnn/Device.py", line 329, in startProc
line: self._startProc(*args, **kwargs)
locals:
self = <local> <Device.Device object at 0x7f89f5416c18>
self._startProc = <local> <bound method Device._startProc of <Device.Device object at 0x7f89f5416c18>>
args = <local> ('gpuZ',)
kwargs = <local> {}
File "/home/deep/Documents/Deep Learning Libraries/returnn/Device.py", line 383, in _startProc
line: interrupt_main()
locals:
interrupt_main = <global> <function interrupt_main at 0x7f89f7750bf8>
File "/home/deep/Documents/Deep Learning Libraries/returnn/Util.py", line 607, in interrupt_main
line: sys.exit(1) # And exit the thread.
locals:
sys = <global> <module 'sys' (built-in)>
sys.exit = <global> <built-in function exit>
SystemExit: 1
KeyboardInterrupt
EXCEPTION
Traceback (most recent call last):
File "../../../rnn.py", line 520, in main
line: init(commandLineOptions=argv[1:])
locals:
init = <global> <function init at 0x7f89f5480d08>
commandLineOptions = <not found>
argv = <local> ['../../../rnn.py', 'config_demo'], _[0]: {len = 15}
File "../../../rnn.py", line 336, in init
line: devices = initDevices()
locals:
devices = <not found>
initDevices = <global> <function initDevices at 0x7f89f5480840>
File "../../../rnn.py", line 152, in initDevices
line: time.sleep(0.25)
locals:
time = <global> <module 'time' (built-in)>
time.sleep = <global> <built-in function sleep>
KeyboardInterrupt
Quitting
from returnn.
pvoigtlaender
commented on July 17, 2024
I made one more fix, please pull and try again
from returnn.
pedro-dreyer
commented on July 17, 2024
Still got an error
deep@deep-System-Product-Name ~/Documents/Deep Learning Libraries/returnn/demos/mdlstm/IAM $ ./go.sh
converting IAM_lines to features/raw/demo.h5
features/raw/demo.h5
0 / 3
CRNN starting up, version 20171010.211508--git-e19cbd3-dirty, pid 25656, cwd /home/deep/Documents/Deep Learning Libraries/returnn/demos/mdlstm/IAM
CRNN command line options: ['config_demo']
Theano: 0.9.0.dev-c697eeab84... (<site-package> in /home/deep/anaconda3/lib/python3.6/site-packages/theano)
pynvml not available, memory information missing
WARNING (theano.sandbox.cuda): The cuda backend is deprecated and will be removed in the next release (v0.10). Please switch to the gpuarray backend. You can get more information about how to switch at this URL:
https://github.com/Theano/Theano/wiki/Converting-to-the-new-gpu-back-end%28gpuarray%29
Using gpu device 0: TITAN Xp (CNMeM is enabled with initial size: 95.0% of memory, cuDNN 5110)
Device gpuX proc starting up, pid 25684
Device gpuX proc: THEANO_FLAGS = 'compiledir_format=compiledir_%(platform)s-%(processor)s-%(python_version)s-%(python_bitwidth)s--dev-gpuZ,device=gpu,force_device=True'
Device gpuX proc exception: ('Shape must be integers', MakeVector{dtype='float32'}.0, 'float32')
Unhandled exception <class 'TypeError'> in thread <_MainThread(MainThread, started 139761854293760)>, proc 25684.
Thread current, main, <_MainThread(MainThread, started 139761854293760)>:
(Excluded thread.)
That were all threads.
EXCEPTION
Traceback (most recent call last):
File "/home/deep/Documents/Deep Learning Libraries/returnn/Device.py", line 977, in process
line: self.process_inner(device, config, self.update_specs, asyncTask)
locals:
self = <local> <Device.Device object at 0x7f1cd29d6908>
self.process_inner = <local> <bound method Device.process_inner of <Device.Device object at 0x7f1cd29d6908>>
device = <local> 'gpuX'
config = <local> <Config.Config object at 0x7f1c91f2b518>
self.update_specs = <local> {'update_rule': 'global', 'update_params': {}, 'layers': [], 'block_size': 0}
asyncTask = <local> <TaskSystem.AsyncTask object at 0x7f1cbd9aef28>
File "/home/deep/Documents/Deep Learning Libraries/returnn/Device.py", line 1030, in process_inner
line: self.initialize(config, update_specs=update_specs)
locals:
self = <local> <Device.Device object at 0x7f1cd29d6908>
self.initialize = <local> <bound method Device.initialize of <Device.Device object at 0x7f1cd29d6908>>
config = <local> <Config.Config object at 0x7f1c91f2b518>
update_specs = <local> {'update_rule': 'global', 'update_params': {}, 'layers': [], 'block_size': 0}
File "/home/deep/Documents/Deep Learning Libraries/returnn/Device.py", line 439, in initialize
line: self.trainnet = LayerNetwork.from_config_topology(config, train_flag=True, eval_flag=False)
locals:
self = <local> <Device.Device object at 0x7f1cd29d6908>
self.trainnet = <local> !AttributeError: 'Device' object has no attribute 'trainnet'
LayerNetwork = <global> <class 'Network.LayerNetwork'>
LayerNetwork.from_config_topology = <global> <bound method LayerNetwork.from_config_topology of <class 'Network.LayerNetwork'>>
config = <local> <Config.Config object at 0x7f1c91f2b518>
train_flag = <not found>
eval_flag = <local> False
File "/home/deep/Documents/Deep Learning Libraries/returnn/Network.py", line 119, in from_config_topology
line: return cls.from_json_and_config(json_content, config, mask=mask, **kwargs)
locals:
cls = <local> <class 'Network.LayerNetwork'>
cls.from_json_and_config = <local> <bound method LayerNetwork.from_json_and_config of <class 'Network.LayerNetwork'>>
json_content = <local> {'classes_source': {'class': 'source', 'data_key': 'sizes', 'from': [], 'n_out': 2, 'dtype': 'int32'}, '1Dto2D': {'class': '1Dto2D', 'from': ['data', 'classes_source']}, 'conv0': {'class': 'conv2', 'n_features': 15, 'filter': [3, 3], 'pool_size': [2, 2], 'from': ['1Dto2D']}, 'mdlstm0': {'class': ..., len = 13
config = <local> <Config.Config object at 0x7f1c91f2b518>
mask = <local> None
kwargs = <local> {'train_flag': True, 'eval_flag': False}
File "/home/deep/Documents/Deep Learning Libraries/returnn/Network.py", line 204, in from_json_and_config
line: network = cls.from_json(json_content, **dict_joined(kwargs, cls.init_args_from_config(config)))
locals:
network = <not found>
cls = <local> <class 'Network.LayerNetwork'>
cls.from_json = <local> <bound method LayerNetwork.from_json of <class 'Network.LayerNetwork'>>
json_content = <local> {'classes_source': {'class': 'source', 'data_key': 'sizes', 'from': [], 'n_out': 2, 'dtype': 'int32'}, '1Dto2D': {'class': '1Dto2D', 'from': ['data', 'classes_source']}, 'conv0': {'class': 'conv2', 'n_features': 15, 'filter': [3, 3], 'pool_size': [2, 2], 'from': ['1Dto2D']}, 'mdlstm0': {'class': ..., len = 13
dict_joined = <global> <function dict_joined at 0x7f1c9c4d3e18>
kwargs = <local> {'mask': None, 'train_flag': True, 'eval_flag': False}
cls.init_args_from_config = <local> <bound method LayerNetwork.init_args_from_config of <class 'Network.LayerNetwork'>>
config = <local> <Config.Config object at 0x7f1c91f2b518>
File "/home/deep/Documents/Deep Learning Libraries/returnn/Network.py", line 422, in from_json
line: traverse(json_content, layer_name, trg, index)
locals:
traverse = <local> <function LayerNetwork.from_json.<locals>.traverse at 0x7f1c919a2d08>
json_content = <local> {'classes_source': {'class': 'source', 'data_key': 'sizes', 'from': [], 'n_out': 2, 'dtype': 'int32'}, '1Dto2D': {'class': '1Dto2D', 'from': ['data', 'classes_source']}, 'conv0': {'class': 'conv2', 'n_features': 15, 'filter': [3, 3], 'pool_size': [2, 2], 'from': ['1Dto2D']}, 'mdlstm0': {'class': ..., len = 13
layer_name = <local> 'output', len = 6
trg = <local> 'classes', len = 7
index = <local> j_classes
File "/home/deep/Documents/Deep Learning Libraries/returnn/Network.py", line 350, in traverse
line: index = traverse(content, prev, target, index)
locals:
index = <local> j_classes
traverse = <local> <function LayerNetwork.from_json.<locals>.traverse at 0x7f1c919a2d08>
content = <local> {'classes_source': {'class': 'source', 'data_key': 'sizes', 'from': [], 'n_out': 2, 'dtype': 'int32'}, '1Dto2D': {'class': '1Dto2D', 'from': ['data', 'classes_source']}, 'conv0': {'class': 'conv2', 'n_features': 15, 'filter': [3, 3], 'pool_size': [2, 2], 'from': ['1Dto2D']}, 'mdlstm0': {'class': ..., len = 13
prev = <local> 'mdlstm4', len = 7
target = <local> 'classes', len = 7
File "/home/deep/Documents/Deep Learning Libraries/returnn/Network.py", line 350, in traverse
line: index = traverse(content, prev, target, index)
locals:
index = <local> j_classes
traverse = <local> <function LayerNetwork.from_json.<locals>.traverse at 0x7f1c919a2d08>
content = <local> {'classes_source': {'class': 'source', 'data_key': 'sizes', 'from': [], 'n_out': 2, 'dtype': 'int32'}, '1Dto2D': {'class': '1Dto2D', 'from': ['data', 'classes_source']}, 'conv0': {'class': 'conv2', 'n_features': 15, 'filter': [3, 3], 'pool_size': [2, 2], 'from': ['1Dto2D']}, 'mdlstm0': {'class': ..., len = 13
prev = <local> 'conv4'
target = <local> 'classes', len = 7
File "/home/deep/Documents/Deep Learning Libraries/returnn/Network.py", line 350, in traverse
line: index = traverse(content, prev, target, index)
locals:
index = <local> j_classes
traverse = <local> <function LayerNetwork.from_json.<locals>.traverse at 0x7f1c919a2d08>
content = <local> {'classes_source': {'class': 'source', 'data_key': 'sizes', 'from': [], 'n_out': 2, 'dtype': 'int32'}, '1Dto2D': {'class': '1Dto2D', 'from': ['data', 'classes_source']}, 'conv0': {'class': 'conv2', 'n_features': 15, 'filter': [3, 3], 'pool_size': [2, 2], 'from': ['1Dto2D']}, 'mdlstm0': {'class': ..., len = 13
prev = <local> 'mdlstm3', len = 7
target = <local> 'classes', len = 7
File "/home/deep/Documents/Deep Learning Libraries/returnn/Network.py", line 350, in traverse
line: index = traverse(content, prev, target, index)
locals:
index = <local> j_classes
traverse = <local> <function LayerNetwork.from_json.<locals>.traverse at 0x7f1c919a2d08>
content = <local> {'classes_source': {'class': 'source', 'data_key': 'sizes', 'from': [], 'n_out': 2, 'dtype': 'int32'}, '1Dto2D': {'class': '1Dto2D', 'from': ['data', 'classes_source']}, 'conv0': {'class': 'conv2', 'n_features': 15, 'filter': [3, 3], 'pool_size': [2, 2], 'from': ['1Dto2D']}, 'mdlstm0': {'class': ..., len = 13
prev = <local> 'conv3'
target = <local> 'classes', len = 7
File "/home/deep/Documents/Deep Learning Libraries/returnn/Network.py", line 350, in traverse
line: index = traverse(content, prev, target, index)
locals:
index = <local> j_classes
traverse = <local> <function LayerNetwork.from_json.<locals>.traverse at 0x7f1c919a2d08>
content = <local> {'classes_source': {'class': 'source', 'data_key': 'sizes', 'from': [], 'n_out': 2, 'dtype': 'int32'}, '1Dto2D': {'class': '1Dto2D', 'from': ['data', 'classes_source']}, 'conv0': {'class': 'conv2', 'n_features': 15, 'filter': [3, 3], 'pool_size': [2, 2], 'from': ['1Dto2D']}, 'mdlstm0': {'class': ..., len = 13
prev = <local> 'mdlstm2', len = 7
target = <local> 'classes', len = 7
File "/home/deep/Documents/Deep Learning Libraries/returnn/Network.py", line 350, in traverse
line: index = traverse(content, prev, target, index)
locals:
index = <local> j_classes
traverse = <local> <function LayerNetwork.from_json.<locals>.traverse at 0x7f1c919a2d08>
content = <local> {'classes_source': {'class': 'source', 'data_key': 'sizes', 'from': [], 'n_out': 2, 'dtype': 'int32'}, '1Dto2D': {'class': '1Dto2D', 'from': ['data', 'classes_source']}, 'conv0': {'class': 'conv2', 'n_features': 15, 'filter': [3, 3], 'pool_size': [2, 2], 'from': ['1Dto2D']}, 'mdlstm0': {'class': ..., len = 13
prev = <local> 'conv2'
target = <local> 'classes', len = 7
File "/home/deep/Documents/Deep Learning Libraries/returnn/Network.py", line 350, in traverse
line: index = traverse(content, prev, target, index)
locals:
index = <local> j_classes
traverse = <local> <function LayerNetwork.from_json.<locals>.traverse at 0x7f1c919a2d08>
content = <local> {'classes_source': {'class': 'source', 'data_key': 'sizes', 'from': [], 'n_out': 2, 'dtype': 'int32'}, '1Dto2D': {'class': '1Dto2D', 'from': ['data', 'classes_source']}, 'conv0': {'class': 'conv2', 'n_features': 15, 'filter': [3, 3], 'pool_size': [2, 2], 'from': ['1Dto2D']}, 'mdlstm0': {'class': ..., len = 13
prev = <local> 'mdlstm1', len = 7
target = <local> 'classes', len = 7
File "/home/deep/Documents/Deep Learning Libraries/returnn/Network.py", line 350, in traverse
line: index = traverse(content, prev, target, index)
locals:
index = <local> j_classes
traverse = <local> <function LayerNetwork.from_json.<locals>.traverse at 0x7f1c919a2d08>
content = <local> {'classes_source': {'class': 'source', 'data_key': 'sizes', 'from': [], 'n_out': 2, 'dtype': 'int32'}, '1Dto2D': {'class': '1Dto2D', 'from': ['data', 'classes_source']}, 'conv0': {'class': 'conv2', 'n_features': 15, 'filter': [3, 3], 'pool_size': [2, 2], 'from': ['1Dto2D']}, 'mdlstm0': {'class': ..., len = 13
prev = <local> 'conv1'
target = <local> 'classes', len = 7
File "/home/deep/Documents/Deep Learning Libraries/returnn/Network.py", line 350, in traverse
line: index = traverse(content, prev, target, index)
locals:
index = <local> j_classes
traverse = <local> <function LayerNetwork.from_json.<locals>.traverse at 0x7f1c919a2d08>
content = <local> {'classes_source': {'class': 'source', 'data_key': 'sizes', 'from': [], 'n_out': 2, 'dtype': 'int32'}, '1Dto2D': {'class': '1Dto2D', 'from': ['data', 'classes_source']}, 'conv0': {'class': 'conv2', 'n_features': 15, 'filter': [3, 3], 'pool_size': [2, 2], 'from': ['1Dto2D']}, 'mdlstm0': {'class': ..., len = 13
prev = <local> 'mdlstm0', len = 7
target = <local> 'classes', len = 7
File "/home/deep/Documents/Deep Learning Libraries/returnn/Network.py", line 350, in traverse
line: index = traverse(content, prev, target, index)
locals:
index = <local> j_classes
traverse = <local> <function LayerNetwork.from_json.<locals>.traverse at 0x7f1c919a2d08>
content = <local> {'classes_source': {'class': 'source', 'data_key': 'sizes', 'from': [], 'n_out': 2, 'dtype': 'int32'}, '1Dto2D': {'class': '1Dto2D', 'from': ['data', 'classes_source']}, 'conv0': {'class': 'conv2', 'n_features': 15, 'filter': [3, 3], 'pool_size': [2, 2], 'from': ['1Dto2D']}, 'mdlstm0': {'class': ..., len = 13
prev = <local> 'conv0'
target = <local> 'classes', len = 7
File "/home/deep/Documents/Deep Learning Libraries/returnn/Network.py", line 350, in traverse
line: index = traverse(content, prev, target, index)
locals:
index = <local> j_classes
traverse = <local> <function LayerNetwork.from_json.<locals>.traverse at 0x7f1c919a2d08>
content = <local> {'classes_source': {'class': 'source', 'data_key': 'sizes', 'from': [], 'n_out': 2, 'dtype': 'int32'}, '1Dto2D': {'class': '1Dto2D', 'from': ['data', 'classes_source']}, 'conv0': {'class': 'conv2', 'n_features': 15, 'filter': [3, 3], 'pool_size': [2, 2], 'from': ['1Dto2D']}, 'mdlstm0': {'class': ..., len = 13
prev = <local> '1Dto2D', len = 6
target = <local> 'classes', len = 7
File "/home/deep/Documents/Deep Learning Libraries/returnn/Network.py", line 408, in traverse
line: return network.add_layer(layer_class(**params)).index
locals:
network = <local> <Network.LayerNetwork object at 0x7f1c91f3d748>
network.add_layer = <local> <bound method LayerNetwork.add_layer of <Network.LayerNetwork object at 0x7f1c91f3d748>>
layer_class = <local> <class 'NetworkTwoDLayer.OneDToTwoDLayer'>
params = <local> {'sources': [<<class 'NetworkBaseLayer.SourceLayer'> class:source name:data>, <<class 'NetworkBaseLayer.SourceLayer'> class:source name:classes_source>], 'dropout': 0.0, 'name': '1Dto2D', 'train_flag': True, 'eval_flag': False, 'network': <Network.LayerNetwork object at 0x7f1c91f3d748>, 'mask': N..., len = 9
index = <local> j_sizes
File "/home/deep/Documents/Deep Learning Libraries/returnn/NetworkTwoDLayer.py", line 51, in __init__
line: sizes = sizes.reshape((2, sizes.size / 2)).dimshuffle(1, 0)
locals:
sizes = <local> Elemwise{Cast{float32}}.0
sizes.reshape = <local> <bound method _tensor_py_operators.reshape of Elemwise{Cast{float32}}.0>
sizes.size = <local> Prod{axis=None, dtype='int64', acc_dtype='int64'}.0
dimshuffle = <not found>
File "/home/deep/anaconda3/lib/python3.6/site-packages/theano/tensor/var.py", line 321, in reshape
line: return theano.tensor.basic.reshape(self, shape, ndim=ndim)
locals:
theano = <global> <module 'theano' from '/home/deep/anaconda3/lib/python3.6/site-packages/theano/__init__.py'>
theano.tensor = <global> <module 'theano.tensor' from '/home/deep/anaconda3/lib/python3.6/site-packages/theano/tensor/__init__.py'>
theano.tensor.basic = <global> <module 'theano.tensor.basic' from '/home/deep/anaconda3/lib/python3.6/site-packages/theano/tensor/basic.py'>
theano.tensor.basic.reshape = <global> <function reshape at 0x7f1cb0b25a60>
self = <local> Elemwise{Cast{float32}}.0
shape = <local> (2, Elemwise{true_div,no_inplace}.0)
ndim = <local> None
File "/home/deep/anaconda3/lib/python3.6/site-packages/theano/tensor/basic.py", line 4910, in reshape
line: rval = op(x, newshape)
locals:
rval = <not found>
op = <local> <theano.tensor.basic.Reshape object at 0x7f1c918033c8>
x = <local> Elemwise{Cast{float32}}.0
newshape = <local> MakeVector{dtype='float32'}.0
File "/home/deep/anaconda3/lib/python3.6/site-packages/theano/gof/op.py", line 615, in __call__
line: node = self.make_node(*inputs, **kwargs)
locals:
node = <not found>
self = <local> <theano.tensor.basic.Reshape object at 0x7f1c918033c8>
self.make_node = <local> <bound method Reshape.make_node of <theano.tensor.basic.Reshape object at 0x7f1c918033c8>>
inputs = <local> (Elemwise{Cast{float32}}.0, MakeVector{dtype='float32'}.0)
kwargs = <local> {}
File "/home/deep/anaconda3/lib/python3.6/site-packages/theano/tensor/basic.py", line 4748, in make_node
line: raise TypeError("Shape must be integers", shp, shp.dtype)
locals:
TypeError = <builtin> <class 'TypeError'>
shp = <local> MakeVector{dtype='float32'}.0
shp.dtype = <local> 'float32', len = 7
TypeError: ('Shape must be integers', MakeVector{dtype='float32'}.0, 'float32')
Device proc gpuX (gpuZ) died: ProcConnectionDied('recv_bytes EOFError: ',)
Theano flags: compiledir_format=compiledir_%(platform)s-%(processor)s-%(python_version)s-%(python_bitwidth)s--dev-gpuZ,device=gpu,force_device=True
EXCEPTION
Traceback (most recent call last):
File "/home/deep/Documents/Deep Learning Libraries/returnn/Device.py", line 329, in startProc
line: self._startProc(*args, **kwargs)
locals:
self = <local> <Device.Device object at 0x7f22b23924e0>
self._startProc = <local> <bound method Device._startProc of <Device.Device object at 0x7f22b23924e0>>
args = <local> ('gpuZ',)
kwargs = <local> {}
File "/home/deep/Documents/Deep Learning Libraries/returnn/Device.py", line 383, in _startProc
line: interrupt_main()
locals:
interrupt_main = <global> <function interrupt_main at 0x7f22b7609ae8>
File "/home/deep/Documents/Deep Learning Libraries/returnn/Util.py", line 607, in interrupt_main
line: sys.exit(1) # And exit the thread.
locals:
sys = <global> <module 'sys' (built-in)>
sys.exit = <global> <built-in function exit>
SystemExit: 1
KeyboardInterrupt
EXCEPTION
Traceback (most recent call last):
File "../../../rnn.py", line 520, in main
line: init(commandLineOptions=argv[1:])
locals:
init = <global> <function init at 0x7f22b23ab950>
commandLineOptions = <not found>
argv = <local> ['../../../rnn.py', 'config_demo'], _[0]: {len = 15}
File "../../../rnn.py", line 336, in init
line: devices = initDevices()
locals:
devices = <not found>
initDevices = <global> <function initDevices at 0x7f22b23ab488>
File "../../../rnn.py", line 152, in initDevices
line: time.sleep(0.25)
locals:
time = <global> <module 'time' (built-in)>
time.sleep = <global> <built-in function sleep>
KeyboardInterrupt
Quitting
from returnn.
pvoigtlaender
commented on July 17, 2024
Sorry, I didn't push correctly before my last post. The change is now pushed, please try again.
from returnn.
pedro-dreyer
commented on July 17, 2024
Another error.
deep@deep-System-Product-Name ~/Documents/Deep Learning Libraries/returnn/demos/mdlstm/IAM $ ./go.sh
converting IAM_lines to features/raw/demo.h5
features/raw/demo.h5
0 / 3
CRNN starting up, version 20171011.180023--git-9ee583e-dirty, pid 6406, cwd /home/deep/Documents/Deep Learning Libraries/returnn/demos/mdlstm/IAM
CRNN command line options: ['config_demo']
Theano: 0.9.0.dev-c697eeab84... (<site-package> in /home/deep/anaconda3/lib/python3.6/site-packages/theano)
pynvml not available, memory information missing
WARNING (theano.sandbox.cuda): The cuda backend is deprecated and will be removed in the next release (v0.10). Please switch to the gpuarray backend. You can get more information about how to switch at this URL:
https://github.com/Theano/Theano/wiki/Converting-to-the-new-gpu-back-end%28gpuarray%29
Using gpu device 0: TITAN Xp (CNMeM is enabled with initial size: 95.0% of memory, cuDNN 5110)
Device gpuX proc starting up, pid 6434
Device gpuX proc: THEANO_FLAGS = 'compiledir_format=compiledir_%(platform)s-%(processor)s-%(python_version)s-%(python_bitwidth)s--dev-gpuZ,device=gpu,force_device=True'
Device gpuX proc exception: (MultiDirectionalTwoDLSTMOp(GpuContiguous.0, GpuContiguous.0, GpuContiguous.0, GpuContiguous.0, GpuContiguous.0, GpuContiguous.0, GpuContiguous.0, GpuContiguous.0, GpuContiguous.0, GpuContiguous.0, GpuContiguous.0, GpuContiguous.0, GpuContiguous.0, GpuContiguous.0, GpuContiguous.0, GpuContiguous.0, GpuContiguous.0, Join.0), 'float32')
Unhandled exception <class 'AssertionError'> in thread <_MainThread(MainThread, started 140316415391488)>, proc 6434.
Thread current, main, <_MainThread(MainThread, started 140316415391488)>:
(Excluded thread.)
That were all threads.
EXCEPTION
Traceback (most recent call last):
File "/home/deep/Documents/Deep Learning Libraries/returnn/Device.py", line 977, in process
line: self.process_inner(device, config, self.update_specs, asyncTask)
locals:
self = <local> <Device.Device object at 0x7f9df1083908>
self.process_inner = <local> <bound method Device.process_inner of <Device.Device object at 0x7f9df1083908>>
device = <local> 'gpuX'
config = <local> <Config.Config object at 0x7f9db9859518>
self.update_specs = <local> {'update_rule': 'global', 'update_params': {}, 'layers': [], 'block_size': 0}
asyncTask = <local> <TaskSystem.AsyncTask object at 0x7f9ddc05bf28>
File "/home/deep/Documents/Deep Learning Libraries/returnn/Device.py", line 1030, in process_inner
line: self.initialize(config, update_specs=update_specs)
locals:
self = <local> <Device.Device object at 0x7f9df1083908>
self.initialize = <local> <bound method Device.initialize of <Device.Device object at 0x7f9df1083908>>
config = <local> <Config.Config object at 0x7f9db9859518>
update_specs = <local> {'update_rule': 'global', 'update_params': {}, 'layers': [], 'block_size': 0}
File "/home/deep/Documents/Deep Learning Libraries/returnn/Device.py", line 439, in initialize
line: self.trainnet = LayerNetwork.from_config_topology(config, train_flag=True, eval_flag=False)
locals:
self = <local> <Device.Device object at 0x7f9df1083908>
self.trainnet = <local> !AttributeError: 'Device' object has no attribute 'trainnet'
LayerNetwork = <global> <class 'Network.LayerNetwork'>
LayerNetwork.from_config_topology = <global> <bound method LayerNetwork.from_config_topology of <class 'Network.LayerNetwork'>>
config = <local> <Config.Config object at 0x7f9db9859518>
train_flag = <not found>
eval_flag = <local> False
File "/home/deep/Documents/Deep Learning Libraries/returnn/Network.py", line 119, in from_config_topology
line: return cls.from_json_and_config(json_content, config, mask=mask, **kwargs)
locals:
cls = <local> <class 'Network.LayerNetwork'>
cls.from_json_and_config = <local> <bound method LayerNetwork.from_json_and_config of <class 'Network.LayerNetwork'>>
json_content = <local> {'classes_source': {'class': 'source', 'data_key': 'sizes', 'from': [], 'n_out': 2, 'dtype': 'int32'}, '1Dto2D': {'class': '1Dto2D', 'from': ['data', 'classes_source']}, 'conv0': {'class': 'conv2', 'n_features': 15, 'filter': [3, 3], 'pool_size': [2, 2], 'from': ['1Dto2D']}, 'mdlstm0': {'class': ..., len = 13
config = <local> <Config.Config object at 0x7f9db9859518>
mask = <local> None
kwargs = <local> {'train_flag': True, 'eval_flag': False}
File "/home/deep/Documents/Deep Learning Libraries/returnn/Network.py", line 204, in from_json_and_config
line: network = cls.from_json(json_content, **dict_joined(kwargs, cls.init_args_from_config(config)))
locals:
network = <not found>
cls = <local> <class 'Network.LayerNetwork'>
cls.from_json = <local> <bound method LayerNetwork.from_json of <class 'Network.LayerNetwork'>>
json_content = <local> {'classes_source': {'class': 'source', 'data_key': 'sizes', 'from': [], 'n_out': 2, 'dtype': 'int32'}, '1Dto2D': {'class': '1Dto2D', 'from': ['data', 'classes_source']}, 'conv0': {'class': 'conv2', 'n_features': 15, 'filter': [3, 3], 'pool_size': [2, 2], 'from': ['1Dto2D']}, 'mdlstm0': {'class': ..., len = 13
dict_joined = <global> <function dict_joined at 0x7f9dc0041e18>
kwargs = <local> {'mask': None, 'train_flag': True, 'eval_flag': False}
cls.init_args_from_config = <local> <bound method LayerNetwork.init_args_from_config of <class 'Network.LayerNetwork'>>
config = <local> <Config.Config object at 0x7f9db9859518>
File "/home/deep/Documents/Deep Learning Libraries/returnn/Network.py", line 422, in from_json
line: traverse(json_content, layer_name, trg, index)
locals:
traverse = <local> <function LayerNetwork.from_json.<locals>.traverse at 0x7f9db92d0d08>
json_content = <local> {'classes_source': {'class': 'source', 'data_key': 'sizes', 'from': [], 'n_out': 2, 'dtype': 'int32'}, '1Dto2D': {'class': '1Dto2D', 'from': ['data', 'classes_source']}, 'conv0': {'class': 'conv2', 'n_features': 15, 'filter': [3, 3], 'pool_size': [2, 2], 'from': ['1Dto2D']}, 'mdlstm0': {'class': ..., len = 13
layer_name = <local> 'output', len = 6
trg = <local> 'classes', len = 7
index = <local> j_classes
File "/home/deep/Documents/Deep Learning Libraries/returnn/Network.py", line 350, in traverse
line: index = traverse(content, prev, target, index)
locals:
index = <local> j_classes
traverse = <local> <function LayerNetwork.from_json.<locals>.traverse at 0x7f9db92d0d08>
content = <local> {'classes_source': {'class': 'source', 'data_key': 'sizes', 'from': [], 'n_out': 2, 'dtype': 'int32'}, '1Dto2D': {'class': '1Dto2D', 'from': ['data', 'classes_source']}, 'conv0': {'class': 'conv2', 'n_features': 15, 'filter': [3, 3], 'pool_size': [2, 2], 'from': ['1Dto2D']}, 'mdlstm0': {'class': ..., len = 13
prev = <local> 'mdlstm4', len = 7
target = <local> 'classes', len = 7
File "/home/deep/Documents/Deep Learning Libraries/returnn/Network.py", line 350, in traverse
line: index = traverse(content, prev, target, index)
locals:
index = <local> j_classes
traverse = <local> <function LayerNetwork.from_json.<locals>.traverse at 0x7f9db92d0d08>
content = <local> {'classes_source': {'class': 'source', 'data_key': 'sizes', 'from': [], 'n_out': 2, 'dtype': 'int32'}, '1Dto2D': {'class': '1Dto2D', 'from': ['data', 'classes_source']}, 'conv0': {'class': 'conv2', 'n_features': 15, 'filter': [3, 3], 'pool_size': [2, 2], 'from': ['1Dto2D']}, 'mdlstm0': {'class': ..., len = 13
prev = <local> 'conv4'
target = <local> 'classes', len = 7
File "/home/deep/Documents/Deep Learning Libraries/returnn/Network.py", line 350, in traverse
line: index = traverse(content, prev, target, index)
locals:
index = <local> j_classes
traverse = <local> <function LayerNetwork.from_json.<locals>.traverse at 0x7f9db92d0d08>
content = <local> {'classes_source': {'class': 'source', 'data_key': 'sizes', 'from': [], 'n_out': 2, 'dtype': 'int32'}, '1Dto2D': {'class': '1Dto2D', 'from': ['data', 'classes_source']}, 'conv0': {'class': 'conv2', 'n_features': 15, 'filter': [3, 3], 'pool_size': [2, 2], 'from': ['1Dto2D']}, 'mdlstm0': {'class': ..., len = 13
prev = <local> 'mdlstm3', len = 7
target = <local> 'classes', len = 7
File "/home/deep/Documents/Deep Learning Libraries/returnn/Network.py", line 350, in traverse
line: index = traverse(content, prev, target, index)
locals:
index = <local> j_classes
traverse = <local> <function LayerNetwork.from_json.<locals>.traverse at 0x7f9db92d0d08>
content = <local> {'classes_source': {'class': 'source', 'data_key': 'sizes', 'from': [], 'n_out': 2, 'dtype': 'int32'}, '1Dto2D': {'class': '1Dto2D', 'from': ['data', 'classes_source']}, 'conv0': {'class': 'conv2', 'n_features': 15, 'filter': [3, 3], 'pool_size': [2, 2], 'from': ['1Dto2D']}, 'mdlstm0': {'class': ..., len = 13
prev = <local> 'conv3'
target = <local> 'classes', len = 7
File "/home/deep/Documents/Deep Learning Libraries/returnn/Network.py", line 350, in traverse
line: index = traverse(content, prev, target, index)
locals:
index = <local> j_classes
traverse = <local> <function LayerNetwork.from_json.<locals>.traverse at 0x7f9db92d0d08>
content = <local> {'classes_source': {'class': 'source', 'data_key': 'sizes', 'from': [], 'n_out': 2, 'dtype': 'int32'}, '1Dto2D': {'class': '1Dto2D', 'from': ['data', 'classes_source']}, 'conv0': {'class': 'conv2', 'n_features': 15, 'filter': [3, 3], 'pool_size': [2, 2], 'from': ['1Dto2D']}, 'mdlstm0': {'class': ..., len = 13
prev = <local> 'mdlstm2', len = 7
target = <local> 'classes', len = 7
File "/home/deep/Documents/Deep Learning Libraries/returnn/Network.py", line 350, in traverse
line: index = traverse(content, prev, target, index)
locals:
index = <local> j_classes
traverse = <local> <function LayerNetwork.from_json.<locals>.traverse at 0x7f9db92d0d08>
content = <local> {'classes_source': {'class': 'source', 'data_key': 'sizes', 'from': [], 'n_out': 2, 'dtype': 'int32'}, '1Dto2D': {'class': '1Dto2D', 'from': ['data', 'classes_source']}, 'conv0': {'class': 'conv2', 'n_features': 15, 'filter': [3, 3], 'pool_size': [2, 2], 'from': ['1Dto2D']}, 'mdlstm0': {'class': ..., len = 13
prev = <local> 'conv2'
target = <local> 'classes', len = 7
File "/home/deep/Documents/Deep Learning Libraries/returnn/Network.py", line 408, in traverse
line: return network.add_layer(layer_class(**params)).index
locals:
network = <local> <Network.LayerNetwork object at 0x7f9db986b710>
network.add_layer = <local> <bound method LayerNetwork.add_layer of <Network.LayerNetwork object at 0x7f9db986b710>>
layer_class = <local> <class 'NetworkTwoDLayer.ConvPoolLayer2'>
params = <local> {'sources': [<<class 'NetworkTwoDLayer.TwoDLSTMLayer'> class:mdlstm name:mdlstm1>], 'dropout': 0.25, 'name': 'conv2', 'train_flag': True, 'eval_flag': False, 'network': <Network.LayerNetwork object at 0x7f9db986b710>, 'n_features': 75, 'filter': [3, 3], 'pool_size': [2, 2], 'mask': None, 'index':..., len = 12
index = <local> InplaceDimShuffle{1,0}.0
File "/home/deep/Documents/Deep Learning Libraries/returnn/NetworkTwoDLayer.py", line 450, in __init__
line: fixed_X, _ = theano.scan(pad_fn, [self.X.dimshuffle(2,0,1,3), T.cast(sizes_raw, "int32")])
locals:
fixed_X = <not found>
_ = <not found>
theano = <global> <module 'theano' from '/home/deep/anaconda3/lib/python3.6/site-packages/theano/__init__.py'>
theano.scan = <global> <function scan at 0x7f9dcd954ea0>
pad_fn = <local> <function ConvPoolLayer2.__init__.<locals>.pad_fn at 0x7f9db8e2eea0>
self = <local> <<class 'NetworkTwoDLayer.ConvPoolLayer2'> class:conv2 name:conv2>
self.X = <local> Y
self.X.dimshuffle = <local> <bound method _tensor_py_operators.dimshuffle of Y>
T = <global> <module 'theano.tensor' from '/home/deep/anaconda3/lib/python3.6/site-packages/theano/tensor/__init__.py'>
T.cast = <global> <function cast at 0x7f9dcf22c598>
sizes_raw = <local> Join.0
File "/home/deep/anaconda3/lib/python3.6/site-packages/theano/scan_module/scan.py", line 845, in scan
line: dummy_f = function(dummy_args,
dummy_outs,
updates=updates,
mode=compile.mode.Mode(linker='py',
optimizer=None),
on_unused_input='ignore',
profile=False)
locals:
dummy_f = <not found>
function = <global> <function function at 0x7f9df0244a60>
dummy_args = <local> [<TensorType(float32, 3D)>, <TensorType(int32, vector)>, y_sizes, x]
dummy_outs = <local> [IncSubtensor{Set;:int32:, :int32:}.0]
updates = <local> OrderedDict(), len = 0
mode = <local> None
compile = <global> <module 'theano.compile' from '/home/deep/anaconda3/lib/python3.6/site-packages/theano/compile/__init__.py'>
compile.mode = <global> <module 'theano.compile.mode' from '/home/deep/anaconda3/lib/python3.6/site-packages/theano/compile/mode.py'>
compile.mode.Mode = <global> <class 'theano.compile.mode.Mode'>
linker = <not found>
optimizer = <not found>
on_unused_input = <not found>
profile = <local> False
File "/home/deep/anaconda3/lib/python3.6/site-packages/theano/compile/function.py", line 326, in function
line: fn = pfunc(params=inputs,
outputs=outputs,
mode=mode,
updates=updates,
givens=givens,
no_default_updates=no_default_updates,
accept_inplace=accept_inplace, name=name,
rebuild_strict=rebuild_strict,
allow_input_downcast=allow_input_downcast,
on_unused_input=on_unused_input,
profile=profile,
output_keys=output_keys)
locals:
fn = <not found>
pfunc = <global> <function pfunc at 0x7f9df0296840>
params = <not found>
inputs = <local> [<TensorType(float32, 3D)>, <TensorType(int32, vector)>, y_sizes, x]
outputs = <local> [IncSubtensor{Set;:int32:, :int32:}.0]
mode = <local> <theano.compile.mode.Mode object at 0x7f9dabc7c828>
updates = <local> OrderedDict(), len = 0
givens = <local> []
no_default_updates = <local> False
accept_inplace = <local> False
name = <local> '/home/deep/anaconda3/lib/python3.6/site-packages/theano/scan_module/scan.py:845', len = 79
rebuild_strict = <local> True
allow_input_downcast = <local> None
on_unused_input = <local> 'ignore', len = 6
profile = <local> False
output_keys = <local> None
File "/home/deep/anaconda3/lib/python3.6/site-packages/theano/compile/pfunc.py", line 486, in pfunc
line: return orig_function(inputs, cloned_outputs, mode,
accept_inplace=accept_inplace, name=name,
profile=profile, on_unused_input=on_unused_input,
output_keys=output_keys)
locals:
orig_function = <global> <function orig_function at 0x7f9df027b268>
inputs = <local> [In(<TensorType(float32, 3D)>), In(<TensorType(int32, vector)>), In(y_sizes), In(x), In(<CudaNdarrayType(float32, vector)> -> GPU_mrg_uniform{CudaNdarrayType(float32, vector),no_inplace}.0), In(<CudaNdarrayType(float32, vector)> -> GPU_mrg_uniform{CudaNdarrayType(float32, vector),no_inplace}.0), ..., len = 42
cloned_outputs = <local> [IncSubtensor{Set;:int32:, :int32:}.0]
mode = <local> <theano.compile.mode.Mode object at 0x7f9dabc7c828>
accept_inplace = <local> False
name = <local> '/home/deep/anaconda3/lib/python3.6/site-packages/theano/scan_module/scan.py:845', len = 79
profile = <local> False
on_unused_input = <local> 'ignore', len = 6
output_keys = <local> None
File "/home/deep/anaconda3/lib/python3.6/site-packages/theano/compile/function_module.py", line 1795, in orig_function
line: fn = Maker(inputs,
outputs,
mode,
accept_inplace=accept_inplace,
profile=profile,
on_unused_input=on_unused_input,
output_keys=output_keys).create(
defaults)
locals:
fn = <not found>
Maker = <local> <class 'theano.compile.function_module.FunctionMaker'>
inputs = <local> [In(<TensorType(float32, 3D)>), In(<TensorType(int32, vector)>), In(y_sizes), In(x), In(<CudaNdarrayType(float32, vector)> -> GPU_mrg_uniform{CudaNdarrayType(float32, vector),no_inplace}.0), In(<CudaNdarrayType(float32, vector)> -> GPU_mrg_uniform{CudaNdarrayType(float32, vector),no_inplace}.0), ..., len = 42
outputs = <local> [Out(IncSubtensor{Set;:int32:, :int32:}.0,False)]
mode = <local> <theano.compile.mode.Mode object at 0x7f9dabc7c828>
accept_inplace = <local> False
profile = <local> False
on_unused_input = <local> 'ignore', len = 6
output_keys = <local> None
create = <not found>
defaults = <local> [None, None, None, None, <b'CudaNdarray([ 1.08880891e-42 1.08880891e-42 1.08880891e-42 1.08880891e-42\n 1.08880891e-42 1.08880891e-42 9.41522460e+29 3.97754038e-21\n 9.17507681e-10 2.55124480e-01 1.69304648e+19 8.61674954e-35\n 3.21907123e+09 1.25503928e+10 2.45272753..., len = 42
File "/home/deep/anaconda3/lib/python3.6/site-packages/theano/compile/function_module.py", line 1661, in create
line: _fn, _i, _o = self.linker.make_thunk(
input_storage=input_storage_lists, storage_map=storage_map)
locals:
_fn = <not found>
_i = <not found>
_o = <not found>
self = <local> <theano.compile.function_module.FunctionMaker object at 0x7f9db8a1d4a8>
self.linker = <local> <theano.gof.link.PerformLinker object at 0x7f9db8b6a518>
self.linker.make_thunk = <local> <bound method LocalLinker.make_thunk of <theano.gof.link.PerformLinker object at 0x7f9db8b6a518>>
input_storage = <local> [None, None, None, None, <b'CudaNdarray([ 1.08880891e-42 1.08880891e-42 1.08880891e-42 1.08880891e-42\n 1.08880891e-42 1.08880891e-42 9.41522460e+29 3.97754038e-21\n 9.17507681e-10 2.55124480e-01 1.69304648e+19 8.61674954e-35\n 3.21907123e+09 1.25503928e+10 2.45272753..., len = 42
input_storage_lists = <local> [[None], [None], [None], [None], [b'CudaNdarray([ 1.08880891e-42 1.08880891e-42 1.08880891e-42 1.08880891e-42\n 1.08880891e-42 1.08880891e-42 9.41522460e+29 3.97754038e-21\n 9.17507681e-10 2.55124480e-01 1.69304648e+19 8.61674954e-35\n 3.21907123e+09 1.25503928e+10 2...., len = 42
storage_map = <local> None
File "/home/deep/anaconda3/lib/python3.6/site-packages/theano/gof/link.py", line 699, in make_thunk
line: return self.make_all(input_storage=input_storage,
output_storage=output_storage,
storage_map=storage_map)[:3]
locals:
self = <local> <theano.gof.link.PerformLinker object at 0x7f9db8b6a518>
self.make_all = <local> <bound method PerformLinker.make_all of <theano.gof.link.PerformLinker object at 0x7f9db8b6a518>>
input_storage = <local> [[None], [None], [None], [None], [b'CudaNdarray([ 1.08880891e-42 1.08880891e-42 1.08880891e-42 1.08880891e-42\n 1.08880891e-42 1.08880891e-42 9.41522460e+29 3.97754038e-21\n 9.17507681e-10 2.55124480e-01 1.69304648e+19 8.61674954e-35\n 3.21907123e+09 1.25503928e+10 2...., len = 42
output_storage = <local> None
storage_map = <local> None
File "/home/deep/anaconda3/lib/python3.6/site-packages/theano/gof/link.py", line 830, in make_all
line: thunks += [node.op.make_thunk(node,
storage_map,
compute_map,
no_recycling,
'py')]
locals:
thunks = <local> [<function Op.make_py_thunk.<locals>.rval at 0x7f9db83929d8>, <function Op.make_py_thunk.<locals>.rval at 0x7f9db8179c80>, <function Op.make_py_thunk.<locals>.rval at 0x7f9db8f91a60>, <function Op.make_py_thunk.<locals>.rval at 0x7f9db8e1e1e0>, <function Op.make_py_thunk.<locals>.rval at 0x7f9db8..., len = 283
node = <local> for{cpu,scan_fn}(Elemwise{minimum,no_inplace}.0, Subtensor{:int64:}.0, Subtensor{:int64:}.0, IncSubtensor{Set;:int64:}.0, Elemwise{minimum,no_inplace}.0, HostFromGpu.0, HostFromGpu.0, HostFromGpu.0, HostFromGpu.0, HostFromGpu.0, HostFromGpu.0, HostFromGpu.0, HostFromGpu.0, HostFromGpu.0, HostFrom...
node.op = <local> <theano.scan_module.scan_op.Scan object at 0x7f9db813e780>
node.op.make_thunk = <local> <bound method Scan.make_thunk of <theano.scan_module.scan_op.Scan object at 0x7f9db813e780>>
storage_map = <local> {<TensorType(float32, 3D)>: [None], <TensorType(int32, vector)>: [None], y_sizes: [None], x: [None], <CudaNdarrayType(float32, vector)>: [b'CudaNdarray([ 1.08880891e-42 1.08880891e-42 1.08880891e-42 1.08880891e-42\n 1.08880891e-42 1.08880891e-42 9.41522460e+29 3.97754038e-21\n 9...., len = 434
compute_map = <local> {<TensorType(float32, 3D)>: [True], <TensorType(int32, vector)>: [True], y_sizes: [True], x: [True], <CudaNdarrayType(float32, vector)>: [True], <CudaNdarrayType(float32, vector)>: [True], W_conv0: [True], b_conv0: [True], W1_mdlstm0: [True], W2_mdlstm0: [True], W3_mdlstm0: [True], W4_mdlstm0: [T..., len = 434
no_recycling = <local> {IncSubtensor{Set;:int32:, :int32:}.0, GPU_mrg_uniform{CudaNdarrayType(float32, vector),no_inplace}.0, GPU_mrg_uniform{CudaNdarrayType(float32, vector),no_inplace}.0}, len = 3
File "/home/deep/anaconda3/lib/python3.6/site-packages/theano/scan_module/scan_op.py", line 892, in make_thunk
line: self.fn = function(wrapped_inputs,
wrapped_outputs,
mode=compilation_mode,
name=self.name,
profile=profile,
on_unused_input='ignore')
locals:
self = <local> <theano.scan_module.scan_op.Scan object at 0x7f9db813e780>
self.fn = <local> !AttributeError: 'Scan' object has no attribute 'fn'
function = <global> <function function at 0x7f9df0244a60>
wrapped_inputs = <local> [In(<TensorType(float32, 3D)>), In(<TensorType(int32, vector)>), In(<TensorType(float32, vector)>), In(W_conv0_copy), In(b_conv0_copy), In(W1_mdlstm0_copy), In(W2_mdlstm0_copy), In(W3_mdlstm0_copy), In(W4_mdlstm0_copy), In(U1_mdlstm0_copy), In(U2_mdlstm0_copy), In(U3_mdlstm0_copy), In(U4_mdlstm0_..., len = 23
wrapped_outputs = <local> [Out(HostFromGpu.0,True), Out(IncSubtensor{Set;:int32:, :int32:}.0,True)]
mode = <not found>
compilation_mode = <local> <theano.compile.mode.Mode object at 0x7f9dabbafb00>
name = <not found>
self.name = <local> 'scan_fn', len = 7
profile = <local> None
on_unused_input = <not found>
File "/home/deep/anaconda3/lib/python3.6/site-packages/theano/compile/function.py", line 326, in function
line: fn = pfunc(params=inputs,
outputs=outputs,
mode=mode,
updates=updates,
givens=givens,
no_default_updates=no_default_updates,
accept_inplace=accept_inplace, name=name,
rebuild_strict=rebuild_strict,
allow_input_downcast=allow_input_downcast,
on_unused_input=on_unused_input,
profile=profile,
output_keys=output_keys)
locals:
fn = <not found>
pfunc = <global> <function pfunc at 0x7f9df0296840>
params = <not found>
inputs = <local> [In(<TensorType(float32, 3D)>), In(<TensorType(int32, vector)>), In(<TensorType(float32, vector)>), In(W_conv0_copy), In(b_conv0_copy), In(W1_mdlstm0_copy), In(W2_mdlstm0_copy), In(W3_mdlstm0_copy), In(W4_mdlstm0_copy), In(U1_mdlstm0_copy), In(U2_mdlstm0_copy), In(U3_mdlstm0_copy), In(U4_mdlstm0_..., len = 23
outputs = <local> [Out(HostFromGpu.0,True), Out(IncSubtensor{Set;:int32:, :int32:}.0,True)]
mode = <local> <theano.compile.mode.Mode object at 0x7f9dabbafb00>
updates = <local> []
givens = <local> []
no_default_updates = <local> False
accept_inplace = <local> False
name = <local> 'scan_fn', len = 7
rebuild_strict = <local> True
allow_input_downcast = <local> None
on_unused_input = <local> 'ignore', len = 6
profile = <local> None
output_keys = <local> None
File "/home/deep/anaconda3/lib/python3.6/site-packages/theano/compile/pfunc.py", line 486, in pfunc
line: return orig_function(inputs, cloned_outputs, mode,
accept_inplace=accept_inplace, name=name,
profile=profile, on_unused_input=on_unused_input,
output_keys=output_keys)
locals:
orig_function = <global> <function orig_function at 0x7f9df027b268>
inputs = <local> [In(<TensorType(float32, 3D)>), In(<TensorType(int32, vector)>), In(<TensorType(float32, vector)>), In(W_conv0_copy), In(b_conv0_copy), In(W1_mdlstm0_copy), In(W2_mdlstm0_copy), In(W3_mdlstm0_copy), In(W4_mdlstm0_copy), In(U1_mdlstm0_copy), In(U2_mdlstm0_copy), In(U3_mdlstm0_copy), In(U4_mdlstm0_..., len = 23
cloned_outputs = <local> [Out(HostFromGpu.0,True), Out(IncSubtensor{Set;:int32:, :int32:}.0,True)]
mode = <local> <theano.compile.mode.Mode object at 0x7f9dabbafb00>
accept_inplace = <local> False
name = <local> 'scan_fn', len = 7
profile = <local> None
on_unused_input = <local> 'ignore', len = 6
output_keys = <local> None
File "/home/deep/anaconda3/lib/python3.6/site-packages/theano/compile/function_module.py", line 1794, in orig_function
line: fn = Maker(inputs,
outputs,
mode,
accept_inplace=accept_inplace,
profile=profile,
on_unused_input=on_unused_input,
output_keys=output_keys).create(
locals:
fn = <not found>
Maker = <local> <class 'theano.compile.function_module.FunctionMaker'>
inputs = <local> [In(<TensorType(float32, 3D)>), In(<TensorType(int32, vector)>), In(<TensorType(float32, vector)>), In(W_conv0_copy), In(b_conv0_copy), In(W1_mdlstm0_copy), In(W2_mdlstm0_copy), In(W3_mdlstm0_copy), In(W4_mdlstm0_copy), In(U1_mdlstm0_copy), In(U2_mdlstm0_copy), In(U3_mdlstm0_copy), In(U4_mdlstm0_..., len = 23
outputs = <local> [Out(HostFromGpu.0,True), Out(IncSubtensor{Set;:int32:, :int32:}.0,True)]
mode = <local> <theano.compile.mode.Mode object at 0x7f9dabbafb00>
accept_inplace = <local> False
profile = <local> None
on_unused_input = <local> 'ignore', len = 6
output_keys = <local> None
create = <not found>
File "/home/deep/anaconda3/lib/python3.6/site-packages/theano/compile/function_module.py", line 1474, in __init__
line: optimizer_profile = optimizer(fgraph)
locals:
optimizer_profile = <not found>
optimizer = <local> [<theano.gof.opt.TopoOptimizer object at 0x7f9df02236d8>, <theano.gof.opt.MergeOptimizer object at 0x7f9df600d320>, <theano.gof.opt.EquilibriumOptimizer object at 0x7f9dabb46320>, <theano.tensor.opt.ShapeOptimizer object at 0x7f9dcefbb1d0>, <theano.gof.opt.TopoOptimizer object at 0x7f9db9934780>,..., len = 79
fgraph = <local> [HostFromGpu(*10 -> GPU_mrg_uniform{CudaNdarrayType(float32, vector),no_inplace}(GpuFromHost(<TensorType(float32, vector)>), TensorConstant{(1,) of 15})), IncSubtensor{Set;:int32:, :int32:}(Alloc(TensorConstant{0.0}, Subtensor{int64}(*4 -> Elemwise{Cast{int32}}(MaxAndArgmax{axis=(0,)}(if{}(Print{...
File "/home/deep/anaconda3/lib/python3.6/site-packages/theano/gof/opt.py", line 98, in __call__
line: return self.optimize(fgraph)
locals:
self = <local> [<theano.gof.opt.TopoOptimizer object at 0x7f9df02236d8>, <theano.gof.opt.MergeOptimizer object at 0x7f9df600d320>, <theano.gof.opt.EquilibriumOptimizer object at 0x7f9dabb46320>, <theano.tensor.opt.ShapeOptimizer object at 0x7f9dcefbb1d0>, <theano.gof.opt.TopoOptimizer object at 0x7f9db9934780>,..., len = 79
self.optimize = <local> <bound method Optimizer.optimize of [<theano.gof.opt.TopoOptimizer object at 0x7f9df02236d8>, <theano.gof.opt.MergeOptimizer object at 0x7f9df600d320>, <theano.gof.opt.EquilibriumOptimizer object at 0x7f9dabb46320>, <theano.tensor.opt.ShapeOptimizer object at 0x7f9dcefbb1d0>, <theano.gof.opt.Topo...
fgraph = <local> [HostFromGpu(*10 -> GPU_mrg_uniform{CudaNdarrayType(float32, vector),no_inplace}(GpuFromHost(<TensorType(float32, vector)>), TensorConstant{(1,) of 15})), IncSubtensor{Set;:int32:, :int32:}(Alloc(TensorConstant{0.0}, Subtensor{int64}(*4 -> Elemwise{Cast{int32}}(MaxAndArgmax{axis=(0,)}(if{}(Print{...
File "/home/deep/anaconda3/lib/python3.6/site-packages/theano/gof/opt.py", line 87, in optimize
line: ret = self.apply(fgraph, *args, **kwargs)
locals:
ret = <not found>
self = <local> [<theano.gof.opt.TopoOptimizer object at 0x7f9df02236d8>, <theano.gof.opt.MergeOptimizer object at 0x7f9df600d320>, <theano.gof.opt.EquilibriumOptimizer object at 0x7f9dabb46320>, <theano.tensor.opt.ShapeOptimizer object at 0x7f9dcefbb1d0>, <theano.gof.opt.TopoOptimizer object at 0x7f9db9934780>,..., len = 79
self.apply = <local> <bound method SeqOptimizer.apply of [<theano.gof.opt.TopoOptimizer object at 0x7f9df02236d8>, <theano.gof.opt.MergeOptimizer object at 0x7f9df600d320>, <theano.gof.opt.EquilibriumOptimizer object at 0x7f9dabb46320>, <theano.tensor.opt.ShapeOptimizer object at 0x7f9dcefbb1d0>, <theano.gof.opt.Topo...
fgraph = <local> [HostFromGpu(*10 -> GPU_mrg_uniform{CudaNdarrayType(float32, vector),no_inplace}(GpuFromHost(<TensorType(float32, vector)>), TensorConstant{(1,) of 15})), IncSubtensor{Set;:int32:, :int32:}(Alloc(TensorConstant{0.0}, Subtensor{int64}(*4 -> Elemwise{Cast{int32}}(MaxAndArgmax{axis=(0,)}(if{}(Print{...
args = <local> ()
kwargs = <local> {}
File "/home/deep/anaconda3/lib/python3.6/site-packages/theano/gof/opt.py", line 235, in apply
line: sub_prof = optimizer.optimize(fgraph)
locals:
sub_prof = <local> (<theano.gof.opt.EquilibriumOptimizer object at 0x7f9dabb46320>, [0.01727128028869629, 0.011417388916015625, 0.00965261459350586], [{[<theano.gof.opt.TopoOptimizer object at 0x7f9dcd949fd0>, <theano.scan_module.scan_opt.PushOutNonSeqScan object at 0x7f9dcd958198>, <theano.scan_module.scan_opt.Pus..., len = 12
optimizer = <local> <theano.tensor.opt.ShapeOptimizer object at 0x7f9dcefbb1d0>
optimizer.optimize = <local> <bound method Optimizer.optimize of <theano.tensor.opt.ShapeOptimizer object at 0x7f9dcefbb1d0>>
fgraph = <local> [HostFromGpu(*10 -> GPU_mrg_uniform{CudaNdarrayType(float32, vector),no_inplace}(GpuFromHost(<TensorType(float32, vector)>), TensorConstant{(1,) of 15})), IncSubtensor{Set;:int32:, :int32:}(Alloc(TensorConstant{0.0}, Subtensor{int64}(*4 -> Elemwise{Cast{int32}}(MaxAndArgmax{axis=(0,)}(if{}(Print{...
File "/home/deep/anaconda3/lib/python3.6/site-packages/theano/gof/opt.py", line 83, in optimize
line: self.add_requirements(fgraph)
locals:
self = <local> <theano.tensor.opt.ShapeOptimizer object at 0x7f9dcefbb1d0>
self.add_requirements = <local> <bound method ShapeOptimizer.add_requirements of <theano.tensor.opt.ShapeOptimizer object at 0x7f9dcefbb1d0>>
fgraph = <local> [HostFromGpu(*10 -> GPU_mrg_uniform{CudaNdarrayType(float32, vector),no_inplace}(GpuFromHost(<TensorType(float32, vector)>), TensorConstant{(1,) of 15})), IncSubtensor{Set;:int32:, :int32:}(Alloc(TensorConstant{0.0}, Subtensor{int64}(*4 -> Elemwise{Cast{int32}}(MaxAndArgmax{axis=(0,)}(if{}(Print{...
File "/home/deep/anaconda3/lib/python3.6/site-packages/theano/tensor/opt.py", line 1482, in add_requirements
line: fgraph.attach_feature(ShapeFeature())
locals:
fgraph = <local> [HostFromGpu(*10 -> GPU_mrg_uniform{CudaNdarrayType(float32, vector),no_inplace}(GpuFromHost(<TensorType(float32, vector)>), TensorConstant{(1,) of 15})), IncSubtensor{Set;:int32:, :int32:}(Alloc(TensorConstant{0.0}, Subtensor{int64}(*4 -> Elemwise{Cast{int32}}(MaxAndArgmax{axis=(0,)}(if{}(Print{...
fgraph.attach_feature = <local> <bound method FunctionGraph.attach_feature of [HostFromGpu(*10 -> GPU_mrg_uniform{CudaNdarrayType(float32, vector),no_inplace}(GpuFromHost(<TensorType(float32, vector)>), TensorConstant{(1,) of 15})), IncSubtensor{Set;:int32:, :int32:}(Alloc(TensorConstant{0.0}, Subtensor{int64}(*4 -> Elemwise{Ca...
ShapeFeature = <global> <class 'theano.tensor.opt.ShapeFeature'>
File "/home/deep/anaconda3/lib/python3.6/site-packages/theano/gof/fg.py", line 541, in attach_feature
line: attach(self)
locals:
attach = <local> <bound method ShapeFeature.on_attach of <theano.tensor.opt.ShapeFeature object at 0x7f9dabb46da0>>
self = <local> [HostFromGpu(*10 -> GPU_mrg_uniform{CudaNdarrayType(float32, vector),no_inplace}(GpuFromHost(<TensorType(float32, vector)>), TensorConstant{(1,) of 15})), IncSubtensor{Set;:int32:, :int32:}(Alloc(TensorConstant{0.0}, Subtensor{int64}(*4 -> Elemwise{Cast{int32}}(MaxAndArgmax{axis=(0,)}(if{}(Print{...
File "/home/deep/anaconda3/lib/python3.6/site-packages/theano/tensor/opt.py", line 1299, in on_attach
line: self.on_import(fgraph, node, reason='on_attach')
locals:
self = <local> <theano.tensor.opt.ShapeFeature object at 0x7f9dabb46da0>
self.on_import = <local> <bound method ShapeFeature.on_import of <theano.tensor.opt.ShapeFeature object at 0x7f9dabb46da0>>
fgraph = <local> [HostFromGpu(*10 -> GPU_mrg_uniform{CudaNdarrayType(float32, vector),no_inplace}(GpuFromHost(<TensorType(float32, vector)>), TensorConstant{(1,) of 15})), IncSubtensor{Set;:int32:, :int32:}(Alloc(TensorConstant{0.0}, Subtensor{int64}(*4 -> Elemwise{Cast{int32}}(MaxAndArgmax{axis=(0,)}(if{}(Print{...
node = <local> MultiDirectionalTwoDLSTMOp(GpuContiguous.0, GpuContiguous.0, GpuContiguous.0, GpuContiguous.0, GpuContiguous.0, GpuContiguous.0, GpuContiguous.0, GpuContiguous.0, GpuContiguous.0, GpuContiguous.0, GpuContiguous.0, GpuContiguous.0, GpuContiguous.0, GpuContiguous.0, GpuContiguous.0, GpuContiguous.0...
reason = <not found>
File "/home/deep/anaconda3/lib/python3.6/site-packages/theano/tensor/opt.py", line 1347, in on_import
line: assert d.dtype in theano.tensor.discrete_dtypes, (node, d.dtype)
locals:
d = <local> Elemwise{true_div,no_inplace}.0
d.dtype = <local> 'float32', len = 7
theano = <global> <module 'theano' from '/home/deep/anaconda3/lib/python3.6/site-packages/theano/__init__.py'>
theano.tensor = <global> <module 'theano.tensor' from '/home/deep/anaconda3/lib/python3.6/site-packages/theano/tensor/__init__.py'>
theano.tensor.discrete_dtypes = <global> ['bool', 'int8', 'int16', 'int32', 'int64', 'uint8', 'uint16', 'uint32', 'uint64'], len = 9
node = <local> MultiDirectionalTwoDLSTMOp(GpuContiguous.0, GpuContiguous.0, GpuContiguous.0, GpuContiguous.0, GpuContiguous.0, GpuContiguous.0, GpuContiguous.0, GpuContiguous.0, GpuContiguous.0, GpuContiguous.0, GpuContiguous.0, GpuContiguous.0, GpuContiguous.0, GpuContiguous.0, GpuContiguous.0, GpuContiguous.0...
AssertionError: (MultiDirectionalTwoDLSTMOp(GpuContiguous.0, GpuContiguous.0, GpuContiguous.0, GpuContiguous.0, GpuContiguous.0, GpuContiguous.0, GpuContiguous.0, GpuContiguous.0, GpuContiguous.0, GpuContiguous.0, GpuContiguous.0, GpuContiguous.0, GpuContiguous.0, GpuContiguous.0, GpuContiguous.0, GpuContiguous.0, GpuContiguous.0, Join.0), 'float32')
Device proc gpuX (gpuZ) died: ProcConnectionDied('recv_bytes EOFError: ',)
Theano flags: compiledir_format=compiledir_%(platform)s-%(processor)s-%(python_version)s-%(python_bitwidth)s--dev-gpuZ,device=gpu,force_device=True
EXCEPTION
Traceback (most recent call last):
File "/home/deep/Documents/Deep Learning Libraries/returnn/Device.py", line 329, in startProc
line: self._startProc(*args, **kwargs)
locals:
self = <local> <Device.Device object at 0x7f1808dbf4e0>
self._startProc = <local> <bound method Device._startProc of <Device.Device object at 0x7f1808dbf4e0>>
args = <local> ('gpuZ',)
kwargs = <local> {}
File "/home/deep/Documents/Deep Learning Libraries/returnn/Device.py", line 383, in _startProc
line: interrupt_main()
locals:
interrupt_main = <global> <function interrupt_main at 0x7f180e034ae8>
File "/home/deep/Documents/Deep Learning Libraries/returnn/Util.py", line 607, in interrupt_main
line: sys.exit(1) # And exit the thread.
locals:
sys = <global> <module 'sys' (built-in)>
sys.exit = <global> <built-in function exit>
SystemExit: 1
KeyboardInterrupt
EXCEPTION
Traceback (most recent call last):
File "../../../rnn.py", line 520, in main
line: init(commandLineOptions=argv[1:])
locals:
init = <global> <function init at 0x7f1808dcf950>
commandLineOptions = <not found>
argv = <local> ['../../../rnn.py', 'config_demo'], _[0]: {len = 15}
File "../../../rnn.py", line 336, in init
line: devices = initDevices()
locals:
devices = <not found>
initDevices = <global> <function initDevices at 0x7f1808dcf488>
File "../../../rnn.py", line 152, in initDevices
line: time.sleep(0.25)
locals:
time = <global> <module 'time' (built-in)>
time.sleep = <global> <built-in function sleep>
KeyboardInterrupt
Quitting
from returnn.
pvoigtlaender
commented on July 17, 2024
Unfortunately I don't have python3 with theano set up at the moment, that's why we're only fixing one problem at a time :D
I pushed another fix, please try again.
from returnn.
pedro-dreyer
commented on July 17, 2024
No worries. I am glad that you taking your time to do the fixes anyway :)
deep@deep-System-Product-Name ~/Documents/Deep Learning Libraries/returnn/demos/mdlstm/IAM $ ./go.sh
converting IAM_lines to features/raw/demo.h5
features/raw/demo.h5
0 / 3
CRNN starting up, version 20171012.181826--git-7c2155f-dirty, pid 12672, cwd /home/deep/Documents/Deep Learning Libraries/returnn/demos/mdlstm/IAM
CRNN command line options: ['config_demo']
Theano: 0.9.0.dev-c697eeab84... (<site-package> in /home/deep/anaconda3/lib/python3.6/site-packages/theano)
pynvml not available, memory information missing
WARNING (theano.sandbox.cuda): The cuda backend is deprecated and will be removed in the next release (v0.10). Please switch to the gpuarray backend. You can get more information about how to switch at this URL:
https://github.com/Theano/Theano/wiki/Converting-to-the-new-gpu-back-end%28gpuarray%29
Using gpu device 0: TITAN Xp (CNMeM is enabled with initial size: 95.0% of memory, cuDNN 5110)
Device gpuX proc starting up, pid 12700
Device gpuX proc: THEANO_FLAGS = 'compiledir_format=compiledir_%(platform)s-%(processor)s-%(python_version)s-%(python_bitwidth)s--dev-gpuZ,device=gpu,force_device=True'
Device train-network: Used data keys: ['classes', 'data', 'sizes']
using adam with nag and momentum schedule
Device gpuX proc exception: /home/deep/.theano/compiledir_Linux-4.8.0-53-generic-x86_64-with-debian-stretch-sid-x86_64-3.6.1-64--dev-gpuZ/tmp29yq2k9e/mcbc70be4d8fab58f670977cc113044f1.so: undefined symbol: _ZTVNSt7__cxx1118basic_stringstreamIcSt11char_traitsIcESaIcEEE
Unhandled exception <class 'ImportError'> in thread <_MainThread(MainThread, started 140580834899712)>, proc 12700.
Thread current, main, <_MainThread(MainThread, started 140580834899712)>:
(Excluded thread.)
That were all threads.
EXCEPTION
Traceback (most recent call last):
File "/home/deep/Documents/Deep Learning Libraries/returnn/Device.py", line 980, in process
line: self.process_inner(device, config, self.update_specs, asyncTask)
locals:
self = <local> <Device.Device object at 0x7fdb81a9b898>
self.process_inner = <local> <bound method Device.process_inner of <Device.Device object at 0x7fdb81a9b898>>
device = <local> 'gpuX'
config = <local> <Config.Config object at 0x7fdb4a2a4390>
self.update_specs = <local> {'update_rule': 'global', 'update_params': {}, 'layers': [], 'block_size': 0}
asyncTask = <local> <TaskSystem.AsyncTask object at 0x7fdb6ca73eb8>
File "/home/deep/Documents/Deep Learning Libraries/returnn/Device.py", line 1033, in process_inner
line: self.initialize(config, update_specs=update_specs)
locals:
self = <local> <Device.Device object at 0x7fdb81a9b898>
self.initialize = <local> <bound method Device.initialize of <Device.Device object at 0x7fdb81a9b898>>
config = <local> <Config.Config object at 0x7fdb4a2a4390>
update_specs = <local> {'update_rule': 'global', 'update_params': {}, 'layers': [], 'block_size': 0}
File "/home/deep/Documents/Deep Learning Libraries/returnn/Device.py", line 576, in initialize
line: self.trainer = theano.function(inputs=[self.block_start, self.block_end],
outputs=outputs,
givens=train_givens,
updates=self.updater.getUpdateList(),
on_unused_input=config.value('theano_on_unused_input', 'ignore'),
no_default_updates=exclude,
name="train_and_updater")
locals:
self = <local> <Device.Device object at 0x7fdb81a9b898>
self.trainer = <local> !AttributeError: 'Device' object has no attribute 'trainer'
theano = <local> <module 'theano' from '/home/deep/anaconda3/lib/python3.6/site-packages/theano/__init__.py'>
theano.function = <local> <function function at 0x7fdb80c5da60>
inputs = <not found>
self.block_start = <local> <TensorType(int64, scalar)>
self.block_end = <local> <TensorType(int64, scalar)>
outputs = <local> [Sum{acc_dtype=float64}.0, Sum{axis=[0], acc_dtype=float64}.0]
givens = <not found>
train_givens = <local> [(y_classes, Subtensor{::, int64:int64:}.0), (x, Subtensor{::, int64:int64:}.0), (y_sizes, Subtensor{::, int64:int64:}.0), (j_classes, Subtensor{::, int64:int64:}.0), (i, Subtensor{::, int64:int64:}.0), (j_sizes, Subtensor{::, int64:int64:}.0), (TensorConstant{0}, epoch_var), (tags, tags_var)], len = 8
updates = <not found>
self.updater = <local> <Updater.Updater object at 0x7fdb0a334828>
self.updater.getUpdateList = <local> <bound method Updater.getUpdateList of <Updater.Updater object at 0x7fdb0a334828>>
on_unused_input = <not found>
config = <local> <Config.Config object at 0x7fdb4a2a4390>
config.value = <local> <bound method Config.value of <Config.Config object at 0x7fdb4a2a4390>>
no_default_updates = <not found>
exclude = <local> []
name = <not found>
File "/home/deep/anaconda3/lib/python3.6/site-packages/theano/compile/function.py", line 326, in function
line: fn = pfunc(params=inputs,
outputs=outputs,
mode=mode,
updates=updates,
givens=givens,
no_default_updates=no_default_updates,
accept_inplace=accept_inplace, name=name,
rebuild_strict=rebuild_strict,
allow_input_downcast=allow_input_downcast,
on_unused_input=on_unused_input,
profile=profile,
output_keys=output_keys)
locals:
fn = <not found>
pfunc = <global> <function pfunc at 0x7fdb80caf840>
params = <not found>
inputs = <local> [<TensorType(int64, scalar)>, <TensorType(int64, scalar)>]
outputs = <local> [Sum{acc_dtype=float64}.0, Sum{axis=[0], acc_dtype=float64}.0]
mode = <local> None
updates = <local> [(counter, Elemwise{add,no_inplace}.0), (prev_epoch, TensorConstant{0}), (updater_epoch, Elemwise{switch,no_inplace}.0), (momemtum_cache, Elemwise{mul,no_inplace}.0), (nadam_m_W_conv0, Elemwise{add,no_inplace}.0), (nadam_v_W_conv0, Elemwise{add,no_inplace}.0), (momemtum_cache, Elemwise{mul,no_inp..., len = 372
givens = <local> [(y_classes, Subtensor{::, int64:int64:}.0), (x, Subtensor{::, int64:int64:}.0), (y_sizes, Subtensor{::, int64:int64:}.0), (j_classes, Subtensor{::, int64:int64:}.0), (i, Subtensor{::, int64:int64:}.0), (j_sizes, Subtensor{::, int64:int64:}.0), (TensorConstant{0}, epoch_var), (tags, tags_var)], len = 8
no_default_updates = <local> []
accept_inplace = <local> False
name = <local> 'train_and_updater', len = 17
rebuild_strict = <local> True
allow_input_downcast = <local> None
on_unused_input = <local> 'ignore', len = 6
profile = <local> None
output_keys = <local> None
File "/home/deep/anaconda3/lib/python3.6/site-packages/theano/compile/pfunc.py", line 486, in pfunc
line: return orig_function(inputs, cloned_outputs, mode,
accept_inplace=accept_inplace, name=name,
profile=profile, on_unused_input=on_unused_input,
output_keys=output_keys)
locals:
orig_function = <global> <function orig_function at 0x7fdb80c94268>
inputs = <local> [In(<TensorType(int64, scalar)>), In(<TensorType(int64, scalar)>), In(y_classes), In(y_data), In(y_sizes), In(j_classes), In(j_data), In(j_sizes), In(epoch_var), In(tags_var), In(b_output -> GpuFromHost.0), In(<CudaNdarrayType(float32, vector)> -> GPU_mrg_uniform{CudaNdarrayType(float32, vector),..., len = 389
cloned_outputs = <local> [Sum{acc_dtype=float64}.0, Sum{axis=[0], acc_dtype=float64}.0]
mode = <local> None
accept_inplace = <local> False
name = <local> 'train_and_updater', len = 17
profile = <local> None
on_unused_input = <local> 'ignore', len = 6
output_keys = <local> None
File "/home/deep/anaconda3/lib/python3.6/site-packages/theano/compile/function_module.py", line 1795, in orig_function
line: fn = Maker(inputs,
outputs,
mode,
accept_inplace=accept_inplace,
profile=profile,
on_unused_input=on_unused_input,
output_keys=output_keys).create(
defaults)
locals:
fn = <not found>
Maker = <local> <class 'theano.compile.function_module.FunctionMaker'>
inputs = <local> [In(<TensorType(int64, scalar)>), In(<TensorType(int64, scalar)>), In(y_classes), In(y_data), In(y_sizes), In(j_classes), In(j_data), In(j_sizes), In(epoch_var), In(tags_var), In(b_output -> GpuFromHost.0), In(<CudaNdarrayType(float32, vector)> -> GPU_mrg_uniform{CudaNdarrayType(float32, vector),..., len = 389
outputs = <local> [Out(Sum{acc_dtype=float64}.0,False), Out(Sum{axis=[0], acc_dtype=float64}.0,False)]
mode = <local> <theano.compile.mode.Mode object at 0x7fdb53b69080>
accept_inplace = <local> False
profile = <local> None
on_unused_input = <local> 'ignore', len = 6
output_keys = <local> None
create = <not found>
defaults = <local> [None, None, <array([[0]], dtype=int32)>, <b'CudaNdarray([[[ 0.]]])'>, <array([[0]], dtype=int32)>, <array([[0]], dtype=int8)>, <array([[0]], dtype=int8)>, <array([[0]], dtype=int8)>, <array(0, dtype=int32)>, <array([], shape=(0, 0), dtype=int8)>, <b'CudaNdarray([-4.38202667 -4.38202667 -4.382026..., len = 389
File "/home/deep/anaconda3/lib/python3.6/site-packages/theano/compile/function_module.py", line 1661, in create
line: _fn, _i, _o = self.linker.make_thunk(
input_storage=input_storage_lists, storage_map=storage_map)
locals:
_fn = <not found>
_i = <not found>
_o = <not found>
self = <local> <theano.compile.function_module.FunctionMaker object at 0x7fdb02b0b5f8>
self.linker = <local> <theano.gof.vm.VM_Linker object at 0x7fdb0037e0b8>
self.linker.make_thunk = <local> <bound method LocalLinker.make_thunk of <theano.gof.vm.VM_Linker object at 0x7fdb0037e0b8>>
input_storage = <local> [None, None, <array([[0]], dtype=int32)>, <b'CudaNdarray([[[ 0.]]])'>, <array([[0]], dtype=int32)>, <array([[0]], dtype=int8)>, <array([[0]], dtype=int8)>, <array([[0]], dtype=int8)>, <array(0, dtype=int32)>, <array([], shape=(0, 0), dtype=int8)>, <b'CudaNdarray([-4.38202667 -4.38202667 -4.382026..., len = 389
input_storage_lists = <local> [[None], [None], [array([[0]], dtype=int32)], [b'CudaNdarray([[[ 0.]]])'], [array([[0]], dtype=int32)], [array([[0]], dtype=int8)], [array([[0]], dtype=int8)], [array([[0]], dtype=int8)], [array(0, dtype=int32)], [array([], shape=(0, 0), dtype=int8)], [b'CudaNdarray([-4.38202667 -4.38202667 -4.38..., len = 389
storage_map = <local> None
File "/home/deep/anaconda3/lib/python3.6/site-packages/theano/gof/link.py", line 699, in make_thunk
line: return self.make_all(input_storage=input_storage,
output_storage=output_storage,
storage_map=storage_map)[:3]
locals:
self = <local> <theano.gof.vm.VM_Linker object at 0x7fdb0037e0b8>
self.make_all = <local> <bound method VM_Linker.make_all of <theano.gof.vm.VM_Linker object at 0x7fdb0037e0b8>>
input_storage = <local> [[None], [None], [array([[0]], dtype=int32)], [b'CudaNdarray([[[ 0.]]])'], [array([[0]], dtype=int32)], [array([[0]], dtype=int8)], [array([[0]], dtype=int8)], [array([[0]], dtype=int8)], [array(0, dtype=int32)], [array([], shape=(0, 0), dtype=int8)], [b'CudaNdarray([-4.38202667 -4.38202667 -4.38..., len = 389
output_storage = <local> None
storage_map = <local> None
File "/home/deep/anaconda3/lib/python3.6/site-packages/theano/gof/vm.py", line 1047, in make_all
line: thunks.append(node.op.make_thunk(node,
storage_map,
compute_map,
no_recycling,
impl=impl))
locals:
thunks = <local> [<function Op.make_c_thunk.<locals>.rval at 0x7fdaf8d110d0>, <function Op.make_c_thunk.<locals>.rval at 0x7fdaf8d11bf8>, <function Op.make_c_thunk.<locals>.rval at 0x7fdaf8d11048>, <function Op.make_c_thunk.<locals>.rval at 0x7fdaf8d11488>, <function Op.make_c_thunk.<locals>.rval at 0x7fdaf8d11d0..., len = 1020
thunks.append = <local> <built-in method append of list object at 0x7fdaf421df48>
node = <local> CuDNNConvHWBCOp{border_mode='valid'}(GpuContiguous.0, GpuContiguous.0, GpuContiguous.0)
node.op = <local> <cuda_implementation.CuDNNConvHWBCOp.CuDNNConvHWBCOp object at 0x7fdb50085780>
node.op.make_thunk = <local> <bound method Op.make_thunk of <cuda_implementation.CuDNNConvHWBCOp.CuDNNConvHWBCOp object at 0x7fdb50085780>>
storage_map = <local> {<TensorType(int64, scalar)>: [None], <TensorType(int64, scalar)>: [None], y_classes: [array([[0]], dtype=int32)], y_data: [b'CudaNdarray([[[ 0.]]])'], y_sizes: [array([[0]], dtype=int32)], j_classes: [array([[0]], dtype=int8)], j_data: [array([[0]], dtype=int8)], j_sizes: [array([[0]], dtype=int..., len = 3228
compute_map = <local> {<TensorType(int64, scalar)>: [True], <TensorType(int64, scalar)>: [True], y_classes: [True], y_data: [True], y_sizes: [True], j_classes: [True], j_data: [True], j_sizes: [True], epoch_var: [True], tags_var: [True], b_output: [True], <CudaNdarrayType(float32, vector)>: [True], <CudaNdarrayType(fl..., len = 3228
no_recycling = <local> {GpuDimShuffle{x}.0, GpuContiguous.0, GpuElemwise{Composite{(i0 + ((i1 * (Switch(i2, i3, (i4 / i5)) + (i6 * (i7 / i8)))) / (i9 + sqrt((i10 / i11)))))},no_inplace}.0, GpuContiguous.0, GpuElemwise{Composite{((i0 * i1) + (i2 * sqr(Switch(i3, i4, i5))))}}[(0, 1)].0, GpuContiguous.0, GpuContiguous.0, ..., len = 553
impl = <local> None
File "/home/deep/anaconda3/lib/python3.6/site-packages/theano/gof/op.py", line 935, in make_thunk
line: return self.make_c_thunk(node, storage_map, compute_map,
no_recycling)
locals:
self = <local> <cuda_implementation.CuDNNConvHWBCOp.CuDNNConvHWBCOp object at 0x7fdb50085780>
self.make_c_thunk = <local> <bound method Op.make_c_thunk of <cuda_implementation.CuDNNConvHWBCOp.CuDNNConvHWBCOp object at 0x7fdb50085780>>
node = <local> CuDNNConvHWBCOp{border_mode='valid'}(GpuContiguous.0, GpuContiguous.0, GpuContiguous.0)
storage_map = <local> {<TensorType(int64, scalar)>: [None], <TensorType(int64, scalar)>: [None], y_classes: [array([[0]], dtype=int32)], y_data: [b'CudaNdarray([[[ 0.]]])'], y_sizes: [array([[0]], dtype=int32)], j_classes: [array([[0]], dtype=int8)], j_data: [array([[0]], dtype=int8)], j_sizes: [array([[0]], dtype=int..., len = 3228
compute_map = <local> {<TensorType(int64, scalar)>: [True], <TensorType(int64, scalar)>: [True], y_classes: [True], y_data: [True], y_sizes: [True], j_classes: [True], j_data: [True], j_sizes: [True], epoch_var: [True], tags_var: [True], b_output: [True], <CudaNdarrayType(float32, vector)>: [True], <CudaNdarrayType(fl..., len = 3228
no_recycling = <local> {GpuDimShuffle{x}.0, GpuContiguous.0, GpuElemwise{Composite{(i0 + ((i1 * (Switch(i2, i3, (i4 / i5)) + (i6 * (i7 / i8)))) / (i9 + sqrt((i10 / i11)))))},no_inplace}.0, GpuContiguous.0, GpuElemwise{Composite{((i0 * i1) + (i2 * sqr(Switch(i3, i4, i5))))}}[(0, 1)].0, GpuContiguous.0, GpuContiguous.0, ..., len = 553
File "/home/deep/anaconda3/lib/python3.6/site-packages/theano/gof/op.py", line 839, in make_c_thunk
line: outputs = cl.make_thunk(input_storage=node_input_storage,
output_storage=node_output_storage)
locals:
outputs = <not found>
cl = <local> <theano.gof.cc.CLinker object at 0x7fdaf7a3c630>
cl.make_thunk = <local> <bound method CLinker.make_thunk of <theano.gof.cc.CLinker object at 0x7fdaf7a3c630>>
input_storage = <not found>
node_input_storage = <local> [[None], [None], [None]]
output_storage = <not found>
node_output_storage = <local> [[None]]
File "/home/deep/anaconda3/lib/python3.6/site-packages/theano/gof/cc.py", line 1190, in make_thunk
line: cthunk, in_storage, out_storage, error_storage = self.__compile__(
input_storage, output_storage, storage_map,
keep_lock=keep_lock)
locals:
cthunk = <not found>
in_storage = <not found>
out_storage = <not found>
error_storage = <not found>
self = <local> <theano.gof.cc.CLinker object at 0x7fdaf7a3c630>
self.__compile__ = <local> <bound method CLinker.__compile__ of <theano.gof.cc.CLinker object at 0x7fdaf7a3c630>>
input_storage = <local> [[None], [None], [None]]
output_storage = <local> [[None]]
storage_map = <local> None
keep_lock = <local> False
File "/home/deep/anaconda3/lib/python3.6/site-packages/theano/gof/cc.py", line 1131, in __compile__
line: thunk = self.cthunk_factory(error_storage,
input_storage,
output_storage,
storage_map,
keep_lock=keep_lock)
locals:
thunk = <not found>
self = <local> <theano.gof.cc.CLinker object at 0x7fdaf7a3c630>
self.cthunk_factory = <local> <bound method CLinker.cthunk_factory of <theano.gof.cc.CLinker object at 0x7fdaf7a3c630>>
error_storage = <local> [None, None, None]
input_storage = <local> ([None], [None], [None])
output_storage = <local> ([None],)
storage_map = <local> None
keep_lock = <local> False
File "/home/deep/anaconda3/lib/python3.6/site-packages/theano/gof/cc.py", line 1586, in cthunk_factory
line: module = get_module_cache().module_from_key(
key=key, lnk=self, keep_lock=keep_lock)
locals:
module = <not found>
get_module_cache = <global> <function get_module_cache at 0x7fdb819798c8>
module_from_key = <not found>
key = <local> (((3, 3), (3,), (3,), (3,), (3,)), ('CLinker.cmodule_key', ('-DCUDA_NDARRAY_CUH=mc72d035fdf91890f3b36710688069b2e', '-DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION', '-O3', '-Wno-unused-label', '-Wno-unused-variable', '-Wno-write-strings', '-arch=sm_61', '-fno-math-errno'), ('cublas', 'cuda_ndarray'...
lnk = <not found>
self = <local> <theano.gof.cc.CLinker object at 0x7fdaf7a3c630>
keep_lock = <local> False
File "/home/deep/anaconda3/lib/python3.6/site-packages/theano/gof/cmodule.py", line 1159, in module_from_key
line: module = lnk.compile_cmodule(location)
locals:
module = <local> None
lnk = <local> <theano.gof.cc.CLinker object at 0x7fdaf7a3c630>
lnk.compile_cmodule = <local> <bound method CLinker.compile_cmodule of <theano.gof.cc.CLinker object at 0x7fdaf7a3c630>>
location = <local> '/home/deep/.theano/compiledir_Linux-4.8.0-53-generic-x86_64-with-debian-stretch-sid-x86_64-3.6.1-64--dev-gpuZ/tmp29yq2k9e', len = 121
File "/home/deep/anaconda3/lib/python3.6/site-packages/theano/gof/cc.py", line 1489, in compile_cmodule
line: module = c_compiler.compile_str(
module_name=mod.code_hash,
src_code=src_code,
location=location,
include_dirs=self.header_dirs(),
lib_dirs=self.lib_dirs(),
libs=libs,
preargs=preargs)
locals:
module = <not found>
c_compiler = <local> <class 'theano.sandbox.cuda.nvcc_compiler.NVCC_compiler'>
c_compiler.compile_str = <local> <function NVCC_compiler.compile_str at 0x7fdb5e30f048>
module_name = <not found>
mod = <local> <theano.gof.cmodule.DynamicModule object at 0x7fdaf7a3cac8>
mod.code_hash = <local> 'mcbc70be4d8fab58f670977cc113044f1', len = 33
src_code = <local> '#include <Python.h>\n#include <iostream>\n#include "theano_mod_helper.h"\n#include "cuda_ndarray.cuh"\n//////////////////////\n//// Support Code\n//////////////////////\n#include <thrust/device_vector.h>\n#include <vector>\n#include <string>\n#include <cublas_v2.h>\nusing namespace std;\n\n//al..., len = 26227
location = <local> '/home/deep/.theano/compiledir_Linux-4.8.0-53-generic-x86_64-with-debian-stretch-sid-x86_64-3.6.1-64--dev-gpuZ/tmp29yq2k9e', len = 121
include_dirs = <not found>
self = <local> <theano.gof.cc.CLinker object at 0x7fdaf7a3c630>
self.header_dirs = <local> <bound method CLinker.header_dirs of <theano.gof.cc.CLinker object at 0x7fdaf7a3c630>>
lib_dirs = <not found>
self.lib_dirs = <local> <bound method CLinker.lib_dirs of <theano.gof.cc.CLinker object at 0x7fdaf7a3c630>>
libs = <local> ['cudart', 'cublas', 'cuda_ndarray', 'cudnn'], _[0]: {len = 6}
preargs = <local> ['-O3', '-fno-math-errno', '-Wno-unused-label', '-Wno-unused-variable', '-Wno-write-strings', '-DCUDA_NDARRAY_CUH=mc72d035fdf91890f3b36710688069b2e', '-DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION', '-arch=sm_61'], len = 8
File "/home/deep/anaconda3/lib/python3.6/site-packages/theano/sandbox/cuda/nvcc_compiler.py", line 417, in compile_str
line: return dlimport(lib_filename)
locals:
dlimport = <global> <function dlimport at 0x7fdb8168c620>
lib_filename = <local> '/home/deep/.theano/compiledir_Linux-4.8.0-53-generic-x86_64-with-debian-stretch-sid-x86_64-3.6.1-64--dev-gpuZ/tmp29yq2k9e/mcbc70be4d8fab58f670977cc113044f1.so', len = 158
File "/home/deep/anaconda3/lib/python3.6/site-packages/theano/gof/cmodule.py", line 302, in dlimport
line: rval = __import__(module_name, {}, {}, [module_name])
locals:
rval = <local> None
__import__ = <builtin> <built-in function __import__>
module_name = <local> 'tmp29yq2k9e.mcbc70be4d8fab58f670977cc113044f1', len = 45
ImportError: /home/deep/.theano/compiledir_Linux-4.8.0-53-generic-x86_64-with-debian-stretch-sid-x86_64-3.6.1-64--dev-gpuZ/tmp29yq2k9e/mcbc70be4d8fab58f670977cc113044f1.so: undefined symbol: _ZTVNSt7__cxx1118basic_stringstreamIcSt11char_traitsIcESaIcEEE
Device proc gpuX (gpuZ) died: ProcConnectionDied('recv_bytes EOFError: ',)
Theano flags: compiledir_format=compiledir_%(platform)s-%(processor)s-%(python_version)s-%(python_bitwidth)s--dev-gpuZ,device=gpu,force_device=True
EXCEPTION
Traceback (most recent call last):
File "/home/deep/Documents/Deep Learning Libraries/returnn/Device.py", line 332, in startProc
line: self._startProc(*args, **kwargs)
locals:
self = <local> <Device.Device object at 0x7f92efaf0780>
self._startProc = <local> <bound method Device._startProc of <Device.Device object at 0x7f92efaf0780>>
args = <local> ('gpuZ',)
kwargs = <local> {}
File "/home/deep/Documents/Deep Learning Libraries/returnn/Device.py", line 386, in _startProc
line: interrupt_main()
locals:
interrupt_main = <global> <function interrupt_main at 0x7f92f4d3cae8>
File "/home/deep/Documents/Deep Learning Libraries/returnn/Util.py", line 607, in interrupt_main
line: sys.exit(1) # And exit the thread.
locals:
sys = <global> <module 'sys' (built-in)>
sys.exit = <global> <built-in function exit>
SystemExit: 1
KeyboardInterrupt
EXCEPTION
Traceback (most recent call last):
File "../../../rnn.py", line 520, in main
line: init(commandLineOptions=argv[1:])
locals:
init = <global> <function init at 0x7f92efaf1950>
commandLineOptions = <not found>
argv = <local> ['../../../rnn.py', 'config_demo'], _[0]: {len = 15}
File "../../../rnn.py", line 336, in init
line: devices = initDevices()
locals:
devices = <not found>
initDevices = <global> <function initDevices at 0x7f92efaf1488>
File "../../../rnn.py", line 152, in initDevices
line: time.sleep(0.25)
locals:
time = <global> <module 'time' (built-in)>
time.sleep = <global> <built-in function sleep>
KeyboardInterrupt
Quitting
from returnn.
albertz
commented on July 17, 2024
This looks like some C++ header mixup.
There are many people who have similar problems, eg. here, here, here, etc.
from returnn.
albertz
commented on July 17, 2024
Not sure if this is up-to-date. Probably the error is not Returnn-related, so I'm closing this now.
from returnn.
Related Issues (20)
- PyTorch distributed training CPU OOM with sync_on_cpu HOT 1
- Support `torch.compile` for RF
- RF backend: PyTorch code
- Different effective learning rate reported over gpus HOT 11
- CUDA error: initialization error HOT 3
- MultiProcDataset inside PyTorch DataLoader with num_workers>0, multiple issues HOT 4
- RuntimeError: CUDA error: unspecified launch failure HOT 2
- NonDaemonicSpawnProcess hangs at exit HOT 2
- High memory usage with datasets (specifically when multi procs are used)
- Hang at exit in TDL worker in multiprocessing `_run_finalizers`, deadlock in `_wait_for_tstate_lock`? HOT 6
- Hang HOT 2
- Returnn Native after using different apptainer uses old compilation HOT 6
- MetaDataset with sequence list filter file
- HDFDataset (or generic dataset) post processing HOT 15
- Dataset batching like ESPnet support
- torch.nn.functional.conv2d: RuntimeError: GET was unable to find an engine to execute this computation HOT 1
- TensorFlow 2.14 degradation in WER HOT 2
- Updates for recent TensorFlow version
- Hang in dataset iterator HOT 5
- Log GPU device for torch backend HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from returnn.