Comments (3)
Stack trace returned 10 entries:
[bt] (0) /home/shel/mxnet/python/mxnet/../../lib/libmxnet.so(_ZN5mxnet7storage23GPUPooledStorageManager5AllocEm+0x3e4) [0x7f6e47f1fde4]
[bt] (1) /home/shel/mxnet/python/mxnet/../../lib/libmxnet.so(_ZN5mxnet11StorageImpl5AllocEmNS_7ContextE+0x57) [0x7f6e47f23817]
[bt] (2) /home/sheldon606/mxnet/python/mxnet/../../lib/libmxnet.so(+0x1433443) [0x7f6e47f72443]
[bt] (3) /home/shel/mxnet/python/mxnet/../../lib/libmxnet.so(_ZN5mxnet4exec13GraphExecutor19InitDataEntryMemoryEPSt6vectorINS_7NDArrayESaIS3_EE+0x2284) [0x7f6e47f7b1c4]
[bt] (4) /home/shel/mxnet/python/mxnet/../../lib/libmxnet.so(_ZN5mxnet4exec13GraphExecutor4InitEN4nnvm6SymbolERKNS_7ContextERKSt3mapISsS4_St4lessISsESaISt4pairIKSsS4_EEERKSt6vectorINS_7NDArrayESaISI_EESM_RKSH_INS_9OpReqTypeESaISN_EESM_PNS_8ExecutorERKSt13unordered_mapINS2_9NodeEntryESI_NS2_13NodeEntryHashENS2_14NodeEntryEqualESaISA_IKSV_SI_EEE+0x5bb) [0x7f6e47f8154b]
[bt] (5) /home/shel/mxnet/python/mxnet/../../lib/libmxnet.so(ZN5mxnet8Executor4BindEN4nnvm6SymbolERKNS_7ContextERKSt3mapISsS3_St4lessISsESaISt4pairIKSsS3_EEERKSt6vectorINS_7NDArrayESaISH_EESL_RKSG_INS_9OpReqTypeESaISM_EESL_PS0+0x684) [0x7f6e47f81c74]
[bt] (6) /home/shel/mxnet/python/mxnet/../../lib/libmxnet.so(MXExecutorBindEX+0x2425) [0x7f6e47f506f5]
[bt] (7) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call_unix64+0x4c) [0x7f6e7cfcfe40]
[bt] (8) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call+0x2eb) [0x7f6e7cfcf8ab]
[bt] (9) /usr/lib/python2.7/lib-dynload/_ctypes.x86_64-linux-gnu.so(_ctypes_callproc+0x48f) [0x7f6e5fdd43df]
Traceback (most recent call last):
File "experiments/fcis/fcis_end2end_train_test.py", line 13, in
train_end2end.main()
File "experiments/fcis/../../fcis/train_end2end.py", line 181, in main
config.TRAIN.lr, config.TRAIN.lr_step)
File "experiments/fcis/../../fcis/train_end2end.py", line 173, in train_net
arg_params=arg_params, aux_params=aux_params, begin_epoch=begin_epoch, num_epoch=end_epoch)
File "experiments/fcis/../../fcis/core/module.py", line 932, in fit
for_training=True, force_rebind=force_rebind)
File "experiments/fcis/../../fcis/core/module.py", line 840, in bind
for_training, inputs_need_grad, force_rebind=False, shared_module=None)
File "experiments/fcis/../../fcis/core/module.py", line 397, in bind
state_names=self._state_names)
File "experiments/fcis/../../fcis/core/DataParallelExecutorGroup.py", line 178, in init
self.bind_exec(data_shapes, label_shapes, shared_group)
File "experiments/fcis/../../fcis/core/DataParallelExecutorGroup.py", line 278, in bind_exec
shared_group))
File "experiments/fcis/../../fcis/core/DataParallelExecutorGroup.py", line 613, in _bind_ith_exec
grad_req=self.grad_req, shared_exec=shared_exec)
File "/home/shel/mxnet/python/mxnet/symbol.py", line 1413, in bind
ctypes.byref(handle)))
File "/home/shel/mxnet/python/mxnet/base.py", line 85, in check_call
raise MXNetError(py_str(_LIB.MXGetLastError()))
mxnet.base.MXNetError: [10:32:26] src/storage/./pooled_storage_manager.h:84: cudaMalloc failed: out of memory
Stack trace returned 10 entries:
[bt] (0) /home/shel/mxnet/python/mxnet/../../lib/libmxnet.so(_ZN5mxnet7storage23GPUPooledStorageManager5AllocEm+0x3e4) [0x7f6e47f1fde4]
[bt] (1) /home/shel/mxnet/python/mxnet/../../lib/libmxnet.so(_ZN5mxnet11StorageImpl5AllocEmNS_7ContextE+0x57) [0x7f6e47f23817]
[bt] (2) /home/shel/mxnet/python/mxnet/../../lib/libmxnet.so(+0x1433443) [0x7f6e47f72443]
[bt] (3) /home/sheld/mxnet/python/mxnet/../../lib/libmxnet.so(_ZN5mxnet4exec13GraphExecutor19InitDataEntryMemoryEPSt6vectorINS_7NDArrayESaIS3_EE+0x2284) [0x7f6e47f7b1c4]
[bt] (4) /home/shel/mxnet/python/mxnet/../../lib/libmxnet.so(_ZN5mxnet4exec13GraphExecutor4InitEN4nnvm6SymbolERKNS_7ContextERKSt3mapISsS4_St4lessISsESaISt4pairIKSsS4_EEERKSt6vectorINS_7NDArrayESaISI_EESM_RKSH_INS_9OpReqTypeESaISN_EESM_PNS_8ExecutorERKSt13unordered_mapINS2_9NodeEntryESI_NS2_13NodeEntryHashENS2_14NodeEntryEqualESaISA_IKSV_SI_EEE+0x5bb) [0x7f6e47f8154b]
[bt] (5) /home/shel/mxnet/python/mxnet/../../lib/libmxnet.so(ZN5mxnet8Executor4BindEN4nnvm6SymbolERKNS_7ContextERKSt3mapISsS3_St4lessISsESaISt4pairIKSsS3_EEERKSt6vectorINS_7NDArrayESaISH_EESL_RKSG_INS_9OpReqTypeESaISM_EESL_PS0+0x684) [0x7f6e47f81c74]
[bt] (6) /home/sheldon606/mxnet/python/mxnet/../../lib/libmxnet.so(MXExecutorBindEX+0x2425) [0x7f6e47f506f5]
[bt] (7) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call_unix64+0x4c) [0x7f6e7cfcfe40]
[bt] (8) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call+0x2eb) [0x7f6e7cfcf8ab]
[bt] (9) /usr/lib/python2.7/lib-dynload/_ctypes.x86_64-linux-gnu.so(_ctypes_callproc+0x48f) [0x7f6e5fdd43df]
from fcis.
Floor 2 shows the error after compiling the mxnet not using cudnn. Demo works ok for me, but the train process doesn't work any more.
from fcis.
@sheldon606 have you solved this problem??
from fcis.
Related Issues (20)
- TypeError: _update_params_on_kvstore() takes exactly 4 arguments (3 given) HOT 1
- there is an error when trained on ResNet-50 model HOT 1
- Index out of bounds error in line 143 of proposal_annotator.py HOT 1
- Some problems encountered when using deconvolution
- mxnet.base.MXNetError: [15:20:32] src/engine/threaded_engine.cc:320: Check failed: device_count_ > 0 (-1 vs. 0) GPU usage requires at least 1 GPU HOT 1
- The result finetune with cityscapes is very poor
- Network structure
- When I train my own coco-like datasets,I successfully train and test the model,but the result of all AP is 0. HOT 5
- If my data is small,such as 6000 images,is it difficult to train the model? HOT 3
- my own data set ,test map is 0
- 有哪位遇到过训练成功,测试map=0的情况吗 HOT 3
- mxnet.base.MXNetError: [16:31:18] src/operator/nn/./cudnn/cudnn_convolution-inl.h:449: Check failed: e == CUDNN_STATUS_SUCCESS (3 vs. 0) cuDNN: CUDNN_STATUS_BAD_PARAM HOT 1
- pip install -r requirements.txt HOT 3
- unable to install ./init.sh error HOT 4
- Problems when launching demo.py
- Is it possible to run FCIS on AMD?
- Trouble building on Windows 10 (TypeError: expected str, bytes or os.PathLike object, not NoneType) HOT 2
- In-depth guide to getting FCIS working on Windows 10 HOT 3
- [HELP] problem with cpu_nms HOT 1
- Result file for COCO test-dev instance segmentation
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from fcis.