paddlepaddle / paddledetection Goto Github PK

2019-12-08 09:36:34,183-INFO: 1076 samples in file dataset/coco/annotations/instances_val2017.json
2019-12-08 09:36:34,186-INFO: places would be ommited when DataLoader is not iterable
W1208 09:36:36.099746   317 device_context.cc:235] Please NOTE: device: 0, CUDA Capability: 60, Driver API Version: 10.0, Runtime API Version: 10.0
W1208 09:36:36.412878   317 device_context.cc:243] device: 0, cuDNN Version: 7.6.
2019-12-08 09:36:40,108-INFO: Loading checkpoint from output/faster_rcnn_dcn_x101_vd_64x4d_fpn_1x/27000...
2019-12-08 09:36:46,786-INFO: 20764 samples in file dataset/coco/annotations/instances_train2017.json
2019-12-08 09:36:46,893-INFO: places would be ommited when DataLoader is not iterable
I1208 09:36:47.311914   317 parallel_executor.cc:421] The number of CUDAPlace, which is used in ParallelExecutor, is 2. And the Program will be copied 2 copies
I1208 09:36:49.598440   317 graph_pattern_detector.cc:96] ---  detected 40 subgraphs
I1208 09:36:49.672873   317 graph_pattern_detector.cc:96] ---  detected 37 subgraphs
W1208 09:36:49.861197   317 fuse_all_reduce_op_pass.cc:72] Find all_reduce operators: 183. To make the speed faster, some all_reduce ops are fused during training, after fusion, the number of all_reduce ops is 153.
I1208 09:36:49.873278   317 build_strategy.cc:363] SeqOnlyAllReduceOps:0, num_trainers:1
I1208 09:36:50.447363   317 parallel_executor.cc:285] Inplace strategy is enabled, when build_strategy.enable_inplace = True
I1208 09:36:50.538653   317 parallel_executor.cc:368] Garbage collection strategy is enabled, when FLAGS_eager_delete_tensor_gb = 0
2019-12-08 09:37:11,792-INFO: iter: 27020, lr: 0.002500, 'loss_cls': '0.107255', 'loss_bbox': '0.027132', 'loss_rpn_cls': '0.029164', 'loss_rpn_bbox': '0.004501', 'loss': '0.162709', time: 1.196, eta: 9 days, 14:16:56
2019-12-08 09:37:33,380-INFO: iter: 27040, lr: 0.002500, 'loss_cls': '0.160085', 'loss_bbox': '0.037842', 'loss_rpn_cls': '0.029905', 'loss_rpn_bbox': '0.005318', 'loss': '0.248189', time: 1.069, eta: 8 days, 13:42:18
2019-12-08 09:37:56,325-INFO: iter: 27060, lr: 0.002500, 'loss_cls': '0.111756', 'loss_bbox': '0.042657', 'loss_rpn_cls': '0.022942', 'loss_rpn_bbox': '0.003554', 'loss': '0.176668', time: 1.153, eta: 9 days, 5:55:00
2019-12-08 09:38:18,266-INFO: iter: 27080, lr: 0.002500, 'loss_cls': '0.117590', 'loss_bbox': '0.037847', 'loss_rpn_cls': '0.028765', 'loss_rpn_bbox': '0.007621', 'loss': '0.183682', time: 1.103, eta: 8 days, 20:15:40
2019-12-08 09:38:38,424-INFO: iter: 27100, lr: 0.002500, 'loss_cls': '0.109035', 'loss_bbox': '0.036450', 'loss_rpn_cls': '0.020142', 'loss_rpn_bbox': '0.002935', 'loss': '0.167680', time: 1.001, eta: 8 days, 0:35:44
2019-12-08 09:39:01,073-INFO: iter: 27120, lr: 0.002500, 'loss_cls': '0.112554', 'loss_bbox': '0.030803', 'loss_rpn_cls': '0.027392', 'loss_rpn_bbox': '0.007306', 'loss': '0.210294', time: 1.125, eta: 9 days, 0:33:28
2019-12-08 09:39:22,075-INFO: iter: 27140, lr: 0.002500, 'loss_cls': '0.119142', 'loss_bbox': '0.032270', 'loss_rpn_cls': '0.024191', 'loss_rpn_bbox': '0.004826', 'loss': '0.176259', time: 1.051, eta: 8 days, 10:21:23
2019-12-08 09:39:43,539-INFO: iter: 27160, lr: 0.002500, 'loss_cls': '0.146193', 'loss_bbox': '0.047863', 'loss_rpn_cls': '0.028642', 'loss_rpn_bbox': '0.007621', 'loss': '0.238028', time: 1.064, eta: 8 days, 12:47:36
2019-12-08 09:40:05,350-INFO: iter: 27180, lr: 0.002500, 'loss_cls': '0.095980', 'loss_bbox': '0.035866', 'loss_rpn_cls': '0.022334', 'loss_rpn_bbox': '0.004785', 'loss': '0.186832', time: 1.113, eta: 8 days, 22:10:17
2019-12-08 09:40:27,442-INFO: iter: 27200, lr: 0.002500, 'loss_cls': '0.103658', 'loss_bbox': '0.030443', 'loss_rpn_cls': '0.023747', 'loss_rpn_bbox': '0.005142', 'loss': '0.174824', time: 1.100, eta: 8 days, 19:36:49
2019-12-08 09:40:48,639-INFO: iter: 27220, lr: 0.002500, 'loss_cls': '0.123171', 'loss_bbox': '0.041430', 'loss_rpn_cls': '0.026343', 'loss_rpn_bbox': '0.007074', 'loss': '0.197138', time: 1.065, eta: 8 days, 12:56:42
2019-12-08 09:41:09,553-INFO: iter: 27240, lr: 0.002500, 'loss_cls': '0.126446', 'loss_bbox': '0.030707', 'loss_rpn_cls': '0.021942', 'loss_rpn_bbox': '0.004218', 'loss': '0.185318', time: 1.044, eta: 8 days, 8:53:24
2019-12-08 09:41:29,916-INFO: iter: 27260, lr: 0.002500, 'loss_cls': '0.109329', 'loss_bbox': '0.035236', 'loss_rpn_cls': '0.029849', 'loss_rpn_bbox': '0.008709', 'loss': '0.188167', time: 1.012, eta: 8 days, 2:44:33
2019-12-08 09:41:51,827-INFO: iter: 27280, lr: 0.002500, 'loss_cls': '0.117947', 'loss_bbox': '0.035055', 'loss_rpn_cls': '0.022669', 'loss_rpn_bbox': '0.004247', 'loss': '0.183069', time: 1.100, eta: 8 days, 19:38:23
2019-12-08 09:42:13,266-INFO: iter: 27300, lr: 0.002500, 'loss_cls': '0.104928', 'loss_bbox': '0.032062', 'loss_rpn_cls': '0.031705', 'loss_rpn_bbox': '0.006268', 'loss': '0.186234', time: 1.072, eta: 8 days, 14:19:22
2019-12-08 09:42:35,486-INFO: iter: 27320, lr: 0.002500, 'loss_cls': '0.110162', 'loss_bbox': '0.032131', 'loss_rpn_cls': '0.037423', 'loss_rpn_bbox': '0.009977', 'loss': '0.179894', time: 1.113, eta: 8 days, 22:07:25
2019-12-08 09:42:57,989-INFO: iter: 27340, lr: 0.002500, 'loss_cls': '0.098853', 'loss_bbox': '0.036495', 'loss_rpn_cls': '0.022839', 'loss_rpn_bbox': '0.007259', 'loss': '0.177353', time: 1.103, eta: 8 days, 20:13:59
2019-12-08 09:43:18,957-INFO: iter: 27360, lr: 0.002500, 'loss_cls': '0.136675', 'loss_bbox': '0.037647', 'loss_rpn_cls': '0.024854', 'loss_rpn_bbox': '0.005528', 'loss': '0.243505', time: 1.051, eta: 8 days, 10:15:43
2019-12-08 09:43:40,055-INFO: iter: 27380, lr: 0.002500, 'loss_cls': '0.107223', 'loss_bbox': '0.036117', 'loss_rpn_cls': '0.025308', 'loss_rpn_bbox': '0.005180', 'loss': '0.191671', time: 1.074, eta: 8 days, 14:33:44
2019-12-08 09:44:01,801-INFO: iter: 27400, lr: 0.002500, 'loss_cls': '0.114322', 'loss_bbox': '0.032498', 'loss_rpn_cls': '0.021028', 'loss_rpn_bbox': '0.003865', 'loss': '0.180020', time: 1.088, eta: 8 days, 17:13:38
2019-12-08 09:44:23,235-INFO: iter: 27420, lr: 0.002500, 'loss_cls': '0.101029', 'loss_bbox': '0.029264', 'loss_rpn_cls': '0.028364', 'loss_rpn_bbox': '0.007907', 'loss': '0.181565', time: 1.067, eta: 8 days, 13:19:37
2019-12-08 09:44:44,491-INFO: iter: 27440, lr: 0.002500, 'loss_cls': '0.097345', 'loss_bbox': '0.030726', 'loss_rpn_cls': '0.029091', 'loss_rpn_bbox': '0.005249', 'loss': '0.170476', time: 1.069, eta: 8 days, 13:35:30
2019-12-08 09:45:06,253-INFO: iter: 27460, lr: 0.002500, 'loss_cls': '0.107105', 'loss_bbox': '0.038378', 'loss_rpn_cls': '0.025624', 'loss_rpn_bbox': '0.005551', 'loss': '0.181381', time: 1.087, eta: 8 days, 17:10:51
2019-12-08 09:45:28,042-INFO: iter: 27480, lr: 0.002500, 'loss_cls': '0.108549', 'loss_bbox': '0.029579', 'loss_rpn_cls': '0.024967', 'loss_rpn_bbox': '0.006547', 'loss': '0.215194', time: 1.064, eta: 8 days, 12:39:42
2019-12-08 09:45:49,157-INFO: iter: 27500, lr: 0.002500, 'loss_cls': '0.112655', 'loss_bbox': '0.038301', 'loss_rpn_cls': '0.025386', 'loss_rpn_bbox': '0.006977', 'loss': '0.210186', time: 1.081, eta: 8 days, 15:58:03
2019-12-08 09:46:10,362-INFO: iter: 27520, lr: 0.002500, 'loss_cls': '0.115814', 'loss_bbox': '0.031053', 'loss_rpn_cls': '0.022635', 'loss_rpn_bbox': '0.004697', 'loss': '0.195093', time: 1.056, eta: 8 days, 11:02:29
2019-12-08 09:46:32,177-INFO: iter: 27540, lr: 0.002500, 'loss_cls': '0.116692', 'loss_bbox': '0.036010', 'loss_rpn_cls': '0.027288', 'loss_rpn_bbox': '0.005695', 'loss': '0.193081', time: 1.066, eta: 8 days, 13:02:31
2019-12-08 09:46:54,590-INFO: iter: 27560, lr: 0.002500, 'loss_cls': '0.128424', 'loss_bbox': '0.034904', 'loss_rpn_cls': '0.025524', 'loss_rpn_bbox': '0.004873', 'loss': '0.208800', time: 1.151, eta: 9 days, 5:23:29
2019-12-08 09:47:16,635-INFO: iter: 27580, lr: 0.002500, 'loss_cls': '0.107845', 'loss_bbox': '0.033880', 'loss_rpn_cls': '0.027985', 'loss_rpn_bbox': '0.005409', 'loss': '0.170375', time: 1.102, eta: 8 days, 19:58:25
2019-12-08 09:47:37,165-INFO: iter: 27600, lr: 0.002500, 'loss_cls': '0.096917', 'loss_bbox': '0.027007', 'loss_rpn_cls': '0.023711', 'loss_rpn_bbox': '0.005217', 'loss': '0.149012', time: 1.022, eta: 8 days, 4:34:20
2019-12-08 09:47:59,157-INFO: iter: 27620, lr: 0.002500, 'loss_cls': '0.148022', 'loss_bbox': '0.041588', 'loss_rpn_cls': '0.034595', 'loss_rpn_bbox': '0.008284', 'loss': '0.243836', time: 1.097, eta: 8 days, 18:56:47
2019-12-08 09:48:21,268-INFO: iter: 27640, lr: 0.002500, 'loss_cls': '0.102161', 'loss_bbox': '0.033482', 'loss_rpn_cls': '0.022656', 'loss_rpn_bbox': '0.003083', 'loss': '0.176066', time: 1.113, eta: 8 days, 22:04:28
2019-12-08 09:48:42,021-INFO: iter: 27660, lr: 0.002500, 'loss_cls': '0.116623', 'loss_bbox': '0.031174', 'loss_rpn_cls': '0.021008', 'loss_rpn_bbox': '0.006698', 'loss': '0.181974', time: 1.032, eta: 8 days, 6:25:45

2019-12-08 10:03:30,895-INFO: iter: 28260, lr: 0.002500, 'loss_cls': '0.120618', 'loss_bbox': '0.036356', 'loss_rpn_cls': '0.028862', 'loss_rpn_bbox': '0.006762', 'loss': '0.196524', time: 1.062, eta: 8 days, 12:08:16
2019-12-08 10:03:54,389-INFO: iter: 28280, lr: 0.002500, 'loss_cls': '0.116200', 'loss_bbox': '0.037455', 'loss_rpn_cls': '0.026497', 'loss_rpn_bbox': '0.006574', 'loss': '0.179571', time: 1.176, eta: 9 days, 9:52:23
2019-12-08 10:04:16,664-INFO: iter: 28300, lr: 0.002500, 'loss_cls': '0.090389', 'loss_bbox': '0.022946', 'loss_rpn_cls': '0.028943', 'loss_rpn_bbox': '0.005928', 'loss': '0.150511', time: 1.106, eta: 8 days, 20:34:48
2019-12-08 10:04:39,204-INFO: iter: 28320, lr: 0.002500, 'loss_cls': '0.099024', 'loss_bbox': '0.030282', 'loss_rpn_cls': '0.027065', 'loss_rpn_bbox': '0.006674', 'loss': '0.180380', time: 1.125, eta: 9 days, 0:10:30
2019-12-08 10:05:00,137-INFO: iter: 28340, lr: 0.002500, 'loss_cls': '0.104252', 'loss_bbox': '0.031640', 'loss_rpn_cls': '0.023287', 'loss_rpn_bbox': '0.004962', 'loss': '0.179577', time: 1.055, eta: 8 days, 10:42:35
2019-12-08 10:05:20,870-INFO: iter: 28360, lr: 0.002500, 'loss_cls': '0.106998', 'loss_bbox': '0.036490', 'loss_rpn_cls': '0.027223', 'loss_rpn_bbox': '0.006033', 'loss': '0.196109', time: 1.037, eta: 8 days, 7:08:30

2019-12-08 11:26:26,361-INFO: iter: 32040, lr: 0.002500, 'loss_cls': '0.093410', 'loss_bbox': '0.025801', 'loss_rpn_cls': '0.020566', 'loss_rpn_bbox': '0.004328', 'loss': '0.155523', time: 1.130, eta: 8 days, 23:52:36
2019-12-08 11:26:48,282-INFO: iter: 32060, lr: 0.002500, 'loss_cls': '0.116218', 'loss_bbox': '0.030774', 'loss_rpn_cls': '0.028761', 'loss_rpn_bbox': '0.008732', 'loss': '0.198504', time: 1.111, eta: 8 days, 20:15:33
2019-12-08 11:27:10,345-INFO: iter: 32080, lr: 0.002500, 'loss_cls': '0.130258', 'loss_bbox': '0.032691', 'loss_rpn_cls': '0.036417', 'loss_rpn_bbox': '0.007845', 'loss': '0.220228', time: 1.090, eta: 8 days, 16:15:38
2019-12-08 11:27:32,476-INFO: iter: 32100, lr: 0.002500, 'loss_cls': '0.108925', 'loss_bbox': '0.037187', 'loss_rpn_cls': '0.021106', 'loss_rpn_bbox': '0.007838', 'loss': '0.178534', time: 1.119, eta: 8 days, 21:54:37
2019-12-08 11:27:54,745-INFO: iter: 32120, lr: 0.002500, 'loss_cls': '0.112486', 'loss_bbox': '0.036119', 'loss_rpn_cls': '0.026041', 'loss_rpn_bbox': '0.005284', 'loss': '0.181879', time: 1.102, eta: 8 days, 18:36:12
2019-12-08 11:28:17,569-INFO: iter: 32140, lr: 0.002500, 'loss_cls': '0.105735', 'loss_bbox': '0.034030', 'loss_rpn_cls': '0.024859', 'loss_rpn_bbox': '0.003535', 'loss': '0.172869', time: 1.147, eta: 9 days, 3:05:29
2019-12-08 11:28:38,749-INFO: iter: 32160, lr: 0.002500, 'loss_cls': '0.097556', 'loss_bbox': '0.032315', 'loss_rpn_cls': '0.027183', 'loss_rpn_bbox': '0.004453', 'loss': '0.168935', time: 1.063, eta: 8 days, 11:03:53
2019-12-08 11:29:01,167-INFO: iter: 32180, lr: 0.002500, 'loss_cls': '0.106844', 'loss_bbox': '0.027867', 'loss_rpn_cls': '0.027673', 'loss_rpn_bbox': '0.007006', 'loss': '0.180671', time: 1.103, eta: 8 days, 18:40:46
2019-12-08 11:29:22,644-INFO: iter: 32200, lr: 0.002500, 'loss_cls': '0.111045', 'loss_bbox': '0.039628', 'loss_rpn_cls': '0.023468', 'loss_rpn_bbox': '0.004395', 'loss': '0.191487', time: 1.086, eta: 8 days, 15:34:46
2019-12-08 11:29:44,434-INFO: iter: 32220, lr: 0.002500, 'loss_cls': '0.124690', 'loss_bbox': '0.027931', 'loss_rpn_cls': '0.019735', 'loss_rpn_bbox': '0.005108', 'loss': '0.175699', time: 1.097, eta: 8 days, 17:33:22
2019-12-08 11:30:05,626-INFO: iter: 32240, lr: 0.002500, 'loss_cls': '0.095456', 'loss_bbox': '0.035033', 'loss_rpn_cls': '0.022073', 'loss_rpn_bbox': '0.003227', 'loss': '0.156538', time: 1.059, eta: 8 days, 10:13:40
2019-12-08 11:30:26,897-INFO: iter: 32260, lr: 0.002500, 'loss_cls': '0.119746', 'loss_bbox': '0.046080', 'loss_rpn_cls': '0.019381', 'loss_rpn_bbox': '0.004442', 'loss': '0.182005', time: 1.066, eta: 8 days, 11:36:56
2019-12-08 11:30:48,440-INFO: iter: 32280, lr: 0.002500, 'loss_cls': '0.115690', 'loss_bbox': '0.028565', 'loss_rpn_cls': '0.028244', 'loss_rpn_bbox': '0.005486', 'loss': '0.200766', time: 1.057, eta: 8 days, 9:58:26
2019-12-08 11:31:09,633-INFO: iter: 32300, lr: 0.002500, 'loss_cls': '0.107902', 'loss_bbox': '0.033771', 'loss_rpn_cls': '0.021195', 'loss_rpn_bbox': '0.004134', 'loss': '0.175219', time: 1.079, eta: 8 days, 14:09:52
2019-12-08 11:31:30,260-INFO: iter: 32320, lr: 0.002500, 'loss_cls': '0.098927', 'loss_bbox': '0.037162', 'loss_rpn_cls': '0.021459', 'loss_rpn_bbox': '0.005403', 'loss': '0.158387', time: 1.029, eta: 8 days, 4:37:34
2019-12-08 11:31:53,241-INFO: iter: 32340, lr: 0.002500, 'loss_cls': '0.107896', 'loss_bbox': '0.033144', 'loss_rpn_cls': '0.033767', 'loss_rpn_bbox': '0.004966', 'loss': '0.197591', time: 1.142, eta: 9 days, 2:11:20

2019-12-08 11:32:13,843-INFO: iter: 32360, lr: 0.002500, 'loss_cls': '0.107879', 'loss_bbox': '0.041093', 'loss_rpn_cls': '0.033133', 'loss_rpn_bbox': '0.011063', 'loss': '0.211041', time: 1.035, eta: 8 days, 5:37:19
2019-12-08 11:32:34,803-INFO: iter: 32380, lr: 0.002500, 'loss_cls': '0.110426', 'loss_bbox': '0.039289', 'loss_rpn_cls': '0.027911', 'loss_rpn_bbox': '0.005713', 'loss': '0.184945', time: 1.052, eta: 8 days, 8:53:31
2019-12-08 11:32:55,430-INFO: iter: 32400, lr: 0.002500, 'loss_cls': '0.128478', 'loss_bbox': '0.041886', 'loss_rpn_cls': '0.026493', 'loss_rpn_bbox': '0.006556', 'loss': '0.188470', time: 1.031, eta: 8 days, 4:53:08
2019-12-08 11:33:17,293-INFO: iter: 32420, lr: 0.002500, 'loss_cls': '0.110536', 'loss_bbox': '0.032399', 'loss_rpn_cls': '0.030574', 'loss_rpn_bbox': '0.007749', 'loss': '0.206734', time: 1.086, eta: 8 days, 15:30:35
2019-12-08 11:33:38,599-INFO: iter: 32440, lr: 0.002500, 'loss_cls': '0.095937', 'loss_bbox': '0.035684', 'loss_rpn_cls': '0.020276', 'loss_rpn_bbox': '0.007348', 'loss': '0.159013', time: 1.070, eta: 8 days, 12:17:27
2019-12-08 11:34:00,011-INFO: iter: 32460, lr: 0.002500, 'loss_cls': '0.120759', 'loss_bbox': '0.040102', 'loss_rpn_cls': '0.025354', 'loss_rpn_bbox': '0.005780', 'loss': '0.201859', time: 1.072, eta: 8 days, 12:43:34
2019-12-08 11:34:20,885-INFO: iter: 32480, lr: 0.002500, 'loss_cls': '0.091575', 'loss_bbox': '0.030783', 'loss_rpn_cls': '0.027161', 'loss_rpn_bbox': '0.004689', 'loss': '0.153618', time: 1.039, eta: 8 days, 6:27:44
2019-12-08 11:34:41,995-INFO: iter: 32500, lr: 0.002500, 'loss_cls': '0.131100', 'loss_bbox': '0.042119', 'loss_rpn_cls': '0.024955', 'loss_rpn_bbox': '0.005969', 'loss': '0.209270', time: 1.054, eta: 8 days, 9:20:55
2019-12-08 11:35:03,078-INFO: iter: 32520, lr: 0.002500, 'loss_cls': '0.108701', 'loss_bbox': '0.025954', 'loss_rpn_cls': '0.019844', 'loss_rpn_bbox': '0.004977', 'loss': '0.180352', time: 1.053, eta: 8 days, 9:05:40
2019-12-08 11:35:24,807-INFO: iter: 32540, lr: 0.002500, 'loss_cls': '0.100763', 'loss_bbox': '0.031137', 'loss_rpn_cls': '0.025281', 'loss_rpn_bbox': '0.004916', 'loss': '0.175427', time: 1.068, eta: 8 days, 11:56:12
2019-12-08 11:35:47,071-INFO: iter: 32560, lr: 0.002500, 'loss_cls': '0.103370', 'loss_bbox': '0.027407', 'loss_rpn_cls': '0.025815', 'loss_rpn_bbox': '0.007054', 'loss': '0.167267', time: 1.132, eta: 9 days, 0:06:07
2019-12-08 11:36:09,177-INFO: iter: 32580, lr: 0.002500, 'loss_cls': '0.093084', 'loss_bbox': '0.023440', 'loss_rpn_cls': '0.023047', 'loss_rpn_bbox': '0.006587', 'loss': '0.151249', time: 1.100, eta: 8 days, 18:06:41
2019-12-08 11:36:30,414-INFO: iter: 32600, lr: 0.002500, 'loss_cls': '0.092012', 'loss_bbox': '0.028003', 'loss_rpn_cls': '0.021767', 'loss_rpn_bbox': '0.006627', 'loss': '0.159122', time: 1.050, eta: 8 days, 8:25:08
2019-12-08 11:36:52,008-INFO: iter: 32620, lr: 0.002500, 'loss_cls': '0.118538', 'loss_bbox': '0.029654', 'loss_rpn_cls': '0.031362', 'loss_rpn_bbox': '0.007302', 'loss': '0.238050', time: 1.106, eta: 8 days, 19:08:26
2019-12-08 11:37:13,449-INFO: iter: 32640, lr: 0.002500, 'loss_cls': '0.072994', 'loss_bbox': '0.026526', 'loss_rpn_cls': '0.022744', 'loss_rpn_bbox': '0.004115', 'loss': '0.134926', time: 1.055, eta: 8 days, 9:23:13
2019-12-08 11:37:34,177-INFO: iter: 32660, lr: 0.002500, 'loss_cls': '0.097421', 'loss_bbox': '0.024360', 'loss_rpn_cls': '0.023935', 'loss_rpn_bbox': '0.006053', 'loss': '0.161697', time: 1.053, eta: 8 days, 9:03:48
2019-12-08 11:37:55,721-INFO: iter: 32680, lr: 0.002500, 'loss_cls': '0.104007', 'loss_bbox': '0.027913', 'loss_rpn_cls': '0.023525', 'loss_rpn_bbox': '0.008149', 'loss': '0.181776', time: 1.053, eta: 8 days, 9:06:54
2019-12-08 11:38:17,213-INFO: iter: 32700, lr: 0.002500, 'loss_cls': '0.115015', 'loss_bbox': '0.037233', 'loss_rpn_cls': '0.026973', 'loss_rpn_bbox': '0.007489', 'loss': '0.198169', time: 1.099, eta: 8 days, 17:47:08
2019-12-08 11:38:37,801-INFO: iter: 32720, lr: 0.002500, 'loss_cls': '0.107442', 'loss_bbox': '0.030250', 'loss_rpn_cls': '0.026453', 'loss_rpn_bbox': '0.006128', 'loss': '0.170408', time: 1.023, eta: 8 days, 3:16:08
2019-12-08 11:38:58,985-INFO: iter: 32740, lr: 0.002500, 'loss_cls': '0.115347', 'loss_bbox': '0.037755', 'loss_rpn_cls': '0.022134', 'loss_rpn_bbox': '0.003413', 'loss': '0.182344', time: 1.065, eta: 8 days, 11:21:58
2019-12-08 11:39:20,945-INFO: iter: 32760, lr: 0.002500, 'loss_cls': '0.100408', 'loss_bbox': '0.030180', 'loss_rpn_cls': '0.031180', 'loss_rpn_bbox': '0.006408', 'loss': '0.176443', time: 1.091, eta: 8 days, 16:18:49
2019-12-08 11:39:42,146-INFO: iter: 32780, lr: 0.002500, 'loss_cls': '0.090289', 'loss_bbox': '0.019610', 'loss_rpn_cls': '0.024948', 'loss_rpn_bbox': '0.003633', 'loss': '0.158710', time: 1.066, eta: 8 days, 11:26:46
2019-12-08 11:40:03,092-INFO: iter: 32800, lr: 0.002500, 'loss_cls': '0.096883', 'loss_bbox': '0.028336', 'loss_rpn_cls': '0.028111', 'loss_rpn_bbox': '0.007136', 'loss': '0.154402', time: 1.048, eta: 8 days, 7:57:52
2019-12-08 11:40:25,261-INFO: iter: 32820, lr: 0.002500, 'loss_cls': '0.127951', 'loss_bbox': '0.041875', 'loss_rpn_cls': '0.027269', 'loss_rpn_bbox': '0.006420', 'loss': '0.202579', time: 1.100, eta: 8 days, 17:53:07
2019-12-08 11:40:47,515-INFO: iter: 32840, lr: 0.002500, 'loss_cls': '0.105165', 'loss_bbox': '0.026979', 'loss_rpn_cls': '0.026087', 'loss_rpn_bbox': '0.006410', 'loss': '0.165277', time: 1.106, eta: 8 days, 19:01:51
2019-12-08 11:41:10,274-INFO: iter: 32860, lr: 0.002500, 'loss_cls': '0.113844', 'loss_bbox': '0.035535', 'loss_rpn_cls': '0.021415', 'loss_rpn_bbox': '0.005199', 'loss': '0.179511', time: 1.148, eta: 9 days, 3:02:05
2019-12-08 11:41:32,032-INFO: iter: 32880, lr: 0.002500, 'loss_cls': '0.107139', 'loss_bbox': '0.034098', 'loss_rpn_cls': '0.023092', 'loss_rpn_bbox': '0.005283', 'loss': '0.178474', time: 1.094, eta: 8 days, 16:51:14
2019-12-08 11:41:53,913-INFO: iter: 32900, lr: 0.002500, 'loss_cls': '0.107068', 'loss_bbox': '0.033079', 'loss_rpn_cls': '0.025516', 'loss_rpn_bbox': '0.005883', 'loss': '0.179267', time: 1.096, eta: 8 days, 17:05:30
2019-12-08 11:42:15,265-INFO: iter: 32920, lr: 0.002500, 'loss_cls': '0.084352', 'loss_bbox': '0.025512', 'loss_rpn_cls': '0.026543', 'loss_rpn_bbox': '0.005927', 'loss': '0.174963', time: 1.060, eta: 8 days, 10:17:00
2019-12-08 11:42:35,988-INFO: iter: 32940, lr: 0.002500, 'loss_cls': '0.094424', 'loss_bbox': '0.026548', 'loss_rpn_cls': '0.028392', 'loss_rpn_bbox': '0.004338', 'loss': '0.166381', time: 1.043, eta: 8 days, 6:58:18
2019-12-08 11:42:57,952-INFO: iter: 32960, lr: 0.002500, 'loss_cls': '0.077417', 'loss_bbox': '0.029750', 'loss_rpn_cls': '0.025627', 'loss_rpn_bbox': '0.006614', 'loss': '0.165685', time: 1.098, eta: 8 days, 17:34:56
2019-12-08 11:43:19,212-INFO: iter: 32980, lr: 0.002500, 'loss_cls': '0.104759', 'loss_bbox': '0.029914', 'loss_rpn_cls': '0.024789', 'loss_rpn_bbox': '0.005739', 'loss': '0.170735', time: 1.056, eta: 8 days, 9:36:00
2019-12-08 11:43:39,756-INFO: iter: 33000, lr: 0.002500, 'loss_cls': '0.116220', 'loss_bbox': '0.036502', 'loss_rpn_cls': '0.019375', 'loss_rpn_bbox': '0.004633', 'loss': '0.182570', time: 1.028, eta: 8 days, 4:06:34
2019-12-08 11:43:39,761-INFO: Save model to output/faster_rcnn_dcn_x101_vd_64x4d_fpn_1x/33000.
2019-12-08 11:43:55,919-INFO: Test iter 0
2019-12-08 11:44:14,859-INFO: Test iter 100
2019-12-08 11:44:33,713-INFO: Test iter 200
2019-12-08 11:44:53,128-INFO: Test iter 300
2019-12-08 11:45:12,317-INFO: Test iter 400
2019-12-08 11:45:30,595-INFO: Test iter 500
2019-12-08 11:45:48,932-INFO: Test iter 600
2019-12-08 11:46:07,444-INFO: Test iter 700
2019-12-08 11:46:25,915-INFO: Test iter 800
2019-12-08 11:46:44,161-INFO: Test iter 900
2019-12-08 11:47:02,660-INFO: Test iter 1000
2019-12-08 11:47:16,320-INFO: Test finish iter 1076
2019-12-08 11:47:16,321-INFO: Total number of images: 1076, inference time: 5.363484210675358 fps.
2019-12-08 11:47:16,698-INFO: Start evaluate...
2019-12-08 11:47:19,054-INFO: Best test box ap: 0.03341993076262028, in iter: 30000
2019-12-08 11:47:40,249-INFO: iter: 33020, lr: 0.002500, 'loss_cls': '0.108531', 'loss_bbox': '0.043173', 'loss_rpn_cls': '0.017900', 'loss_rpn_bbox': '0.004340', 'loss': '0.169767', time: 12.031, eta: 95 days, 15:54:32
2019-12-08 11:48:01,339-INFO: iter: 33040, lr: 0.002500, 'loss_cls': '0.079204', 'loss_bbox': '0.025323', 'loss_rpn_cls': '0.019523', 'loss_rpn_bbox': '0.004204', 'loss': '0.132759', time: 1.051, eta: 8 days, 8:30:30
2019-12-08 11:48:23,153-INFO: iter: 33060, lr: 0.002500, 'loss_cls': '0.090260', 'loss_bbox': '0.036725', 'loss_rpn_cls': '0.027528', 'loss_rpn_bbox': '0.006311', 'loss': '0.179389', time: 1.095, eta: 8 days, 17:01:20
2019-12-08 11:48:44,463-INFO: iter: 33080, lr: 0.002500, 'loss_cls': '0.093552', 'loss_bbox': '0.034076', 'loss_rpn_cls': '0.018074', 'loss_rpn_bbox': '0.007104', 'loss': '0.152736', time: 1.065, eta: 8 days, 11:12:37
2019-12-08 11:49:06,337-INFO: iter: 33100, lr: 0.002500, 'loss_cls': '0.136960', 'loss_bbox': '0.034778', 'loss_rpn_cls': '0.019313', 'loss_rpn_bbox': '0.004328', 'loss': '0.193764', time: 1.087, eta: 8 days, 15:29:17

2019-12-08 11:49:27,727-INFO: iter: 33120, lr: 0.002500, 'loss_cls': '0.096826', 'loss_bbox': '0.027477', 'loss_rpn_cls': '0.027583', 'loss_rpn_bbox': '0.003562', 'loss': '0.162150', time: 1.071, eta: 8 days, 12:25:21


2019-12-08 11:49:48,957-INFO: iter: 33140, lr: 0.002500, 'loss_cls': '0.111899', 'loss_bbox': '0.039398', 'loss_rpn_cls': '0.017612', 'loss_rpn_bbox': '0.005294', 'loss': '0.181398', time: 1.044, eta: 8 days, 7:10:46
2019-12-08 11:50:09,371-INFO: iter: 33160, lr: 0.002500, 'loss_cls': '0.114998', 'loss_bbox': '0.040129', 'loss_rpn_cls': '0.021477', 'loss_rpn_bbox': '0.005812', 'loss': '0.186288', time: 1.042, eta: 8 days, 6:47:26
2019-12-08 11:50:30,408-INFO: iter: 33180, lr: 0.002500, 'loss_cls': '0.088878', 'loss_bbox': '0.026715', 'loss_rpn_cls': '0.027719', 'loss_rpn_bbox': '0.007981', 'loss': '0.171643', time: 1.040, eta: 8 days, 6:19:52
2019-12-08 11:50:51,979-INFO: iter: 33200, lr: 0.002500, 'loss_cls': '0.127711', 'loss_bbox': '0.037682', 'loss_rpn_cls': '0.028521', 'loss_rpn_bbox': '0.004866', 'loss': '0.217983', time: 1.069, eta: 8 days, 11:54:39
2019-12-08 11:51:13,415-INFO: iter: 33220, lr: 0.002500, 'loss_cls': '0.135208', 'loss_bbox': '0.038123', 'loss_rpn_cls': '0.029232', 'loss_rpn_bbox': '0.005568', 'loss': '0.215429', time: 1.088, eta: 8 days, 15:28:55
2019-12-08 11:51:34,174-INFO: iter: 33240, lr: 0.002500, 'loss_cls': '0.124688', 'loss_bbox': '0.035213', 'loss_rpn_cls': '0.030661', 'loss_rpn_bbox': '0.007142', 'loss': '0.207017', time: 1.044, eta: 8 days, 7:04:49
2019-12-08 11:51:55,334-INFO: iter: 33260, lr: 0.002500, 'loss_cls': '0.107978', 'loss_bbox': '0.040368', 'loss_rpn_cls': '0.029550', 'loss_rpn_bbox': '0.005072', 'loss': '0.180974', time: 1.060, eta: 8 days, 10:09:37
2019-12-08 11:52:16,021-INFO: iter: 33280, lr: 0.002500, 'loss_cls': '0.117390', 'loss_bbox': '0.036064', 'loss_rpn_cls': '0.022095', 'loss_rpn_bbox': '0.004055', 'loss': '0.184817', time: 1.032, eta: 8 days, 4:57:10
2019-12-08 11:52:39,507-INFO: iter: 33300, lr: 0.002500, 'loss_cls': '0.099554', 'loss_bbox': '0.028488', 'loss_rpn_cls': '0.030340', 'loss_rpn_bbox': '0.004249', 'loss': '0.173735', time: 1.168, eta: 9 days, 6:50:47
2019-12-08 11:53:00,719-INFO: iter: 33320, lr: 0.002500, 'loss_cls': '0.091982', 'loss_bbox': '0.025819', 'loss_rpn_cls': '0.026813', 'loss_rpn_bbox': '0.004227', 'loss': '0.173283', time: 1.058, eta: 8 days, 9:50:57
/home/admin/.local/lib/python3.6/site-packages/paddle/fluid/executor.py:774: UserWarning: The following exception is not an EOF exception.
  "The following exception is not an EOF exception.")
Traceback (most recent call last):
  File "tools/train.py", line 340, in <module>
    main()
  File "tools/train.py", line 246, in main
    outs = exe.run(compiled_train_prog, fetch_list=train_values)
  File "/home/admin/.local/lib/python3.6/site-packages/paddle/fluid/executor.py", line 775, in run
    six.reraise(*sys.exc_info())
  File "/opt/conda/lib/python3.6/site-packages/six.py", line 693, in reraise
    raise value
  File "/home/admin/.local/lib/python3.6/site-packages/paddle/fluid/executor.py", line 770, in run
    use_program_cache=use_program_cache)
  File "/home/admin/.local/lib/python3.6/site-packages/paddle/fluid/executor.py", line 829, in _run_impl
    return_numpy=return_numpy)
  File "/home/admin/.local/lib/python3.6/site-packages/paddle/fluid/executor.py", line 669, in _run_parallel
    tensors = exe.run(fetch_var_names)._move_to_list()
paddle.fluid.core_avx.EnforceNotMet:

--------------------------------------------
C++ Call Stacks (More useful to developers):
--------------------------------------------
0   std::string paddle::platform::GetTraceBackString<std::string const&>(std::string const&, char const*, int)
1   paddle::platform::EnforceNotMet::EnforceNotMet(std::string const&, char const*, int)
2   void paddle::operators::GPUGather<float, int>(paddle::platform::DeviceContext const&, paddle::framework::Tensor const&, paddle::framework::Tensor const&, paddle::framework::Tensor*)
3   paddle::operators::GatherOpCUDAKernel<float>::Compute(paddle::framework::ExecutionContext const&) const
4   std::_Function_handler<void (paddle::framework::ExecutionContext const&), paddle::framework::OpKernelRegistrarFunctor<paddle::platform::CUDAPlace, false, 0ul, paddle::operators::GatherOpCUDAKernel<float>, paddle::operators::GatherOpCUDAKernel<double>, paddle::operators::GatherOpCUDAKernel<long>, paddle::operators::GatherOpCUDAKernel<int>, paddle::operators::GatherOpCUDAKernel<paddle::platform::float16> >::operator()(char const*, char const*, int) const::{lambda(paddle::framework::ExecutionContext const&)#1}>::_M_invoke(std::_Any_data const&, paddle::framework::ExecutionContext const&)
5   paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_,boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&, paddle::framework::RuntimeContext*) const
6   paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_,boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&) const
7   paddle::framework::OperatorBase::Run(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&)
8   paddle::framework::details::ComputationOpHandle::RunImpl()
9   paddle::framework::details::FastThreadedSSAGraphExecutor::RunOpSync(paddle::framework::details::OpHandleBase*)
10  paddle::framework::details::FastThreadedSSAGraphExecutor::RunOp(paddle::framework::details::OpHandleBase*, std::shared_ptr<paddle::framework::BlockingQueue<unsigned long> > const&, unsigned long*)
11  std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, void> >::_M_invoke(std::_Any_data const&)
12  std::__future_base::_State_base::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>&, bool&)
13  ThreadPool::ThreadPool(unsigned long)::{lambda()#1}::operator()() const

------------------------------------------
Python Call Stacks (More useful to users):
------------------------------------------
  File "/home/admin/.local/lib/python3.6/site-packages/paddle/fluid/framework.py", line 2459, in append_op
    attrs=kwargs.get("attrs", None))
  File "/home/admin/.local/lib/python3.6/site-packages/paddle/fluid/layer_helper.py", line 43, in append_op
    return self.main_program.current_block().append_op(*args, **kwargs)
  File "/home/admin/.local/lib/python3.6/site-packages/paddle/fluid/layers/nn.py", line 10806, in gather
    attrs={'overwrite': overwrite})
  File "/home/admin/.local/lib/python3.6/site-packages/paddle/fluid/layers/detection.py", line 428, in rpn_target_assign
    predicted_cls_logits = nn.gather(cls_logits, score_index)
  File "/data/nas/workspace/jupyter/PaddleDetection-release-0.1/ppdet/core/workspace.py", line 113, in partial_apply
    return op(*args, **kwargs_)
  File "/data/nas/workspace/jupyter/PaddleDetection-release-0.1/ppdet/modeling/anchor_heads/rpn_head.py", line 227, in get_loss
    im_info=im_info)
  File "/data/nas/workspace/jupyter/PaddleDetection-release-0.1/ppdet/modeling/architectures/faster_rcnn.py", line 100, in build
    rpn_loss = self.rpn_head.get_loss(im_info, gt_box, is_crowd)
  File "/data/nas/workspace/jupyter/PaddleDetection-release-0.1/ppdet/modeling/architectures/faster_rcnn.py", line 196, in train
    return self.build(feed_vars, 'train')
  File "tools/train.py", line 128, in main
    train_fetches = model.train(feed_vars)
  File "tools/train.py", line 340, in <module>
    main()

----------------------
Error Message Summary:
----------------------
PaddleCheckError: Expected index.dims()[0] > 0, but received index.dims()[0]:0 <= 0:0.
The index of gather_op should not be empty when the index's rank is 1. at [/paddle/paddle/fluid/operators/gather.cu.h:82]
  [operator < gather > error]
terminate called without an active exception
W1208 11:53:03.514448   366 init.cc:205] *** Aborted at 1575777183 (unix time) try "date -d @1575777183" if you are using GNU date ***
W1208 11:53:03.517763   366 init.cc:205] PC: @                0x0 (unknown)
W1208 11:53:03.519001   366 init.cc:205] *** SIGABRT (@0x1f90000013d) received by PID 317 (TID 0x7f0f2b4bf700) from PID 317; stack trace: ***
W1208 11:53:03.525950   366 init.cc:205]     @     0x7f139613d100 (unknown)
W1208 11:53:03.542176   366 init.cc:205]     @     0x7f1395da15f7 __GI_raise
W1208 11:53:03.555480   366 init.cc:205]     @     0x7f1395da2ce8 __GI_abort
W1208 11:53:03.578831   366 init.cc:205]     @     0x7f137646d84a __gnu_cxx::__verbose_terminate_handler()
W1208 11:53:03.585741   366 init.cc:205]     @     0x7f137646bf47 __cxxabiv1::__terminate()
W1208 11:53:03.608299   366 init.cc:205]     @     0x7f137646bf7d std::terminate()
W1208 11:53:03.621994   366 init.cc:205]     @     0x7f137646bc5a __gxx_personality_v0
W1208 11:53:03.625728   366 init.cc:205]     @     0x7f1388dd3b97 _Unwind_ForcedUnwind_Phase2
W1208 11:53:03.629989   366 init.cc:205]     @     0x7f1388dd3e7d _Unwind_ForcedUnwind
W1208 11:53:03.634865   366 init.cc:205]     @     0x7f139613bd60 __GI___pthread_unwind
W1208 11:53:03.639715   366 init.cc:205]     @     0x7f1396136dd5 __pthread_exit
W1208 11:53:03.664770   366 init.cc:205]     @     0x559fb4fe2289 PyThread_exit_thread
W1208 11:53:03.670189   366 init.cc:205]     @     0x559fb4e7447a PyEval_RestoreThread.cold.736
W1208 11:53:03.674211   366 init.cc:205]     @     0x7f13427e65b9 pybind11::gil_scoped_release::~gil_scoped_release()
W1208 11:53:03.675829   366 init.cc:205]     @     0x7f134279af23 _ZZN8pybind1112cpp_function10initializeIZN6paddle6pybindL22pybind11_init_core_avxERNS_6moduleEEUlRNS2_9operators6reader22LoDTensorBlockingQueueERKSt6vectorINS2_9framework9LoDTensorESaISC_EEE58_bIS9_SG_EINS_4nameENS_9is_methodENS_7siblingEEEEvOT_PFT0_DpT1_EDpRKT2_ENUlRNS_6detail13function_callEE1_4_FUNESY_
W1208 11:53:03.678143   366 init.cc:205]     @     0x7f13427fa6e6 pybind11::cpp_function::dispatcher()
W1208 11:53:03.705709   366 init.cc:205]     @     0x559fb4f23fd4 _PyCFunction_FastCallDict
W1208 11:53:03.722385   366 init.cc:205]     @     0x559fb4fb1d3e call_function
W1208 11:53:03.749676   366 init.cc:205]     @     0x559fb4fd619a _PyEval_EvalFrameDefault
W1208 11:53:03.776196   366 init.cc:205]     @     0x559fb4fac8c8 PyEval_EvalCodeEx
W1208 11:53:03.791987   366 init.cc:205]     @     0x559fb4fad456 function_call
W1208 11:53:03.819322   366 init.cc:205]     @     0x559fb4f23dde PyObject_Call
W1208 11:53:03.846752   366 init.cc:205]     @     0x559fb4fd7994 _PyEval_EvalFrameDefault
W1208 11:53:03.861977   366 init.cc:205]     @     0x559fb4fab7db fast_function
W1208 11:53:03.877820   366 init.cc:205]     @     0x559fb4fb1cc5 call_function
W1208 11:53:03.905719   366 init.cc:205]     @     0x559fb4fd619a _PyEval_EvalFrameDefault
W1208 11:53:03.921274   366 init.cc:205]     @     0x559fb4fab7db fast_function
W1208 11:53:03.937258   366 init.cc:205]     @     0x559fb4fb1cc5 call_function
W1208 11:53:03.964187   366 init.cc:205]     @     0x559fb4fd619a _PyEval_EvalFrameDefault
W1208 11:53:03.989130   366 init.cc:205]     @     0x559fb4fabe4b _PyFunction_FastCallDict
W1208 11:53:04.013437   366 init.cc:205]     @     0x559fb4f2439f _PyObject_FastCallDict
W1208 11:53:04.037919   366 init.cc:205]     @     0x559fb4f28ff3 _PyObject_Call_Prepend

def load_inference_model(args):
    use_gpu =False
    # 设置AnalysisConfig
    config = AnalysisConfig(os.path.join(args.weights, "model"), os.path.join(args.weights,"params"))
    if use_gpu:
        print("use gpu infer")
        config.enable_use_gpu(memory_pool_init_size_mb=3000)
    else:
        print("use cpu infer")
        config.disable_gpu()
        thread_num = 20
        config.set_cpu_math_library_num_threads(thread_num)
        config.enable_mkldnn()
    # 创建PaddlePredictor
    predictor = create_paddle_predictor(config)
    return predictor

#####
def infer(args):
       data_list = []
       for dd in data:
           dd = np.array(dd)
           image = PaddleTensor()
           image.name = "data"
           image.shape = [1] + list(dd.shape)
           print("image.shape", image.shape)
           image.dtype = PaddleDType.FLOAT32
           image.data = PaddleBuf(
              dd.astype("float32").flatten().tolist())
           data_list.append(image)
       print("data_list", data_list)
       print("shapes", [dd.shape for dd in data_list])
       ####
       postprocess_conf["im_height"] = im_height
       postprocess_conf["im_width"] = im_width
       outputs = predictor.run(data_list)
       print("outputs", outputs[0].shape)

W1204 02:16:48.881902  2325 naive_executor.cc:43] The NaiveExecutor can not work properly if the cmake flag ON_INFER is not set.
W1204 02:16:48.881958  2325 naive_executor.cc:45] Unlike the training phase, all the scopes and variables will be reused to save the allocation overhead.
W1204 02:16:48.881970  2325 naive_executor.cc:48] Please re-compile the inference library by setting the cmake flag ON_INFER=ON if you are running Paddle Inference
Traceback (most recent call last):
  File "infer_demo_mkldnn.py", line 233, in <module>
    infer(args)
  File "infer_demo_mkldnn.py", line 166, in infer
    outputs = predictor.run(data_list)
paddle.fluid.core_avx.EnforceNotMet:

--------------------------------------------
C++ Call Stacks (More useful to developers):
--------------------------------------------
0   std::string paddle::platform::GetTraceBackString<std::string const&>(std::string const&, char const*, int)
1   paddle::platform::EnforceNotMet::EnforceNotMet(std::string const&, char const*, int)
2   paddle::operators::ConcatMKLDNNOpKernel<float>::Compute(paddle::framework::ExecutionContext const&) const
3   std::_Function_handler<void (paddle::framework::ExecutionContext const&), paddle::framework::OpKernelRegistrarFunctor<paddle::platform::CPUPlace, false, 0ul, paddle::operators::ConcatMKLDNNOpKernel<float>, paddle::operators::ConcatMKLDNNOpKernel<signed char>, paddle::operators::ConcatMKLDNNOpKernel<unsigned char> >::operator()(char const*, char const*, int) const::{lambda(paddle::framework::ExecutionContext const&)#1}>::_M_invoke(std::_Any_data const&, paddle::framework::ExecutionContext const&)
4   paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&, paddle::framework::RuntimeContext*) const
5   paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&) const
6   paddle::framework::OperatorBase::Run(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&)
7   paddle::framework::NaiveExecutor::Run()
8   paddle::AnalysisPredictor::Run(std::vector<paddle::PaddleTensor, std::allocator<paddle::PaddleTensor> > const&, std::vector<paddle::PaddleTensor, std::allocator<paddle::PaddleTensor> >*, int)

------------------------------------------
Python Call Stacks (More useful to users):
------------------------------------------
  File "/usr/local/lib/python2.7/dist-packages/paddle/fluid/framework.py", line 1771, in append_op
    attrs=kwargs.get("attrs", None))
  File "/usr/local/lib/python2.7/dist-packages/paddle/fluid/layer_helper.py", line 43, in append_op
    return self.main_program.current_block().append_op(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/paddle/fluid/layers/tensor.py", line 210, in concat
    attrs={'axis': axis})
  File "/tmp/code/user_custom.py", line 822, in __call__
    roi_feat_shuffle = fluid.layers.concat(roi_out_list)
  File "/tmp/code/user_custom.py", line 194, in create_model
    roi_feat = roi_extractor(body_feats, rois, spatial_scale)
  File "command.py", line 103, in main
    wfw_obj = wfw_cls(workflow_conf)
  File "command.py", line 108, in <module>
    main()

----------------------
Error Message Summary:
----------------------
PaddleCheckError: Expected input->layout() == DataLayout::kMKLDNN, but received input->layout():NCHW != DataLayout::kMKLDNN:MKLDNNLAYOUT.
Wrong layout set for Input tensor at [/paddle/paddle/fluid/operators/mkldnn/concat_mkldnn_op.cc:34]
  [operator < concat > error]

[pid: 3791][tid: 139684194371328][INFO][2019-12-03 15:43:02,095][handler.py:171] Failed infering, EnforceNotMet:

cd PaddleDetection
export PYTHONPATH=`pwd`:$PYTHONPATH

python tools/export_model.py -c configs/dcn/yolov3_r50vd_dcn.yml \
        -o weights=https://paddlemodels.bj.bcebos.com/object_detection/yolov3_r50vd_dcn_imagenet.tar \
        --output_dir=inference_model \

import paddle.fluid as fluid
import numpy as np
import cv2
from PIL import Image, ImageDraw
from ppdet.utils.coco_eval import get_category_info

def Permute(im, channel_first=True, to_bgr=False):
    if channel_first:
        im = np.swapaxes(im, 1, 2)
        im = np.swapaxes(im, 1, 0)
    if to_bgr:
        im = im[[2, 1, 0], :, :]
    return im


def DecodeImage(im_path):
    with open(im_path, 'rb') as f:
        im = f.read()
    data = np.frombuffer(im, dtype='uint8')
    im = cv2.imdecode(data, 1)  # BGR mode, but need RGB mode
    im = cv2.cvtColor(im, cv2.COLOR_BGR2RGB)
    return im


def ResizeImage(im, target_size=608, max_size=0):
    if len(im.shape) != 3:
        raise ImageError('image is not 3-dimensional.')
    im_shape = im.shape
    print(im_shape)
    im_size_min = np.min(im_shape[0:2])
    im_size_max = np.max(im_shape[0:2])
    if float(im_size_min) == 0:
        raise ZeroDivisionError('min size of image is 0')
    if max_size != 0:
        im_scale = float(target_size) / float(im_size_min)
        # Prevent the biggest axis from being more than max_size
        if np.round(im_scale * im_size_max) > max_size:
            im_scale = float(max_size) / float(im_size_max)
        im_scale_x = im_scale
        im_scale_y = im_scale
    else:
        im_scale_x = float(target_size) / float(im_shape[1])
        im_scale_y = float(target_size) / float(im_shape[0])
    
    im = cv2.resize(
             im,
             None,
             None,
             fx=im_scale_x,
             fy=im_scale_y,
             interpolation=2)
    return im


def NormalizeImage(im,mean = [0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225], is_scale=True):
    """Normalize the image.
    Operators:
        1.(optional) Scale the image to [0,1]
        2. Each pixel minus mean and is divided by std
    """
    im = im.astype(np.float32, copy=False)
    mean = np.array(mean)[np.newaxis, np.newaxis, :]
    std = np.array(std)[np.newaxis, np.newaxis, :]
    if is_scale:
        im = im / 255.0
    im -= mean
    im /= std
    return im


def Prepocess(img_path):
    test_img = DecodeImage(img_path)
    img_shape = test_img.shape[:2]
    test_img = ResizeImage(test_img)
    test_img = NormalizeImage(test_img)
    test_img = Permute(test_img)
    test_img = test_img[np.newaxis,:]#.reshape(1, 3, 608, 608)
    return test_img, img_shape

def colormap(rgb=False):
    """
    Get colormap
    """
    color_list = np.array([
        0.000, 0.447, 0.741, 0.850, 0.325, 0.098, 0.929, 0.694, 0.125, 0.494,
        0.184, 0.556, 0.466, 0.674, 0.188, 0.301, 0.745, 0.933, 0.635, 0.078,
        0.184, 0.300, 0.300, 0.300, 0.600, 0.600, 0.600, 1.000, 0.000, 0.000,
        1.000, 0.500, 0.000, 0.749, 0.749, 0.000, 0.000, 1.000, 0.000, 0.000,
        0.000, 1.000, 0.667, 0.000, 1.000, 0.333, 0.333, 0.000, 0.333, 0.667,
        0.000, 0.333, 1.000, 0.000, 0.667, 0.333, 0.000, 0.667, 0.667, 0.000,
        0.667, 1.000, 0.000, 1.000, 0.333, 0.000, 1.000, 0.667, 0.000, 1.000,
        1.000, 0.000, 0.000, 0.333, 0.500, 0.000, 0.667, 0.500, 0.000, 1.000,
        0.500, 0.333, 0.000, 0.500, 0.333, 0.333, 0.500, 0.333, 0.667, 0.500,
        0.333, 1.000, 0.500, 0.667, 0.000, 0.500, 0.667, 0.333, 0.500, 0.667,
        0.667, 0.500, 0.667, 1.000, 0.500, 1.000, 0.000, 0.500, 1.000, 0.333,
        0.500, 1.000, 0.667, 0.500, 1.000, 1.000, 0.500, 0.000, 0.333, 1.000,
        0.000, 0.667, 1.000, 0.000, 1.000, 1.000, 0.333, 0.000, 1.000, 0.333,
        0.333, 1.000, 0.333, 0.667, 1.000, 0.333, 1.000, 1.000, 0.667, 0.000,
        1.000, 0.667, 0.333, 1.000, 0.667, 0.667, 1.000, 0.667, 1.000, 1.000,
        1.000, 0.000, 1.000, 1.000, 0.333, 1.000, 1.000, 0.667, 1.000, 0.167,
        0.000, 0.000, 0.333, 0.000, 0.000, 0.500, 0.000, 0.000, 0.667, 0.000,
        0.000, 0.833, 0.000, 0.000, 1.000, 0.000, 0.000, 0.000, 0.167, 0.000,
        0.000, 0.333, 0.000, 0.000, 0.500, 0.000, 0.000, 0.667, 0.000, 0.000,
        0.833, 0.000, 0.000, 1.000, 0.000, 0.000, 0.000, 0.167, 0.000, 0.000,
        0.333, 0.000, 0.000, 0.500, 0.000, 0.000, 0.667, 0.000, 0.000, 0.833,
        0.000, 0.000, 1.000, 0.000, 0.000, 0.000, 0.143, 0.143, 0.143, 0.286,
        0.286, 0.286, 0.429, 0.429, 0.429, 0.571, 0.571, 0.571, 0.714, 0.714,
        0.714, 0.857, 0.857, 0.857, 1.000, 1.000, 1.000
    ]).astype(np.float32)
    color_list = color_list.reshape((-1, 3)) * 255
    if not rgb:
        color_list = color_list[:, ::-1]
    return color_list

def draw_bbox(image, catid2name, bboxes, threshold):
    """
    Draw bbox on image
    """
    draw = ImageDraw.Draw(image)

    catid2color = {}
    color_list = colormap(rgb=True)[:40]
    for dt in np.array(bboxes):
        catid, bbox, score = dt['category_id'], dt['bbox'], dt['score']
        if score < threshold:
            continue

        xmin, ymin, w, h = bbox
        xmax = xmin + w
        ymax = ymin + h

        if catid not in catid2color:
            idx = np.random.randint(len(color_list))
            catid2color[catid] = color_list[idx]
        color = tuple(catid2color[catid])

        # draw bbox
        draw.line(
            [(xmin, ymin), (xmin, ymax), (xmax, ymax), (xmax, ymin),
             (xmin, ymin)],
            width=2,
            fill=color)

        # draw label
        text = "{} {:.2f}".format(catid2name[catid], score)
        tw, th = draw.textsize(text)
        draw.rectangle(
            [(xmin + 1, ymin - th), (xmin + tw + 1, ymin)], fill=color)
        draw.text((xmin + 1, ymin - th), text, fill=(255, 255, 255))

    return image

def clip_bbox(bbox):
    xmin = max(min(bbox[0], 1.), 0.)
    ymin = max(min(bbox[1], 1.), 0.)
    xmax = max(min(bbox[2], 1.), 0.)
    ymax = max(min(bbox[3], 1.), 0.)
    return xmin, ymin, xmax, ymax

def bbox2out(results, clsid2catid, is_bbox_normalized=False):
    """
    Args:
        results: request a dict, should include: `bbox`, `im_id`,
                 if is_bbox_normalized=True, also need `im_shape`.
        clsid2catid: class id to category id map of COCO2017 dataset.
        is_bbox_normalized: whether or not bbox is normalized.
    """
    xywh_res = []
    for t in results:
        bboxes = t['bbox'][0]
        lengths = t['bbox'][1][0]
        if bboxes.shape == (1, 1) or bboxes is None:
            continue

        k = 0
        for i in range(len(lengths)):
            num = lengths[i]
            for j in range(num):
                dt = bboxes[k]
                clsid, score, xmin, ymin, xmax, ymax = dt.tolist()
                catid = (clsid2catid[int(clsid)])

                if is_bbox_normalized:
                    xmin, ymin, xmax, ymax = \
                            clip_bbox([xmin, ymin, xmax, ymax])
                    w = xmax - xmin
                    h = ymax - ymin
                    im_height, im_width = t['im_shape'][i].tolist()
                    xmin *= im_width
                    ymin *= im_height
                    w *= im_width
                    h *= im_height
                else:
                    w = xmax - xmin + 1
                    h = ymax - ymin + 1

                bbox = [xmin, ymin, w, h]
                coco_res = {
                    'category_id': catid,
                    'bbox': bbox,
                    'score': score
                }
                xywh_res.append(coco_res)
                k += 1
    return xywh_res

def test():
    infer_prog = fluid.Program()
    startup_prog = fluid.Program()
    
    place = fluid.CUDAPlace(0)
    exe = fluid.Executor(place)
    exe.run(startup_prog)
    
    path = "inference_model/yolov3_r50vd_dcn"
    img_path = "demo/000000014439.jpg"
    
    test_img, img_shape = Prepocess(img_path)
    print("shape of test_img:", test_img.shape)
    img_shape = np.array(img_shape).reshape(1, 2)
    img_shape = img_shape.astype('int32')
    print(img_shape.dtype)
    #exit()
    [inference_program, feed_target_names, fetch_targets] = (fluid.io.load_inference_model(
        dirname=path, executor=exe, model_filename='__model__', params_filename='__params__'))
    print(feed_target_names, test_img.shape, img_shape.shape)
    outs = exe.run(inference_program,
              feed={feed_target_names[0]: test_img, feed_target_names[1]: img_shape},
              fetch_list=fetch_targets,
              return_numpy=False)
    print(img_shape)
    res = {
             'bbox': (np.array(outs[0]), outs[0].recursive_sequence_lengths()),
              'im_shape': img_shape
          }

    clsid2catid, catid2name = get_category_info(None, False, True)
    bbox_results = bbox2out([res], clsid2catid, False)
    print(bbox_results)

    image = Image.open(img_path).convert('RGB')
    image = draw_bbox(image, catid2name, bbox_results, 0.5)
    image.save('test_demo.jpg', quality=95)

if __name__ == '__main__':
    test()

Traceback (most recent call last):
  File "tools/train.py", line 340, in <module>
    main()
  File "tools/train.py", line 246, in main
    outs = exe.run(compiled_train_prog, fetch_list=train_values)
  File "/opt/_internal/cpython-3.6.0/lib/python3.6/site-packages/paddle/fluid/executor.py", line 774, in run
    six.reraise(*sys.exc_info())
  File "/opt/_internal/cpython-3.6.0/lib/python3.6/site-packages/six.py", line 693, in reraise
    raise value
  File "/opt/_internal/cpython-3.6.0/lib/python3.6/site-packages/paddle/fluid/executor.py", line 769, in run
    use_program_cache=use_program_cache)
  File "/opt/_internal/cpython-3.6.0/lib/python3.6/site-packages/paddle/fluid/executor.py", line 828, in _run_impl
    return_numpy=return_numpy)
  File "/opt/_internal/cpython-3.6.0/lib/python3.6/site-packages/paddle/fluid/executor.py", line 668, in _run_parallel
    tensors = exe.run(fetch_var_names)._move_to_list()
paddle.fluid.core_avx.EnforceNotMet:

--------------------------------------------
C++ Call Stacks (More useful to developers):
--------------------------------------------
0   std::string paddle::platform::GetTraceBackString<std::string const&>(std::string const&&&, char const*, int)
1   paddle::platform::EnforceNotMet::EnforceNotMet(std::string const&, char const*, int)
2   void paddle::operators::GPUGather<float, int>(paddle::platform::DeviceContext const&, paddle::framework::Tensor const&, paddle::framework::Tensor const&, paddle::framework::Tensor*)
3   paddle::operators::CUDAGenerateProposalsKernel<paddle::platform::CUDADeviceContext, float>::Compute(paddle::framework::ExecutionContext const&) const
4   _ZNSt17_Function_handlerIFvRKN6paddle9framework16ExecutionContextEEZNKS1_24OpKernelRegistrarFunctorINS0_8platform9CUDAPlaceELb0ELm0EJNS0_9operators27CUDAGenerateProposalsKernelINS7_17CUDADeviceContextEfEEEEclEPKcSF_iEUlS4_E_E9_M_invokeERKSt9_Any_dataS4_
5   paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, paddle::platform::Place const&, paddle::framework::RuntimeContext*) const
6   paddle::framework::OperatorWithKernel::RunImpl(paddle::framework::Scope const&, paddle::platform::Place const&) const
7   paddle::framework::OperatorBase::Run(paddle::framework::Scope const&, paddle::platform::Place const&)
8   paddle::framework::details::ComputationOpHandle::RunImpl()
9   paddle::framework::details::FastThreadedSSAGraphExecutor::RunOpSync(paddle::framework::details::OpHandleBase*)
10  paddle::framework::details::FastThreadedSSAGraphExecutor::RunOp(paddle::framework::details::OpHandleBase*, std::shared_ptr<paddle::framework::BlockingQueue<unsigned long> > const&, unsigned long*)
11  std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()(), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, void> >::_M_invoke(std::_Any_data const&)
12  std::__future_base::_State_base::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()()>&, bool&)
13  _ZZN10ThreadPoolC1EmENKUlvE_clEv

------------------------------------------
Python Call Stacks (More useful to users):
------------------------------------------
  File "/opt/_internal/cpython-3.6.0/lib/python3.6/site-packages/paddle/fluid/framework.py", line 2508, in append_op
    attrs=kwargs.get("attrs", None))
  File "/opt/_internal/cpython-3.6.0/lib/python3.6/site-packages/paddle/fluid/layer_helper.py", line 43, in append_op
    return self.main_program.current_block().append_op(*args, **kwargs)
  File "/opt/_internal/cpython-3.6.0/lib/python3.6/site-packages/paddle/fluid/layers/detection.py", line 2810, in generate_proposals
    'RpnRoiProbs': rpn_roi_probs})
  File "/ssd1/xiege/PaddleDetection/ppdet/core/workspace.py", line 113, in partial_apply
    return op(*args, **kwargs_)
  File "/ssd1/xiege/PaddleDetection/ppdet/modeling/anchor_heads/rpn_head.py", line 438, in _get_single_proposals
    variances=self.anchor_var)
  File "/ssd1/xiege/PaddleDetection/ppdet/modeling/anchor_heads/rpn_head.py", line 462, in get_proposals
    fpn_feat, im_info, lvl, mode)
  File "/ssd1/xiege/PaddleDetection/ppdet/modeling/architectures/cascade_rcnn_cls_aware.py", line 95, in build
    rpn_rois = self.rpn_head.get_proposals(body_feats, im_info, mode=mode)
  File "/ssd1/xiege/PaddleDetection/ppdet/modeling/architectures/cascade_rcnn_cls_aware.py", line 178, in train
    return self.build(feed_vars, 'train')
  File "tools/train.py", line 128, in main
    train_fetches = model.train(feed_vars)
  File "tools/train.py", line 340, in <module>
    main()

----------------------
Error Message Summary:
----------------------
Error: The index of gather_op should not be empty when the index's rank is 1.
  [Hint: Expected index.dims()[0] > 0, but received index.dims()[0]:0 <= 0:0.] at (/ssd3/xiege/paddle/Paddle/paddle/fluid/operators/gather.cu.h:82)
  [operator < generate_proposals > error]
terminate called without an active exception
W1202 12:22:18.564357  5013 init.cc:205] *** Aborted at 1575289338 (unix time) try "date -d @1575289338" if you are using GNU date ***
W1202 12:22:18.566856  5013 init.cc:205] PC: @                0x0 (unknown)
W1202 12:22:18.567122  5013 init.cc:205] *** SIGABRT (@0x12c2) received by PID 4802 (TID 0x7f25b6bfd700) from PID 4802; stack trace: ***
W1202 12:22:18.569178  5013 init.cc:205]     @     0x7f2c6a0647e0 (unknown)
W1202 12:22:18.571429  5013 init.cc:205]     @     0x7f2c694604f5 __GI_raise
W1202 12:22:18.573428  5013 init.cc:205]     @     0x7f2c69461cd5 __GI_abort
W1202 12:22:18.574733  5013 init.cc:205]     @     0x7f2bd822aa8d __gnu_cxx::__verbose_terminate_handler()
W1202 12:22:18.575976  5013 init.cc:205]     @     0x7f2bd8228be6 (unknown)
W1202 12:22:18.577111  5013 init.cc:205]     @     0x7f2bd8228c13 std::terminate()
W1202 12:22:18.578280  5013 init.cc:205]     @     0x7f2bd82288a6 __gxx_personality_v0
W1202 12:22:18.579454  5013 init.cc:205]     @     0x7f2c154bb1ce (unknown)
W1202 12:22:18.580602  5013 init.cc:205]     @     0x7f2c154bb2b4 _Unwind_ForcedUnwind
W1202 12:22:18.582444  5013 init.cc:205]     @     0x7f2c6a062f60 __GI___pthread_unwind
W1202 12:22:18.584290  5013 init.cc:205]     @     0x7f2c6a05d175 __pthread_exit
W1202 12:22:18.586236  5013 init.cc:205]     @     0x7f2c6a426c5f PyThread_exit_thread
W1202 12:22:18.588135  5013 init.cc:205]     @     0x7f2c6a3d32aa PyEval_RestoreThread
W1202 12:22:18.590368  5013 init.cc:205]     @     0x7f2baac61dd9 pybind11::gil_scoped_release::~gil_scoped_release()
W1202 12:22:18.591140  5013 init.cc:205]     @     0x7f2baac14643 _ZZN8pybind1112cpp_function10initializeIZN6paddle6pybindL22pybind11_init_core_avxERNS_6moduleEEUlRNS2_9operators6reader22LoDTensorBlockingQueueERKSt6vectorINS2_9framework9LoDTensorESaISC_EEE58_bIS9_SG_EINS_4nameENS_9is_methodENS_7siblingEEEEvOT_PFT0_DpT1_EDpRKT2_ENUlRNS_6detail13function_callEE1_4_FUNESY_
W1202 12:22:18.593189  5013 init.cc:205]     @     0x7f2baac83fb1 pybind11::cpp_function::dispatcher()
W1202 12:22:18.595247  5013 init.cc:205]     @     0x7f2c6a33cfbd _PyCFunction_FastCallDict
W1202 12:22:18.597225  5013 init.cc:205]     @     0x7f2c6a3d4138 (unknown)
W1202 12:22:18.599215  5013 init.cc:205]     @     0x7f2c6a3d7acc _PyEval_EvalFrameDefault
W1202 12:22:18.601167  5013 init.cc:205]     @     0x7f2c6a3d3fce (unknown)
W1202 12:22:18.603116  5013 init.cc:205]     @     0x7f2c6a3d45f3 PyEval_EvalCodeEx
W1202 12:22:18.605052  5013 init.cc:205]     @     0x7f2c6a3163f3 (unknown)
W1202 12:22:18.606988  5013 init.cc:205]     @     0x7f2c6a2e434a PyObject_Call
W1202 12:22:18.608937  5013 init.cc:205]     @     0x7f2c6a3d7e55 _PyEval_EvalFrameDefault
W1202 12:22:18.610893  5013 init.cc:205]     @     0x7f2c6a3d3660 (unknown)
W1202 12:22:18.612833  5013 init.cc:205]     @     0x7f2c6a3d4584 (unknown)
W1202 12:22:18.614792  5013 init.cc:205]     @     0x7f2c6a3d7acc _PyEval_EvalFrameDefault
W1202 12:22:18.616705  5013 init.cc:205]     @     0x7f2c6a3d3660 (unknown)
W1202 12:22:18.618620  5013 init.cc:205]     @     0x7f2c6a3d4584 (unknown)
W1202 12:22:18.620504  5013 init.cc:205]     @     0x7f2c6a3d7acc _PyEval_EvalFrameDefault
W1202 12:22:18.622380  5013 init.cc:205]     @     0x7f2c6a3d3660 (unknown)
W1202 12:22:18.624182  5013 init.cc:205]     @     0x7f2c6a3dcc66 _PyFunction_FastCallDict