shaohua0116 / novelviewsynthesis-tensorflow Goto Github PK

Hi,
I am encountering a strange error when training with my own dataset. The training works for a while then gives me this error:

`
Traceback (most recent call last):
File "/home/ebartrum/anaconda3/envs/novel_view/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1334, in _do_call
return fn(*args)
File "/home/ebartrum/anaconda3/envs/novel_view/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1319, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/home/ebartrum/anaconda3/envs/novel_view/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1407, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.OutOfRangeError: RandomShuffleQueue '_1_shuffle_batch/random_shuffle_queue' is closed and has insufficient elements (requested 12, cur
rent size 0)
[[{{node shuffle_batch}} = QueueDequeueManyV2[component_types=[DT_FLOAT, DT_STRING, DT_FLOAT], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"]
(shuffle_batch/random_shuffle_queue, shuffle_batch/n)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "trainer.py", line 185, in
main()
File "trainer.py", line 182, in main
trainer.train()
File "trainer.py", line 113, in train
self.run_single_step(self.batch_train, step=s, is_train=True)
File "trainer.py", line 130, in run_single_step
batch_chunk = self.session.run(batch)
File "/home/ebartrum/anaconda3/envs/novel_view/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 929, in run
run_metadata_ptr)
File "/home/ebartrum/anaconda3/envs/novel_view/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1152, in _run
feed_dict_tensor, options, run_metadata)
File "/home/ebartrum/anaconda3/envs/novel_view/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run
run_metadata)
File "/home/ebartrum/anaconda3/envs/novel_view/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.OutOfRangeError: RandomShuffleQueue '_1_shuffle_batch/random_shuffle_queue' is closed and has insufficient elements (requested 12, current size 0)
[[node shuffle_batch (defined at /home/ebartrum/Documents/repos/NovelViewSynthesis-TensorFlow/input_ops.py:76) = QueueDequeueManyV2[component_types=[DT_FLOAT, DT_STRING, DT_FLOAT], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](shuffle_batch/random_shuffle_queue, shuffle_batch/n)]]

Caused by op 'shuffle_batch', defined at:
File "trainer.py", line 185, in
main()
File "trainer.py", line 179, in main
trainer = Trainer(config, model, dataset_train, dataset_test)
File "trainer.py", line 39, in init
dataset, self.batch_size, is_training=True)
File "/home/ebartrum/Documents/repos/NovelViewSynthesis-TensorFlow/input_ops.py", line 76, in create_input_ops
min_after_dequeue=min_capacity,
File "/home/ebartrum/anaconda3/envs/novel_view/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 306, in new_func
return func(*args, **kwargs)
File "/home/ebartrum/anaconda3/envs/novel_view/lib/python3.6/site-packages/tensorflow/python/training/input.py", line 1344, in shuffle_batch
name=name)
File "/home/ebartrum/anaconda3/envs/novel_view/lib/python3.6/site-packages/tensorflow/python/training/input.py", line 871, in _shuffle_batch
dequeued = queue.dequeue_many(batch_size, name=name)
File "/home/ebartrum/anaconda3/envs/novel_view/lib/python3.6/site-packages/tensorflow/python/ops/data_flow_ops.py", line 478, in dequeue_many
self._queue_ref, n=n, component_types=self._dtypes, name=name)
File "/home/ebartrum/anaconda3/envs/novel_view/lib/python3.6/site-packages/tensorflow/python/ops/gen_data_flow_ops.py", line 3487, in queue_dequeue_many_v2
component_types=component_types, timeout_ms=timeout_ms, name=name)
File "/home/ebartrum/anaconda3/envs/novel_view/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/home/ebartrum/anaconda3/envs/novel_view/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
return func(*args, **kwargs)
File "/home/ebartrum/anaconda3/envs/novel_view/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3274, in create_op
op_def=op_def)
File "/home/ebartrum/anaconda3/envs/novel_view/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1770, in init
self._traceback = tf_stack.extract_stack()

OutOfRangeError (see above for traceback): RandomShuffleQueue '_1_shuffle_batch/random_shuffle_queue' is closed and has insufficient elements (requested 12, current size 0)
[[node shuffle_batch (defined at /home/ebartrum/Documents/repos/NovelViewSynthesis-TensorFlow/input_ops.py:76) = QueueDequeueManyV2[component_types=[DT_FLOAT, DT_STRING, DT_FLOAT], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](shuffle_batch/random_shuffle_queue, shuffle_batch/n)]]
`
Do you know what this means?

Hey,
I have tried to adapt your code to run on my dataset which contains images of size 128,128. I changed output_shapes accordingly in object_loader.py. The training runs but doesnt converge to a solution, the output images are uniformly black (which is the background colour of my dataset). Do you know why this would be and how I could fix it?
Thanks

Hi, first of all, thanks for the open-sourcing of this project, it is really fantastic on the part of your team. However, I tried to use tensorboard directly from your code however it says that no dashboard are currently active. To lanunch, I used tensorboard --logdir train_dir/folder_with_train_data
Is it right?

I am reproducing the paper of you,but I found the error of training does not converge.The test error and similarity did not reach your results,on the dataset of kitti,the training error is about 0.15 and cannot go down.As to the dataset of chair,the training error is about 0.1 and cannot go down.In this condition,the error of testing was poor.In this case could you give me some guidance?Thank you very much!

Hi!
Are there any checkpoints available for further fine-tuning?
(especially interested in the one trained on KITTI)