How to deal with huge data? about tensorflow-101 HOT 10 OPEN

you can use the input queues to read from the filesystem instead of loading everything to memory, this is discussed in this tutorial https://ischlag.github.io/2016/06/19/tensorflow-input-pipeline-example/

from tensorflow-101.

Thanks for the response @aditbiswas1 . If my current dataset directory structure is like this:
Dataset
->Category A
--->Photos
->Category B
--->Photos
->Category C
--->Photos
->Category D
--->Photos

should i make the csv file containing the path to the datasets first or is there any other way to deal with that kind of problem?

from tensorflow-101.

hmm, ya using a csv sounds like a reasonably well way to solve the problem 👍

from tensorflow-101.

is there any tutorial that you know to create csv out of it? sorry i'm new at this @aditbiswas1

from tensorflow-101.

hi @aditbiswas1 i've learn on how to do things from the tutorial you gave, but i was wondering if i wanted to resize my image first, in which step i can do it? this is the tutorial btw https://ischlag.github.io/2016/06/19/tensorflow-input-pipeline-example/

from tensorflow-101.

oh hi, sorry didnt see the previous notification. hmm, you can create a csv in multiple ways, I would normally try to find a way to create a python list from all the fileobject names and then just dump that into a csv using the inbuilt csv module, you'll most likely want to just write some loops to do this. you can get a list of what files are present in the directory by using os.listdir("directory")

regarding resizing of the images. this can be part of your preprocessing right after you've decoded the image file. an example for this can be found in the docs here
https://www.tensorflow.org/api_docs/python/image/resizing#resize_images

from tensorflow-101.

since you also wanted to do some stuff like grouping by labels, you probably want to look at something like pandas which is super useful for manipulating datasets. http://pandas.pydata.org/pandas-docs/version/0.13.1/index.html

from tensorflow-101.

hi @aditbiswas1 i've managed to enter the training step, but at certain epoch the training is stopped. It said that there is an Out of Range Error because of FIFO batch. Here is the error:

OutOfRangeError (see above for traceback): FIFOQueue '_3_batch/fifo_queue' is closed and has insufficient elements (requested 8, current size 5)
[[Node: batch = QueueDequeueMany[_class=["loc:@batch/fifo_queue"], component_types=[DT_FLOAT, DT_INT32], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/cpu:0"](batch/fifo_queue, batch/n)]]

I'm following the tutorial you gave, but i uncomment the num_thread line in the tf.train.batch

from tensorflow-101.

from tensorflow-101.

yes i've managed to reach 600-ish iterations, and i got the error, and i set the batch size into 8, and i also suspect that the last batch is not dividable by 8, but i've used this line of code : allow_smaller_final_batch=True
but still didn't work

from tensorflow-101.