slow IO speed when using ImageFolder on ImageNet-100 about pycil HOT 1 CLOSED

It is indeed possible to load all the data from the ImageNet-100 dataset into memory at the beginning, and I have successfully done this before. However, implementing this approach can be quite challenging due to the varying sizes of images in ImageNet. The use of numpy arrays is not suitable for loading images with different sizes. If we were to use lists instead of numpy arrays, we would lose several convenient features, such as the ability to index elements using a list of indices (e.g., a[[1, 3, 5]]).

To achieve this loading of data into memory at the start, modifications would need to be made to the data_manager.py and base.py files. However, these changes would require a significant amount of time and effort."

Please let me know if there's anything else I can help you with!

from pycil.