tudyx / ai-dataloader Goto Github PK

Hello,

It could be really nice to add error handling while fetching.
eg. For Dataset that reads data from IO file/http .....
Currently we can only unwrap/expect.
One solution could be to allow fetch to return a Result. And then collat if all Result are Ok
Iterator then return a Result<Batch, Self::E>

Hello,

I was wondering if we can create a python wrapper to this library using pyO3/maturin and ship it on Pypi.
This way python user could also benefit from this crate ie no more process fork but real multithreading.

Right now we have benchmarks to compare with PyTorch but not really to compare ai-dataloader with itself.
It will be very useful to have some, to be able to make choices based on real data. For instance, it would help with #17, #13 and #1

There are some benchmark using the criterion library but, at least on my computer, they are too noisy to be useful. It will be useful to improve them or maybe try a different approach.

The rustls benchmarks seems very qualitative and could be a good source of inspiration.

Hello,
I think it could be great to pass a variable (pin_memory) to the dataloader to allow pin memory for the TorchCollate case. Wdyt ?

Hello,

Thank you for your awesome work !!!
As you may know there is no dataloader feature in tch-rs, it could be really cool to have it with ai-dataloader and maybe with multi-threading handling in further steps.

Thank you :)

In order to reduce compile time / complexity.
It could be nice to make tch-rs and ndarray option feature.
WDYT ?

I was wondering how to build a DataLoader-usable dataset consisting of multiple tch::Tensor, and the error I encountered was not implementing Collate< (Tensor), (Tensor), (Tensor), (Tensor) > for DefaultCollate