tudyx / ai-dataloader Goto Github PK
View Code? Open in Web Editor NEWA rust port of pytorch dataloader
License: Apache License 2.0
A rust port of pytorch dataloader
License: Apache License 2.0
Hello,
It could be really nice to add error handling while fetching.
eg. For Dataset that reads data from IO file/http .....
Currently we can only unwrap/expect.
One solution could be to allow fetch to return a Result. And then collat if all Result are Ok
Iterator then return a Result<Batch, Self::E>
Hello,
I was wondering if we can create a python wrapper to this library using pyO3/maturin and ship it on Pypi.
This way python user could also benefit from this crate ie no more process fork but real multithreading.
Right now we have benchmarks to compare with PyTorch but not really to compare ai-dataloader
with itself.
It will be very useful to have some, to be able to make choices based on real data. For instance, it would help with #17, #13 and #1
There are some benchmark using the criterion library but, at least on my computer, they are too noisy to be useful. It will be useful to improve them or maybe try a different approach.
The rustls benchmarks seems very qualitative and could be a good source of inspiration.
Hello,
I think it could be great to pass a variable (pin_memory) to the dataloader to allow pin memory for the TorchCollate case. Wdyt ?
Hello,
Thank you for your awesome work !!!
As you may know there is no dataloader feature in tch-rs, it could be really cool to have it with ai-dataloader and maybe with multi-threading handling in further steps.
Thank you :)
In order to reduce compile time / complexity.
It could be nice to make tch-rs and ndarray option feature.
WDYT ?
I was wondering how to build a DataLoader-usable dataset consisting of multiple tch::Tensor, and the error I encountered was not implementing Collate< (Tensor), (Tensor), (Tensor), (Tensor) > for DefaultCollate
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.