Comments (2)
awolant
commented on May 28, 2024
Hello @cyanic-selkie, nice to hear from you again.
Thanks a lot for your work to run these tests.
Is this something that can be easily fixed? How difficult would it be to implement the experimental_fetch_to_device option for other strategies, most notably the MultiWorkerMirroredStrategy. We need this strategy in particular since our infrastructure is made out of A100 MiGs which cannot be addressed individually on a single host.
To best answer this I need to contact some other teams at NVIDIA who we collaborated with on this feature when it was developed. I will get back to you on that.
One important question here is what is your exact software stack? Mainly, do you use upstream TensorFlow or the NGC container?
from dali.
cyanic-selkie
commented on May 28, 2024
One important question here is what is your exact software stack? Mainly, do you use upstream TensorFlow or the NGC container?
We use custom containers with nvidia/cuda:11.8.0-cudnn8-devel-ubuntu22.04 as the base image. We then simply install Tensorflow/DALI via pip inside it.
One important thing that I completely forgot about is that we use tcmalloc.
Our driver version is 525.60.13.
from dali.
Related Issues (20)
- Can I use DALI only for data augmentation? HOT 2
- Checkpoint support for readers HOT 7
- How to use Dali tool for image sharpening operation HOT 4
- Ran of GPU memory when using Imagenet but not COCO-Stuff 2017 HOT 20
- how to do a image zoom? HOT 9
- Unrecognized image format HOT 1
- Inference Model without converting TensorGPU to TensorCPU HOT 2
- Stack a batch in one batch of this shape HOT 4
- How to get center crop HOT 6
- GitHub Roadmap 2024 HOT 4
- Why is the val_loss curve trained through Dali data loading method oscillating？ HOT 2
- NumPy decoder HOT 8
- Using JPEG hardware decoder with DALI on A100 GPU
- Extracting properties from a list of DataNodes HOT 5
- A100 hardware decoder HOT 1
- Extract motion vectors HOT 7
- Segmentation fault when using 'mixed' HOT 5
- Bbox Pruning Too Aggressive? HOT 5
- Indexing video with binary mask HOT 1
- source_info tensor not guaranteed to contain correct data HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dali.