Comments (3)
@jpfeil Thank you for reporting this.
A few questions:
- Does the image get decoded eventually or your pipeline fails? Note that in DALI there is a hierarchy of decoders and when one fails the next one tries to handle the image. This error message comes from one of them (libjpeg-turbo based), but there's one more (OpenCV based), so I am curious if that one failed as well.
- Could you please share the image that fails so that I can check on my end?
from dali.
Thanks, @jantonguirao. I didn't realize DALI could still process the larger images. I just figured it was skipping the larger images. The code block mentions failing fast to prevent an ungraceful exit, but I'm not really familiar with the DALI codebase. Let me check if the images are getting processed eventually.
` // Determine the output image size before attempting decompress to prevent
// OOM'ing doing the decompress
jpeg_calc_output_dimensions(&cinfo);
int64 total_size = static_cast(cinfo.output_height) *
static_cast(cinfo.output_width) *
static_cast(cinfo.num_components);
// Some of the internal routines do not gracefully handle ridiculously
// large images, so fail fast.
if (cinfo.output_width <= 0 || cinfo.output_height <= 0) {
ERROR_LOG << "Invalid image size: " << cinfo.output_width << " x "
<< cinfo.output_height << std::endl;
return nullptr;
}
if (total_size >= (1LL << 29)) {
ERROR_LOG << "Image too large: " << total_size << std::endl;
return nullptr;
}`
from dali.
Yes, this decoder fails, but after that there's an opencv based decoder, so I am curious if that one is able to decode the large image
from dali.
Related Issues (20)
- Segmentation fault when using 'mixed' HOT 5
- Bbox Pruning Too Aggressive? HOT 5
- Indexing video with binary mask HOT 1
- source_info tensor not guaranteed to contain correct data HOT 1
- 16 bit gray scale Image read error HOT 1
- COCO Reader pixelwise_masks Emtpy Output HOT 7
- Dali on Jetson: nvidia.dali.fn.readers.video_resize is missing HOT 4
- Numpy reader test (GDS) HOT 4
- How to add a scalar value to the loader? HOT 1
- Can DALI be integrated into HuggingFace Trainer? HOT 9
- Bug in creating `TensorGPU` when `stream` key is `None` in CUDA array interface HOT 2
- Webdataset reader behavior with many sources HOT 1
- ModuleNotFoundError: No module named 'nvidia.dali.python_function_plugin' HOT 3
- Speed up Dino with DALI HOT 3
- error using webdataset
- webdataset cannot stop cycling at end of epoch HOT 11
- Get audio data from external data sources and start iteration HOT 7
- Encountered some issues when using mirror flip HOT 2
- Can AArch64 Numba tests be re-renabled? HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dali.