Comments (1)
Having looked into this further it seems the core of the issue is with two different formats in the same repo.
When the parquet
config is first, the WebDataset
s are loaded as parquet
, if the WebDataset
configs are first, the parquet
is loaded as WebDataset
.
A workaround in my case would be to just turn the parquet
into a WebDataset
, although I'd still need the Dataset Viewer config limit increasing. In other cases using the same format may not be possible.
Relevant code:
from datasets.
Related Issues (20)
- Missing pyspark dependency causes the testsuite to error out, instead of a few tests to be skipped
- The test test_move_script_doesnt_change_hash fails because it runs the 'python' command while the python executable has a different name
- load_dataset with multiple jsonlines files interprets datastructure too early HOT 5
- Add Arabic Docs to datasets
- Some of DownloadConfig's properties are always being overridden in load.py
- IterableDataset: cannot resolve features from list of numpy arrays
- Slow iteration speeds when using IterableDataset.shuffle with load_dataset(data_files=..., streaming=True) HOT 2
- load_dataset broken in 2.21.0 HOT 4
- website broken: Create a new dataset repository, doesn't create a new repo in Firefox HOT 4
- ConnectionError for gated datasets and unauthenticated users
- CI is broken for numpy-2: Failed to fetch wheel: llvmlite==0.34.0 HOT 2
- cudf-cu12 24.4.1, ibis-framework 8.0.0 requires pyarrow<15.0.0a0,>=14.0.1,pyarrow<16,>=2 and datasets 2.21.0 requires pyarrow>=15.0.0 HOT 1
- Stream dataset does not iterate if the batch size is larger than the dataset size (related to drop_last_batch) HOT 1
- module 'pyarrow.lib' has no attribute 'ListViewType' HOT 1
- datasets cannot handle nested json if features is given. HOT 1
- Audio dataset load everything in RAM and is very slow HOT 3
- [interleave_dataset] sample batches from a single source at a time
- Make dataset viewer more flexible in displaying metadata alongside images
- Caching shuffles by np.random.Generator results in unintiutive behavior HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from datasets.