Comments (9)
Sorry for the late response, I was on holidays.
I looked into xarray in the beginning and decided against because of the missing sparse data support and the plain fact, that things like scikit-learn only accept numpy arrays and sparse matrices as input.
These days, we're putting a lot of thought in improving the backed infrastructure of anndata for chunked calculations. We might return to xarray for that reason. I can also keep you posted on the benchmarks, soon, here.
from anndata.
However, as soon as xarray fully supports sparse arrays, it should handle this wrapping by itself.
Well, to be clear -- it could handle the wrapping by itself. We would need to define a metadata convention (but this should be pretty simple/straightforward).
from anndata.
we now support zarr, which is feature-comparable so i guess this can be closed
from anndata.
@Hoeze, do you have a sense of how sparse data could be handled with netCDF or if anyone is working on it? I saw you had mentioned this on the xarray sparse issue, but haven't been able to find out too much myself.
If we could conform more to a standard like netCDF, that could help with interchange as mentioned here: ivirshup/sc-interchange#5.
from anndata.
@ivirshup Yes, there are some things ongoing.
The best bet for native sparse array support in xarray will be pydata/sparse.
However, you should talk to @shoyer for the native integration into xarray.
It would be awesome if someone would push this!
This solution will likely only support COO format for some time until pydata/sparse supports CSD (see pydata/sparse#258).
However, a lot of frameworks like TileDB or Tensorflow support only COO anyway.
In the meantime you can still save the data in sparse format and wrap it yourself.
I.e. take the coordinate index and the data array from your sparse matrix and save this as NetCDF4.
This of course requires some wrapping inside AnnData or any other framework you want to use.
IMHO, if possible I would prefer a dense matrix over a sparse one.
Everything with a sparsity ratio lower than 90-95% will very likely cost more processing power to decode than you can theoretically save. Especially in cases where you have to convert it to dense format anyway.
Also, compression algorithms can save comparable amounts of storage.
In each case, you save a lot of engineering effort.
However, @falexwolf might have another opinion, as he did a lot of benchmarking on this.
from anndata.
Thanks for the feedback!
There were some very cool PRs over the weekend that make this seem closer to reality, like pydata/sparse#261.
However, a lot of frameworks like TileDB or Tensorflow support only COO anyway.
I think this is fine. On the fly conversion from COO to CSR or CSC should be easy enough. The main issue with COO right now is that scipy.sparse
's version doesn't have subsetting, which makes it a pain to use here.
I.e. take the coordinate index and the data array from your sparse matrix and save this as NetCDF4.
This of course requires some wrapping inside AnnData or any other framework you want to use.
I'm not entirely sure what this entails. Will I be able to have a COO array and dense array with shared coordinates in a netcdf file? Or is that the wrapping you were referring to?
IMHO, if possible I would prefer a dense matrix over a sparse one.
I don't think one in unequivocally better that the other for all operations. In my experience, reading the whole matrix into memory is much faster when it's sparse on disk. This may be less of an issue with more modern compression algorithms, but support is limited with hdf5.
To me, the main pain points with sparse representation are random access along non-compressed dimensions, library support (though this is fairly good for in-memory data), and chunking.
from anndata.
There were some very cool PRs over the weekend that make this seem closer to reality, like pydata/sparse#261.
Yes, with pydata/xarray#3117 this could finally happen soon!
I.e. take the coordinate index and the data array from your sparse matrix and save this as NetCDF4.
This of course requires some wrapping inside AnnData or any other framework you want to use.I'm not entirely sure what this entails. Will I be able to have a COO array and dense array with shared coordinates in a netcdf file? Or is that the wrapping you were referring to?
Yes, that's the wrapping problem:
NetCDF does not have (as far as i know) any conventions about storing sparse structures, trees, etc.
This means you have to store e.g. a sparse COO matrix as a coordinate matrix and a value vector.
When reading this data, you then have to wrap it with e.g. pydata/sparse, scipy.sparse or another language-dependent library.
However, as soon as xarray fully supports sparse arrays, it should handle this wrapping by itself.
IMHO, if possible I would prefer a dense matrix over a sparse one.
I don't think one in unequivocally better that the other for all operations. In my experience, reading the whole matrix into memory is much faster when it's sparse on disk. This may be less of an issue with more modern compression algorithms, but support is limited with hdf5.
To me, the main pain points with sparse representation are random access along non-compressed dimensions, library support (though this is fairly good for in-memory data), and chunking.
TileDB will be very useful in this case. It is multithreaded and stores data in chunks. I.e. even non-compressed dimension lookups should be quite fast.
Unfortunately, TileDB's Python and R library are still in its infancy.
from anndata.
@shoyer, what would the goals of an NetCDF-storable sparse array be for xarray? Would you just want to target reading the whole array into memory at once via xarray?
I see how this would be straight forward. If partial/chunked access for dask or keeping the data compatible with NetCDF libraries are goals I think it get's more complicated. Are these cases in scope for an xarray solution, or would this have to happen downstream?
from anndata.
Reading whole sparse arrays from a netCDF file at once seems like a good start, and something that could easily be done in xarray.
Eventually, it would probably be nice to have chunked/partial access, but that does seem much more complicated. I'm not sure netCDF is the right file format in that case, since you probably want a more intelligent (tree like) on-disk indexing scheme and netCDF's compression filters are not very flexible. Maybe this could be done more easily with zarr? Either way, xarray could wrap a third-party library that implements sparse arrays on disk.
from anndata.
Related Issues (20)
- Upcoming Scipy 1.13 incompatible with file-backed sparse matrices
- CZI files' `var` count changd
- `/` in column names makes AnnData Zarr object unreadable on windows HOT 12
- String indexes in var lead to UserWarning HOT 1
- Move tests out of the package
- Refactor `BaseCompressedSparseDataset` and subclasses to remove `backed_{csr,csc}_matrix`
- concat_on_disk should use int64 indptrs by default so that it can concatenate large files HOT 4
- TypeError: Can't implicitly convert non-string objects to strings HOT 4
- Error concatenating scRNA with visium dataset
- Error: Expected a python object, received a character HOT 1
- memory usage of concat HOT 3
- Using `Pint` for units HOT 9
- UMAP of gene subset of adata HOT 2
- Error using sc.pl.stacked_violin HOT 1
- NotImplementedError with concat_on_disk
- Issue when setting anndata.X to numpy array HOT 3
- Does anndata.raw can be automatically modified after define it?
- Make the call to `_check_uniqueness()` optional in `_init_as_actual()` (and `__init__()`) HOT 1
- `concat_on_disk` outer join
- `concat_on_disk` merge strategies are untested/not implemented
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from anndata.