How does dask-ml handle large datasets? about dask-ml HOT 7 CLOSED

It seems like a brief page describing the scalability of different approaches might be in-scope.

from dask-ml.

Several of the algorithms mentioned in the documentation are parallel implementations on large datasets. See http://dask-ml.readthedocs.io/en/latest/

from dask-ml.

Can we please explicitly list them here? Are the GLMs a custom implementation in dask? Future plans?

from dask-ml.

Can we please explicitly list them here?

I'm not sure I undestand the question. They are all listed in the documentation. Perhaps you want the API reference? http://dask-ml.readthedocs.io/en/latest/modules/api.html

If you want to list them here that would be fine with me.

Are the GLMs a custom implementation in dask?

Yes. This is already answered in the documentation. See http://dask-ml.readthedocs.io/en/latest/glm.html

If the documentation is not sufficiently clear it would be great if you would collaborate to improve it.

from dask-ml.

Sorry, I was typing quickly to finish my thoughts before a call and didn't do a good job. I think I'm asking a question that wouldn't necessarily be expected to be covered by the docs.

What I meant was: What are all the custom algorithm implementations from scratch in dask-ml? I picked up from the docs that GLMs are indeed a custom implementation from scratch. Others (?):

• Preprocessors
• Spectral clustering

The rest seem to wrap existing implementations in some way to better parallelize them, e.g., for parameter search or CV. Is that a correct characterization?

from dask-ml.

I think that a comparasion of the different approaches in the same way of the Spectral Clustering Example would be great. It could answer questions like: "how well is dask grid search implementation when compared to sklearn?"

from dask-ml.

It looks like @TomAugspurger is doing some work to describe scalability here: #111

I encourage people to engage on that PR

from dask-ml.