sklearn-xarray compatability about dask-ml HOT 4 OPEN

Thanks @phausamann, I tried with

import xarray as xr
from sklearn_xarray import wrap
from sklearn_xarray.data import load_dummy_dataarray
from dask_ml.preprocessing import StandardScaler

X = load_dummy_dataarray()
X.to_netcdf("/tmp/X.nc")
X2 = xr.open_mfdataset("/tmp/X.nc").to_array()[0]

X3 = xr.concat([X2] * 1000, dim='sample')

Xt = wrap(StandardScaler()).fit_transform(X3)
Xt

And everything seemed to work correctly.

from dask-ml.

In the end, I even made a new release (0.1.3) which is available through pip as of now.

With this update, the above code should work, as well as (hopefully) other dask-ml estimators. Feel free to open an issue in the sklearn-xarray repository if anything doesn't work as expected.

from dask-ml.

I've looked into it and I believe that for the simple case of using dask-ml estimators on the underlying data of DataArrays and Datasets it is sufficient to make sure the wrapper always passes the .data attribute to the estimator.

I'll push a commit to the develop branch of sklearn-xarray later today that should make the following code work:

from sklearn_xarray import wrap
from dask_ml.preprocessing import StandardScaler
import xarray as xr
import numpy as np
import dask.array as da

X = xr.DataArray(
        da.from_array(np.random.random((100, 10)), chunks=(10, 10)),
        coords={'sample': range(100), 'feature': range(10)},
        dims=('sample', 'feature')
    )

Xt = wrap(StandardScaler()).fit_transform(X)

In []: type(Xt.data)
Out[]: dask.array.core.Array

Now, for compatibility with fully dasked-backed Datasets there's a lot more to be done in order to make sure that on-disk coordinates are not completely loaded into memory during the coordinate update...

from dask-ml.

Having dask-ml and sklearn-xarray "compatible" would be a great plus for both libraries as it would allow a simple mechanism for scaling stat methods to large datasets !

from dask-ml.