We mirror scipy.fft, but not anymore numpy.fft.

scipy.fft is mostly an overload of numpy.fft so this should be trivial.

• Mirror np.fft
• Add utests

I see in the comment this is due to some poetry bug, but you really should not force such a dependency on the casual users which may not even use poetry, but simply want to install xrscipy into some environment with pip. For instance, on some older environments (in scientific environments on clusters one does not always have a choice) urllib>2 clasheswith some conda dependencies.

This could perhaps be in the dev requirements if you need it.

From #4 :

For example, interpolate module should be deprecated as xarray now natively supports it.

The doc-build is now failing because of the following reason.

Read the Docs build information
Build id: 9895546
Project: xr-scipy
Version: latest
Commit: 0a1a4af1d68384105fbdbd83d8e5fe9e8fcb873b
Date: 2019-11-01T10:05:46.878426Z
State: finished
Success: False


[rtd-command-info] start-time: 2019-11-01T10:05:47.322490Z, end-time: 2019-11-01T10:05:47.883210Z, duration: 0, exit-code: 0
git clone --no-single-branch --depth 50 https://github.com/fujiisoup/xr-scipy.git .
Cloning into '.'...

[rtd-command-info] start-time: 2019-11-01T10:05:48.190562Z, end-time: 2019-11-01T10:05:48.201085Z, duration: 0, exit-code: 0
git checkout --force origin/master
Note: checking out 'origin/master'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at 0a1a4af Merge xarray-dsp (#5)

[rtd-command-info] start-time: 2019-11-01T10:05:48.263911Z, end-time: 2019-11-01T10:05:48.271728Z, duration: 0, exit-code: 0
git clean -d -f -f


[rtd-command-info] start-time: 2019-11-01T10:05:52.658558Z, end-time: 2019-11-01T10:06:58.074382Z, duration: 65, exit-code: 137
conda env create --quiet --name latest --file doc/environment.yml
Collecting package metadata: ...working... done
Solving environment: ...working... Killed


Command killed due to excessive memory consumption

As we have discussed in the mailing group, it might be reasonable to merge this project with my smaller project smartass101/xarray-dsp.

The xarray_dsp package in the project currently features mostly functions for frequency and other filters and spectral analysis and a few other wrappers. It mostly targets time-domain analysis.

There are also a few wrappers for wavelets from pycwt, but I guess those wouldn't really fit into xr-scipy since pycwt is not included in scipy. It's also a question whether you want to restrict this project to scipy functions.

I also found kaipak/xrsigproc which also uses some smoothing functions from scipy.ndimage and scipy.convolve for (probably) spatial filtering.
There is also serazing/xscale which also does some signal processing and FFT using mostly scipy again.
Maybe we could contact these projects as well.

Hello,

I'm working with dask, and I stumbled across the need to specify the dask="parallelized" argument to xr.apply_ufunc in frequency_filter. Currently, there is no way to do that.

One way to do it is to unpack apply_kwargs. I'm curious what the intent of that variable is.

Before:

            ret = xr.apply_ufunc(
                sosfiltfilt,
                sos,
                darray,
                input_core_dims=[[], [dim]],
                output_core_dims=[[dim]],
                kwargs=apply_kwargs,
            )

After:

            ret = xr.apply_ufunc(
                sosfiltfilt,
                sos,
                darray,
                input_core_dims=[[], [dim]],
                output_core_dims=[[dim]],
                **apply_kwargs,
            )

Hello @hippalectryon-0, I was wondering if it would be possible to relax the python requirements, preferably matching the one in xarray which is:

requires-python = ">=3.9"

I appreciate you took over the maintenance burden @hippalectryon-0, I really do.
Is there a good reason why you moved the signal wrappers into the "other" subpackage? They do follow the scipy naming, so putting them under other is rather confusing for users, especially to those who used the older xrscipy.signal module and now wonder where it went.

from numpy

[ ] fft
[ ] histogram

from scipy

[ ] interpolation
[ ] fftpack

xrscipy used to work just fine with older xarray versions, so unless you really need xrscipy>=2023, please don't introduce such a strict dependency which prevents simple xrscipy installation in older environments (in scientific environments and clusters one does not always have a choice).

I was using xr-scipy recently and I got the following error:

/home/filipito/.local/lib/python3.8/site-packages/xarray_dsp/filters.py:379: AccessorRegistrationWarning: registration of accessor <class 'xarray_dsp.filters.FilterAccessor'> under name 'filt' for type <class 'xarray.core.dataarray.DataArray'> is overriding a preexisting attribute with the same name.
  class FilterAccessor(object):
Traceback (most recent call last):
  File "/home/filipito/skola/diplomka/data/processing_main.py", line 105, in <module>
    analyze(39796)
  File "/home/filipito/skola/diplomka/data/processing_main.py", line 82, in analyze
    spectrogram = dsp.spectrogram(sel_plasma, nperseg=512, dim='time')
  File "/home/filipito/anaconda3/envs/tjii_data/lib/python3.8/site-packages/xrscipy/signal/spectral.py", line 322, in spectrogram
    Pxx = crossspectrogram(darray, darray, fs, seglen, overlap_ratio, window,
  File "/home/filipito/anaconda3/envs/tjii_data/lib/python3.8/site-packages/xrscipy/signal/spectral.py", line 150, in crossspectrogram
    f, t, Pxy = scipy.signal.spectral._spectral_helper(d_val,
  File "/home/filipito/.local/lib/python3.8/site-packages/scipy/signal/spectral.py", line 24, in __getattr__
    raise AttributeError(
AttributeError: scipy.signal.spectral is deprecated and has no attribute _spectral_helper. Try looking in scipy.signal instead. 

It seems to be introduced with scipy version 1.8.0 where the scipy.signal.spectral.py became deprecated but the _spectral_helper was moved to scipy.signal._spectral_py.py (effectively it was just a rename of spectral.py to _spectral_py.py).
I have found two ways to fix this error, I do not know which one of them is preferable so I have not created a pull request.

1. change the setup to require scipy version <= 1.7.3
2. or change line 149 in the referenced file to this:
   f, t, Pxy = scipy.signal._spectral_py._spectral_helper(d_val,

https://github.com/fujiisoup/xr-scipy/blob/ee7e8fb438c3c2e0bdd73885f39df862d1b9f114/xrscipy/signal/spectral.py#L149-L154

From #4 :

@fujiisoup, would you be up for transferring the project to the xarray-contrib organization? I am thinking that It would help increase the overall visibility of the project and in the process, some folks may chip in to help with the development/maintenance of the project moving forward if need be.

Continuation of  #4

Projects to consider:

• xarray-dsp

There are also a few wrappers for wavelets from pycwt, but I guess those wouldn't really fit into xr-scipy since pycwt is not included in scipy. It's also a question whether you want to restrict this project to scipy functions.

I also found kaipak/xrsigproc which also uses some smoothing functions from scipy.ndimage and scipy.convolve for (probably) spatial filtering.
There is also serazing/xscale which also does some signal processing and FFT using mostly scipy again.
Maybe we could contact these projects as well.

Also, I know crusaderky/xarray_extras.
There must be much more small packages that have been developed individually.
For a limited list, see FAQ page of xarray's document page

Hi!
I have created a library that has wrappers for scipy.stats (among other things): https://xarray-einstats.readthedocs.io/en/latest/.
It currently has:
wrappers for many numpy.linalg functions
wrappers for scipy.stats distributions plus a few functions
wrappers for einops functions
a numba.guvectorize-decorated version of numpy.histogram for dataarrays
I have mostly added wrappers for things I personally use, also trying to not overlap with your library nor xarray-extras, so it might not make much sense as a group, but the modules are completely independent between them and could be reorganized into independent packages or merged into existing ones. I don't know what your roadmap is nor if there is interest within the xarray for scipy wrappers or similar things, but it could be interesting to collaborate.
In my case, I might add some very thin wrappers on a few special functions that I use often (i.e. expit, logsumexp), and some more guvectorized things, but the things above are mostly what I'll use and what I am willing to maintain.
Also, feedback very welcome!

Related:

• xrft

Hi there,

I wanted to find out how to install this package? Do I just have to do a Git pull? Or can I use conda or the like?

Best,
Avik

In version 1.6.0, scipy renamed some methods in scipy.integrate. It has kept the old methods for backward compatibility, but changed their docstrings. This is a problem for xr-scipy, because it assumes that the docstrings contain parameter y https://github.com/fujiisoup/xr-scipy/blob/8ff001ded269c37a7edafd09535c49c04670a48f/xrscipy/integrate.py#L47 which is not true for the deprecated methods.

As a result, import of xr-scipy with scipy >= 1.6.0 fails with KeyError: 'y'.

I can send a PR that fixes this issue and adds the new method names into xr-scipy, if you are interested.

Xarray.apply_ufunc now disallow to change the dimension size of input_core_dims.
xrscipy.fft uses this. needs update.

Also, it might be reasonable to merge fft and fftn function.

@fujiisoup I've finally managed to fix most utests, docs and modules (I think ^^). I've put a temporary new readthedocs here, but I think it's better for everyone if we keep the same page for the docs and the pipy.

Could you also transfer the ownership / share the rights of the readthedocs project and the pipy project ?

Thanks !

Just instnalled the xrscipy using the toml file, and it can be imported successfully. But seems I cannot call any xrscipy functions.

When using xrscipy is using default calling of apply_ufunc with dask="forbidden". This disallows using the package with dask arrays what is needed for example when using preprocess option in open_mfdataset.

Not sure whether this is a design decision; however, would it be possible to pass some kwargs for xrscipy internals?

On the other hand, adding .load() in the preprocess function resolves this issue.

Any opinion @smartass101, @fujiisoup ?

Thank you,
Lukas

Example

import xrscipy
import numpy as np


# Create example datasets: 
Nt = 256; Nx = 10
xs = np.arange(Nx)
ds_list = []

for x in xs: 
    ds = xr.Dataset({"time_data": ("time", np.random.rand((Nt)))},
                   coords={
                       "time": np.arange(Nt), 
                       "x": x
                   })
    ds_list.append(ds)

# Seve as mf_dataset:

xr.save_mfdataset(datasets=ds_list,
                  paths=[f"/tmp/ds_{x}.nc" for x in xs])


# Processing function (this is important): 

func = lambda ds: xrscipy.fft.fft(ds.time_data, "time")

# This resolves the issue
# func = lambda ds: xrscipy.fft.fft(ds.time_data.load(), "time")

ds_mf = xr.open_mfdataset([f"/tmp/ds_{x}.nc" for x in xs],
                           concat_dim="x",
                           parallel=True,
                           combine="nested",
                           preprocess=func)

Error

<ipython-input-35-1f7125289f88> in <lambda>(ds)
     24 # Processing function:
     25 
---> 26 func = lambda ds: xrscipy.fft.fft(ds.time_data, "time")
     27 
     28 ds_mf = xr.open_mfdataset([f"/tmp/ds_{x}.nc" for x in xs],

/sw/python/3.8-anaconda-2020.07/lib/python3.8/site-packages/xrscipy/fft.py in _wrap1d(func, freq_func, y, coord, **kwargs)
     44         return result.set_dims(v.dims)
     45 
---> 46     ds = utils.wrap_dataset(apply_func, y, dim, keep_coords='drop')
     47 
     48     # attach frequency coordinate

/sw/python/3.8-anaconda-2020.07/lib/python3.8/site-packages/xrscipy/utils.py in wrap_dataset(func, y, *dims, **kwargs)
     26 
     27     if isinstance(y, xr.DataArray):
---> 28         result = wrap_dataset(func, y._to_temp_dataset(), *dims,
     29                               keep_coords=keep_coords)
     30         # Drop unnecessary coordinate.

/sw/python/3.8-anaconda-2020.07/lib/python3.8/site-packages/xrscipy/utils.py in wrap_dataset(func, y, *dims, **kwargs)
     37         for key in y.data_vars:
     38             if any(d in y[key].dims for d in dims):
---> 39                 ds[key] = func(y[key].variable)
     40             else:
     41                 ds[key] = y[key]

/sw/python/3.8-anaconda-2020.07/lib/python3.8/site-packages/xrscipy/fft.py in apply_func(v)
     39     def apply_func(v):
     40         # v: xr.Varaible
---> 41         result = xr.apply_ufunc(
     42             func, v, input_core_dims=[[dim]],
     43             output_core_dims=[output_core_dim], kwargs=kwargs)

/sw/python/3.8-anaconda-2020.07/lib/python3.8/site-packages/xarray/core/computation.py in apply_ufunc(func, input_core_dims, output_core_dims, exclude_dims, vectorize, join, dataset_join, dataset_fill_value, keep_attrs, kwargs, dask, output_dtypes, output_sizes, meta, dask_gufunc_kwargs, *args)
   1179     # feed Variables directly through apply_variable_ufunc
   1180     elif any(isinstance(a, Variable) for a in args):
-> 1181         return variables_vfunc(*args)
   1182     else:
   1183         # feed anything else through apply_array_ufunc

/sw/python/3.8-anaconda-2020.07/lib/python3.8/site-packages/xarray/core/computation.py in apply_variable_ufunc(func, signature, exclude_dims, dask, output_dtypes, vectorize, keep_attrs, dask_gufunc_kwargs, *args)
    650     if any(is_duck_dask_array(array) for array in input_data):
    651         if dask == "forbidden":
--> 652             raise ValueError(
    653                 "apply_ufunc encountered a dask array on an "
    654                 "argument, but handling for dask arrays has not "

ValueError: apply_ufunc encountered a dask array on an argument, but handling for dask arrays has not been enabled. Either set the ``dask`` argument or load your data into memory first with ``.load()`` or ``.compute()``