Deion of the problem Following up on question

Here are the results of some tests, for lmax=5-130 Fedora Asah

Thanks for this, I can add the following: macOS (M2 Max, pytho

issue when calling pyshtools.spectralanalysis.SHReturnTapersMap about shtools HOT 6 OPEN

siwill22 commented on June 24, 2024

issue when calling pyshtools.spectralanalysis.SHReturnTapersMap

from shtools.

Comments (6)

siwill22 commented on June 24, 2024

In case it helps, I can confirm that I still get the error when running within a 'clean' conda environment created as follows:

conda create --name shtest -c conda-forge pyshtools

which installs as follows:

Channels:

conda-forge
defaults
Platform: osx-arm64
Collecting package metadata (repodata.json): done
Solving environment: done

Package Plan

environment location: /Users/simon/anaconda3/envs/shtest

added / updated specs:
- pyshtools

The following packages will be downloaded:

package                    |            build
---------------------------|-----------------
astropy-6.0.0              |  py311h9ea6feb_0         8.7 MB  conda-forge
astropy-iers-data-0.2024.2.5.0.30.52|     pyhd8ed1ab_0         1.2 MB  conda-forge
azure-core-cpp-1.10.3      |       he231e37_1         282 KB  conda-forge
azure-storage-blobs-cpp-12.10.0|       h6aa02a4_0         401 KB  conda-forge
azure-storage-common-cpp-12.5.0|       h607ffeb_2         104 KB  conda-forge
brotli-python-1.1.0        |  py311ha891d26_1         335 KB  conda-forge
bzip2-1.0.8                |       h93a5062_5         119 KB  conda-forge
c-ares-1.26.0              |       h93a5062_0         142 KB  conda-forge
cartopy-0.22.0             |  py311h6e08293_1         1.8 MB  conda-forge
certifi-2024.2.2           |     pyhd8ed1ab_0         157 KB  conda-forge
cfitsio-4.3.1              |       h808cd33_0         743 KB  conda-forge
cftime-1.6.3               |  py311h9ea6feb_0         201 KB  conda-forge
contourpy-1.2.0            |  py311hd03642b_0         235 KB  conda-forge
curl-8.5.0                 |       h2d989ff_0         148 KB  conda-forge
ducc0-0.33.0               |  py311h92babd0_0         1.4 MB  conda-forge
font-ttf-ubuntu-0.83       |       h77eed37_1         1.5 MB  conda-forge
fonttools-4.48.1           |  py311h05b510d_0         2.6 MB  conda-forge
gdal-3.8.3                 |  py311h43f0207_2         1.6 MB  conda-forge
geos-3.12.1                |       h965bd2d_0         1.3 MB  conda-forge
geotiff-1.7.1              |      h7bcba05_15         114 KB  conda-forge
ghostscript-10.02.1        |       h965bd2d_0        56.4 MB  conda-forge
gmt-6.5.0                  |       haf49670_0        35.2 MB  conda-forge
hdf5-1.14.3                |nompi_h5bb55e9_100         3.3 MB  conda-forge
idna-3.6                   |     pyhd8ed1ab_0          49 KB  conda-forge
importlib-metadata-7.0.1   |     pyha770c72_0          26 KB  conda-forge
kealib-1.5.3               |       h210d843_0         136 KB  conda-forge
kiwisolver-1.4.5           |  py311he4fd1f5_1          60 KB  conda-forge
lcms2-2.16                 |       ha0e7c42_0         207 KB  conda-forge
libarchive-3.7.2           |       hcacb583_1         765 KB  conda-forge
libblas-3.9.0              |21_osxarm64_openblas          15 KB  conda-forge
libboost-headers-1.84.0    |       hce30654_0        13.2 MB  conda-forge
libcblas-3.9.0             |21_osxarm64_openblas          14 KB  conda-forge
libcurl-8.5.0              |       h2d989ff_0         342 KB  conda-forge
libev-4.33                 |       h93a5062_2         105 KB  conda-forge
libgdal-3.8.3              |       hb4bbca3_2         8.1 MB  conda-forge
libgfortran-5.0.0          |13_2_0_hd922786_3         108 KB  conda-forge
libgfortran5-13.2.0        |       hf226fd6_3         974 KB  conda-forge
libglib-2.78.3             |       hb438215_0         2.3 MB  conda-forge
libgoogle-cloud-2.12.0     |       h49bbb43_5        29.9 MB  conda-forge
libgrpc-1.60.0             |       hfc68871_1         4.0 MB  conda-forge
libiconv-1.17              |       h0d3ecfb_2         661 KB  conda-forge
liblapack-3.9.0            |21_osxarm64_openblas          14 KB  conda-forge
libnetcdf-4.9.2            |nompi_h291a7c2_113         666 KB  conda-forge
libnghttp2-1.58.0          |       ha4dd798_1         552 KB  conda-forge
libopenblas-0.3.26         |openmp_h6c19121_0         2.8 MB  conda-forge
libpng-1.6.42              |       h091b4b1_0         258 KB  conda-forge
libpq-16.1                 |       h0f8b458_7         2.3 MB  conda-forge
libprotobuf-4.25.1         |       h810fc01_1         2.1 MB  conda-forge
librttopo-1.1.0            |      hc8f776e_15         188 KB  conda-forge
libspatialite-5.1.0        |       h69abc6b_4         3.9 MB  conda-forge
libsqlite-3.44.2           |       h091b4b1_0         796 KB  conda-forge
libxml2-2.12.5             |       h0d0cfa8_0         574 KB  conda-forge
llvm-openmp-17.0.6         |       hcd81f8e_0         268 KB  conda-forge
matplotlib-base-3.8.2      |  py311hfdba5f6_0         7.5 MB  conda-forge
minizip-4.0.4              |       hc35e051_0          77 KB  conda-forge
netcdf4-1.6.5              |nompi_py311ha6bebe6_100         449 KB  conda-forge
nss-3.97                   |       h5ce2875_0         1.7 MB  conda-forge
numpy-1.26.4               |  py311h7125741_0         6.3 MB  conda-forge
pandas-2.2.0               |  py311hfbe21a1_0        14.1 MB  conda-forge
pcre2-10.42                |       h26f9a81_0         605 KB  conda-forge
pillow-10.2.0              |  py311hb9c5795_0        39.7 MB  conda-forge
pip-24.0                   |     pyhd8ed1ab_0         1.3 MB  conda-forge
pixman-0.43.2              |       hebf3989_0         194 KB  conda-forge
platformdirs-4.2.0         |     pyhd8ed1ab_0          20 KB  conda-forge
poppler-24.02.0            |       h896e6cb_0         1.4 MB  conda-forge
postgresql-16.1            |       hc6ab77f_7         4.2 MB  conda-forge
proj-9.3.1                 |       h93d94ba_0         2.5 MB  conda-forge
pyerfa-2.0.1.1             |  py311h9ea6feb_0         344 KB  conda-forge
pygmt-0.11.0               |     pyhd8ed1ab_0         148 KB  conda-forge
pyproj-3.6.1               |  py311h9a031f7_5         482 KB  conda-forge
pyshtools-4.10.4           |  py311h667c9c3_0         1.0 MB  conda-forge
python-3.11.7              |hdf0ec26_1_cpython        13.9 MB  conda-forge
python-tzdata-2023.4       |     pyhd8ed1ab_0         143 KB  conda-forge
python_abi-3.11            |          4_cp311           6 KB  conda-forge
pytz-2024.1                |     pyhd8ed1ab_0         184 KB  conda-forge
pyyaml-6.0.1               |  py311heffc1b2_1         183 KB  conda-forge
scipy-1.12.0               |  py311h4f9446f_2        14.8 MB  conda-forge
setuptools-69.0.3          |     pyhd8ed1ab_0         460 KB  conda-forge
shapely-2.0.2              |  py311h0815064_1         528 KB  conda-forge
sqlite-3.44.2              |       hf2abe2d_0         784 KB  conda-forge
tiledb-2.19.1              |       h49d9ff7_0         4.5 MB  conda-forge
tk-8.6.13                  |       h5083fa2_1         3.0 MB  conda-forge
tzcode-2024a               |       h93a5062_0          62 KB  conda-forge
tzdata-2024a               |       h0c530f3_0         117 KB  conda-forge
urllib3-2.2.0              |     pyhd8ed1ab_0          92 KB  conda-forge
wheel-0.42.0               |     pyhd8ed1ab_0          56 KB  conda-forge
xarray-2024.1.1            |     pyhd8ed1ab_0         714 KB  conda-forge
xerces-c-3.2.5             |       hf393695_0         1.2 MB  conda-forge
------------------------------------------------------------
                                       Total:       315.9 MB

from shtools.

MarkWieczorek commented on June 24, 2024

Could you let me know if this error occurs every time for the same value of lmax on a given machine? Or if this only occurs sometimes?

from shtools.

MarkWieczorek commented on June 24, 2024

It turns out that I get the same error using lmax=110 on an Apple M1 max. The error occurred three successive times, so it is reproducible for me. I will try to find the lowest lmax that this crashes on my Mac, and then try to reproduce this on linux. If you could do the same, that would be great.

from shtools.

siwill22 commented on June 24, 2024

I ran 5 iterations on Apple M2 Max.
The function is successful consistently for lmax=69, and returns the error consistently for lmax=70

from shtools.

MarkWieczorek commented on June 24, 2024

Here are the results of some tests, for lmax=5-130

Fedora Asahi M1 (Python 3.11 and 3.12, pyshtools installed from source) : fails at LMAX=114 with DSTEGR info = 22
macOS (M1 Max, python 3.11, pyshtools installed from source) : fails at LMAX = 103 with DSTEGR info = 22
macOS (M1 Max, python 3.12, pyshtools installed from source) : fails at LMAX = 121 with DSTEGR info = 22
macOS (intel, python 3.11, pyshtools installed with conda) : fails for LMAX>=70 with DSTEGR info = 15
macOS (M1 max, python 3.11, pyshtools installed with conda) : fails for LMAX>=71

So, the error occurs on all machines, but at slightly different values of lmax. Even on the same machine, the failure can occur at different values when using different versions of python. The value at which it fails is the worst when installing pyshtools by conda (python 3.11).

I am not really sure what this means. The good thing is that it is repeatable on a given machine, so it probably isn't a memory related problem. I suspect that the problem might be related to slightly different ways of compiling LAPACK/BLAS that cause the computations to be performed slightly differently (such as a different optimization, or some option like fast-math). To be honest, I am a little surprised that determining the eigenvectors of a 1000x1000 matrix works at all! For such large matrices, it is possible that there is a better way of doing this than the way I chose. If this is important to you, then perhaps we should look into it.

from shtools.

siwill22 commented on June 24, 2024

Thanks for this, I can add the following:

macOS (M2 Max, python 3.12, pyshtools installed from source): fails at LMAX = 115 with DSTEGR info = 22

One other thing I noticed - the execution time appears to be significantly different between cases. The above config takes close to double the time of the config with python3.11 / pyshtools from conda.

For now, it is helpful to know that the issue is reproducible and that I can get better results with the different python versions and installation from source. (it was a little frustrating to upgrade from an intel Mac only to find the problem appeared worse based on the setup I was using initially on M2)

As for importance - my specific use case involves Earth's lithospheric magnetic field. Since LCS-1 is defined to degree 185, it would be nice to be able to do localisations to this degree. However, for the masks that I want to use (different from the test case) I am now able to get results for LMAX > 140, a lot higher than with the conda install. (In one case LMAX = 170 works, the failure point is somewhere between 170 and 185 - impressive indeed). Therefore, I am happy to work around this for now, and don't have a sense of how important this would be for other users. If you do look into this at some point I'd be happy to help run further tests.

from shtools.

issue when calling pyshtools.spectralanalysis.SHReturnTapersMap about shtools HOT 6 OPEN

Comments (6)

Package Plan

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent