astropy / astroscrappy Goto Github PK

View Code? Open in Web Editor NEW

67.0 9.0 34.0 762 KB

Speedy Cosmic Ray Annihilation Package in Python

Python 14.92% C 48.92% Cython 36.16%

astroscrappy's Introduction

Astro-SCRAPPY: The Speedy Cosmic Ray Annihilation Package in Python

An optimized cosmic ray detector.

Author:	Curtis McCully

Astro-SCRAPPY is designed to detect cosmic rays in images (numpy arrays), based on Pieter van Dokkum's L.A.Cosmic algorithm.

Much of this was originally adapted from cosmics.py written by Malte Tewes. I have ported all of the slow functions to Cython/C, and optimized where I can. This is designed to be as fast as possible so some of the readability has been sacrificed, specifically in the C code.

If you use this code, please cite the Zendo DOI: https://zenodo.org/record/1482019

Please cite the original paper which can be found at: http://www.astro.yale.edu/dokkum/lacosmic/

van Dokkum 2001, PASP, 113, 789, 1420 (article : http://adsabs.harvard.edu/abs/2001PASP..113.1420V)

This code requires Cython, preferably version >= 0.21.

Parallelization is achieved using OpenMP. This code should compile (although the Cython files may have issues) using a compiler that does not support OMP, e.g. clang.

Notes

There are some differences from original LA Cosmic:

Automatic recognition of saturated stars. This avoids treating such stars as large cosmic rays.
I have tried to optimize all of the code as much as possible while maintaining the integrity of the algorithm. One of the key speedups is to use a separable median filter instead of the true median filter. While these are not identical, they produce comparable results and the separable version is much faster.
This implementation is much faster than the Python by as much as a factor of ~17 depending on the given parameters, even without running multiple threads. With multiple threads, this can be increased easily by another factor of 2. This implementation is much faster than the original IRAF version, improvment by a factor of ~90.

The arrays always must be C-contiguous, thus all loops are y outer, x inner. This follows the astropy.io.fits (pyfits) convention.

scipy is required for certain tests to pass, but the code itself does not depend on scipy.

License

This project is Copyright (c) Astropy Developers and licensed under the terms of the BSD 3-Clause license. See the licenses folder for more information.

astroscrappy's People

Contributors

Stargazers

Watchers

astroscrappy's Issues

python setup.py egg_info should not install astropy

I noticed on Travis that the python setup.py egg_info builds are installing astropy, which should not happen:

...
Searching for astropy
Reading https://pypi.python.org/simple/astropy/
Reading http://astropy.org
Best match: astropy 1.0.4
Downloading https://pypi.python.org/packages/source/a/astropy/astropy-1.0.4.tar.gz#md5=6d4696a695129fb3566ddc9643a2a944

No dependencies should be installed when running egg_info.

Failed building wheel for astroscrappy

Can anyone help me ?

`C:\Windows\system32>pip3 install --user --upgrade pip setuptools
Requirement already satisfied: pip in c:\users\user\appdata\local\programs\python\python39\lib\site-packages (21.1.3)
Requirement already satisfied: setuptools in c:\users\user\appdata\local\programs\python\python39\lib\site-packages (57.1.0)

C:\Windows\system32>pip3 install wheel
Collecting wheel
Using cached wheel-0.36.2-py2.py3-none-any.whl (35 kB)
Installing collected packages: wheel
Successfully installed wheel-0.36.2

C:\Windows\system32>pip install astroscrappy
Collecting astroscrappy
Using cached astroscrappy-1.0.8.tar.gz (449 kB)
Collecting astropy
Downloading astropy-4.2.1-cp39-cp39-win_amd64.whl (6.2 MB)
|████████████████████████████████| 6.2 MB 1.7 MB/s
Requirement already satisfied: numpy in c:\users\user\appdata\local\programs\python\python39\lib\site-packages (from astroscrappy) (1.20.3)
Collecting pyerfa
Downloading pyerfa-2.0.0-cp39-cp39-win_amd64.whl (366 kB)
|████████████████████████████████| 366 kB 544 kB/s
Building wheels for collected packages: astroscrappy
Building wheel for astroscrappy (setup.py) ... error
ERROR: Command errored out with exit status 1:
command: 'c:\users\user\appdata\local\programs\python\python39\python.exe' -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\Users\USER\AppData\Local\Temp\pip-install-smf3c_lx\astroscrappy_f576f84cc7f14594918d4ed7b70dc496\setup.py'"'"'; file='"'"'C:\Users\USER\AppData\Local\Temp\pip-install-smf3c_lx\astroscrappy_f576f84cc7f14594918d4ed7b70dc496\setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(file) if os.path.exists(file) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' bdist_wheel -d 'C:\Users\USER\AppData\Local\Temp\pip-wheel-q20hxdn8'
cwd: C:\Users\USER\AppData\Local\Temp\pip-install-smf3c_lx\astroscrappy_f576f84cc7f14594918d4ed7b70dc496
Complete output (31 lines):
running bdist_wheel
running build
running build_py
creating build
creating build\lib.win-amd64-3.9
creating build\lib.win-amd64-3.9\astroscrappy
copying astroscrappy\conftest.py -> build\lib.win-amd64-3.9\astroscrappy
copying astroscrappy\cython_version.py -> build\lib.win-amd64-3.9\astroscrappy
copying astroscrappy\version.py -> build\lib.win-amd64-3.9\astroscrappy
copying astroscrappy_astropy_init.py -> build\lib.win-amd64-3.9\astroscrappy
copying astroscrappy_init_.py -> build\lib.win-amd64-3.9\astroscrappy
creating build\lib.win-amd64-3.9\astroscrappy\tests
copying astroscrappy\tests\fake_data.py -> build\lib.win-amd64-3.9\astroscrappy\tests
copying astroscrappy\tests\setup_package.py -> build\lib.win-amd64-3.9\astroscrappy\tests
copying astroscrappy\tests\test_astroscrappy.py -> build\lib.win-amd64-3.9\astroscrappy\tests
copying astroscrappy\tests\test_cleaning.py -> build\lib.win-amd64-3.9\astroscrappy\tests
copying astroscrappy\tests\test_utils.py -> build\lib.win-amd64-3.9\astroscrappy\tests
copying astroscrappy\tests_init_.py -> build\lib.win-amd64-3.9\astroscrappy\tests
creating build\lib.win-amd64-3.9\astroscrappy\utils
copying astroscrappy\utils\setup_package.py -> build\lib.win-amd64-3.9\astroscrappy\utils
copying astroscrappy\utils_init_.py -> build\lib.win-amd64-3.9\astroscrappy\utils
copying astroscrappy\astroscrappy.c -> build\lib.win-amd64-3.9\astroscrappy
copying astroscrappy_compiler.c -> build\lib.win-amd64-3.9\astroscrappy
copying astroscrappy\utils\image_utils.c -> build\lib.win-amd64-3.9\astroscrappy\utils
copying astroscrappy\utils\imutils.c -> build\lib.win-amd64-3.9\astroscrappy\utils
copying astroscrappy\utils\median_utils.c -> build\lib.win-amd64-3.9\astroscrappy\utils
copying astroscrappy\utils\medutils.c -> build\lib.win-amd64-3.9\astroscrappy\utils
copying astroscrappy\tests\coveragerc -> build\lib.win-amd64-3.9\astroscrappy\tests
running build_ext
building 'astroscrappy._compiler' extension
error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/

ERROR: Failed building wheel for astroscrappy
Running setup.py clean for astroscrappy
Failed to build astroscrappy
Installing collected packages: pyerfa, astropy, astroscrappy
Running setup.py install for astroscrappy ... error
ERROR: Command errored out with exit status 1:
command: 'c:\users\user\appdata\local\programs\python\python39\python.exe' -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\Users\USER\AppData\Local\Temp\pip-install-smf3c_lx\astroscrappy_f576f84cc7f14594918d4ed7b70dc496\setup.py'"'"'; file='"'"'C:\Users\USER\AppData\Local\Temp\pip-install-smf3c_lx\astroscrappy_f576f84cc7f14594918d4ed7b70dc496\setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(file) if os.path.exists(file) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record 'C:\Users\USER\AppData\Local\Temp\pip-record-8m6qj_69\install-record.txt' --single-version-externally-managed --compile --install-headers 'c:\users\user\appdata\local\programs\python\python39\Include\astroscrappy'
cwd: C:\Users\USER\AppData\Local\Temp\pip-install-smf3c_lx\astroscrappy_f576f84cc7f14594918d4ed7b70dc496
Complete output (31 lines):
running install
running build
running build_py
creating build
creating build\lib.win-amd64-3.9
creating build\lib.win-amd64-3.9\astroscrappy
copying astroscrappy\conftest.py -> build\lib.win-amd64-3.9\astroscrappy
copying astroscrappy\cython_version.py -> build\lib.win-amd64-3.9\astroscrappy
copying astroscrappy\version.py -> build\lib.win-amd64-3.9\astroscrappy
copying astroscrappy_astropy_init.py -> build\lib.win-amd64-3.9\astroscrappy
copying astroscrappy_init_.py -> build\lib.win-amd64-3.9\astroscrappy
creating build\lib.win-amd64-3.9\astroscrappy\tests
copying astroscrappy\tests\fake_data.py -> build\lib.win-amd64-3.9\astroscrappy\tests
copying astroscrappy\tests\setup_package.py -> build\lib.win-amd64-3.9\astroscrappy\tests
copying astroscrappy\tests\test_astroscrappy.py -> build\lib.win-amd64-3.9\astroscrappy\tests
copying astroscrappy\tests\test_cleaning.py -> build\lib.win-amd64-3.9\astroscrappy\tests
copying astroscrappy\tests\test_utils.py -> build\lib.win-amd64-3.9\astroscrappy\tests
copying astroscrappy\tests_init_.py -> build\lib.win-amd64-3.9\astroscrappy\tests
creating build\lib.win-amd64-3.9\astroscrappy\utils
copying astroscrappy\utils\setup_package.py -> build\lib.win-amd64-3.9\astroscrappy\utils
copying astroscrappy\utils_init_.py -> build\lib.win-amd64-3.9\astroscrappy\utils
copying astroscrappy\astroscrappy.c -> build\lib.win-amd64-3.9\astroscrappy
copying astroscrappy_compiler.c -> build\lib.win-amd64-3.9\astroscrappy
copying astroscrappy\utils\image_utils.c -> build\lib.win-amd64-3.9\astroscrappy\utils
copying astroscrappy\utils\imutils.c -> build\lib.win-amd64-3.9\astroscrappy\utils
copying astroscrappy\utils\median_utils.c -> build\lib.win-amd64-3.9\astroscrappy\utils
copying astroscrappy\utils\medutils.c -> build\lib.win-amd64-3.9\astroscrappy\utils
copying astroscrappy\tests\coveragerc -> build\lib.win-amd64-3.9\astroscrappy\tests
running build_ext
building 'astroscrappy._compiler' extension
error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/visual-cpp-build-tools/
----------------------------------------
ERROR: Command errored out with exit status 1: 'c:\users\user\appdata\local\programs\python\python39\python.exe' -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\Users\USER\AppData\Local\Temp\pip-install-smf3c_lx\astroscrappy_f576f84cc7f14594918d4ed7b70dc496\setup.py'"'"'; file='"'"'C:\Users\USER\AppData\Local\Temp\pip-install-smf3c_lx\astroscrappy_f576f84cc7f14594918d4ed7b70dc496\setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(file) if os.path.exists(file) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record 'C:\Users\USER\AppData\Local\Temp\pip-record-8m6qj_69\install-record.txt' --single-version-externally-managed --compile --install-headers 'c:\users\user\appdata\local\programs\python\python39\Include\astroscrappy' Check the logs for full command output.`

Image gain restoration

I remember previously astroscrappy.detect_cosmics returned the cleaned array multiplied by gain factor, but now it does not. (I could not find CHANGELOG.. )

astroscrappy.detect_cosmics(np.ones((100,100)), gain=3)[1]

array([[1., 1., 1., ..., 1., 1., 1.],
       [1., 1., 1., ..., 1., 1., 1.],
       [1., 1., 1., ..., 1., 1., 1.],
       ...,
       [1., 1., 1., ..., 1., 1., 1.],
       [1., 1., 1., ..., 1., 1., 1.],
       [1., 1., 1., ..., 1., 1., 1.]], dtype=float32)

It used to be

array([[3., 3., 3., ..., 3., 3., 3.],
       [3., 3., 3., ..., 3., 3., 3.],
       [3., 3., 3., ..., 3., 3., 3.],
       ...,
       [3., 3., 3., ..., 3., 3., 3.],
       [3., 3., 3., ..., 3., 3., 3.],
       [3., 3., 3., ..., 3., 3., 3.]], dtype=float32)

if my memory is correct, and this is why I used my custom function to do nccd.data = cleanarr / gain.

Now I find my processed images have extremely small counts & SNR (as I have gain ~ 10).

How can I write a code which incorporates both versions of astroscrappy within a single function?
(e.g., for inbkg, bkg, pssl stuff, I could use try-except clause)

Unable to install astroscrappy with pip-install

Hi everyone!
I'm failing to install astroscrappy using pip3 (Python 3.9) on a Mac with MacOSX Big Sur (11.1) and Catalina (10.15.7). Hereafter the error trace:

ciuser@CiRiduz GUI % pip3 install astroscrappy
Collecting astroscrappy
  Using cached astroscrappy-1.0.8.tar.gz (449 kB)
Requirement already satisfied: astropy in /Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages (from astroscrappy) (4.2)
Requirement already satisfied: numpy in /Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages (from astroscrappy) (1.19.4)
Requirement already satisfied: pyerfa in /Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages (from astropy->astroscrappy) (1.7.1.1)
Building wheels for collected packages: astroscrappy
  Building wheel for astroscrappy (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /Library/Frameworks/Python.framework/Versions/3.9/bin/python3.9 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/h2/5mydv0ns1_d4b0c_hkhw5bsw0000gn/T/pip-install-yn2_h3lf/astroscrappy_10c05a1c9b454c2c91100ae523b25b31/setup.py'"'"'; __file__='"'"'/private/var/folders/h2/5mydv0ns1_d4b0c_hkhw5bsw0000gn/T/pip-install-yn2_h3lf/astroscrappy_10c05a1c9b454c2c91100ae523b25b31/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /private/var/folders/h2/5mydv0ns1_d4b0c_hkhw5bsw0000gn/T/pip-wheel-d8p3sv87
       cwd: /private/var/folders/h2/5mydv0ns1_d4b0c_hkhw5bsw0000gn/T/pip-install-yn2_h3lf/astroscrappy_10c05a1c9b454c2c91100ae523b25b31/
  Complete output (38 lines):
  OpenMP was not found. astroscrappy will be compiled without OpenMP. (Use the "-v" option of setup.py for more details.)
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.macosx-10.9-universal2-3.9
  creating build/lib.macosx-10.9-universal2-3.9/astroscrappy
  copying astroscrappy/conftest.py -> build/lib.macosx-10.9-universal2-3.9/astroscrappy
  copying astroscrappy/version.py -> build/lib.macosx-10.9-universal2-3.9/astroscrappy
  copying astroscrappy/_astropy_init.py -> build/lib.macosx-10.9-universal2-3.9/astroscrappy
  copying astroscrappy/__init__.py -> build/lib.macosx-10.9-universal2-3.9/astroscrappy
  copying astroscrappy/cython_version.py -> build/lib.macosx-10.9-universal2-3.9/astroscrappy
  creating build/lib.macosx-10.9-universal2-3.9/astroscrappy/tests
  copying astroscrappy/tests/test_utils.py -> build/lib.macosx-10.9-universal2-3.9/astroscrappy/tests
  copying astroscrappy/tests/test_astroscrappy.py -> build/lib.macosx-10.9-universal2-3.9/astroscrappy/tests
  copying astroscrappy/tests/fake_data.py -> build/lib.macosx-10.9-universal2-3.9/astroscrappy/tests
  copying astroscrappy/tests/__init__.py -> build/lib.macosx-10.9-universal2-3.9/astroscrappy/tests
  copying astroscrappy/tests/test_cleaning.py -> build/lib.macosx-10.9-universal2-3.9/astroscrappy/tests
  copying astroscrappy/tests/setup_package.py -> build/lib.macosx-10.9-universal2-3.9/astroscrappy/tests
  creating build/lib.macosx-10.9-universal2-3.9/astroscrappy/utils
  copying astroscrappy/utils/__init__.py -> build/lib.macosx-10.9-universal2-3.9/astroscrappy/utils
  copying astroscrappy/utils/setup_package.py -> build/lib.macosx-10.9-universal2-3.9/astroscrappy/utils
  copying astroscrappy/astroscrappy.c -> build/lib.macosx-10.9-universal2-3.9/astroscrappy
  copying astroscrappy/_compiler.c -> build/lib.macosx-10.9-universal2-3.9/astroscrappy
  copying astroscrappy/utils/image_utils.c -> build/lib.macosx-10.9-universal2-3.9/astroscrappy/utils
  copying astroscrappy/utils/medutils.c -> build/lib.macosx-10.9-universal2-3.9/astroscrappy/utils
  copying astroscrappy/utils/median_utils.c -> build/lib.macosx-10.9-universal2-3.9/astroscrappy/utils
  copying astroscrappy/utils/imutils.c -> build/lib.macosx-10.9-universal2-3.9/astroscrappy/utils
  copying astroscrappy/tests/coveragerc -> build/lib.macosx-10.9-universal2-3.9/astroscrappy/tests
  running build_ext
  building 'astroscrappy._compiler' extension
  creating build/temp.macosx-10.9-universal2-3.9
  creating build/temp.macosx-10.9-universal2-3.9/astroscrappy
  gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch arm64 -arch x86_64 -g -I/usr/local/opt/llvm/include -I/Library/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c astroscrappy/_compiler.c -o build/temp.macosx-10.9-universal2-3.9/astroscrappy/_compiler.o
  gcc -bundle -undefined dynamic_lookup -arch arm64 -arch x86_64 -g -L/usr/local/opt/llvm/lib -I/usr/local/opt/llvm/include build/temp.macosx-10.9-universal2-3.9/astroscrappy/_compiler.o -o build/lib.macosx-10.9-universal2-3.9/astroscrappy/_compiler.cpython-39-darwin.so
  ld: in '/usr/local/opt/llvm/lib/libunwind.dylib', building for macOS-arm64 but attempting to link with file built for macOS-x86_64
  clang: error: linker command failed with exit code 1 (use -v to see invocation)
  error: command '/usr/bin/gcc' failed with exit code 1
  ----------------------------------------
  ERROR: Failed building wheel for astroscrappy
  Running setup.py clean for astroscrappy
Failed to build astroscrappy
Installing collected packages: astroscrappy
    Running setup.py install for astroscrappy ... error
    ERROR: Command errored out with exit status 1:
     command: /Library/Frameworks/Python.framework/Versions/3.9/bin/python3.9 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/h2/5mydv0ns1_d4b0c_hkhw5bsw0000gn/T/pip-install-yn2_h3lf/astroscrappy_10c05a1c9b454c2c91100ae523b25b31/setup.py'"'"'; __file__='"'"'/private/var/folders/h2/5mydv0ns1_d4b0c_hkhw5bsw0000gn/T/pip-install-yn2_h3lf/astroscrappy_10c05a1c9b454c2c91100ae523b25b31/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/h2/5mydv0ns1_d4b0c_hkhw5bsw0000gn/T/pip-record-modb2__s/install-record.txt --single-version-externally-managed --compile --install-headers /Library/Frameworks/Python.framework/Versions/3.9/include/python3.9/astroscrappy
         cwd: /private/var/folders/h2/5mydv0ns1_d4b0c_hkhw5bsw0000gn/T/pip-install-yn2_h3lf/astroscrappy_10c05a1c9b454c2c91100ae523b25b31/
    Complete output (38 lines):
    OpenMP was not found. astroscrappy will be compiled without OpenMP. (Use the "-v" option of setup.py for more details.)
    running install
    running build
    running build_py
    creating build
    creating build/lib.macosx-10.9-universal2-3.9
    creating build/lib.macosx-10.9-universal2-3.9/astroscrappy
    copying astroscrappy/conftest.py -> build/lib.macosx-10.9-universal2-3.9/astroscrappy
    copying astroscrappy/version.py -> build/lib.macosx-10.9-universal2-3.9/astroscrappy
    copying astroscrappy/_astropy_init.py -> build/lib.macosx-10.9-universal2-3.9/astroscrappy
    copying astroscrappy/__init__.py -> build/lib.macosx-10.9-universal2-3.9/astroscrappy
    copying astroscrappy/cython_version.py -> build/lib.macosx-10.9-universal2-3.9/astroscrappy
    creating build/lib.macosx-10.9-universal2-3.9/astroscrappy/tests
    copying astroscrappy/tests/test_utils.py -> build/lib.macosx-10.9-universal2-3.9/astroscrappy/tests
    copying astroscrappy/tests/test_astroscrappy.py -> build/lib.macosx-10.9-universal2-3.9/astroscrappy/tests
    copying astroscrappy/tests/fake_data.py -> build/lib.macosx-10.9-universal2-3.9/astroscrappy/tests
    copying astroscrappy/tests/__init__.py -> build/lib.macosx-10.9-universal2-3.9/astroscrappy/tests
    copying astroscrappy/tests/test_cleaning.py -> build/lib.macosx-10.9-universal2-3.9/astroscrappy/tests
    copying astroscrappy/tests/setup_package.py -> build/lib.macosx-10.9-universal2-3.9/astroscrappy/tests
    creating build/lib.macosx-10.9-universal2-3.9/astroscrappy/utils
    copying astroscrappy/utils/__init__.py -> build/lib.macosx-10.9-universal2-3.9/astroscrappy/utils
    copying astroscrappy/utils/setup_package.py -> build/lib.macosx-10.9-universal2-3.9/astroscrappy/utils
    copying astroscrappy/astroscrappy.c -> build/lib.macosx-10.9-universal2-3.9/astroscrappy
    copying astroscrappy/_compiler.c -> build/lib.macosx-10.9-universal2-3.9/astroscrappy
    copying astroscrappy/utils/image_utils.c -> build/lib.macosx-10.9-universal2-3.9/astroscrappy/utils
    copying astroscrappy/utils/medutils.c -> build/lib.macosx-10.9-universal2-3.9/astroscrappy/utils
    copying astroscrappy/utils/median_utils.c -> build/lib.macosx-10.9-universal2-3.9/astroscrappy/utils
    copying astroscrappy/utils/imutils.c -> build/lib.macosx-10.9-universal2-3.9/astroscrappy/utils
    copying astroscrappy/tests/coveragerc -> build/lib.macosx-10.9-universal2-3.9/astroscrappy/tests
    running build_ext
    building 'astroscrappy._compiler' extension
    creating build/temp.macosx-10.9-universal2-3.9
    creating build/temp.macosx-10.9-universal2-3.9/astroscrappy
    gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -arch arm64 -arch x86_64 -g -I/usr/local/opt/llvm/include -I/Library/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c astroscrappy/_compiler.c -o build/temp.macosx-10.9-universal2-3.9/astroscrappy/_compiler.o
    gcc -bundle -undefined dynamic_lookup -arch arm64 -arch x86_64 -g -L/usr/local/opt/llvm/lib -I/usr/local/opt/llvm/include build/temp.macosx-10.9-universal2-3.9/astroscrappy/_compiler.o -o build/lib.macosx-10.9-universal2-3.9/astroscrappy/_compiler.cpython-39-darwin.so
    ld: in '/usr/local/opt/llvm/lib/libunwind.dylib', building for macOS-arm64 but attempting to link with file built for macOS-x86_64
    clang: error: linker command failed with exit code 1 (use -v to see invocation)
    error: command '/usr/bin/gcc' failed with exit code 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /Library/Frameworks/Python.framework/Versions/3.9/bin/python3.9 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/h2/5mydv0ns1_d4b0c_hkhw5bsw0000gn/T/pip-install-yn2_h3lf/astroscrappy_10c05a1c9b454c2c91100ae523b25b31/setup.py'"'"'; __file__='"'"'/private/var/folders/h2/5mydv0ns1_d4b0c_hkhw5bsw0000gn/T/pip-install-yn2_h3lf/astroscrappy_10c05a1c9b454c2c91100ae523b25b31/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/h2/5mydv0ns1_d4b0c_hkhw5bsw0000gn/T/pip-record-modb2__s/install-record.txt --single-version-externally-managed --compile --install-headers /Library/Frameworks/Python.framework/Versions/3.9/include/python3.9/astroscrappy Check the logs for full command output.

This installation was in the framework of ccdproc.
I had a look to several suggestions an solution, also here (e.g. issue #45), but none of them worked.
Could someone can help me to overcome this problem, please? Thanks a lot!!

(It's the first time I open an issue here, so I beg your pardon if I miss something o I compiled it not properly.)

Needs documentation

It looks to me like there's as yet no narrative docs for astroscrappy. Is that right @cmccully, or is it somewhere that I missed? If the former, there should definitely at least be an API reference and some quick usage examples/basictutorials. I'm happy to set up readthedocs to get the builds running once there's something to build.

Counts subtracted across whole image

Hi! I'm updating a previous spectral reduction code to use astroscrappy instead of cosmics.py, and am encountering an issue with the cleaned image. When I run both cosmics.py and astroscrappy on the same image with the same parameters, the cosmic ray mask is essentially the same, but the cleaned image is different. The result from cosmics.py seems right -- the cleaned image is different where the cosmic rays are. However, the result from astroscrappy is not the same as the input image anywhere, and has hundreds of counts subtracted in every pixel across the entire frame. I'm not sure if there is a background subtraction issue or something, but I do not input a background image. Has anyone encountered something like this, or have any insight on what the problem might be? Thanks!!

[1.0.3] Cython not called for missing .c files

Hi, I am currently preparing a Debian package for astroscrappy; and this needs to rebuild everything from scratch (invoking cython for the *.pyx files). To ensure this, I removed the three .c files before the build. Cython and cython3 are installed.

The build process however fails with the error

running build_ext
error: [Errno 2] Could not find C/C++ file astroscrappy/utils/median_utils.(c/cpp)
    for Cython file astroscrappy/utils/median_utils.pyx
    when building extension astroscrappy.utils.median_utils.
    Cython must be installed to build from a git checkout.: 'astroscrappy/utils/median_utils.c'
E: pybuild pybuild:274: build: plugin distutils failed with:
    exit code=1: /usr/bin/python setup.py build

Otherwise, the package builds fine.

Best regards

Ole

Missing testrunner in travis build

@eteq @embray @astrofrog Do you know why these travis builds are failing with the following error:

Traceback (most recent call last):
File "", line 1, in
File "astroscrappy/_astropy_init.py", line 106, in test
test_runner = _get_test_runner()
File "astroscrappy/_astropy_init.py", line 28, in _get_test_runner
from astropy.tests.helper import TestRunner
ImportError: cannot import name TestRunner
The command "python setup.py $SETUP_CMD" exited with 1.

Document build procedure on Windows + python 2.7

The problem is that stdint.h is not distributed with the default MSVC compiler for python 2.7. Prompted by #11 .

Infrastructure updates notice from Astropy Project

Hello from Astropy Project!

The following updates to your package might be necessary to ensure compatibility with the latest stable version of Astropy:

MPLBACKEND is now set to Agg in ci-helpers, packages expecting interactive plotting should override it in .travis.yml
Astropy 3.1 is not compatible with Numpy <1.13 versions. If you want to keep testing these older Numpy versions, please use ASTROPY_VERSION=3.0 or ASTROPY_VERSION=LTS in your Travis CI matrix.
Add sphinx-astropy as a package dependency if you are using astropy-helpers 3.1 or later. Otherwise, your documentation build (e.g., on ReadTheDocs) might fail.
If you are using six that is bundled with Astropy, please consider using the standalone six package instead.

If these are no longer applicable to you, please close the issue.

This is an automated issue for packages that opted in for automated astropy-helpers update. If this is opened in error, please let @pllim know.

xref astropy/astropy-tools#108

Coverage report is incorrect

At the moment, the only files included in the 100% coverage number are two __init__.py files. I think there are ways to include Cython/C code in the coverage (this is done in Astropy core I believe)

Investigate test_median_clean failures

test_median_clean is occasionally failing on some architectures: #71 (mipsel), #82 (windows wheels), #66 (comment) (i686), #85 (cp311-win_amd64 wheel):

Was marked as xfail in #85.

      def test_median_clean(testdata):
          imdata, crmask = testdata
          # Because our image only contains single cosmics, turn off
          # neighbor detection. Also, our cosmic rays are high enough
          # contrast that we can turn our detection threshold up.
          _mask, clean = detect_cosmics(imdata, readnoise=10., gain=1.0,
                                        sigclip=6, sigfrac=1.0, cleantype='median')
      
          assert (clean[crmask] != imdata[crmask]).sum() == crmask.sum()
      
          # Run it again on the clean data. We shouldn't find any new cosmic rays
          _mask2, _clean2 = detect_cosmics(clean, readnoise=10., gain=1.0,
                                           sigclip=6, sigfrac=1.0, cleantype='median')
  >       assert _mask2.sum() == 0
  E       assert 100 == 0

cosmic rays at image edges

Hi @cmccully -- have you run into any problems with cosmic rays at edges? I'm using the default parameters, and it seems to be having some problems detecting cosmic rays at the edge of the image. Is there something else I should be setting or is there something obvious I am missing?

I can send some example code if that helps and or explore it a little further.

Deprecation warning

I've been getting warnings like this a lot lately.

This particular example is from build a docker image.

  DEPRECATION: astroscrappy was installed using the legacy 'setup.py install' method, because a wheel could not be built for it. pip 21.0 will remove support for this functionality. A possible replacement is to fix the wheel build issue reported above. You can find discussion regarding this at https://github.com/pypa/pip/issues/8368.

This is the full log section.

Building wheels for collected packages: ccdproc, pyinotify, astroscrappy, click-didyoumean
  Building wheel for ccdproc (PEP 517): started
  Building wheel for ccdproc (PEP 517): finished with status 'done'
  Created wheel for ccdproc: filename=ccdproc-2.1.0-py3-none-any.whl size=100410 sha256=f17d5744c7b7a80b10cec9af3a744ed267b22a8101d144e73d634540bb1cc212
  Stored in directory: /root/.cache/pip/wheels/7e/42/8a/9c7af08da12f1fe3415c901c5153a955802a5be253fcd4b626
  Building wheel for pyinotify (setup.py): started
  Building wheel for pyinotify (setup.py): finished with status 'done'
  Created wheel for pyinotify: filename=pyinotify-0.9.6-py3-none-any.whl size=25339 sha256=b046095d419919e66d89686b2bf6fc7e4a58c5fe5583379c581c5690fa7a804c
  Stored in directory: /root/.cache/pip/wheels/9d/a0/4b/1a80814e4ad0b035c07831ea1b06b691046198492bbc5769b6
  Building wheel for astroscrappy (setup.py): started
  Building wheel for astroscrappy (setup.py): finished with status 'error'
  ERROR: Command errored out with exit status 1:
   command: /usr/local/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-3las_u1a/astroscrappy/setup.py'"'"'; __file__='"'"'/tmp/pip-install-3las_u1a/astroscrappy/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-x08cy2ke                                                            
       cwd: /tmp/pip-install-3las_u1a/astroscrappy/                                                                                                                                                                                          
  Complete output (58 lines):                                                                                                                                                                                                                
  running bdist_wheel                                                                                                                                                                                                                        
  running build                                                                                                                                                                                                                              
  running build_py                                                                                                                                                                                                                           
  creating build                                                                                                                                                                                                                             
  creating build/lib.linux-x86_64-3.8                                                                                                                                                                                                        
  creating build/lib.linux-x86_64-3.8/astroscrappy                                                                                                                                                                                           
  copying astroscrappy/__init__.py -> build/lib.linux-x86_64-3.8/astroscrappy                                                                                                                                                                
  copying astroscrappy/_astropy_init.py -> build/lib.linux-x86_64-3.8/astroscrappy                                                                                                                                                           
  copying astroscrappy/version.py -> build/lib.linux-x86_64-3.8/astroscrappy                                                                                                                                                                 
  copying astroscrappy/conftest.py -> build/lib.linux-x86_64-3.8/astroscrappy                                                                                                                                                                
  copying astroscrappy/cython_version.py -> build/lib.linux-x86_64-3.8/astroscrappy                                                                                                                                                          
  creating build/lib.linux-x86_64-3.8/astroscrappy/utils                                                                                                                                                                                     
  copying astroscrappy/utils/__init__.py -> build/lib.linux-x86_64-3.8/astroscrappy/utils                                                                                                                                                    
  copying astroscrappy/utils/setup_package.py -> build/lib.linux-x86_64-3.8/astroscrappy/utils                                                                                                                                               
  creating build/lib.linux-x86_64-3.8/astroscrappy/tests                                                                                                                                                                                     
  copying astroscrappy/tests/__init__.py -> build/lib.linux-x86_64-3.8/astroscrappy/tests                                                                                                                                                    
  copying astroscrappy/tests/fake_data.py -> build/lib.linux-x86_64-3.8/astroscrappy/tests                                                                                                                                                   
  copying astroscrappy/tests/setup_package.py -> build/lib.linux-x86_64-3.8/astroscrappy/tests                                                                                                                                               
  copying astroscrappy/tests/test_astroscrappy.py -> build/lib.linux-x86_64-3.8/astroscrappy/tests                                                                                                                                           
  copying astroscrappy/tests/test_cleaning.py -> build/lib.linux-x86_64-3.8/astroscrappy/tests                                                                                                                                               
  copying astroscrappy/tests/test_utils.py -> build/lib.linux-x86_64-3.8/astroscrappy/tests                                                                                                                                                  
  copying astroscrappy/_compiler.c -> build/lib.linux-x86_64-3.8/astroscrappy                                                                                                                                                                
  copying astroscrappy/astroscrappy.c -> build/lib.linux-x86_64-3.8/astroscrappy                                                                                                                                                             
  copying astroscrappy/utils/image_utils.c -> build/lib.linux-x86_64-3.8/astroscrappy/utils                                                                                                                                                  
  copying astroscrappy/utils/medutils.c -> build/lib.linux-x86_64-3.8/astroscrappy/utils                                                                                                                                                     
  copying astroscrappy/utils/median_utils.c -> build/lib.linux-x86_64-3.8/astroscrappy/utils                                                                                                                                                 
  copying astroscrappy/utils/imutils.c -> build/lib.linux-x86_64-3.8/astroscrappy/utils                                                                                                                                                      
  copying astroscrappy/tests/coveragerc -> build/lib.linux-x86_64-3.8/astroscrappy/tests                                                                                                                                                     
  running build_ext                                                                                                                                                                                                                          
  Traceback (most recent call last):                                                                                                                                                                                                         
    File "<string>", line 1, in <module>                                                                                                                                                                                                     
    File "/tmp/pip-install-3las_u1a/astroscrappy/setup.py", line 133, in <module>                                                                                                                                                            
      setup(name=PACKAGENAME,                                                                                                                                                                                                                
    File "/usr/local/lib/python3.8/site-packages/setuptools/__init__.py", line 145, in setup                                                                                                                                                 
      return distutils.core.setup(**attrs)                                                                                                                                                                                                   
    File "/usr/local/lib/python3.8/distutils/core.py", line 148, in setup                                                                                                                                                                    
      dist.run_commands()                                                                                                                                                                                                                    
    File "/usr/local/lib/python3.8/distutils/dist.py", line 966, in run_commands                                                                                                                                                             
      self.run_command(cmd)                                                                                                                                                                                                                  
    File "/usr/local/lib/python3.8/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/usr/local/lib/python3.8/site-packages/wheel/bdist_wheel.py", line 192, in run
      self.run_command('build')
    File "/usr/local/lib/python3.8/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/usr/local/lib/python3.8/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/usr/local/lib/python3.8/distutils/command/build.py", line 135, in run
      self.run_command(cmd_name)
    File "/usr/local/lib/python3.8/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "/usr/local/lib/python3.8/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/tmp/pip-install-3las_u1a/astroscrappy/.eggs/astropy_helpers-2.0.11-py3.8.egg/astropy_helpers/commands/build_ext.py", line 296, in run
      np_include = get_numpy_include_path()
    File "/tmp/pip-install-3las_u1a/astroscrappy/.eggs/astropy_helpers-2.0.11-py3.8.egg/astropy_helpers/utils.py", line 89, in get_numpy_include_path
      import numpy
  ModuleNotFoundError: No module named 'numpy'
  ----------------------------------------
  ERROR: Failed building wheel for astroscrappy
  Running setup.py clean for astroscrappy
  Building wheel for click-didyoumean (setup.py): started
  Building wheel for click-didyoumean (setup.py): finished with status 'done'
  Created wheel for click-didyoumean: filename=click_didyoumean-0.0.3-py3-none-any.whl size=2147 sha256=58bcd55a73356b4bb8f9d42454abc917fb28e27d2e9f2aceb49f5ce2b2544923
  Stored in directory: /root/.cache/pip/wheels/c2/97/1e/c7dda0b8e9616dcd8df2a2882e556ef62879967ac31e0e3d04
Successfully built ccdproc pyinotify click-didyoumean
Failed to build astroscrappy
Installing collected packages: numpy, astropy, asgiref, astroscrappy, tifffile, scipy, decorator, networkx, pillow, six, cycler, certifi, kiwisolver, pyparsing, python-dateutil, matplotlib, imageio, PyWavelets, scikit-image, astropy-healpix, reproject, ccdproc, vine, click, pytz, billiard, amqp, kombu, wcwidth, prompt-toolkit, click-repl, click-didyoumean, celery, pyinotify, idna, chardet, urllib3, requests, tzdata, websockets
    Running setup.py install for astroscrappy: started
    Running setup.py install for astroscrappy: finished with status 'done'
  DEPRECATION: astroscrappy was installed using the legacy 'setup.py install' method, because a wheel could not be built for it. pip 21.0 will remove support for this functionality. A possible replacement is to fix the wheel build issue reported above. You can find discussion regarding this at https://github.com/pypa/pip/issues/8368.

The Dockerfile content is:

FROM python:3
ENV PYTHONUNBUFFERED 1
RUN apt-get update && apt-get clean
RUN pip install --upgrade pip
RUN mkdir /build
WORKDIR /build
COPY requirements.txt /build
RUN pip install -r requirements.txt

and the requirements.txt is:

astropy
asgiref
ccdproc
celery
pyinotify
requests
tzdata
websockets

and the image is built with docker build --no-cache . -t <tag> from within the folder that contains the Dockerfile and requirements.txt. Then is pushed to a registry.

New release needed

The last release is from 2018 and uses astropy-helpers. It currently fails to install in a clean environment as at the time it did not use pyproject to define build-time dependencies. It might be time for a new release?

Fails to compile with OS X Xcode cc due to -fopenmp compiler option

I attempted to install ccdproc using pip, under a Ureka installation, and the installation failed while building astroscrappy with this message:
...

creating build/temp.macosx-10.6-x86_64-2.7/astroscrappy/utils

cc -fno-strict-aliasing -I/Users/bjw/software/ureka/Ureka/python/include - I/Users/bjw/software/ureka/Ureka/python/include -DNDEBUG -I/Users/bjw/software/ureka/Ureka/python/include -I/Users/bjw/software/ureka/Ureka/python/include -I/Users/bjw/software/ureka/Ureka/python/lib/python2.7/site-packages/numpy/core/include -Iastroscrappy/utils -I/Users/bjw/software/ureka/Ureka/python/include/python2.7 -c astroscrappy/utils/median_utils.c -o build/temp.macosx-10.6-x86_64-2.7/astroscrappy/utils/median_utils.o -g -O3 -funroll-loops -ffast-math -fopenmp -Wp,-w -Wno-unused-function

clang: error: unsupported option '-fopenmp'

cc --version prints:

Apple LLVM version 7.3.0 (clang-703.0.29)
Target: x86_64-apple-darwin15.3.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

I haven't installed another C compiler yet. Is it possible to test for the legality of this compiler option before requiring it?

Travis Fails to build Astroscrappy

I'm having trouble with the builds in travis, "apparently" due to astroscrappy. I say apparently because I don't think astroscrappy itself is the problem but I know it is the point of failure for travis at least. Running locally it works OK.

Any feedback is appreciated. I apologize in advance if this issue does not correspond here.

My .travis.yml file is like this:

language: python
python:
- '3.5'
- '3.6'

branches:
  only:
  - master
  - development

install:
- sudo apt-get update

- wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh;


- bash miniconda.sh -b -p $HOME/miniconda
- export PATH="$HOME/miniconda/bin:$PATH"
- hash -r
- conda config --set always_yes yes --set changeps1 no

- conda config --add channels http://ssb.stsci.edu/astroconda
- conda update -q conda

- conda info -a

- conda env create -q python=$TRAVIS_PYTHON_VERSION -f environment.yml
- source activate goodman

- python setup.py test
- python setup.py install

# removed unnecessary details at the bottom.

The build fails at conda env create -q python=$TRAVIS_PYTHON_VERSION -f environment.yml

and the environment.yml file is:

name: goodman
channels:
  - astroconda
dependencies:
  - numpy
  - pandas
  - matplotlib
  - scipy
  - cython
  - astropy>3
  - pip
  - pip:
    - ccdproc
    - coveralls
    - astroplan
    - sphinx
    - sphinxcontrib.napoleon

Some of the things I've tried so far to fix it.

Specify latest version of cython
Isolate ccdproc installation, thefore astroscrappy's as a dependency
Isolate astroscrappy installation.

The relevan traceback is below and the full build is at https://travis-ci.org/soar-telescope/goodman/jobs/407313483

  creating build/temp.linux-x86_64-3.7/astroscrappy
  gcc -pthread -B /home/travis/miniconda/envs/goodman/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/travis/miniconda/envs/goodman/include/python3.7m -c astroscrappy/_compiler.c -o build/temp.linux-x86_64-3.7/astroscrappy/_compiler.o
  gcc -pthread -shared -B /home/travis/miniconda/envs/goodman/compiler_compat -L/home/travis/miniconda/envs/goodman/lib -Wl,-rpath=/home/travis/miniconda/envs/goodman/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.7/astroscrappy/_compiler.o -o build/lib.linux-x86_64-3.7/astroscrappy/_compiler.cpython-37m-x86_64-linux-gnu.so
  building 'astroscrappy.utils.median_utils' extension
  creating build/temp.linux-x86_64-3.7/astroscrappy/utils
  gcc -pthread -B /home/travis/miniconda/envs/goodman/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/travis/miniconda/envs/goodman/lib/python3.7/site-packages/numpy/core/include -Iastroscrappy/utils -I/home/travis/miniconda/envs/goodman/include/python3.7m -c astroscrappy/utils/median_utils.c -o build/temp.linux-x86_64-3.7/astroscrappy/utils/median_utils.o -g -O3 -funroll-loops -ffast-math -fopenmp -Wp,-w -Wno-unused-function
  gcc -pthread -B /home/travis/miniconda/envs/goodman/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/travis/miniconda/envs/goodman/lib/python3.7/site-packages/numpy/core/include -Iastroscrappy/utils -I/home/travis/miniconda/envs/goodman/include/python3.7m -c astroscrappy/utils/medutils.c -o build/temp.linux-x86_64-3.7/astroscrappy/utils/medutils.o -g -O3 -funroll-loops -ffast-math -fopenmp -Wp,-w -Wno-unused-function
  gcc -pthread -shared -B /home/travis/miniconda/envs/goodman/compiler_compat -L/home/travis/miniconda/envs/goodman/lib -Wl,-rpath=/home/travis/miniconda/envs/goodman/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.7/astroscrappy/utils/median_utils.o build/temp.linux-x86_64-3.7/astroscrappy/utils/medutils.o -o build/lib.linux-x86_64-3.7/astroscrappy/utils/median_utils.cpython-37m-x86_64-linux-gnu.so -g -fopenmp
  building 'astroscrappy.utils.image_utils' extension
  gcc -pthread -B /home/travis/miniconda/envs/goodman/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/travis/miniconda/envs/goodman/lib/python3.7/site-packages/numpy/core/include -Iastroscrappy/utils -I/home/travis/miniconda/envs/goodman/include/python3.7m -c astroscrappy/utils/image_utils.c -o build/temp.linux-x86_64-3.7/astroscrappy/utils/image_utils.o -g -O3 -funroll-loops -ffast-math -fopenmp -Wp,-w -Wno-unused-function
  gcc -pthread -B /home/travis/miniconda/envs/goodman/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/travis/miniconda/envs/goodman/lib/python3.7/site-packages/numpy/core/include -Iastroscrappy/utils -I/home/travis/miniconda/envs/goodman/include/python3.7m -c astroscrappy/utils/imutils.c -o build/temp.linux-x86_64-3.7/astroscrappy/utils/imutils.o -g -O3 -funroll-loops -ffast-math -fopenmp -Wp,-w -Wno-unused-function
  gcc -pthread -shared -B /home/travis/miniconda/envs/goodman/compiler_compat -L/home/travis/miniconda/envs/goodman/lib -Wl,-rpath=/home/travis/miniconda/envs/goodman/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.7/astroscrappy/utils/image_utils.o build/temp.linux-x86_64-3.7/astroscrappy/utils/imutils.o -o build/lib.linux-x86_64-3.7/astroscrappy/utils/image_utils.cpython-37m-x86_64-linux-gnu.so -g -fopenmp
  building 'astroscrappy.astroscrappy' extension
  gcc -pthread -B /home/travis/miniconda/envs/goodman/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/travis/miniconda/envs/goodman/lib/python3.7/site-packages/numpy/core/include -I/home/travis/miniconda/envs/goodman/include/python3.7m -c astroscrappy/astroscrappy.c -o build/temp.linux-x86_64-3.7/astroscrappy/astroscrappy.o -Wp,-w -Wno-unused-function
  astroscrappy/astroscrappy.c: In function ‘__Pyx__ExceptionSave’:
  astroscrappy/astroscrappy.c:27186:19: error: ‘PyThreadState’ has no member named ‘exc_type’
       *type = tstate->exc_type;
                     ^
  astroscrappy/astroscrappy.c:27187:20: error: ‘PyThreadState’ has no member named ‘exc_value’
       *value = tstate->exc_value;
                      ^
  astroscrappy/astroscrappy.c:27188:17: error: ‘PyThreadState’ has no member named ‘exc_traceback’
       *tb = tstate->exc_traceback;
                   ^
  astroscrappy/astroscrappy.c: In function ‘__Pyx__ExceptionReset’:
  astroscrappy/astroscrappy.c:27195:22: error: ‘PyThreadState’ has no member named ‘exc_type’
       tmp_type = tstate->exc_type;
                        ^
  astroscrappy/astroscrappy.c:27196:23: error: ‘PyThreadState’ has no member named ‘exc_value’
       tmp_value = tstate->exc_value;
                         ^
  astroscrappy/astroscrappy.c:27197:20: error: ‘PyThreadState’ has no member named ‘exc_traceback’
       tmp_tb = tstate->exc_traceback;
                      ^
  astroscrappy/astroscrappy.c:27198:11: error: ‘PyThreadState’ has no member named ‘exc_type’
       tstate->exc_type = type;
             ^
  astroscrappy/astroscrappy.c:27199:11: error: ‘PyThreadState’ has no member named ‘exc_value’
       tstate->exc_value = value;
             ^
  astroscrappy/astroscrappy.c:27200:11: error: ‘PyThreadState’ has no member named ‘exc_traceback’
       tstate->exc_traceback = tb;
             ^
  astroscrappy/astroscrappy.c: In function ‘__Pyx__GetException’:
  astroscrappy/astroscrappy.c:27255:22: error: ‘PyThreadState’ has no member named ‘exc_type’
       tmp_type = tstate->exc_type;
                        ^
  astroscrappy/astroscrappy.c:27256:23: error: ‘PyThreadState’ has no member named ‘exc_value’
       tmp_value = tstate->exc_value;
                         ^
  astroscrappy/astroscrappy.c:27257:20: error: ‘PyThreadState’ has no member named ‘exc_traceback’
       tmp_tb = tstate->exc_traceback;
                      ^
  astroscrappy/astroscrappy.c:27258:11: error: ‘PyThreadState’ has no member named ‘exc_type’
       tstate->exc_type = local_type;
             ^
  astroscrappy/astroscrappy.c:27259:11: error: ‘PyThreadState’ has no member named ‘exc_value’
       tstate->exc_value = local_value;
             ^
  astroscrappy/astroscrappy.c:27260:11: error: ‘PyThreadState’ has no member named ‘exc_traceback’
       tstate->exc_traceback = local_tb;
             ^
  astroscrappy/astroscrappy.c: In function ‘__Pyx__ExceptionSwap’:
  astroscrappy/astroscrappy.c:27282:22: error: ‘PyThreadState’ has no member named ‘exc_type’
       tmp_type = tstate->exc_type;
                        ^
  astroscrappy/astroscrappy.c:27283:23: error: ‘PyThreadState’ has no member named ‘exc_value’
       tmp_value = tstate->exc_value;
                         ^
  astroscrappy/astroscrappy.c:27284:20: error: ‘PyThreadState’ has no member named ‘exc_traceback’
       tmp_tb = tstate->exc_traceback;
                      ^
  astroscrappy/astroscrappy.c:27285:11: error: ‘PyThreadState’ has no member named ‘exc_type’
       tstate->exc_type = *type;
             ^
  astroscrappy/astroscrappy.c:27286:11: error: ‘PyThreadState’ has no member named ‘exc_value’
       tstate->exc_value = *value;
             ^
  astroscrappy/astroscrappy.c:27287:11: error: ‘PyThreadState’ has no member named ‘exc_traceback’
       tstate->exc_traceback = *tb;
             ^
  error: command 'gcc' failed with exit status 1
  
  ----------------------------------------
  Failed building wheel for astroscrappy
  Running setup.py clean for astroscrappy

Astropy Affiliated Package Review

This package has been re-reviewed by the Astropy coordination committee in relation to the Astropy affiliated package ecosystem.

We have adopted a review process for affiliated package that includes assigning quantitative ‘scores’ (red/orange/green) for different categories of review. You can read up more about this process here. (This document, currently in Google Docs, will be moved to the documentation in the near future.) For each of the categories below we have listed the score and have included some comments when the score is not green.

Functionality/Scope
No further comments
Integration with Astropy ecosystem
In general, we think it might be worth investigating whether some of the fast convolution code could be combined with astropy.convolution
Documentation
There is currently no link to any documentation, and the documentation in the repo is just an API section. This was the biggest issue we came across during the review.
Testing
There is only a single test of the high-level function, and the test coverage values reported by coveralls is incorrect (it does not include any Cython code). It would be worth enabling coverage testing for Cython/C as is done in the astropy core package
Development status
There hasn't been any activity on the repo for over a year, so we've marked this as 'low activity'.
Python 3 compatibility
No further comments

Summary/Decision: There are some areas in which this package did not score well with the new
review process. As per the review guidelines, these areas should be fixed – if review areas stay in the red for too long, we may consider removing the package from the affiliated package list. But please don't hesitate to let us know how we can help!

If you agree with the above review, please feel free to close this issue. If you have any follow-up questions or disagree with any of the comments above, leave a comment and we can discuss it here. At any point in future you can request a re-review of the package if you believe any of the scores should be updated - contact the coordination committee, and we’ll do a new review. Note that we are in the process of redesigning the http://affiliated.astropy.org page to show these scores (but not the comments). Finally, please keep the title of this issue as-is (“Astropy Affiliated Package Review”) to make it easy to search for affiliated package reviews in future.

Segmentation fault (core dumped)

Hi.

I'm trying to run detect_cosmics on a pretty big array from DECam (26671, 26671). Unfortunately the code always crashes with a Segmentation fault.

Is the array just too big or am I missing something obvious?

Thanks.

-TL

Unable to install astroscrappy with pip-install

Hello, I'm unable to install astroscrappy with,
pip install astroscrappy

Building wheels for collected packages: astroscrappy
  Building wheel for astroscrappy (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: ./workspace/spacetelescope/notebooks/env2/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/_6/w75l5bpn5tgbwm_jg8z1m07h0002ch/T/pip-install-pgqkrk1n/astroscrappy/setup.py'"'"'; __file__='"'"'/private/var/folders/_6/
w75l5bpn5tgbwm_jg8z1m07h0002ch/T/pip-install-pgqkrk1n/astroscrappy/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /private/var/folders/_6/w7
5l5bpn5tgbwm_jg8z1m07h0002ch/T/pip-wheel-j2aryhmz --python-tag cp37
       cwd: /private/var/folders/_6/w75l5bpn5tgbwm_jg8z1m07h0002ch/T/pip-install-pgqkrk1n/astroscrappy/
  Complete output (59 lines):
  OpenMP was not found. astroscrappy will be compiled without OpenMP. (Use the "-v" option of setup.py for more details.)
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.macosx-10.7-x86_64-3.7
  creating build/lib.macosx-10.7-x86_64-3.7/astroscrappy
  copying astroscrappy/conftest.py -> build/lib.macosx-10.7-x86_64-3.7/astroscrappy
  copying astroscrappy/version.py -> build/lib.macosx-10.7-x86_64-3.7/astroscrappy
  copying astroscrappy/_astropy_init.py -> build/lib.macosx-10.7-x86_64-3.7/astroscrappy
  copying astroscrappy/__init__.py -> build/lib.macosx-10.7-x86_64-3.7/astroscrappy
  copying astroscrappy/cython_version.py -> build/lib.macosx-10.7-x86_64-3.7/astroscrappy
  creating build/lib.macosx-10.7-x86_64-3.7/astroscrappy/tests
  copying astroscrappy/tests/test_utils.py -> build/lib.macosx-10.7-x86_64-3.7/astroscrappy/tests
  copying astroscrappy/tests/test_astroscrappy.py -> build/lib.macosx-10.7-x86_64-3.7/astroscrappy/tests
  copying astroscrappy/tests/fake_data.py -> build/lib.macosx-10.7-x86_64-3.7/astroscrappy/tests
  copying astroscrappy/tests/__init__.py -> build/lib.macosx-10.7-x86_64-3.7/astroscrappy/tests
  copying astroscrappy/tests/test_cleaning.py -> build/lib.macosx-10.7-x86_64-3.7/astroscrappy/tests
  copying astroscrappy/tests/setup_package.py -> build/lib.macosx-10.7-x86_64-3.7/astroscrappy/tests
  creating build/lib.macosx-10.7-x86_64-3.7/astroscrappy/utils
  copying astroscrappy/utils/__init__.py -> build/lib.macosx-10.7-x86_64-3.7/astroscrappy/utils
  copying astroscrappy/utils/setup_package.py -> build/lib.macosx-10.7-x86_64-3.7/astroscrappy/utils
  copying astroscrappy/astroscrappy.c -> build/lib.macosx-10.7-x86_64-3.7/astroscrappy
  copying astroscrappy/_compiler.c -> build/lib.macosx-10.7-x86_64-3.7/astroscrappy
  copying astroscrappy/utils/image_utils.c -> build/lib.macosx-10.7-x86_64-3.7/astroscrappy/utils
  copying astroscrappy/utils/medutils.c -> build/lib.macosx-10.7-x86_64-3.7/astroscrappy/utils
  copying astroscrappy/utils/median_utils.c -> build/lib.macosx-10.7-x86_64-3.7/astroscrappy/utils
  copying astroscrappy/utils/imutils.c -> build/lib.macosx-10.7-x86_64-3.7/astroscrappy/utils
  copying astroscrappy/tests/coveragerc -> build/lib.macosx-10.7-x86_64-3.7/astroscrappy/tests
  running build_ext
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/private/var/folders/_6/w75l5bpn5tgbwm_jg8z1m07h0002ch/T/pip-install-pgqkrk1n/astroscrappy/setup.py", line 148, in <module>
      **package_info
    File "./workspace/spacetelescope/notebooks/env2/lib/python3.7/site-packages/setuptools/__init__.py", line 145, in setup
      return distutils.core.setup(**attrs)
    File "./anaconda3/lib/python3.7/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "./anaconda3/lib/python3.7/distutils/dist.py", line 966, in run_commands
      self.run_command(cmd)
    File "./anaconda3/lib/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "./workspace/spacetelescope/notebooks/env2/lib/python3.7/site-packages/wheel/bdist_wheel.py", line 192, in run
      self.run_command('build')
    File "./anaconda3/lib/python3.7/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "./anaconda3/lib/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "./anaconda3/lib/python3.7/distutils/command/build.py", line 135, in run
      self.run_command(cmd_name)
    File "./anaconda3/lib/python3.7/distutils/cmd.py", line 313, in run_command
      self.distribution.run_command(command)
    File "./anaconda3/lib/python3.7/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/private/var/folders/_6/w75l5bpn5tgbwm_jg8z1m07h0002ch/T/pip-install-pgqkrk1n/astroscrappy/.eggs/astropy_helpers-2.0.10-py3.7.egg/astropy_helpers/commands/build_ext.py", line 296, in run
      np_include = get_numpy_include_path()
    File "/private/var/folders/_6/w75l5bpn5tgbwm_jg8z1m07h0002ch/T/pip-install-pgqkrk1n/astroscrappy/.eggs/astropy_helpers-2.0.10-py3.7.egg/astropy_helpers/utils.py", line 89, in get_numpy_include_path
      import numpy
  ModuleNotFoundError: No module named 'numpy'
  ----------------------------------------
  ERROR: Failed building wheel for astroscrappy
  Running setup.py clean for astroscrappy
Failed to build astroscrappy
Installing collected packages: numpy, astropy, astroscrappy
  Running setup.py install for astroscrappy ... done
Successfully installed astropy-3.2.1 astroscrappy-1.0.8 numpy-1.17.2

Conda issue during numpy 1.11 update

Is there any reason the conda build doesn't allow numpy 1.11? I get this error when trying to update:

[Anaconda] C:\Users\Admin>conda install numpy=1.11
Using Anaconda Cloud api site https://api.anaconda.org
Fetching package metadata: ......
Solving package specifications: ....

The following specifications were found to be in conflict:
  - astroscrappy (target=astroscrappy-1.0.3-np110py35_0.tar.bz2) -> numpy 1.10*|1.9*
  - astroscrappy (target=astroscrappy-1.0.3-np110py35_0.tar.bz2) -> python 2.7*|3.4*
  - numpy 1.11*

Use "conda info <package>" to see the dependencies for each package.

[1.0.3] Tests fail on bigendian machines (powerpc)

On Debian bigendian machines, there are many tests failing: test_medfilt5, test_medfilt7, test_sepmedfilt3, test_sepmedfilt5, test_sepmedfilt7, test_sepmedfilt9, test_subsample, test_rebin, test_laplaceconvolve, and test_convolve.
The cause seems to be different; here as one example:

________________________________ test_medfilt5 _________________________________

    def test_medfilt5():
        a = np.ascontiguousarray(np.random.random((1001, 1001))).astype('<f4')
        npmed5 = ndimage.filters.median_filter(a, size=(5, 5), mode='nearest')
        npmed5[:2, :] = a[:2, :]
        npmed5[-2:, :] = a[-2:, :]
        npmed5[:, :2] = a[:, :2]
        npmed5[:, -2:] = a[:, -2:]

        med5 = medfilt5(a)
>       assert np.all(med5 == npmed5)
E       assert <function all at 0xf7d5dcb0>(array([[  1.7...dtype=float32) == array([[ 0.959...dtype=float32)
E        +  where <function all at 0xf7d5dcb0> = np.all
E         Detailed information truncated, use "-vv" to show)

astroscrappy/tests/test_utils.py:67: AssertionError

Full log here. For 64-bit powerpc, where we have big and little endian architectures, only the big endian one fails; the little endian succeeds. For MIPS (32 bit) the situation is the same.
Complete overview on all architectures is here.

ah_bootstrap.py from astropy-headers distributed with tarball needs to be updated

When I tried to build astroscrappy-1.0.8 (from PyPi) as a MacPort package it fails with the following error:

Traceback (most recent call last): File "setup.py", line 8, in <module> import ah_bootstrap File "/opt/local/var/macports/build/_opt_vjpd_macports_ports_python_py-vjpd-astroscrappy/py36-vjpd-astroscrappy/work/astroscrappy-1.0.8/ah_bootstrap.py", line 920, in <module> BOOTSTRAPPER = _Bootstrapper.main() File "/opt/local/var/macports/build/_opt_vjpd_macports_ports_python_py-vjpd-astroscrappy/py36-vjpd-astroscrappy/work/astroscrappy-1.0.8/ah_bootstrap.py", line 223, in main bootstrapper.run() File "/opt/local/var/macports/build/_opt_vjpd_macports_ports_python_py-vjpd-astroscrappy/py36-vjpd-astroscrappy/work/astroscrappy-1.0.8/ah_bootstrap.py", line 321, in run dist = method() File "/opt/local/var/macports/build/_opt_vjpd_macports_ports_python_py-vjpd-astroscrappy/py36-vjpd-astroscrappy/work/astroscrappy-1.0.8/ah_bootstrap.py", line 389, in get_local_directory_dist upgrade = self._do_upgrade(dist) File "/opt/local/var/macports/build/_opt_vjpd_macports_ports_python_py-vjpd-astroscrappy/py36-vjpd-astroscrappy/work/astroscrappy-1.0.8/ah_bootstrap.py", line 569, in _do_upgrade return self._do_download(version=upgrade.version) File "/opt/local/var/macports/build/_opt_vjpd_macports_ports_python_py-vjpd-astroscrappy/py36-vjpd-astroscrappy/work/astroscrappy-1.0.8/ah_bootstrap.py", line 554, in _do_download raise Exception(msg.format(DIST_NAME, source, repr(e))) Exception: Error retrieving astropy-helpers from PyPI: DistutilsError('theallow-hostsoption is not supported when using pip to install requirements.',)

This can be fixed by updating ah_bootstrap.py from the latest distribution of astro-headers (v4.0.1).

Best regards
John Pritchard

P.S. Why is the latest version on PyPi 1.0.8 but only 1.0.5 on GitHub? It's a little disconcerting...

MNT: Rename default branch from master to main

Please rename your default branch from master to main as part of astropy/astropy-project#151 , preferably by 2021-03-22. Also a friendly reminder to check documentation, workflows, etc., and update them accordingly. Please don't forget to communicate this change to your users and stakeholders. To summarize:

Rename branch from master to main, preferably by 2021-03-22.
Update documentation, workflows, etc., accordingly. -- See #62
Communicate this change to your users and stakeholders.

Once this is taken care of, you may close this issue.

This is an automated issue. If this is opened in error, please let @pllim know!

Enable AppVeyor?

AppVeyor doesn't seem to be active for this repository, but it would probably be a good idea to do so given that there is Cython code. If you are happy for this to be the case, we should add the project to the astropy organization on AppVeyor.

Build failing

In my CI tests that require building astroscrappy from sources, the build started failing today. The reported error is:

      ./astroscrappy/astroscrappy.c: In function ‘__pyx_import_star_set’:
      ./astroscrappy/astroscrappy.c:25148:20: error: lvalue required as left operand of assignment
      25148 |     CYTHON_ATOMICS = __Pyx_PyObject_IsTrue(o); if (unlikely((CYTHON_ATOMICS == (int)-1) && PyErr_Occurred())) __PYX_ERR(2, 26, __pyx_L2_error)
            |                    ^
      error: command '/usr/bin/gcc' failed with exit code 1

This problem only occurs when using Python 3.10. More details here.

required numpy array dtype inconsistent with standard image data ranges

The doctext of detect_cosmics says that the input should be a float numpy array, but the parameter satlevel default is given as 65538, which indicates that the function expects a uint16 data-type?
By convention, float dtype image data should be 0.0...1.0 (or sometimes -1.0...1.0) but not use uint16 numbers up to 2**16.
Could you clarify if detect_cosmics requires

a certain input range of numbers to properly work?
a certain numpy dtype as input (help text says float, but as I described above, that somehow doesn't fit with satlevel default, and while testing, it seems to do something properly with a uint8 input?)
both of the above?

Thanks!

CR masking when very long tails

Hello,
thanks for the very good tool you are providing!
I tried to use astroscrappy for CR masking on an EMCCD on which tails are very long (tens to hundreds of pixels, sometimes several lines).
I did not manage to find a way to mask these tails entirely.
Could you tell me what would be the parameters I should try to tune in order to get rid of them?
Thanks a lot for the help.
I add the link to an image if you are interested:
https://nuage.osupytheas.fr/s/D2ALKby4f4HiYQ4

Test failure on Debian architecture mipsel

With the release 1.1.0, I get the following failure on mips64el (MIPS, 32 bit, little endian):

______________________________ test_median_clean _______________________________

    def test_median_clean():
        # Because our image only contains single cosmics, turn off
        # neighbor detection. Also, our cosmic rays are high enough
        # contrast that we can turn our detection threshold up.
        _mask, clean = detect_cosmics(imdata, readnoise=10., gain=1.0,
                                      sigclip=6, sigfrac=1.0, cleantype='median')
    
        assert (clean[crmask] != imdata[crmask]).sum() == crmask.sum()
    
        # Run it again on the clean data. We shouldn't find any new cosmic rays
        _mask2, _clean2 = detect_cosmics(clean, readnoise=10., gain=1.0,
                                         sigclip=6, sigfrac=1.0, cleantype='median')
>       assert _mask2.sum() == 0
E       assert 8780 == 0
E        +  where 8780 = <built-in method sum of numpy.ndarray object at 0xf353f530>()
E        +    where <built-in method sum of numpy.ndarray object at 0xf353f530> = array([[False, False, False, ..., False, False, False],\n       [False, False, False, ..., False, False, False],\n      ...alse],\n       [False, False, False, ..., False, False, False],\n       [False, False, False, ..., False, False, False]]).sum

/usr/lib/python3/dist-packages/astroscrappy/tests/test_cleaning.py:22: AssertionError

Full build log here.

Package versions:

Python 3.9.9 and 3.10
Numpy: 1.21.4
Scipy: 1.7.1
Astropy: 5.0

A similar problem (with _mask2.sum()=8700) happened with armhf (ARM 32 bit little endian) with astropy 4.3.1 (but with 5.0 it succeeded on armhf).