Problem deion I am using lightkurv

I can reproduce problem with TESS SPOC 2 minute lightcurves on Windows, so it is

PR <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id=

Potential memory leak using `lightkurve.read` about lightkurve HOT 3 CLOSED

zclaytor commented on July 22, 2024

Potential memory leak using `lightkurve.read`

from lightkurve.

Comments (3)

orionlee commented on July 22, 2024

I can reproduce problem with TESS SPOC 2 minute lightcurves on Windows, so it is not specific to Kepler Bonus lightcurves / Linux.
the problem seems to be a recent regression. The older lightkurve 2.4.0 does not appear to have the memory leak

The problem exists for TESS SPOC lightcurves:

lightkurve 2.5.0dev
astropy 5.0.4
0 8.715249 12.326077
1 11.507044 17.704528
2 13.629003 19.827207
3 17.910356 24.108518
4 18.42802 24.631339
5 21.177316 27.380927
6 25.452262 31.656041
7 26.010972 32.214751
8 28.761221 34.965
9 33.035526 39.239305
10 33.597524 39.801303
...
100 221.514371 227.757565
101 224.226204 230.469398
102 228.460606 234.7038
103 230.504471 236.747665
104 233.216248 239.459442
105 237.450479 243.693673
...
146 318.471775 324.392998
147 318.974317 324.886945
148 321.538435 327.451063
149 324.656307 329.973692
150 325.023099 329.973692

But the slightly older lightkurve v2.4.0 does not have the problem:

lightkurve 2.4.0
astropy 5.0.4
0 6.542636 10.153735
1 7.216337 13.414637
2 7.222926 13.422954
3 9.420149 15.620135
4 7.856869 15.620135
...
119 40.648573 46.893591
120 36.930138 46.893591
121 39.122917 46.893591
...
197 35.755834 46.893591
198 36.766859 46.893591
199 7.098715 46.893591
...

from lightkurve.

orionlee commented on July 22, 2024

PR #1299 seems to have introduced the problem.

Workarounds:

Downgrade lightkurve to v2.4.0, or
if you use the source, you can change the few lines to use the old v2.4.0 logic

For

lightkurve/src/lightkurve/io/generic.py

Lines 37 to 39 in f8e8c16

 else: 

 with fits.open(filename) as hdulist: 

 hdulist = deepcopy(hdulist)

Replace it with

    else:
        hdulist = fits.open(filename)

Essentially, it revert back to the old logic. The old logic has some issues. But I think they mostly manifest in edge cases.

I don't understand why the new logic cause memory leak though.

from lightkurve.

orionlee commented on July 22, 2024

Confirmed that the memory leak indeed stems from the deecopy(hdulist) in the PR.

A sample script that demonstrates the problem.

from glob import glob
import tracemalloc
import astropy
from astropy.io import fits
import lightkurve as lk
from copy import deepcopy

# import gc

# Show lk.read memory leak issues in
# https://github.com/lightkurve/lightkurve/issues/1388

print('lightkurve',  lk.__version__)
print('astropy', astropy.__version__)


def read_lite(filename):
    # simulate the logic in 
    # https://github.com/lightkurve/lightkurve/pull/1299
    hdulist = None
    with fits.open(filename) as hdulist:
        hdulist = deepcopy(hdulist)
    return hdulist

    
# files is a list of sample SPOC TESS lc
files = sorted(glob("spoc_samples/*_lc.fits"))
tracemalloc.start()

for i, f in enumerate(files[:]):
    l = read_lite((f))
    # print memory used so far in MB
    a, b = tracemalloc.get_traced_memory()
    print(i, a / 1e6, b / 1e6)
    # del l
    # gc.collect()

Similar memory leak:

lightkurve 2.5.0dev
astropy 5.0.4
0 2.450405 4.575374
1 4.578252 6.877436
2 6.697511 8.996654
3 8.81539 11.115467
4 10.934333 13.232662
...
145 297.996694 300.220553
146 300.038592 302.264887
147 302.08049 304.306785
148 304.122388 306.348683
149 306.164286 308.390581

from lightkurve.

Potential memory leak using `lightkurve.read` about lightkurve HOT 3 CLOSED

Comments (3)

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent

	else:
	with fits.open(filename) as hdulist:
	hdulist = deepcopy(hdulist)