Attempts to download from lightkurve results in a KeyError: 'dataURL' about lightkurve HOT 14 CLOSED

Hi there! 👋 Thank you for opening your first Lightkurve issue! 🙏 One of our maintainers will get back to you as soon as possible. 👩‍🚀 You can expect a response within 7 days. 📅 If you haven’t heard anything by then, feel free to ping this thread. 🛎️ We love that you are using Lightkurve and appreciate your feedback. 👍

from lightkurve.

I happen to have just run into something similar:

>>> import lightkurve as lk
>>> q = lk.search_lightcurve('HD 53929', mission='TESS')
>>> lc = q[0].download()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/wball/.local/lib/python3.11/site-packages/lightkurve/utils.py", line 566, in wrapper
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/home/wball/.local/lib/python3.11/site-packages/lightkurve/search.py", line 432, in download
    return self._download_one(
           ^^^^^^^^^^^^^^^^^^^
  File "/home/wball/.local/lib/python3.11/site-packages/lightkurve/search.py", line 351, in _download_one
    download_url = table[:1]["dataURL"][0]
                   ~~~~~~~~~^^^^^^^^^^^
  File "/home/wball/.local/lib/python3.11/site-packages/astropy/table/table.py", line 2055, in __getitem__
    return self.columns[item]
           ~~~~~~~~~~~~^^^^^^
  File "/home/wball/.local/lib/python3.11/site-packages/astropy/table/table.py", line 264, in __getitem__
    return OrderedDict.__getitem__(self, item)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyError: 'dataURL'

I just tried λ Nor and V1175 Cas, both of which failed, so I wonder if there's an upstream issue, perhaps with MAST itself. I haven't much more time to look at this today.

Environment

• platform (e.g. Linux, OSX, Windows): Linux (Fedora 38)
• lightkurve version (e.g. 1.0b6): 2.4.1
• installation method (e.g. pip, conda, source): pip

from lightkurve.

Perhaps the desired field has changed its name to 'dataURI'?

>>> print(q[0].table['dataURL'])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/wball/.local/lib/python3.11/site-packages/astropy/table/table.py", line 2055, in __getitem__
    return self.columns[item]
           ~~~~~~~~~~~~^^^^^^
  File "/home/wball/.local/lib/python3.11/site-packages/astropy/table/table.py", line 264, in __getitem__
    return OrderedDict.__getitem__(self, item)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyError: 'dataURL'
>>> print(q[0].table['dataURI'])
                                             dataURI                                              
--------------------------------------------------------------------------------------------------
mast:HLSP/qlp/s0033/0000/0002/9228/8128/hlsp_qlp_tess_ffi_s0033-0000000292288128_tess_v01_llc.fits

Changing dataURL to dataURI in this line fixed the downloads for me:

from lightkurve.

Hi all,

Just add another data point, I had the same problem, and Warrick's solution also worked for me.

Environment

• platform (e.g. Linux, OSX, Windows): Linux (Mint 21)
• lightkurve version (e.g. 1.0b6): 2.4.1
• installation method (e.g. pip, conda, source): pip

from lightkurve.

Somehow MAST no longer returns dataURL column (it used to return dataURI and dataURL). lightkurve has been using dataURL to download the products.

@scfleming Has MAST dropped dataURL column?

Workaround: Similar to what Warrick suggested, one can also do it without changing lightkurve source code by creating the dataURL column in the search result in application codes:

sr = lk.search_lightcurve("TIC278525028")
sr.table["dataURL"] = sr.table["dataURI"]  # workaround MAST issue
lcc = sr.download_all()

from lightkurve.

Thanks for letting me know, I will inquire with the team. In the mean time, if using dataURI can serve as a temporary work-around I'd recommend doing that until further notice. I can confirm that a get_product_list() from astroquery.mast is only returning a dataURI column in the resultant table.

from lightkurve.

FWIW, it looks like Lightkurve already uses 'dataURI' for Kepler products:

from lightkurve.

So far we haven't noticed anything that would explain this change. Yesterday was a monthly patch for us, but none of the changes we made should have changed this detail. I'm going to keep checking in with folks here, there's a possibility the recommendation would be to change so everything uses dataURI for returned products, but I don't yet understand why this has come up only now, still working through that detail.

from lightkurve.

That seems to work @warrickball. Thanks so much.

from lightkurve.

@scfleming Thanks for looking into it! So should we use dataURI for download in all cases, regardless on whether MAST would resurface dataURL or not?

Even that is the case, please keep us updated. If dataURL is permanently removed from MAST API, we might need to release a patch for the old lightkurve v1.x, in addition to the current v2.x

P.S. I recall some old correspondence on dataURI and dataURL for TASOC lightcurves (the values were different). The recommendation was to use dataURI. It looks like I dropped the ball on following it up.

from lightkurve.

A correction: lightkurve actually uses dataURL for logging purpose only. The logging logic still causes the error nonetheless.

The actual download is done by astroquery's Observations.download_product(), which always uses dataURI.

from lightkurve.

Quick update: we have identiied that this was the unexpected result of a change we made during the monthly patch. We'll be working on a fix to restore previous functionality.

Whether lightkurve should use dataURI or dataURL universally when looking at data product returns, and if so, the code changes to implement that can be treated as a separate lightkurve question. Some of the folks at MAST might be able to help confirm what the best practice should be.

from lightkurve.

In terms of the latter question: best practice is to always rely on dataURI. This is defined for every data product, and what you get after a get_data_products() query, for example.

The dataURL parameter is only available for the top-most (primary) product of an observation. It's a shortcut to check on or retrieve the primary product of an Observation without running a get_data_products() call. It works in some cases for datasets that lightkurve supports because some of them only have one product, but that is not the case generally.

I strongly recommend lightkurve get updated to retrieve all files using only the dataURI fields from a get_data_products()call, and not to rely on joining or using the dataURL from an Observations.query_criteria() search at the Observation level. I would not deem this an urgent issue, but it would make the codebase much more future-proof.

from lightkurve.

Thanks Scott for looking into it from MAST angle.

A summary of the situation:

1. When MAST resolves the issue with the patch, the issue will be resolved for all existing lightkurve users.
2. @christinahedges We should consider to release a patch for lightkurve nevertheless. See PR #1380
3. A clarification of the lightkurve bug: lightkurve misused dataURL in logging logic only. For the actual product download, the correct dataURI has been used via astroquery's Observations.download_product().

from lightkurve.