Cannot get all COOPS stations (e.g. some in Hawaii) about searvey HOT 20 CLOSED

I just figured out there's a nice way to automatically get the station detailed metadata along with a single original metdata request using the url https://api.tidesandcurrents.noaa.gov/mdapi/prod/webapi/stations.json?expand=details. This results in something like:

{
  "count": 301,
  "units": null,
  "stations": [
    {
      "tidal": true,
      "greatlakes": false,
      "shefcode": "NWWH1",
      "details": {
        "id": "1611400",
        "established": "1954-11-24 00:00:00.0",
        "removed": "",
        "noaachart": "19383",
        "timemeridian": 0,
        "timezone": -10.0,
        "origyear": "1991-02-16 00:00:00.0",
        "self": "https://api.tidesandcurrents.noaa.gov/mdapi/prod/webapi/stations/1611400/details.json"
      },

instead of

{
  "count": 301,
  "units": null,
  "stations": [
    {
      "tidal": true,
      "greatlakes": false,
      "shefcode": "NWWH1",
      "details": {
        "self": "https://api.tidesandcurrents.noaa.gov/mdapi/prod/webapi/stations/1611400/details.json"
      },

from searvey.

Code to parse the JSON file:

df_ = pd.read_json("https://api.tidesandcurrents.noaa.gov/mdapi/prod/webapi/stations.json")
df = pd.DataFrame.from_records(df_["stations"])
len(df)

get_coops_stations() return 443 records (including discontinued ones). The JSON file contains 301 stations. The COOPS/NWS seems to mostly be a superset of the main COOPS product but it is not a strict superset. E.g. the following stations do not exist on COOPS/NWS: 8771367, 9755968, 9758066

Do we just merge? Do we create a different function?

Another issue is that the column names are different, so we also need to decide if we want to rename column names or not.

As far as performance goes, I feel that making an extra request per station shouldn't be the default. If it's needed then we should have a boolean argument for it or something.

I think we need to spend a bit more time on this before we decide on a course of action

from searvey.

At least locally, you can disable specific pre-commit checks by defining the SKIP env variable. For example:

SKIP=check-added-large-files git commit -m '...'

I am not sure what to suggest for the CI integration though. If nothing else works, we might have to edit the .pre-commit-config.yml file to disable the check, make the commit and and re-enable it afterwards. Or we could just disable the check alltogether.

source:

• https://pre-commit.com/#temporarily-disabling-hooks
• https://pre-commit.ci/#configuration # check the skip key

from searvey.

I just realized a strange issue on COOPS part. Although the missing stations data single station metadata can be retrieved from the MD API, e.g. 9411399, the station metadata list does not have the station listed! I thought this was only an issue with the COOPS/NWS product, but it seem that their main product metadata list also has the same missing stations! I am adding the main API anyway for now, so that when they fix the issue on their side we get full list.

@AtiehAlipour-NOAA would it be possible that you please follow up this issue with someone on the COOPS side to see why their station metadata API (documented https://api.tidesandcurrents.noaa.gov/mdapi/prod/) is missing those stations?

Correction:
I made a correction about station not being available from metadata API. The correct statement is that the station is missing from the metadata listing while it still is available to view in the metadata API if you check the specific station url!

from searvey.

Thank you very much, @SorooshMani-NOAA for your help. I greatly appreciate it. This is also a relevant and helpful tool to construct different API calls from COOPS: https://tidesandcurrents.noaa.gov/api-helper/url-generator.html

from searvey.

I am not sure this is a searvey issue or a COOPS one.

The stations that are returned by COOPS in the Hawaii region are 7 and they are all returned. This can be checked here: https://access.co-ops.nos.noaa.gov/nwsproducts.html?type=current (filter by H1)

from searvey.

@pmav99 thanks for looking into this. I suspected this could be a coops issue or at least their API issue, but I never had time to check.

I just realized based on the comment in the webpage where we fetch station metadata, it seems it's different from the coops main product, I need to look into this:

(This page is for NOAA CO-OPS/NWS support only, for CO-OPS main products please visit to the CO-OPS web site.)

https://access.co-ops.nos.noaa.gov/nwsproducts.html vs https://tidesandcurrents.noaa.gov/

Update
We should use this instead for getting metadata: https://api.tidesandcurrents.noaa.gov/mdapi/prod/#intro

from searvey.

The main issue I see with using COOPS API for station metadata is the lack of information for whether a station is active or not. To get that information, I need to do separate details query for each station to get the removed date (empty if not applicable). @pmav99 do you suggest I go that route?

This is all of the stations data:
https://api.tidesandcurrents.noaa.gov/mdapi/prod/webapi/stations.json

This is details for one station:
https://api.tidesandcurrents.noaa.gov/mdapi/prod/webapi/stations/9414750/details.json

The API has a lot more data than what we get from the NWS table at https://access.co-ops.nos.noaa.gov/nwsproducts.html, but for now I keep the station metadata limited to what we used to get from the NWS table. The only missing piece now is information about whether a station is active or not, for which I seem to need one query per station!

from searvey.

Thanks for the detailed description of what we're looking at. As far as searvey is concerned, do we want to keep the active/inactive info for stations in the metadata? If not we can make it optional as you suggested.

I actually don't know about the discrepancy in the station counts, maybe I should ask someone at COOPS about difference between their main and NWS products.

@saeed-moghimi-noaa @brey what do you think?

from searvey.

@AtiehAlipour-NOAA noticed another station that is missing from the current metadata url: 8662245

This can be found here: https://api.tidesandcurrents.noaa.gov/mdapi/prod/webapi/stations/8662245.json, but not at https://access.co-ops.nos.noaa.gov/nwsproducts.html?type=current as discussed here before.

from searvey.

stations_noData.txt
@SorooshMani-NOAA, here is the list of ~130 stations with the same issue. Thanks a lot for your help.

from searvey.

@pmav99 given the newfound missing stations, I think it makes more sense to just use the COOPS API metadata rather than the NWS product. In any case we are using the COOPS API for the data, so if it doesn't have the station in metadata, it might not have the data for those stations either (whether current or historical).

from searvey.

fine with me

from searvey.

@pmav99 for #123 I'm adding new tests and renaming older tests. Then I'm recording cassettes for the requests. Now when I try to commit I get this error:

check for added large files..............................................Failed
- hook id: check-added-large-files
- exit code: 1

tests/cassettes/coops_test/test_coops_stations_new_api.yaml (943 KB) exceeds 500 KB.
tests/cassettes/coops_test/test_coops_stations_within_region_new_api.yaml (943 KB) exceeds 500 KB.
tests/cassettes/coops_test/test_coops_stations_within_region_old_api.yaml (1384 KB) exceeds 500 KB.

What should I do?

from searvey.

Thank you for your suggestions! Since a couple of the existing cassettes are larger that 500KB anyway, the best way is to just disable it for cassettes, or change the threshold maybe(?)! I'll try to dig a little bit to see if there's any obvious way to do either.

from searvey.

I just realized a strange issue on COOPS part. Although the missing stations data can be retrieved from the API, e.g. 9411399, the station metadata list does not have the station listed! I thought this was only an issue with the COOPS/NWS product, but it seem that their main product metadata list also has the same missing stations! I am adding the main API anyway for now, so that when they fix the issue on their side we get full list.

@AtiehAlipour-NOAA would it be possible that you please follow up this issue with someone on the COOPS side to see why their station metadata API (documented https://api.tidesandcurrents.noaa.gov/mdapi/prod/) is missing those stations?

Good question !

from searvey.

Looking at the expanded view of the same station 9411399 I see:

"details": {
        "id": "9411399",
        "established": "2012-10-31 00:00:00.0",
        "removed": "2013-03-09 23:59:00.0",
        "noaachart": "18721",
        "timemeridian": -120,
        "timezone": -8.0,
        "origyear": "2012-11-10 00:00:00.0",
        "self": "https://api.tidesandcurrents.noaa.gov/mdapi/prod/webapi/stations/9411399/details.json"
      },

which shows that this station has been removed in 2013. All the stations listed in the metadata API seem to be active (no removed data). So I guess their API for stations don't list the discontinued ones, or maybe we need to query it differently to get those inactive stations. I'm also having a conversation with @AtiehAlipour-NOAA about this, we'll report any relevant outcome here.

from searvey.

Thanks to @AtiehAlipour-NOAA for this information:

By default, if you do not specify a type= in the MDAPI station listing call, you will get the list of active water level stations, i.e. https://api.tidesandcurrents.noaa.gov/mdapi/prod/webapi/stations.json?type=waterlevels&expand=details

Since the station we listed are historic, non-active stations, they would not be available at that URL, but only at the historic WL MDAPI listing

https://api.tidesandcurrents.noaa.gov/mdapi/prod/webapi/stations.xml?type=historicwl&expand=details

from searvey.

After getting the historical data stations, I get ~3000 stations total, but it's still missing ~20 of the 130 stations in the list!

array([8411696, 8414672, 8429489, 8462752, 8462925, 8515421, 8516925,
       8517394, 8518919, 8630440, 8654792, 8579629, 8579381, 8634689,
       8634858, 8675365, 9413745, 9415625, 9418851, 9468132, 8416828])

from searvey.

@mojy3995, if interested, you can test this by calling the new API to get the stations:

bx = box(-180, -18.56, -137.52, 33)
df = coops.get_coops_stations(region=bx, metadata_source='main')
df

which yields:

        nws_id                                      name                     state  ...             removed        status                      geometry
nos_id                                                                              ...
1611400  NWWH1                                Nawiliwili                        HI  ...                 NaT        active   POINT (-159.35609 21.95440)
1612340  OOUH1                                  Honolulu                        HI  ...                 NaT        active   POINT (-157.86453 21.30333)
1612401  PRHH1                              Pearl Harbor                        HI  ...                 NaT        active   POINT (-157.96390 21.36750)
1612480  MOKH1                                  Mokuoloe                        HI  ...                 NaT        active   POINT (-157.78999 21.43306)
1615680  KLIH1                   Kahului, Kahului Harbor                        HI  ...                 NaT        active   POINT (-156.46899 20.89494)
1617433  KWHH1                                  Kawaihae                        HI  ...                 NaT        active   POINT (-155.82941 20.03660)
1617760  ILOH1                 Hilo, Hilo Bay, Kuhio Bay                        HI  ...                 NaT        active   POINT (-155.06000 19.73028)
1619910  SNDP5               Sand Island, Midway Islands  United States of America  ...                 NaT        active   POINT (-177.36000 28.21167)
1770000  NSTP6                 Pago Pago, American Samoa            American Samoa  ...                 NaT        active  POINT (-170.69000 -14.28000)
1611347   <NA>    PORT ALLEN, HANAPEPE BAY, KAUAI ISLAND                        HI  ... 1997-02-12 00:00:00  discontinued   POINT (-159.59200 21.90330)
1612366   <NA>  FORT KAMEHAMEHA, BISHOP POINT, PEARL HBR                        HI  ... 1987-03-24 23:59:00  discontinued   POINT (-157.96700 21.33000)
1612404   <NA>     FORD ISLAND, FERRY DOCK, PEARL HARBOR                        HI  ... 1987-03-23 23:59:00  discontinued   POINT (-157.94000 21.36830)
1612479   <NA>  MOKUOLOE, KANEONE BAY, OAHU IS. (BACKUP)                        HI  ... 1991-01-29 23:59:00  discontinued   POINT (-157.79201 21.43500)
1612702   <NA>                    LAIEMALOO, OAHU ISLAND                        HI  ... 1983-12-01 23:59:00  discontinued   POINT (-157.92200 21.63670)
1613198   <NA>                         Kaunakakai Harbor                        HI  ... 2001-01-28 23:59:00  discontinued   POINT (-157.03200 21.08500)
1614465   <NA>           KAUMALAPAU HARBOR, LANAI ISLAND                        HI  ... 2015-03-23 23:59:00  discontinued   POINT (-156.99500 20.79000)
1619000   <NA>                            Johnston Atoll  United States of America  ... 2003-02-19 23:59:00  discontinued   POINT (-169.53000 16.73830)
1732417   <NA>                            Fare Ute Point                            ... 2002-09-05 23:59:00  discontinued  POINT (-149.57201 -17.53500)

[18 rows x 8 columns]

from searvey.