Implement variables and PointData for Snotel.
Ideally this will use Climata as much as possible https://github.com/heigeo/climata/pulls

Publish to pypi

• Setup pypi for M3Works
• Setup actions to publish to PyPi after successful build of main branch if the tag changed (or on a tag creation?)

• metloom version: 0.2.0
• Python version: 3.8
• Operating System: mac

Description

Sampling snotel station 'CESC1' around 2020-03-08 results in a timezone error
sensor_df["datetime"].apply(self._handle_df_tz) results in nonexistanttimeerror 2020-03-08 02:02:00
the tzinfo is 'America/Los_Angeles'

#21 (comment)

metloom currently assumes that there is only one polygon in a shapefile. This is not a safe assumption. One option could be doing a union of all polygons in a shapefile
#21 (comment)

• metloom version: 0.2.1
• Python version: 3.9
• Operating System: MacOS

Description

Using a small bounding box to grab mesowest stations and received an error

What I Did

Running this I receive this traceback

File "/home/user/envs/loomenv/lib/python3.9/site-packages/metloom/pointdata/mesowest.py", line 285, in points_from_geometry
  data = jdata['STATION']
KeyError: 'STATION'

It would appear mesowest doesn't consider receiving no stations back. The solution here is to return an empty list instead of trying to index on it.

SAIL data lives in the ARM website. They look to have a rest API

https://adc.arm.gov/armlive/
https://adc.arm.gov/discovery/#/

• metloom version: 0.2.2
• Python version: 3.8
• Operating System: OSX

Description

CDEC sometimes makes 15 minute data available when daily or hourly is not. We can resample to get the data we want. Example

https://cdec.water.ca.gov/dynamicapp/req/JSONDataServlet?Stations=FLV&SensorNums=18&dur_code=E&start_date=2021-11-01&end_date=2021-11-04

https://cdec.water.ca.gov/dynamicapp/req/JSONDataServlet?Stations=FLV&SensorNums=18&dur_code=D&start_date=2021-11-01&end_date=2021-11-04

This would broaden the list of sensors that we can actually retrieve data for

Creating this to continue a convo with Joe.

Would be cool to add in the csv data from the Irwin Study plot to be used from metloom.

There doesn't sound like a publicly available API or link but we could still make a class to read the csv that is often emailed around when updated. We could also add a variables class to map variables. Here is what I am picturing.

from metloom.pointdata import IrwinPointData 
from metloom.variables import IrwinVariables

pnt = IrwinPointData(csv=file)
df = pnt.get_daily_data(start, end, [IrwinVariables.SNOWDEPTH])

The variables are a straightforward mapping of their names to a more useful or readable name. e.g.

in variables.py

class IrwinVariables(VariableBase):
    SNOWDEPTH = InstrumentDescription("Sno_Height_M", "SNOWDEPTH")

While this effort wouldnt solve much in the way of getting the data, it would help standardize the use of individual energy balance sites that all seem to do things a scosh different.

NRCS has depth and temp stations available via the Snotel Lite (SNOTLITE) network. This is not documented on the AWDB docs, but is available via the "SNTLT" network code. Add this as something we find by default

• metloom version: 0.3.2
• Python version: 3.8.10
• Operating System: Ubuntu 20.04

Description

Pulling any CDEC station metadata is currently showing 404. After a long arduous discussion with the powers that be, I learned the link that is currently being used they deprecated 2 years ago and had a redirect to somewhere else. 2 Days ago they got rid of that redirect for security reasons. When I asked what is the designated webservice they would like folks to use for metadata they said there isn't one and offer their web interface as a solution.

What I Did

from metloom.pointdata import CDECPointData
pnt = CDECPointData('TUM', 'Tuolumne Meadows')
print(pnt.metadata)

yields an exception

requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http://cdec.water.ca.gov/cdecstation2/CDecServlet/getStationInfo?stationID=TUM

The recommended url to use now is https://cdec.water.ca.gov/dynamicapp/staMeta?station_id=TUM .

• metloom version: 0.2.3
• Python version: 3.8
• Operating System: Ubuntu

Description

Attempting to pull a mesowest station and I misspelled the station id. This results in 'STATION is not in the response data and thus throws an exception.

What I Did

from metloom.pointdata import MesowestPointData
from dateparser import parse

pnt = pnt = MesowestPointData('NOT_REAL_CODE', 'test')
pnt.get_daily_data(parse('2 days ago'), parse('today'), [pnt.ALLOWED_VARIABLES.TEMP])

This throws

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/micah/projects/m3works/metloom/metloom/pointdata/mesowest.py", line 280, in get_daily_data
    df = self._get_data(start_date, end_date, variables, interval='D')
  File "/home/micah/projects/m3works/metloom/metloom/pointdata/mesowest.py", line 119, in _get_data
    sensor_df = self._sensor_response_to_df(
  File "/home/micah/projects/m3works/metloom/metloom/pointdata/mesowest.py", line 183, in _sensor_response_to_df
    station_response = response_data['STATION']
KeyError: 'STATION'

Note the response data is:

{'SUMMARY': {'NUMBER_OF_OBJECTS': 0, 'RESPONSE_CODE': 2, 'VERSION': 'v2.13.8', 'RESPONSE_MESSAGE': 'No stations found for this request.', 'RESPONSE_TIME': 0}}

Question on where the NAD-83 EPSG code information came:

I needed a refresher myself today and came across this in the FAQ:

SNOTEL site coordinates in the WGS84 coordinate system (Latitude/Longitude) are available on our website

Comparing the values for 637:ID:SNTL, it looks like that this is also returned in the SOAP API metadata.

So is this conversion necessary?

• metloom version: 0.2.1
• Python version: 3.9
• Operating System: Mac

Description

Resample dataframe returns an empty dataframe when the original index does not contain values for the resampled index. This will show up when a sensor returns data at odd increments (non of which line up with an hour or day)

What I Did

def test_frequency():
    stat = MesowestPointData("KTRK", "Truckee-Tahoe")
    # returning nothing
    data = stat.get_hourly_data(
        datetime(2021, 11, 4), datetime(2021, 11, 8),
        [stat.ALLOWED_VARIABLES.TEMP]
    )
    print(data)
    assert data is not None

I ran into an issue where 585:WY:SNTL returns duplicate entries of the exact same value for hourly air temp. This should be simple to handle on the metloom side of things

As a newcomer to metloom, it is quite challenging to find specific classes or tests when trying to figure out execution logic. You can use your IDE (or preferred environment) to do a search and it sill leaves my head spinning when trying to navigate. Files also get quite long right now.

I am proposing three architectural changes/conventions:

• File names to match their class
• One class per file
• Each class test file name corresponds with "test_my_awesome_class"

I have followed this pattern in the past and it makes code navigation quite simpler (especially for newcomers). It also makes testing each class simpler and organizes your test suite the same manner as your sources.

What do you guys think?

I am happy to prototype this in a PR

Add a buffer parameter (assume same units as shapefile) when searching for stations to allow searches expanding past the shapefile bounding box

• metloom version: 0.2.11
• Python version: 3..8.10
• Operating System: Ubuntu 20.04

Description

I pulled GroundTemperature from the snotel network using get_daily_data and it will error out.

What I Did

pnt = SnotelPointData('505:CO:SNTL')
df = pnt.get_daily_data(
    datetime(2020, 1,1), datetime(2020, 1,2),
    [SnotelVariables.TEMPGROUND8IN]
)

  File "/home/micah/projects/m3works/metloom/metloom/pointdata/snotel.py", line 120, in _fetch_data_for_variables
    params = extra_params[variable.name]
KeyError: 'GROUND TEMPERATURE -8IN'

Process finished with exit code 1

With the change to the HTML page for CDEC metadata, some stations return a different number of tables. We could handle this in a more robust way because currently we cannot parse the sensor metadata for TRK.

https://cdec.water.ca.gov/dynamicapp/staMeta?station_id=TRK

• Setup M3Works on readthedocs
• Configure readthedocs (https://github.com/SnowEx/snowexsql/blob/master/.readthedocs.yaml)

Looking to add a csv reader for the SnowEx Stations and the Senator Beck stations.

Description

It would be nice to have the same mapped features for using flat files that we encounter with SnowEx and with Senator Beck.

• metloom version: 0.1.0
• Python version: 3
• Operating System: mac, ubuntu

Description

When installing metloom with the latest setuptools, climata fails to build. Climata depends on suds-jurko which uses deprecated setuptools methods. Suds-jurko appears unmaintained. Since we only have limited dependencies on Climata, the best course of action would be to abandon the dependency and rewrite the parts we need with a more current SOAP client

We should:

• Implement hourly data function in the USGS loader
• Improve the search to be for daily and instantaneous variables
• Add resampling fallbacks to all methods

It would be very useful to have a class that handles 7 day forecasts form the NWS

Here is the API https://www.weather.gov/documentation/services-web-api

We can call the points api to get the URL for the forecast https://api.weather.gov/points/42,-119

In this example, 3 forecast URLs are available

"forecast": "https://api.weather.gov/gridpoints/BOI/28,28/forecast",
"forecastHourly": "https://api.weather.gov/gridpoints/BOI/28,28/forecast/hourly",
"forecastGridData": "https://api.weather.gov/gridpoints/BOI/28,28",

forecast can be used to return the 12 hour increments

forecastHourly returns hourly data (hourly in local tz)

forecastGridData returns the 'raw' grid data (hourly UTC). This is likely the easiest version for us to use in the API. We can aggregate to daily as needed.

• metloom version: 0.3.5

Description

2.0 Shapely offers a lot! We should do it just for testing advantages. But looks to be some changes to be aware of. Add it to the summer list!

https://shapely.readthedocs.io/en/stable/migration.html

Looks like the soil moisture could be added to this list for stations?