Giter VIP home page Giter VIP logo

tedpearson / forecastmetrics Goto Github PK

View Code? Open in Web Editor NEW
16.0 2.0 4.0 2.52 MB

Get weather forecasts and write them into VictoriaMetrics (or InfluxDB), for use with Grafana dashboards. Currently supports the National Weather Service (NWS) and VisualCrossing forecast APIs.

License: MIT License

Go 100.00%
weather forecast-data forecast-dashboards victoriametrics national-weather-service visual-crossing grafana golang

forecastmetrics's Introduction

ForecastMetrics

ForecastMetrics is a tool to store forecast data from multiple sources in VictoriaMetrics or InfluxDB.

Features:

  • Hourly forecast updates written to Influx or VictoriaMetrics for multiple locations
  • Http server implementing the prometheus query_range endpoint, allowing it to be used as a Prometheus data source for adhoc forecasts in Grafana.

I currently use VictoriaMetrics as my time series database. Because of that, this project does a few things specifically to support it:

  • Writes hourly sunup information
  • Uses no retention policies (not supported in VictoriaMetrics)
  • No overwrite of metrics
    • Every forecast is a new tag/label, since VictoriaMetrics doesn't support overwriting metrics as Influx does.
    • Past data is written one data point per hour, also because overwriting data is unsupported.
    • An alternative mode can be enabled by setting overwrite_data to true in the config file. In this mode, there will only be one forecast series per source/location.

Currently supported sources:

  • National Weather Service (NWS) (US-only)
  • VisualCrossing (Global)
  • No other sources planned at this time, due to not meeting the below criteria (7 day hourly forecast, reasonably priced or free)
  • Open an issue if you find a worthy source!

Usage:

Install

  • Download a binary from the latest Release if your architecture is available

    curl -O https://github.com/tedpearson/ForecastMetrics/releases/download/v4.1.0/forecastmetrics-linux-arm64
    
  • Make the binary executable

    chmod +x forecastmetrics-linux-arm64
    
  • If your architecture is not avaialable, you'll need to build from source:

    • Clone this repo
    • Install Go
    • cd ForecastMetrics
      go build
      

Config

  • Get the example application and location configs:

    curl https://raw.githubusercontent.com/tedpearson/ForecastMetrics/master/forecastmetrics.example.yaml > forecastmetrics.yaml
    curl https://raw.githubusercontent.com/tedpearson/ForecastMetrics/master/locations.example.yaml > locations.yaml
    
  • Modify the configs with your own values for:

    • locations
    • influxdb/victoriametrics connection
      • if using influxdb, you may set overwrite_data to true, creating only a single series for each source/location.
    • desired influx measurement names (metrics prefixes for victoriametrics)
    • which weather sources to enable
    • add your own key for Visualcrossing, if desired
    • server config for ad-hoc forecasts:
      • Set the port the server should listen on (set to 0 to disable the server)
      • Insert your own Azure Maps Shared Key (requires Azure Maps account. There is a free tier.)

Ad-hoc Forecasts Setup

Since version 4.0, ForecastMetrics supports use as a prometheus data source in grafana for getting ad-hoc weather forecasts for any location.

  • Only simple queries are supported, no functions or other features
  • Queries should look like this: forecast_metricname{source="nws",location="place"}
  • The location tag supports these formats:
    • place name
    • lat,lon
    • place name|nickname
    • lat,lon|nickname
  • An optional tag save is also supported. if save="true", ForecastMetrics will add it to locations.yaml and update the metric every hour.
    • The locations.yaml file needs to be writable by the user running the process for this to work.
  • To add as a data source to Grafana, add as a Prometheus data source. When you save, there will be an error about "404 Not Found - There was an error returned querying the Prometheus API." You can ignore this error and proceed to configuring a dashboard.

Run

Run the binary like this:

./forecastmetrics --config forecastmetrics.yaml --locations locations.yaml

Grafana Dashboard

I've included definitions for my grafana dashboard in the repo, both for InfluxDB and VictoriaMetrics which I now use. Here are screenshots of each in use. I use this dashboard daily for my local weather forecast.

Influx Dashboard

influx grafana dashboard

VictoriaMetrics Dashboard

VictoriaMetrics dashboard

Rationale behind included/planned sources:

I was looking for a replacement for DarkSky, who were bought by Apple and retired their API in 2021 2022 2023. DarkSky had the best forecasts and a generous free version, with 7 days of forecast data available.

I used the DarkSky data to power my own visualizations of my local forecast in Grafana. I find my Grafana graphs of forecast data much more intuitive than any weather app or website out there. I display the 7 day forecast for temps, precip, wind, and clouds, on the same graph with 7 days of actual data history from my Ambient Weather personal weather station, and also the forecast from 24 hours previous. (You can find my Ambient Weather exporter here.)

So when I went looking for replacements I needed these features:

  • At least 7 days of HOURLY forecast data. Daily highs and lows are not very interesting to look at in a graph.
  • I preferred Free APIs or APIs allowing at least 1500 forecasts per month, as I only made <200 calls/day to DarkSky, and paying large amounts for my personal forecast dashboard is just silly.
    • This is why visualcrossing is a supported source, because their free tier supports 250 forecasts/day.

forecastmetrics's People

Contributors

dependabot[bot] avatar tedpearson avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

forecastmetrics's Issues

Question about retention/data overwrite - no issue

First of all thanks a lot of this code. I have been using this on a Raspberry Pi with Influxdb and Grafana and it has been working really well (I copied your graphs, really liked the way you presented the data). I'm in the process of moving the setup to a home-built NAS running OpenMediaVault and putting InfluxDB and Grafana into Docker containers so I want to make sure I do this properly for the long run.

On the RPi I have been running ForecastMetrics as a cron job every hour. It was just something I figured would work. But now that I'm reading up on this a little more I'm confused as to how much data will accumulate in InfluxDB since it doesn't look like old data is being overwritten and there is no retention policy (I'm using 1 year in InfluxDB). Originally when I was setting this up it didn't look like unnecessary data was being accumulated but now I'm not sure. What is your suggestion on how to populate an InfluxDB database?
If the answer is just a cron job I may see if it makes sense to put it inside a container (not sure why, just to see if I can do it as I'm learning to use containers). Any obvious issues with that?

Thanks

H

404 Not Found

Hi
My config is setup as follow
pi@ra-helios-stats:~/weather $ cat config/weather2influxdb.yaml
locations:

  • name: xxxxxx
    latitude: -2x.xxxx
    longitude: 2x.xxxx

influxdb:
host: http://localhost:8086
user: weather
password: xxxxxxxxxxxx
database: weather

forecast:
measurement_name: forecast
history:
enabled: yes
retention_policy: history_rp
measurement_name: forecast_history

astronomy:
enabled: no
measurement_name: astronomy

sources:
enabled:
- visualcrossing
visualcrossing:
key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

http_cache_dir: /tmp/weather2influxdb-cache
pi@ra-helios-stats:~/weather $

The Grafana sources are connected and tested ok
But when I run the binary I get

pi@ra-helios-stats:~/weather $ ./weather2influxdb-linux-arm
2021/04/19 15:55:59 Getting VisualCrossing forecast
2021/04/19 15:56:01 Writing 378 points to "forecast" in InfluxDB for "visualcrossing"
2021/04/19 15:56:01 influxdb2client E! Write error: 404 Not Found: 404 page not found

2021/04/19 15:56:01 404 Not Found: 404 page not found

github.com/tedpearson/weather2influxdb/influx.(*Writer).WriteMeasurements
/home/runner/work/weather2influxdb/weather2influxdb/influx/influx.go:29
main.App.RunForecast
/home/runner/work/weather2influxdb/weather2influxdb/main.go:86
main.main
/home/runner/work/weather2influxdb/weather2influxdb/main.go:53
runtime.main
/opt/hostedtoolcache/go/1.15.7/x64/src/runtime/proc.go:204
runtime.goexit
/opt/hostedtoolcache/go/1.15.7/x64/src/runtime/asm_arm.s:857
2021/04/19 15:56:01 Writing 378 points to "forecast_history" in InfluxDB for "visualcrossing"
2021/04/19 15:56:01 influxdb2client E! Write error: 404 Not Found: 404 page not found

2021/04/19 15:56:01 404 Not Found: 404 page not found

github.com/tedpearson/weather2influxdb/influx.(*Writer).WriteMeasurements
/home/runner/work/weather2influxdb/weather2influxdb/influx/influx.go:29
main.App.RunForecast
/home/runner/work/weather2influxdb/weather2influxdb/main.go:99
main.main
/home/runner/work/weather2influxdb/weather2influxdb/main.go:53
runtime.main
/opt/hostedtoolcache/go/1.15.7/x64/src/runtime/proc.go:204
runtime.goexit
/opt/hostedtoolcache/go/1.15.7/x64/src/runtime/asm_arm.s:857
pi@ra-helios-stats:~/weather $

Could you point me in the right direction please

web key

i think you want to remove your web key from the code.,..

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.