smartemission / smartemission Goto Github PK

View Code? Open in Web Editor NEW

9.0 10.0 6.0 122.53 MB

Smart Emission Data Platform - data harvesting, processing and OGC services (WMS, WFS, SOS, STA) for sensor data

Home Page: http://data.smartemission.nl

License: GNU General Public License v3.0

HTML 19.96% Python 4.86% Shell 40.34% CSS 0.18% JavaScript 0.88% PHP 3.66% PLpgSQL 22.13% Dockerfile 7.98%

air-quality environment ann gis osgeo sensor-network ogc sos grafana-influxdb python

smartemission's Introduction

smartemission

NB all resources/software here is now being moved to a new GitHub Organization for SE: https://github.com/smartemission

"The Smart Emission Data Platform assembles, processes and unlocks sensor data for Air Quality, Sound, and Meteo from citizen-based sensor networks."

This repository contains sources and documentation for

(raw) sensor data publication (push) and acquisition (pull/harvesting)
raw sensor data processing (validation, calibration, aggregation, ETL)
unlocking sensor data as Open Data via standardized (OGC) web-APIs (WMS, WFS, SOS, SensorThings API)

Home page for the SE Data Platform is: http://data.smartemission.nl

The SE Data Platform originates from the Smart Emission Nijmegen project (2015+). The landing page for that project is http://smartemission.nl.

Documentation for the Smartemission Data Platform is generated from this repo and can be found at: http://smartplatform.readthedocs.org

smartemission's People

Contributors

Stargazers

Watchers

Forkers

vermeuleng bertt internetofwater tvsltd keshava leevydanomalik

smartemission's Issues

Check calibration still ok after upgrades

Calibration has been updated on the test server to the latest GitHub version. Also the Stetl Docker image has been rebuilt to the latest SciPi (github version, see Docker file: https://github.com/Geonovum/smartemission/blob/master/docker/stetl/Dockerfile). The data conversion for last values shows quite some new warnings and errors though. May be mismatches between (pickled) versions? See e.g. the log output for generated last values for the SmartApp (http://test.smartemission.nl/smartapp/):

root@SETEST:~/git/etl# ./last.sh 
2016-09-24 17:26:15,093 util INFO Found cStringIO, good!
2016-09-24 17:26:15,109 util INFO Found lxml.etree, native XML parsing, fabulous!
2016-09-24 17:26:15,149 util INFO Found GDAL/OGR Python bindings, super!!
2016-09-24 17:26:15,157 main INFO Stetl version = 1.1.0rc2
2016-09-24 17:26:15,158 main INFO Found args file at: options/docker.args
2016-09-24 17:26:15,158 ETL INFO INIT - Stetl version is 1.1.0rc2
2016-09-24 17:26:15,158 ETL INFO Config/working dir = /root/git/etl
2016-09-24 17:26:15,158 ETL INFO Reading config_file = last.cfg
2016-09-24 17:26:15,158 ETL INFO Substituting 18 args in config file from args_dict: OrderedDict([('__name__', 'asection'), ('raw_device_url', 'http://whale.citygis.nl/sensors/v1'), ('pg_host', 'postgis'), ('pg_user', 'docker'), ('pg_password', 'docker'), ('pg_port', '5432'), ('pg_database', 'gis'), ('pg_schema', 'smartem'), ('pg_schema_rt', 'smartem_rt'), ('pg_schema_raw', 'smartem_raw'), ('pg_schema_refined', 'smartem_refined'), ('refiner_max_input_records', '4000'), ('publisher_max_input_records', '3000'), ('max_proc_time_secs', '600'), ('api_interval_secs', '1'), ('sos_host', 'sos52n'), ('sos_port', '8080'), ('sos_path', '/sos52n/service')])
2016-09-24 17:26:15,159 ETL INFO Substituting args OK
2016-09-24 17:26:15,159 ETL INFO START
2016-09-24 17:26:15,159 util INFO Timer start: total ETL
2016-09-24 17:26:15,159 chain INFO Assembling Chain: input_raw_sensor_last_api|output_postgres_insert...
/usr/local/lib/python2.7/dist-packages/sklearn/base.py:315: UserWarning: Trying to unpickle estimator StandardScaler from version pre-0.18 when using version 0.19.dev0. This might lead to breaking code or invalid results. Use at your own risk.
  UserWarning)
/usr/local/lib/python2.7/dist-packages/sklearn/base.py:315: UserWarning: Trying to unpickle estimator MLPRegressor from version pre-0.18 when using version 0.19.dev0. This might lead to breaking code or invalid results. Use at your own risk.
  UserWarning)
/usr/local/lib/python2.7/dist-packages/sklearn/base.py:315: UserWarning: Trying to unpickle estimator Pipeline from version pre-0.18 when using version 0.19.dev0. This might lead to breaking code or invalid results. Use at your own risk.
  UserWarning)
2016-09-24 17:26:15,405 input INFO cfg = {'url': 'http://whale.citygis.nl/sensors/v1', 'api_interval_secs': '0', 'class': 'rawsensorapi.RawSensorLastInput', 'sensor_names': 'temperature,humidity,pressure,noiseavg,noiselevelavg,co2,co,no2,o3,coraw,no2raw,o3raw'}
2016-09-24 17:26:15,405 httpinput INFO url=http://whale.citygis.nl/sensors/v1 parameters=None
2016-09-24 17:26:15,406 output INFO cfg = {'input_format': 'record_array', 'database': 'gis', 'replace': 'True', 'host': 'postgis', 'user': 'docker', 'key': 'unique_id', 'table': 'last_device_output', 'password': 'docker', 'class': 'outputs.dboutput.PostgresInsertOutput', 'schema': 'smartem_rt'}
2016-09-24 17:26:15,406 chain INFO Running Chain: input_raw_sensor_last_api|output_postgres_insert
2016-09-24 17:26:15,406 RawSensorAPI INFO Init: fetching device list from URL: "http://whale.citygis.nl/sensors/v1/devices" ...
2016-09-24 17:26:15,429 RawSensorAPI INFO Found   64 devices: [0, 1, 7, 8, 9, 10, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 75662, 828231, 3487232]
2016-09-24 17:26:15,429 dboutput INFO Init: connect to DB
2016-09-24 17:26:15,429 postgis INFO Connecting to dbname=gis user=docker host=postgis port=5432
2016-09-24 17:26:15,438 postgis DEBUG Connected to database gis
2016-09-24 17:26:15,525 RawSensorAPI WARNING device_id=8 sensor=temperature: invalid input for s_temperatureambient: detail=s_temperatureambient not present
2016-09-24 17:26:15,525 RawSensorAPI WARNING device_id=8 sensor=humidity: invalid input for s_humidity: detail=s_humidity not present
2016-09-24 17:26:15,525 RawSensorAPI WARNING device_id=8 sensor=pressure: invalid input for s_barometer: detail=s_barometer not present
2016-09-24 17:26:15,525 RawSensorAPI WARNING device_id=8 sensor=noiseavg: invalid input for ['v_audio0', 'v_audioplus1', 'v_audioplus2', 'v_audioplus3', 'v_audioplus4', 'v_audioplus5', 'v_audioplus6', 'v_audioplus7', 'v_audioplus8', 'v_audioplus9']: detail=v_audio0 not present
2016-09-24 17:26:15,525 RawSensorAPI WARNING device_id=8 sensor=noiselevelavg: invalid input for noiseavg: detail=noiseavg not present
2016-09-24 17:26:15,525 RawSensorAPI WARNING device_id=8 sensor=co2: invalid input for s_co2: detail=s_co2 not present
2016-09-24 17:26:15,525 RawSensorAPI WARNING device_id=8 sensor=co: invalid input for ['s_o3resistance', 's_no2resistance', 's_co2', 's_temperatureambient', 's_temperatureunit', 's_humidity', 's_barometer', 's_lightsensorbottom']: detail=s_o3resistance not present
2016-09-24 17:26:15,525 RawSensorAPI WARNING device_id=8 sensor=no2: invalid input for ['s_o3resistance', 's_no2resistance', 's_co2', 's_temperatureambient', 's_temperatureunit', 's_humidity', 's_barometer', 's_lightsensorbottom']: detail=s_o3resistance not present
2016-09-24 17:26:15,525 RawSensorAPI WARNING device_id=8 sensor=o3: invalid input for ['s_o3resistance', 's_no2resistance', 's_co2', 's_temperatureambient', 's_temperatureunit', 's_humidity', 's_barometer', 's_lightsensorbottom']: detail=s_o3resistance not present
2016-09-24 17:26:15,525 RawSensorAPI WARNING device_id=8 sensor=coraw: invalid input for ['s_coresistance']: detail=s_coresistance not present
2016-09-24 17:26:15,526 RawSensorAPI WARNING device_id=8 sensor=no2raw: invalid input for ['s_no2resistance']: detail=s_no2resistance not present
2016-09-24 17:26:15,526 RawSensorAPI WARNING device_id=8 sensor=o3raw: invalid input for ['s_o3resistance']: detail=s_o3resistance not present
2016-09-24 17:26:15,550 RawSensorAPI WARNING device_id=9 sensor=co2: invalid input for s_co2: detail=s_co2 not present
2016-09-24 17:26:15,550 RawSensorAPI WARNING device_id=9 sensor=co: invalid input for ['s_o3resistance', 's_no2resistance', 's_co2', 's_temperatureambient', 's_temperatureunit', 's_humidity', 's_barometer', 's_lightsensorbottom']: detail=s_o3resistance not present
2016-09-24 17:26:15,550 RawSensorAPI WARNING device_id=9 sensor=no2: invalid input for ['s_o3resistance', 's_no2resistance', 's_co2', 's_temperatureambient', 's_temperatureunit', 's_humidity', 's_barometer', 's_lightsensorbottom']: detail=s_o3resistance not present
2016-09-24 17:26:15,550 RawSensorAPI WARNING device_id=9 sensor=o3: invalid input for ['s_o3resistance', 's_no2resistance', 's_co2', 's_temperatureambient', 's_temperatureunit', 's_humidity', 's_barometer', 's_lightsensorbottom']: detail=s_o3resistance not present
2016-09-24 17:26:15,550 RawSensorAPI WARNING device_id=9 sensor=coraw: invalid input for ['s_coresistance']: detail=s_coresistance not present
2016-09-24 17:26:15,550 RawSensorAPI WARNING device_id=9 sensor=no2raw: invalid input for ['s_no2resistance']: detail=s_no2resistance not present
2016-09-24 17:26:15,550 RawSensorAPI WARNING device_id=9 sensor=o3raw: invalid input for ['s_o3resistance']: detail=s_o3resistance not present
2016-09-24 17:26:15,550 dboutput INFO query is INSERT INTO last_device_output (name,point,value_raw,altitude,value,label,unit,value_stale,time,device_name,unique_id,device_id) VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)
2016-09-24 17:26:15,564 dboutput INFO committed 5 records
2016-09-24 17:26:15,587 RawSensorAPI WARNING device_id=10 sensor=noiseavg: invalid input for ['v_audio0', 'v_audioplus1', 'v_audioplus2', 'v_audioplus3', 'v_audioplus4', 'v_audioplus5', 'v_audioplus6', 'v_audioplus7', 'v_audioplus8', 'v_audioplus9']: detail=v_audio0 not present
2016-09-24 17:26:15,587 RawSensorAPI WARNING device_id=10 sensor=noiselevelavg: invalid input for noiseavg: detail=noiseavg not present
2016-09-24 17:26:15,587 RawSensorAPI WARNING device_id=10 sensor=co: invalid input for ['s_o3resistance', 's_no2resistance', 's_co2', 's_temperatureambient', 's_temperatureunit', 's_humidity', 's_barometer', 's_lightsensorbottom']: detail=s_o3resistance not present
2016-09-24 17:26:15,587 RawSensorAPI WARNING device_id=10 sensor=no2: invalid input for ['s_o3resistance', 's_no2resistance', 's_co2', 's_temperatureambient', 's_temperatureunit', 's_humidity', 's_barometer', 's_lightsensorbottom']: detail=s_o3resistance not present
2016-09-24 17:26:15,587 RawSensorAPI WARNING device_id=10 sensor=o3: invalid input for ['s_o3resistance', 's_no2resistance', 's_co2', 's_temperatureambient', 's_temperatureunit', 's_humidity', 's_barometer', 's_lightsensorbottom']: detail=s_o3resistance not present
2016-09-24 17:26:15,587 RawSensorAPI WARNING device_id=10 sensor=coraw: invalid input for ['s_coresistance']: detail=s_coresistance not present
2016-09-24 17:26:15,587 RawSensorAPI WARNING device_id=10 sensor=no2raw: invalid input for ['s_no2resistance']: detail=s_no2resistance not present
2016-09-24 17:26:15,587 RawSensorAPI WARNING device_id=10 sensor=o3raw: invalid input for ['s_o3resistance']: detail=s_o3resistance not present
2016-09-24 17:26:15,609 RawSensorAPI WARNING device_id=12 sensor=temperature: invalid input for s_temperatureambient: detail=s_temperatureambient: val(0) < min(233150)
2016-09-24 17:26:15,609 RawSensorAPI WARNING device_id=12 sensor=pressure: invalid input for s_barometer: detail=s_barometer: val(0) < min(20000)
2016-09-24 17:26:15,609 RawSensorAPI WARNING device_id=12 sensor=co: invalid input for ['s_o3resistance', 's_no2resistance', 's_co2', 's_temperatureambient', 's_temperatureunit', 's_humidity', 's_barometer', 's_lightsensorbottom']: detail=s_temperatureambient: val(0) < min(233150)
2016-09-24 17:26:15,609 RawSensorAPI WARNING device_id=12 sensor=no2: invalid input for ['s_o3resistance', 's_no2resistance', 's_co2', 's_temperatureambient', 's_temperatureunit', 's_humidity', 's_barometer', 's_lightsensorbottom']: detail=s_temperatureambient: val(0) < min(233150)
2016-09-24 17:26:15,610 RawSensorAPI WARNING device_id=12 sensor=o3: invalid input for ['s_o3resistance', 's_no2resistance', 's_co2', 's_temperatureambient', 's_temperatureunit', 's_humidity', 's_barometer', 's_lightsensorbottom']: detail=s_temperatureambient: val(0) < min(233150)
2016-09-24 17:26:15,610 RawSensorAPI WARNING device_id=12 sensor=coraw: invalid input for ['s_coresistance']: detail=s_coresistance: val(2110698) > max(1500000)
2016-09-24 17:26:15,616 dboutput INFO committed 6 records
2016-09-24 17:26:15,636 RawSensorAPI WARNING id=13-no2  invalid output for no2: detail=no2: val(-0.851461350514) < min(0)
2016-09-24 17:26:15,646 dboutput INFO committed 11 records
2016-09-24 17:26:15,691 RawSensorAPI WARNING id=15-co  invalid output for co: detail=co: val(-163.306436844) < min(0)
2016-09-24 17:26:15,701 dboutput INFO committed 11 records
2016-09-24 17:26:15,722 RawSensorAPI WARNING device_id=16 sensor=temperature: invalid input for s_temperatureambient: detail=s_temperatureambient: val(0) < min(233150)
2016-09-24 17:26:15,722 RawSensorAPI WARNING device_id=16 sensor=co: invalid input for ['s_o3resistance', 's_no2resistance', 's_co2', 's_temperatureambient', 's_temperatureunit', 's_humidity', 's_barometer', 's_lightsensorbottom']: detail=s_temperatureambient: val(0) < min(233150)
2016-09-24 17:26:15,723 RawSensorAPI WARNING device_id=16 sensor=no2: invalid input for ['s_o3resistance', 's_no2resistance', 's_co2', 's_temperatureambient', 's_temperatureunit', 's_humidity', 's_barometer', 's_lightsensorbottom']: detail=s_temperatureambient: val(0) < min(233150)
2016-09-24 17:26:15,723 RawSensorAPI WARNING device_id=16 sensor=o3: invalid input for ['s_o3resistance', 's_no2resistance', 's_co2', 's_temperatureambient', 's_temperatureunit', 's_humidity', 's_barometer', 's_lightsensorbottom']: detail=s_temperatureambient: val(0) < min(233150)
2016-09-24 17:26:15,730 dboutput INFO committed 8 records
2016-09-24 17:26:15,753 RawSensorAPI WARNING device_id=17 sensor=co: invalid input for ['s_o3resistance', 's_no2resistance', 's_co2', 's_temperatureambient', 's_temperatureunit', 's_humidity', 's_barometer', 's_lightsensorbottom']: detail=s_no2resistance: val(0) < min(800)
2016-09-24 17:26:15,753 RawSensorAPI WARNING device_id=17 sensor=no2: invalid input for ['s_o3resistance', 's_no2resistance', 's_co2', 's_temperatureambient', 's_temperatureunit', 's_humidity', 's_barometer', 's_lightsensorbottom']: detail=s_no2resistance: val(0) < min(800)
2016-09-24 17:26:15,753 RawSensorAPI WARNING device_id=17 sensor=o3: invalid input for ['s_o3resistance', 's_no2resistance', 's_co2', 's_temperatureambient', 's_temperatureunit', 's_humidity', 's_barometer', 's_lightsensorbottom']: detail=s_no2resistance: val(0) < min(800)
2016-09-24 17:26:15,753 RawSensorAPI WARNING device_id=17 sensor=no2raw: invalid input for ['s_no2resistance']: detail=s_no2resistance: val(0) < min(800)
2016-09-24 17:26:15,760 dboutput INFO committed 8 records
2016-09-24 17:26:15,782 RawSensorAPI WARNING id=18-co  invalid output for co: detail=co: val(-122.669794231) < min(0)
2016-09-24 17:26:15,792 dboutput INFO committed 11 records
2016-09-24 17:26:15,813 RawSensorAPI WARNING id=19-co  invalid output for co: detail=co: val(-131.835533783) < min(0)
2016-09-24 17:26:15,823 dboutput INFO committed 11 records
2016-09-24 17:26:15,845 RawSensorAPI WARNING device_id=20 sensor=temperature: invalid input for s_temperatureambient: detail=s_temperatureambient: val(0) < min(233150)
2016-09-24 17:26:15,846 RawSensorAPI WARNING device_id=20 sensor=co2: invalid input for s_co2: detail=s_co2 not present
2016-09-24 17:26:15,846 RawSensorAPI WARNING device_id=20 sensor=co: invalid input for ['s_o3resistance', 's_no2resistance', 's_co2', 's_temperatureambient', 's_temperatureunit', 's_humidity', 's_barometer', 's_lightsensorbottom']: detail=s_co2 not present
2016-09-24 17:26:15,846 RawSensorAPI WARNING device_id=20 sensor=no2: invalid input for ['s_o3resistance', 's_no2resistance', 's_co2', 's_temperatureambient', 's_temperatureunit', 's_humidity', 's_barometer', 's_lightsensorbottom']: detail=s_co2 not present
2016-09-24 17:26:15,846 RawSensorAPI WARNING device_id=20 sensor=o3: invalid input for ['s_o3resistance', 's_no2resistance', 's_co2', 's_temperatureambient', 's_temperatureunit', 's_humidity', 's_barometer', 's_lightsensorbottom']: detail=s_co2 not present
2016-09-24 17:26:15,852 dboutput INFO committed 7 records
2016-09-24 17:26:15,894 RawSensorAPI WARNING id=22-co  invalid output for co: detail=co: val(-221.881455502) < min(0)
2016-09-24 17:26:15,894 RawSensorAPI WARNING id=22-no2  invalid output for no2: detail=no2: val(-20.7889521532) < min(0)
2016-09-24 17:26:15,904 dboutput INFO committed 10 records
2016-09-24 17:26:15,924 RawSensorAPI WARNING device_id=23 sensor=co2: invalid input for s_co2: detail=s_co2: val(10979000) > max(5000000)
2016-09-24 17:26:15,924 RawSensorAPI WARNING device_id=23 sensor=co: invalid input for ['s_o3resistance', 's_no2resistance', 's_co2', 's_temperatureambient', 's_temperatureunit', 's_humidity', 's_barometer', 's_lightsensorbottom']: detail=s_co2: val(10979000) > max(5000000)
2016-09-24 17:26:15,924 RawSensorAPI WARNING device_id=23 sensor=no2: invalid input for ['s_o3resistance', 's_no2resistance', 's_co2', 's_temperatureambient', 's_temperatureunit', 's_humidity', 's_barometer', 's_lightsensorbottom']: detail=s_co2: val(10979000) > max(5000000)
2016-09-24 17:26:15,924 RawSensorAPI WARNING device_id=23 sensor=o3: invalid input for ['s_o3resistance', 's_no2resistance', 's_co2', 's_temperatureambient', 's_temperatureunit', 's_humidity', 's_barometer', 's_lightsensorbottom']: detail=s_co2: val(10979000) > max(5000000)
2016-09-24 17:26:15,931 dboutput INFO committed 8 records
2016-09-24 17:26:15,952 RawSensorAPI WARNING device_id=24 sensor=temperature: invalid input for s_temperatureambient: detail=s_temperatureambient: val(0) < min(233150)
2016-09-24 17:26:15,952 RawSensorAPI WARNING device_id=24 sensor=humidity: invalid input for s_humidity: detail=s_humidity: val(3283) < min(20000)
2016-09-24 17:26:15,952 RawSensorAPI WARNING device_id=24 sensor=co: invalid input for ['s_o3resistance', 's_no2resistance', 's_co2', 's_temperatureambient', 's_temperatureunit', 's_humidity', 's_barometer', 's_lightsensorbottom']: detail=s_temperatureambient: val(0) < min(233150)
2016-09-24 17:26:15,952 RawSensorAPI WARNING device_id=24 sensor=no2: invalid input for ['s_o3resistance', 's_no2resistance', 's_co2', 's_temperatureambient', 's_temperatureunit', 's_humidity', 's_barometer', 's_lightsensorbottom']: detail=s_temperatureambient: val(0) < min(233150)
2016-09-24 17:26:15,952 RawSensorAPI WARNING device_id=24 sensor=o3: invalid input for ['s_o3resistance', 's_no2resistance', 's_co2', 's_temperatureambient', 's_temperatureunit', 's_humidity', 's_barometer', 's_lightsensorbottom']: detail=s_temperatureambient: val(0) < min(233150)
2016-09-24 17:26:15,959 dboutput INFO committed 7 records
2016-09-24 17:26:15,980 RawSensorAPI WARNING device_id=25 sensor=temperature: invalid input for s_temperatureambient: detail=s_temperatureambient not present
2016-09-24 17:26:15,980 RawSensorAPI WARNING device_id=25 sensor=humidity: invalid input for s_humidity: detail=s_humidity not present
2016-09-24 17:26:15,980 RawSensorAPI WARNING device_id=25 sensor=pressure: invalid input for s_barometer: detail=s_barometer not present
2016-09-24 17:26:15,980 RawSensorAPI WARNING device_id=25 sensor=noiseavg: invalid input for ['v_audio0', 'v_audioplus1', 'v_audioplus2', 'v_audioplus3', 'v_audioplus4', 'v_audioplus5', 'v_audioplus6', 'v_audioplus7', 'v_audioplus8', 'v_audioplus9']: detail=v_audio0 not present
2016-09-24 17:26:15,980 RawSensorAPI WARNING device_id=25 sensor=noiselevelavg: invalid input for noiseavg: detail=noiseavg not present
2016-09-24 17:26:15,980 RawSensorAPI WARNING device_id=25 sensor=co2: invalid input for s_co2: detail=s_co2 not present
2016-09-24 17:26:15,980 RawSensorAPI WARNING device_id=25 sensor=co: invalid input for ['s_o3resistance', 's_no2resistance', 's_co2', 's_temperatureambient', 's_temperatureunit', 's_humidity', 's_barometer', 's_lightsensorbottom']: detail=s_o3resistance not present
2016-09-24 17:26:15,980 RawSensorAPI WARNING device_id=25 sensor=no2: invalid input for ['s_o3resistance', 's_no2resistance', 's_co2', 's_temperatureambient', 's_temperatureunit', 's_humidity', 's_barometer', 's_lightsensorbottom']: detail=s_o3resistance not present
2016-09-24 17:26:15,980 RawSensorAPI WARNING device_id=25 sensor=o3: invalid input for ['s_o3resistance', 's_no2resistance', 's_co2', 's_temperatureambient', 's_temperatureunit', 's_humidity', 's_barometer', 's_lightsensorbottom']: detail=s_o3resistance not present
2016-09-24 17:26:15,980 RawSensorAPI WARNING device_id=25 sensor=coraw: invalid input for ['s_coresistance']: detail=s_coresistance not present
2016-09-24 17:26:15,980 RawSensorAPI WARNING device_id=25 sensor=no2raw: invalid input for ['s_no2resistance']: detail=s_no2resistance not present
2016-09-24 17:26:15,981 RawSensorAPI WARNING device_id=25 sensor=o3raw: invalid input for ['s_o3resistance']: detail=s_o3resistance not present
2016-09-24 17:26:16,002 RawSensorAPI WARNING id=26-co  invalid output for co: detail=co: val(-215.097673326) < min(0)
2016-09-24 17:26:16,019 dboutput INFO committed 11 records
2016-09-24 17:26:16,043 RawSensorAPI WARNING device_id=27 sensor=temperature: invalid input for s_temperatureambient: detail=s_temperatureambient: val(0) < min(233150)
2016-09-24 17:26:16,043 RawSensorAPI WARNING device_id=27 sensor=co: invalid input for ['s_o3resistance', 's_no2resistance', 's_co2', 's_temperatureambient', 's_temperatureunit', 's_humidity', 's_barometer', 's_lightsensorbottom']: detail=s_temperatureambient: val(0) < min(233150)
2016-09-24 17:26:16,043 RawSensorAPI WARNING device_id=27 sensor=no2: invalid input for ['s_o3resistance', 's_no2resistance', 's_co2', 's_temperatureambient', 's_temperatureunit', 's_humidity', 's_barometer', 's_lightsensorbottom']: detail=s_temperatureambient: val(0) < min(233150)
2016-09-24 17:26:16,043 RawSensorAPI WARNING device_id=27 sensor=o3: invalid input for ['s_o3resistance', 's_no2resistance', 's_co2', 's_temperatureambient', 's_temperatureunit', 's_humidity', 's_barometer', 's_lightsensorbottom']: detail=s_temperatureambient: val(0) < min(233150)
2016-09-24 17:26:16,051 dboutput INFO committed 8 records
2016-09-24 17:26:16,082 dboutput INFO committed 12 records
2016-09-24 17:26:16,144 RawSensorAPI WARNING device_id=31 sensor=temperature: invalid input for s_temperatureambient: detail=s_temperatureambient not present
2016-09-24 17:26:16,144 RawSensorAPI WARNING device_id=31 sensor=co: invalid input for ['s_o3resistance', 's_no2resistance', 's_co2', 's_temperatureambient', 's_temperatureunit', 's_humidity', 's_barometer', 's_lightsensorbottom']: detail=s_temperatureambient not present
2016-09-24 17:26:16,144 RawSensorAPI WARNING device_id=31 sensor=no2: invalid input for ['s_o3resistance', 's_no2resistance', 's_co2', 's_temperatureambient', 's_temperatureunit', 's_humidity', 's_barometer', 's_lightsensorbottom']: detail=s_temperatureambient not present
2016-09-24 17:26:16,144 RawSensorAPI WARNING device_id=31 sensor=o3: invalid input for ['s_o3resistance', 's_no2resistance', 's_co2', 's_temperatureambient', 's_temperatureunit', 's_humidity', 's_barometer', 's_lightsensorbottom']: detail=s_temperatureambient not present
2016-09-24 17:26:16,152 dboutput INFO committed 8 records
2016-09-24 17:26:16,171 RawSensorAPI WARNING device_id=32 sensor=temperature: invalid input for s_temperatureambient: detail=s_temperatureambient: val(0) < min(233150)
2016-09-24 17:26:16,172 RawSensorAPI WARNING device_id=32 sensor=co: invalid input for ['s_o3resistance', 's_no2resistance', 's_co2', 's_temperatureambient', 's_temperatureunit', 's_humidity', 's_barometer', 's_lightsensorbottom']: detail=s_temperatureambient: val(0) < min(233150)
2016-09-24 17:26:16,172 RawSensorAPI WARNING device_id=32 sensor=no2: invalid input for ['s_o3resistance', 's_no2resistance', 's_co2', 's_temperatureambient', 's_temperatureunit', 's_humidity', 's_barometer', 's_lightsensorbottom']: detail=s_temperatureambient: val(0) < min(233150)
2016-09-24 17:26:16,172 RawSensorAPI WARNING device_id=32 sensor=o3: invalid input for ['s_o3resistance', 's_no2resistance', 's_co2', 's_temperatureambient', 's_temperatureunit', 's_humidity', 's_barometer', 's_lightsensorbottom']: detail=s_temperatureambient: val(0) < min(233150)
2016-09-24 17:26:16,179 dboutput INFO committed 8 records
2016-09-24 17:26:16,202 RawSensorAPI WARNING id=33-co  invalid output for co: detail=co: val(-290.689024551) < min(0)
2016-09-24 17:26:16,212 dboutput INFO committed 11 records
2016-09-24 17:26:16,234 RawSensorAPI WARNING id=34-co  invalid output for co: detail=co: val(-290.112474534) < min(0)
2016-09-24 17:26:16,245 dboutput INFO committed 11 records
2016-09-24 17:26:16,312 RawSensorAPI WARNING id=37-co  invalid output for co: detail=co: val(-245.020656058) < min(0)
2016-09-24 17:26:16,313 RawSensorAPI WARNING id=37-no2  invalid output for no2: detail=no2: val(-2.3944040892) < min(0)
2016-09-24 17:26:16,322 dboutput INFO committed 10 records
2016-09-24 17:26:16,344 RawSensorAPI WARNING id=38-co  invalid output for co: detail=co: val(-270.390771958) < min(0)
2016-09-24 17:26:16,354 dboutput INFO committed 11 records
2016-09-24 17:26:16,398 RawSensorAPI WARNING id=40-co  invalid output for co: detail=co: val(-66.2058499111) < min(0)
2016-09-24 17:26:16,409 dboutput INFO committed 11 records
2016-09-24 17:26:16,429 RawSensorAPI WARNING device_id=41 sensor=noiseavg: invalid input for ['v_audio0', 'v_audioplus1', 'v_audioplus2', 'v_audioplus3', 'v_audioplus4', 'v_audioplus5', 'v_audioplus6', 'v_audioplus7', 'v_audioplus8', 'v_audioplus9']: detail=v_audio0 not present
2016-09-24 17:26:16,429 RawSensorAPI WARNING device_id=41 sensor=noiselevelavg: invalid input for noiseavg: detail=noiseavg not present
2016-09-24 17:26:16,430 RawSensorAPI WARNING id=41-co  invalid output for co: detail=co: val(-164.658767465) < min(0)
2016-09-24 17:26:16,439 dboutput INFO committed 9 records
2016-09-24 17:26:16,460 RawSensorAPI WARNING id=42-co  invalid output for co: detail=co: val(-100.875816295) < min(0)
2016-09-24 17:26:16,471 dboutput INFO committed 11 records
2016-09-24 17:26:16,493 RawSensorAPI WARNING id=43-co  invalid output for co: detail=co: val(-367.034695059) < min(0)
2016-09-24 17:26:16,493 RawSensorAPI WARNING id=43-no2  invalid output for no2: detail=no2: val(-11.6446581641) < min(0)
2016-09-24 17:26:16,503 dboutput INFO committed 10 records
2016-09-24 17:26:16,524 RawSensorAPI WARNING device_id=44 sensor=pressure: invalid input for s_barometer: detail=s_barometer: val(0) < min(20000)
2016-09-24 17:26:16,524 RawSensorAPI WARNING device_id=44 sensor=co: invalid input for ['s_o3resistance', 's_no2resistance', 's_co2', 's_temperatureambient', 's_temperatureunit', 's_humidity', 's_barometer', 's_lightsensorbottom']: detail=s_barometer: val(0) < min(20000)
2016-09-24 17:26:16,524 RawSensorAPI WARNING device_id=44 sensor=no2: invalid input for ['s_o3resistance', 's_no2resistance', 's_co2', 's_temperatureambient', 's_temperatureunit', 's_humidity', 's_barometer', 's_lightsensorbottom']: detail=s_barometer: val(0) < min(20000)
2016-09-24 17:26:16,525 RawSensorAPI WARNING device_id=44 sensor=o3: invalid input for ['s_o3resistance', 's_no2resistance', 's_co2', 's_temperatureambient', 's_temperatureunit', 's_humidity', 's_barometer', 's_lightsensorbottom']: detail=s_barometer: val(0) < min(20000)
2016-09-24 17:26:16,533 dboutput INFO committed 8 records
2016-09-24 17:26:16,577 RawSensorAPI WARNING id=46-co  invalid output for co: detail=co: val(-17.3055911486) < min(0)
2016-09-24 17:26:16,577 RawSensorAPI WARNING id=46-no2  invalid output for no2: detail=no2: val(-1.4947980253) < min(0)
2016-09-24 17:26:16,588 dboutput INFO committed 10 records
2016-09-24 17:26:16,632 RawSensorAPI WARNING id=48-co  invalid output for co: detail=co: val(-12.7121213695) < min(0)
2016-09-24 17:26:16,651 dboutput INFO committed 11 records
2016-09-24 17:26:16,671 RawSensorAPI WARNING device_id=49 sensor=temperature: invalid input for s_temperatureambient: detail=s_temperatureambient not present
2016-09-24 17:26:16,671 RawSensorAPI WARNING device_id=49 sensor=humidity: invalid input for s_humidity: detail=s_humidity not present
2016-09-24 17:26:16,672 RawSensorAPI WARNING device_id=49 sensor=pressure: invalid input for s_barometer: detail=s_barometer not present
2016-09-24 17:26:16,672 RawSensorAPI WARNING device_id=49 sensor=noiseavg: invalid input for ['v_audio0', 'v_audioplus1', 'v_audioplus2', 'v_audioplus3', 'v_audioplus4', 'v_audioplus5', 'v_audioplus6', 'v_audioplus7', 'v_audioplus8', 'v_audioplus9']: detail=v_audio0 not present
2016-09-24 17:26:16,672 RawSensorAPI WARNING device_id=49 sensor=noiselevelavg: invalid input for noiseavg: detail=noiseavg not present
2016-09-24 17:26:16,672 RawSensorAPI WARNING device_id=49 sensor=co2: invalid input for s_co2: detail=s_co2 not present
2016-09-24 17:26:16,672 RawSensorAPI WARNING device_id=49 sensor=co: invalid input for ['s_o3resistance', 's_no2resistance', 's_co2', 's_temperatureambient', 's_temperatureunit', 's_humidity', 's_barometer', 's_lightsensorbottom']: detail=s_o3resistance not present
2016-09-24 17:26:16,672 RawSensorAPI WARNING device_id=49 sensor=no2: invalid input for ['s_o3resistance', 's_no2resistance', 's_co2', 's_temperatureambient', 's_temperatureunit', 's_humidity', 's_barometer', 's_lightsensorbottom']: detail=s_o3resistance not present
2016-09-24 17:26:16,672 RawSensorAPI WARNING device_id=49 sensor=o3: invalid input for ['s_o3resistance', 's_no2resistance', 's_co2', 's_temperatureambient', 's_temperatureunit', 's_humidity', 's_barometer', 's_lightsensorbottom']: detail=s_o3resistance not present
2016-09-24 17:26:16,672 RawSensorAPI WARNING device_id=49 sensor=coraw: invalid input for ['s_coresistance']: detail=s_coresistance not present
2016-09-24 17:26:16,672 RawSensorAPI WARNING device_id=49 sensor=no2raw: invalid input for ['s_no2resistance']: detail=s_no2resistance not present
2016-09-24 17:26:16,672 RawSensorAPI WARNING device_id=49 sensor=o3raw: invalid input for ['s_o3resistance']: detail=s_o3resistance not present
2016-09-24 17:26:16,755 RawSensorAPI WARNING id=53-no2  invalid output for no2: detail=no2: val(-1.38323300558) < min(0)
2016-09-24 17:26:16,766 dboutput INFO committed 11 records
2016-09-24 17:26:16,853 RawSensorAPI WARNING id=57-co  invalid output for co: detail=co: val(-21.4827841807) < min(0)
2016-09-24 17:26:16,853 RawSensorAPI WARNING id=57-no2  invalid output for no2: detail=no2: val(-3.19902299456) < min(0)
2016-09-24 17:26:16,863 dboutput INFO committed 10 records
2016-09-24 17:26:16,885 RawSensorAPI WARNING id=58-co  invalid output for co: detail=co: val(-185.6378756) < min(0)
2016-09-24 17:26:16,885 RawSensorAPI WARNING id=58-no2  invalid output for no2: detail=no2: val(-5.67803952936) < min(0)
2016-09-24 17:26:16,895 dboutput INFO committed 10 records
2016-09-24 17:26:16,916 RawSensorAPI WARNING id=59-co  invalid output for co: detail=co: val(-41.4279619675) < min(0)
2016-09-24 17:26:16,916 RawSensorAPI WARNING id=59-no2  invalid output for no2: detail=no2: val(-2.19644433821) < min(0)
2016-09-24 17:26:16,925 dboutput INFO committed 10 records
2016-09-24 17:26:16,948 RawSensorAPI WARNING id=60-co  invalid output for co: detail=co not present
2016-09-24 17:26:16,948 RawSensorAPI WARNING id=60-no2  invalid output for no2: detail=no2 not present
2016-09-24 17:26:16,948 RawSensorAPI WARNING id=60-o3  invalid output for o3: detail=o3 not present
2016-09-24 17:26:16,956 dboutput INFO committed 9 records
2016-09-24 17:26:17,006 RawSensorAPI WARNING device_id=62 sensor=co2: invalid input for s_co2: detail=s_co2: val(64636000) > max(5000000)
2016-09-24 17:26:17,006 RawSensorAPI WARNING device_id=62 sensor=co: invalid input for ['s_o3resistance', 's_no2resistance', 's_co2', 's_temperatureambient', 's_temperatureunit', 's_humidity', 's_barometer', 's_lightsensorbottom']: detail=s_co2: val(64636000) > max(5000000)
2016-09-24 17:26:17,006 RawSensorAPI WARNING device_id=62 sensor=no2: invalid input for ['s_o3resistance', 's_no2resistance', 's_co2', 's_temperatureambient', 's_temperatureunit', 's_humidity', 's_barometer', 's_lightsensorbottom']: detail=s_co2: val(64636000) > max(5000000)
2016-09-24 17:26:17,006 RawSensorAPI WARNING device_id=62 sensor=o3: invalid input for ['s_o3resistance', 's_no2resistance', 's_co2', 's_temperatureambient', 's_temperatureunit', 's_humidity', 's_barometer', 's_lightsensorbottom']: detail=s_co2: val(64636000) > max(5000000)
2016-09-24 17:26:17,014 dboutput INFO committed 8 records
2016-09-24 17:26:17,188 RawSensorAPI INFO Processing halted: all devices done
2016-09-24 17:26:17,188 RawSensorAPI INFO Exit
2016-09-24 17:26:17,188 dboutput INFO Exit: disconnect from DB
2016-09-24 17:26:17,189 chain INFO DONE - 65 rounds - chain=input_raw_sensor_last_api|output_postgres_insert 
2016-09-24 17:26:17,189 util INFO Timer end: total ETL time=2.0 sec
2016-09-24 17:26:17,189 ETL INFO ALL DONE

Query jose data from influxdb streaming

A lot of memory is needed to load all the influxdb data at once (e.g. ~30gb). Loading it in pieces, low-pass filtering the pieces immediately, and sampling afterwards saves a lot of memory.

Change CalibrationInfluxDbInput to work with a step size

How to read both jose and rivm sequentially??

Create script file that learns and writes MLPRegressor models from Jose and RIVM data

Automatic creation of CSV file of Jose measurements at location 12 and 14

Columns: s_barometer, s_co2, s_humidity, s_no2resistance, s_o3resistance, s_temperatureambient, s_temperatureunit
Granularity: raw values, 3 times per minute
Maybe seperate files for sensor 12 and 14

SensorThings API (STA) SensorUp uitfaseren en vervangen door GOST

De huidige STA server van SensorUp werkt goed maar heeft volgende nadelen:

proprietary met 1 jaar gratis licentie (loopt binnenkort af)
heeft eigen VM server nodig (kosten Geonovum) in beheer SensorUp
kent maar 1 versie, geen test en productie
werkt niet met Docker, dus "hangt erbij" in platform

Een nieuwe Open Source SensorThings API (STA) server: Geodan GOST met daarbij MQTT en NodeRed gaan we toevoegen als Docker images. Dan kunnen we terug naar 2 servers (Test+Prod) ipv 3. STA/MQTT/NodeRed zouden we ook stand-alone voor kleinere projecten kunnen inzetten. Net als InfluxDB/Grafana overigens.

Calibratie toevoegen aan laatste waarden

Just:

de "last.sh, last.cfg" haalt live waarden op van WHale API (tbv
SmartApp) op en roept direct de sensorconverters.py aan via
sensordefs.py. Dus heeft niet de calibrator initialisatie en geeft
foutmeldingen hierover.

Er zitten "gaten" in de historische data van de 52 North API

Add Grafana Viewer as Service

For viewing InfluxDB Grafana viewer provides an enhanced framework:

"Grafana is an open source metric analytics & visualization suite. It is most commonly used for visualizing time series data for infrastructure and application analytics but many use it in other domains including industrial sensors, home automation, weather, and process control."

See http://grafana.org/

Install as Docker service.

Add (un)certainty measurement to reported values

For example RMSE:
O3: +- 7.4 ug/m3
NO2: +- 12.6 ug/m3
CO: +- 124.5 ug/m3

Omrekening voor gaswaarden CO, NO2, O3

Voor CO, NO2, O3 zijn geen ppm-waarden uit de raw sensors API beschikbaar. Er dient een omrekening te komen van de ruwe bijv weerstandswaarden naar ppm en ug/m3 die nu uit de sensors komen.

SmartApp: JavaScript error bij sluiten info popup

Zie plaatje hieronder

Gebeurt bij sluiten van de popup die verschijnt na klikken op station.

Log warning if calibration model not available

Currently stopping with an error

2017-01-16 11:54:28,417 chain INFO Running Chain: input_raw_sensor_db| (refine_filter|output_postgres_insert) (refine_filter_small|output_influxdb_write)
2017-01-16 11:54:28,417 dbinput INFO Init: connect to DB
2017-01-16 11:54:28,417 postgis INFO Connecting to dbname=gis user=docker host=postgis port=5432
2017-01-16 11:54:28,428 postgis DEBUG Connected to database gis
2017-01-16 11:54:28,436 dbinput INFO read recs: 0
2017-01-16 11:54:28,442 dbinput INFO read recs: 4000
2017-01-16 11:54:28,447 SmartemDbInput INFO read timeseries_recs: 4000
2017-01-16 11:54:28,448 dboutput INFO Init: connect to DB
2017-01-16 11:54:28,448 postgis INFO Connecting to dbname=gis user=docker host=postgis port=5432
2017-01-16 11:54:28,455 postgis DEBUG Connected to database gis
2017-01-16 11:54:28,456 dbinput INFO read recs: 1
2017-01-16 11:54:28,456 dbinput INFO Exit: disconnect from DB
2017-01-16 11:54:28,456 dboutput INFO Exit: disconnect from DB
Traceback (most recent call last):
File "/usr/local/bin/stetl", line 5, in
pkg_resources.run_script('Stetl==1.1.0rc4', 'stetl')
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 528, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 1394, in run_script
execfile(script_filename, namespace, namespace)
File "/usr/local/lib/python2.7/dist-packages/Stetl-1.1.0rc4-py2.7.egg/EGG-INFO/scripts/stetl", line 40, in
main()
File "/usr/local/lib/python2.7/dist-packages/Stetl-1.1.0rc4-py2.7.egg/EGG-INFO/scripts/stetl", line 31, in main
etl.run()
File "/usr/local/lib/python2.7/dist-packages/Stetl-1.1.0rc4-py2.7.egg/stetl/etl.py", line 101, in run
chain.run()
File "/usr/local/lib/python2.7/dist-packages/Stetl-1.1.0rc4-py2.7.egg/stetl/chain.py", line 208, in run
packet = self.first_comp.process(packet)
File "/usr/local/lib/python2.7/dist-packages/Stetl-1.1.0rc4-py2.7.egg/stetl/component.py", line 185, in process
packet = self.next.process(packet)
File "/usr/local/lib/python2.7/dist-packages/Stetl-1.1.0rc4-py2.7.egg/stetl/splitter.py", line 57, in process
comp.process(packet)
File "/usr/local/lib/python2.7/dist-packages/Stetl-1.1.0rc4-py2.7.egg/stetl/component.py", line 174, in process
packet = self.invoke(packet)
File "/home/pieter/Documents/GemeenteNijmegen/smartemission/etl/refiner.py", line 70, in invoke
models = record_in['models']
KeyError: 'models'

Publiceren refined Jose waarden naar InfluxDB

De Refiner (gecalibreerde uurwaarden) schrijft nu naar Postgis DB. Uitbreiden met ook naar InfluxDB schrijven. Kan via Grafana dan visueel met RIVM waarden harvested in rivm tabel vergeleken.

Moet simpel in Stetl refiner.cfg ETL te doen zijn:

chains = input_raw_sensor_db| (refine_filter|output_postgres_insert) (refine_filter_small|output_influxdb_write)

In InfluxDB dan publiceren naar joserefined tabel.

Geen enkele waarde mag None zijn voor een voorspelling

Nu wordt er alleen gecheckt of de gas componenten none zijn

Implement SensorThings API Publisher

In addition to the SOS publisher the project needs a publisher ETL process to publish to the SensorUp SensorThings API (STA) via REST. The implementation of the STAPublisher can be done similar to the SOS publisher: using Stetl with Jinja2 templates and agnostic metadata publication on first observation insert.

Tijdens calibratie alleen rijen met NA verwijderen in kolommen van Jose, niet van RIVM

Nu wordt (denk ik) alle gegevens weggegooid van RIVM stations die niet alle gassen meten

Maak gebruik van temporele relatie tijdens calibratie

Ik heb de code voor low-pass filtering voor de calibratie (en voor toepassen van het calibratie model) in een comment gezet. Dit leverde veel bugs op omdat ik niet op lette.

In de gas metingen (s_no2resistance, s_o3resistance, s_coresistance) zit ruis. Door een running/rolling mean te gebruiken kan dit eruit worden gehaald waardoor de calibratie beter wordt. Voor deze oplossing moet per device/geohash een rolling mean worden bijgehouden voor elk van de gassen.

Een andere oplossing is ook vorige inputs aan het calibratie model meegeven. Dan zou die de huidige RIVM meting kunnen voorspellen aan de hand van de afgelopen n (bijvoorbeeld 10) jose metingen. Hiervoor moet per device/geohash de laaste n metingen worden bijgehouden in een lijst.

Nog een andere oplossing is een recurrent neural network. Zie ook #36. Deze heeft een staat en de volgorde waarin observaties worden aangeboden maakt dus uit. Dus moet er worden gegarandeerd dat observaties chronologische worden aangeleverd. En elke device/geohash moet zijn eigen kopie van het model hebben.

Todo

ANN omrekening gaswaarden - fouten bij running mean en ohm2ugm3

Eerste versie ANN calibratie O3, NO2 en CO is klaar, nu verfijnen debuggen. Onderstaande fout is opgelost maar nog eens goed naar kijken want is mogelijk lapmiddel (zie commit) en de "echte fout" zit elders in de keten, uit mail:

.... kreeg ik in Refiner permanent errors op de gassen
(o3, no2 en co). Ik heb in ieder geval de errors gefixed, zie deze
commit;
462c979c94f567e221fb3
c25b9bd7cd5c91e0d36

.... nog eens daarnaar kijken want weet niet of ik
alles snapte wat er gebeurde, 2 error typen:

- value np array ipv val return
- emtpy new_val in running_mean func

Data sensoren maakt datumsprongen in WFS

Sensor 41 begint bij 25 juni, terwijl in de Whale viewer ook vroegere data te vinden is. Sensor 12 bijvoorbeeld maakt een sprong van april naar juli, terwijl deze data ook gewoon in de Whale viewer te vinden is.

Sensor_defs gebruiken bij selecteren kolommen tijdens calibratie

Automatic creation of csv file of RIVM measurement at Graafseweg and Ruyterstraat

Columns: O3_Waarden, NO2_Waarden, CO_Waarden
Granularity: per hour (can be interpolated)
Maybe separate file per station

iPhone map issue after closing the data view

On iPhone 6 (iOS 9.2.1), after closing the data view window, the returning map behaves strange. The map is not centered on the right location anymore and some tiles seem to be missing.

AirSensEUR support in platform

To easily support AirSensEUR http://www.airsenseur.org/ pushed data Influxdb https://hub.docker.com/_/influxdb/ and Grafana https://grafana.net/ need to be setup.

SmartApp moet echte SOS gebruiken

De SmartApp gebruikt nu de gemuleerde SOS voor de laatste waarden. Dat is historisch ontstaan, omdat de echte SOS er nog niet was. Zie ook emails en http://smartemission.ruhosting.nl/forums/topic/52-north-api/

...voor de stations beter niet naar SmartApp kijken. Waarom: SmartApp laat alleen de huidige/laatste waarden, direct vanuit de ruwe sensordata zien. Als er geen laatste waarden zijn, of berekend kunnen worden, zijn die stations ook niet zichtbaar of zelfs aanwezig.
Het is ook geen echte SOS maar destijds specifiek voor die service ontwikkeld omdat er nog geen historie en echte SOS was. Zie ook plaatje aangehecht: links de SOS rechts de SOS emulator (SmartApp).

De enige SOS voor SE project is:
http://data.smartemission.nl/sos52n/api/v1/stations
Dit zijn alle stations die ooit wat gestuurd hebben. Hoeven dus niet "up" te zijn.

Het achterliggende probleem is "wisselvalligheid" van:

- de stations: soms vallen ze compleet weg (denk WIFI probleem), voorgoed, of komen weer terug (daardoor "gaten" in data of houd na periode op)

- de sensoren in de stations: zelfs als station "up" is kunnen individuele sensoren soms geen of invalide waarden doorgeven, ook dit soms tijdelijk. Met name ontbreekt vaak meteo data bijv temperatuur. Daardoor kunnen de Calibraties (zie Pieter) ook niet gedaan worden en ontbreken gaswaarden m.n. NO2, CO en O3 terwijl de ruwe waarden NO2Raw etc wel aanwezig zijn.

Dus aanwezigheid station zou 3 kleuren kunnen krijgen, bijv:
- groen: "up" en alle indicatoren aanwezig
- oranje: "up" en niet alle indicatoren aanwezig
- rood: "down"

Voor  "down" moet ook criterium zijn nu: laatste 2 uur geen waarden.

TODO voor SmartApp: aansluiten op SOS en "smarter" maken, zelf uitvinden of stations "up" down" of daartussen zijn...

Replace sklearn MLPRegressor by a Keras recurrent neural net

Eliminates the need for manual filtering
Better prediction performance
Use of incremental data and data in batches, i.e. solves memory limit issues

Calibrate multiple gasses in one run

Now three runs are necessary

Support voor InfluxDB publishing raw Josene data

Tbv kalibratie waarden in InfluxDB ipv CSVs. Geeft mogelijkheid externe APIs voor ruwe waarden.

Nieuwe KNMI INSPIRE WMS

Vraag KNMI:

 "Kunnen jullie de volgende service configureren?

http://geoservices.knmi.nl/cgi-bin/inspireviewservice.cgi? DATASET=urn:xkdc:ds:nl.knmi::Actuele10mindataKNMIstations/1/&SERVICE=WMS&

Ook hebben wij de volgende layer toegevoegd ff_dd. Dit is windrichting en snelheid als vector.

Bijvoorbeeld:

http://geoservices.knmi.nl/cgi-bin/inspireviewservice.cgi?DATASET=urn:xkdc:ds:nl.knmi::Actuele10mindataKNMIstations/1/&SERVICE=WMS&&SERVICE=WMS&VERSION=1.3.0&REQUEST=GetMap&LAYERS=baselayer,ff_dd,overlay&WIDTH=1585&HEIGHT=955&CRS=EPSG%3A4326&BBOX=48.12140142490623,0.7452707655566745,55.61451651691333,13.181487960144377&STYLES=windspeed_arrow%2Fbarb&FORMAT=image/png&TRANSPARENT=TRUE&

Kunnen jullie deze ook opnemen?
Mvg,
Maarten Plieger

RIVM WMS in Heronviewer

http://geodata.rivm.nl/geoserver/wms?

Heeft interessante lagen: bijv actueel NO2, O3 etc en wat historie:
vandaag_no2_00, vandaag_no2_03, vandaag_no2_06 etc
Dat is bijv de vespreiding NO2 om de drie uur.

Add ETL for extracting/storing Last values from CityGIS Raw Sensor API

Like the title says. Provide ETL using Stetl.

Merge Jose and RIVM data from Ruyterstraat en Graafseweg to one single array

To be used for calibration

Voeg random hash aan elke set van opgeslagen model parameters toe

Nu is er geen (betrouwbare) manier om te zien welke model parameters bij elkaar horen.

Per keer dat calibrator.sh wordt er een randomized search over de parameters gedaan. Dat levert meerdere sets van parameters op. Nu is er geen mogelijkheid om parameters binnen een set en sets binnen een search aan elkaar te koppelen.

Historische Data download issue: Testbevinding voor 'noiselevelavg' url voor WFS download

Testbevindingen Historische Data beschikbaarstelling via Smart Emission data platform en Heron viewer:

De tijdserie begint op 20 juli, 10 uur en loopt door tot 28 juli =ik dacht even dat ik alleen de laatste dagen kon terugzien=, maar naar beneden scrollend in de lange lijst volgt daarna
de data van 1 april tot en met 20 juli (van day 20160401, hour 19 t/m 20160720, hour 9).
Dit lijkt me verder geen probleem, is handig als in de documentatie staat dat de "oude" data ouder dan 20 juli 2016 onder de andere dataregels verschijnt.

De directe WFS download werkt, voor no2 en 03 en co2 geprobeerd. Alleen voor 'noiselevelavg' krijg ik een foutmelding: Na url manipulatie met noiselevelavg voor bijvoorbeeld sensorstation 58, via deze url ....
http://data.smartemission.nl/geoserver/wfs?request=GetFeature&typename=timeseries_noiselevelavg&outputformat=csv&CQL_FILTER=device_id=58
... krijg ik deze foutmelding:

-<ows:ExceptionReport xsi:schemaLocation="http://www.opengis.net/ows/1.1 http://data.smartemission.nl:80/geoserver/schemas/ows/1.1.0/owsAll.xsd" version="2.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:ows="http://www.opengis.net/ows/1.1" xmlns:xs="http://www.w3.org/2001/XMLSchema">

-<ows:Exception locator="typeName" exceptionCode="InvalidParameterValue">

ows:ExceptionTextFeature type :timeseries_noiselevelavg unknown/ows:ExceptionText

/ows:Exception

/ows:ExceptionReport

Geteste calibratie meta-parameters bewaren

Hierdoor kan op termijn onderzocht worden welke meta-parameters leiden tot goede resultaten.

eenvoudige download

Vraag om een eenvoudige download per sensor. enkele knop, met download laatste 24 uur of download laatste week.

Parameter plots of exponential distributions are wrong

All performances are 0

laagfrequent geluid

Burgers vragen om een aparte view op laagfrequent geluid. <25 DB

Juiste eenheden/min/max waarden tbv validatie: check nodig

In de Refiner ETL worden ruwe sensor-waarden omgezet/gekalibreerd (Gassen) naar nieuwe waarden. Zowel voor als na de omzetting worden input en output waarden gevalideerd op minimale en maximale waarden. De omzetting wordt niet gedaan als een van de input waarden buiten min-max ligt. Dit geldt ook voor de omgezette waarden. Om dit te structureren is er een configuratie-bestand waarin alle metadata per component staat, zowel voor ruwe sensor-indicators (meestal prefix s_ of v_) als omgezette waarden. Bijv s_temperatureambient (milliKelvin) is een ruwe waarde en temperature (Celsius) een omgezette waarde. Deze staan in bestand
https://github.com/Geonovum/smartemission/blob/master/etl/sensordefs.py
, m.n. vanaf regel 76. De vraag is: klopt alle de metadata in dit bestand met name voor de s_ en v_ indicatoren? Iemand van Intemo zou dit beste kunnen nagaan.

Altijd zelfde min/max scale gebruiken bij visualisatie van effect

Niet vergeten...

Memory error bij leren calibratie model

Just:

Geheugen probleem "OSError: [Errno 12] Cannot allocate memory"
calibrator, maar had je aangegegeven:

Error:

2017-01-22 11:51:12,537 InfluxDbInput INFO Querying database: SELECT
time, component, geohash, value FROM joseextract WHERE
(geohash='u1hnvkb' or geohash='u1hnvkc' or geohash='u1hnvm3' or
geohash='u178hw8' or geohash = 'u178kt7') AND time >
'2017-01-22T09:59:59Z' LIMIT 1000000
2017-01-22 11:51:12,539 urllib3.connectionpool INFO Resetting dropped
connection: influxdb
2017-01-22 11:51:12,541 InfluxDbInput INFO Received 0 results
2017-01-22 11:51:13,693 Calibration INFO Received rivm data with shape
(45844, 5)
2017-01-22 11:51:13,693 Calibration INFO Received jose data with shape
(3422682, 10)
2017-01-22 11:51:13,693 Calibration INFO Pre-processing geohash and time
2017-01-22 11:51:19,512 Calibration INFO Interpolating RIVM values
towards jose measurements
2017-01-22 11:52:13,411 Calibration INFO RIVM and Jose are merged, new
shape (5471651, 13)
2017-01-22 11:52:13,412 Calibration INFO Filling and dropping missing values
2017-01-22 11:52:19,353 Calibration INFO Missing values:
{u's_no2resistance': 881144, u's_lightsensortop': 0,
u'carbon_monoxide__air_': 0, u'nitrogen_dioxide__air_': 0,
u's_temperatureunit': 626608, u's_barometer': 626608, u's_o3resistance':
881552, 'time': 0, 'geohash': 0, u'ozone__air_': 0,
u's_temperatureambient': 734759, u's_coresistance': 1488382,
u's_humidity': 29}
2017-01-22 11:52:22,220 Calibration INFO Missing values are removed, new
shape (3922930, 13).
2017-01-22 11:58:50,900 Calibration INFO Returning packet of length 3922930
2017-01-22 11:58:52,254 Calibration INFO Receiving packet of size 3922930
2017-01-22 11:59:49,703 Calibration INFO Created data frame with shape
(3922930, 11)
2017-01-22 12:00:03,091 Calibration INFO Sample data frame, keeping 1
out of every 50 rows. New shape (78459, 9)
2017-01-22 12:00:03,160 Calibration INFO Starting randomized cross
validated search to find best parameters. Running 50 iterations with 10
cross validations of -2 cores. Finding relation from s_barometer,
s_coresistance, s_humidity, s_lightsensortop, s_no2resistance,
s_o3resistance, s_temperatureambient, s_temperatureunit to
carbon_monoxide__air_.
2017-01-22 12:00:12,505 dboutput INFO Exit: disconnect from DB
Traceback (most recent call last):
File "/usr/local/bin/stetl", line 5, in
pkg_resources.run_script('Stetl==1.1.0rc4', 'stetl')
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 528,
in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 1394,
in run_script
execfile(script_filename, namespace, namespace)
File
"/usr/local/lib/python2.7/dist-packages/Stetl-1.1.0rc4-py2.7.egg/EGG-INFO/scripts/stetl",
line 40, in
main()
File
"/usr/local/lib/python2.7/dist-packages/Stetl-1.1.0rc4-py2.7.egg/EGG-INFO/scripts/stetl",
line 31, in main
etl.run()
File
"/usr/local/lib/python2.7/dist-packages/Stetl-1.1.0rc4-py2.7.egg/stetl/etl.py",
line 101, in run
chain.run()
File
"/usr/local/lib/python2.7/dist-packages/Stetl-1.1.0rc4-py2.7.egg/stetl/chain.py",
line 208, in run
packet = self.first_comp.process(packet)
File
"/usr/local/lib/python2.7/dist-packages/Stetl-1.1.0rc4-py2.7.egg/stetl/component.py",
line 185, in process
packet = self.next.process(packet)
File
"/usr/local/lib/python2.7/dist-packages/Stetl-1.1.0rc4-py2.7.egg/stetl/component.py",
line 185, in process
packet = self.next.process(packet)
File
"/usr/local/lib/python2.7/dist-packages/Stetl-1.1.0rc4-py2.7.egg/stetl/component.py",
line 185, in process
packet = self.next.process(packet)
File
"/usr/local/lib/python2.7/dist-packages/Stetl-1.1.0rc4-py2.7.egg/stetl/component.py",
line 174, in process
packet = self.invoke(packet)
File "/root/git/etl/calibration.py", line 217, in invoke
gs.fit(x, y)
File
"/usr/local/lib/python2.7/dist-packages/sklearn/model_selection/_search.py",
line 1185, in fit
return self._fit(X, y, groups, sampled_params)
File
"/usr/local/lib/python2.7/dist-packages/sklearn/model_selection/_search.py",
line 562, in _fit
for parameters in parameter_iterable
File
"/usr/local/lib/python2.7/dist-packages/sklearn/externals/joblib/parallel.py",
line 728, in call
n_jobs = self._initialize_backend()
File
"/usr/local/lib/python2.7/dist-packages/sklearn/externals/joblib/parallel.py",
line 540, in _initialize_backend
**self._backend_args)
File
"/usr/local/lib/python2.7/dist-packages/sklearn/externals/joblib/_parallel_backends.py",
line 307, in configure
self._pool = MemmapingPool(n_jobs, **backend_args)
File
"/usr/local/lib/python2.7/dist-packages/sklearn/externals/joblib/pool.py",
line 600, in init
super(MemmapingPool, self).init(**poolargs)
File
"/usr/local/lib/python2.7/dist-packages/sklearn/externals/joblib/pool.py",
line 420, in init
super(PicklingPool, self).init(**poolargs)
File "/usr/lib/python2.7/multiprocessing/pool.py", line 159, in init
self._repopulate_pool()
File "/usr/lib/python2.7/multiprocessing/pool.py", line 223, in
_repopulate_pool
w.start()
File "/usr/lib/python2.7/multiprocessing/process.py", line 130, in start
self._popen = Popen(self)
File "/usr/lib/python2.7/multiprocessing/forking.py", line 121, in
init
self.pid = os.fork()
OSError: [Errno 12] Cannot allocate memory

Laatste calibratie model publiceseren (in /calibration)

Het beschikbaren maken van het calibratie model maakt offline gebruik en testen makkelijker. Mogelijk ook transparanter voor anderen die willen ontdekken wat er precies geleerd is.

InfluxDB crash - out of memory

[2417457.246864] Out of memory: Kill process 18104 (influxd) score 757 or sacrifice child
[2417457.246875] Killed process 18104 (influxd) total-vm:9868536kB, anon-rss:7502348kB, file-rss:0kB

Eenmalig sklearn pipeline objecten inlezen ipv elke keer als er een voorspelling wordt gemaakt

Meteen inzoomen op sensor via link smartapp

Is het mogelijk om met een directe link per sensor direct in de smartapp in te zoomen (en de sensorinfo te openen) op de sensor waar de link van is?

Save input order for calibration models in db

For later use and debugging

Alle RIVM stations worden gebruikt voor de calibratie van alle gassen

Althans, zo lijkt het bij de histogram van geohashes:

Dit is ofwel een fout in de visualizatie, ofwel een fout in het leren van het model.

ETL start niet door ontbreken "picklegit" Python module

Na de commit eb7f8e1 starten de ETL processen niet meer. Zie onder. Mogelijk ontbrekend package/module, maar welke? Moet in Stetl Docker file toegevoegd.

2016-07-21 11:42:01,574 main INFO Stetl version = 1.1.0rc1
2016-07-21 11:42:01,575 main INFO Found args file at: options/docker.args
2016-07-21 11:42:01,576 ETL INFO INIT - Stetl version is 1.1.0rc1
2016-07-21 11:42:01,576 ETL INFO Config/working dir = /opt/geonovum/smartem/git/etl
2016-07-21 11:42:01,576 ETL INFO Reading config_file = last.cfg
.
.
2016-07-21 11:42:01,576 ETL INFO START
2016-07-21 11:42:01,576 util INFO Timer start: total ETL
2016-07-21 11:42:01,576 chain INFO Assembling Chain: input_raw_sensor_last_api|output_postgres_insert...
2016-07-21 11:42:01,601 factory ERROR cannot create class 'rawsensorapi.RawSensorLastInput'
2016-07-21 11:42:01,601 factory ERROR cannot create object instance from class 'rawsensorapi.RawSensorLastInput' e=No module named picklegit
Traceback (most recent call last):
  File "/usr/local/bin/stetl", line 5, in <module>
    pkg_resources.run_script('Stetl==1.1.0rc1', 'stetl')
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 528, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 1394, in run_script
    execfile(script_filename, namespace, namespace)
  File "/usr/local/lib/python2.7/dist-packages/Stetl-1.1.0rc1-py2.7.egg/EGG-INFO/scripts/stetl", line 40, in <module>
    main()
  File "/usr/local/lib/python2.7/dist-packages/Stetl-1.1.0rc1-py2.7.egg/EGG-INFO/scripts/stetl", line 31, in main
    etl.run()
  File "/usr/local/lib/python2.7/dist-packages/Stetl-1.1.0rc1-py2.7.egg/stetl/etl.py", line 98, in run
    chain.assemble()
  File "/usr/local/lib/python2.7/dist-packages/Stetl-1.1.0rc1-py2.7.egg/stetl/chain.py", line 42, in assemble
    etl_comp = factory.create_obj(self.config_dict, etl_section_name)
  File "/usr/local/lib/python2.7/dist-packages/Stetl-1.1.0rc1-py2.7.egg/stetl/factory.py", line 29, in create_obj
    raise e
ImportError: No module named picklegit

Sensor 71 & 73 geven geen no2 data in de actuele data in SOSEMU

Check if RIVM SOS uses well-defined time

Mail Just:

Bij RIVM LML XML via http://www.lml.rivm.nl/sos is in verleden wel eens onduidelijk geweest welke tijdzone aangehouden werd, ik dacht NL tijd en zomertijd/wintertijd zelfs. Daarom
hoop ik wel dat de RIVM SOS de juiste tijd (timestamps) geeft. Het lijkt soms of RIVM data bijv NO2 in trends achterlopen (SE voor), kan dus met uren mismatch te maken hebben.
Ik weet bijna 100% zeker dat SE wel goed met tijd omgaat: alles intern UTC en als UTC
opgeslagen, maar je weet maar nooit....

Problemen met Whale server http opvragingen

De laatste weken komen er errors uit de Whale server, m.n. bij opvragen timeseries, bijv
http://<whale-url>//sensors/v1/devices/22/timeseries (whale url symbolisch):

{
Message: "An error has occurred."
}

Dit is niet permanent: een volgende aanroep kan weer sucessvol zijn:

{
id: "22",
days: [
"/sensors/v1/devices/22/timeseries/20160402",
"/sensors/v1/devices/22/timeseries/20160403",
"/sensors/v1/devices/22/timeseries/20160404",
"/sensors/v1/devices/22/timeseries/20160405",
"/sensors/v1/devices/22/timeseries/20160406",
"/sensors/v1/devices/22/timeseries/20160407",
"/sensors/v1/devices/22/timeseries/20160408",
"/sensors/v1/devices/22/timeseries/20160409",
"/sensors/v1/devices/22/timeseries/20160410",
"/sensors/v1/devices/22/timeseries/20160411",
.
.

Gebeurt ook bij andere aanroepen, niet alleen timeseries. In ieder geval zorgt dit ervoor dat de Smart Emission Harvester en andere ETL processen, zoals "Last Values" niet "door kunnen lopen" bijv logfile:

2017-03-29 12:25:02,798 RawSensorAPI INFO processable hours for device 7 day 20160731: []
2017-03-29 12:25:02,798 RawSensorAPI INFO Init: fetching timeseries days list from URL: "http://<whale url>/sensors/v1/devices/8/timeseries" ...
2017-03-29 12:25:03,247 httpinput ERROR HTTPError fetching from URL http://<whale url>/sensors/v1/devices/8/timeseries: code=500 e=HTTP Error 500: Internal Server Error
2017-03-29 12:25:03,247 RawSensorAPI INFO Exit
2017-03-29 12:25:03,248 dboutput INFO Exit: disconnect from DB
Traceback (most recent call last):
  File "/usr/local/bin/stetl", line 5, in <module>
    pkg_resources.run_script('Stetl==1.1.0rc4', 'stetl')
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 528, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 1394, in run_script
    execfile(script_filename, namespace, namespace)
  File "/usr/local/lib/python2.7/dist-packages/Stetl-1.1.0rc4-py2.7.egg/EGG-INFO/scripts/stetl", line 40, in <module>
    main()
  File "/usr/local/lib/python2.7/dist-packages/Stetl-1.1.0rc4-py2.7.egg/EGG-INFO/scripts/stetl", line 31, in main
    etl.run()
  File "/usr/local/lib/python2.7/dist-packages/Stetl-1.1.0rc4-py2.7.egg/stetl/etl.py", line 101, in run
    chain.run()
  File "/usr/local/lib/python2.7/dist-packages/Stetl-1.1.0rc4-py2.7.egg/stetl/chain.py", line 208, in run
    packet = self.first_comp.process(packet)
  File "/usr/local/lib/python2.7/dist-packages/Stetl-1.1.0rc4-py2.7.egg/stetl/component.py", line 168, in process
    result = self.before_invoke(packet)
  File "/opt/geonovum/smartem/git/etl/rawsensorapi.py", line 568, in before_invoke
    self.next_hour()
  File "/opt/geonovum/smartem/git/etl/rawsensorapi.py", line 535, in next_hour
    self.next_day()
  File "/opt/geonovum/smartem/git/etl/rawsensorapi.py", line 521, in next_day
    self.fetch_ts_days()
  File "/opt/geonovum/smartem/git/etl/rawsensorapi.py", line 432, in fetch_ts_days
    json_str = self.read_from_url(ts_days_url)
  File "/usr/local/lib/python2.7/dist-packages/Stetl-1.1.0rc4-py2.7.egg/stetl/inputs/httpinput.py", line 93, in read_from_url
    raise e
urllib2.HTTPError: HTTP Error 500: Internal Server Error

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.