connectedhumber / air-quality-web Goto Github PK

Currently fetching data from the API requires a datetime parameter to be passed to specify the date and time for which readings are required. It would be cool if we could pass a latest parameter instead which retrieves the latest readings from sensors.

Something like this for instance https://aq.connectedhumber.org/api.php?action=fetch-data&latest&reading_type=PM25 would provide the most recent reading for each sensor.

Data exists in the database but some devices are reporting no data in the 1 day window when it definitely exists. Devices 10, 18 and 39 are all incorrectly showing no data. One of these is a paid for device and the owner regularly reviews his data on the map.

We would like to be able to display mobile devices on the map.

• We can display mobile device markers with their last-known location
• We can display a devices route by looking at last-known readings - but only if a marker is clicked
• Mobile devices will never have a device location in the DB
• We can interpolate to guess the location of a reading based on the next / previous readings in the client-side front-end

We'll need some real data coming in to test it though.

I know Mysql stores data as UTC but this can mask the actual time of day that something happens. In the Uk UTC is broadly the same as GMT so it's not an issue. However if this system were to be used globally the chart time scales would be affected.

Perhaps we need to include time-zone as an option on the URL for API calls - and maybe even browser time-zone for the map display?

I will raise the point with Robin and maybe add an extra tz field for devices.

Apparently, before doing an sql select issuing set time_zone='time zone' forces mysql to convert dates to the requested time zone.

I think this merits discussion.

Currently the tab-data display (graph and device info popup for a sensor) default to PM2.5.

Change this to be the currently displayed informaion.
E.g. Reading Type set to Temperature, the graph should pop up and display temperature.

Usability feature as that is what I would assume they may want to look at.

Is the new server an issue?

I think these should be three separate documents - easier to maintain and read.

By default, we display the last 24 hours worth of data. We should add the ability change & manipulate this.

• Quick change last: 1h / 1d / 1w / 1m / 3m etc.
• Advanced custom input

We'll have to figure out the size of the average we need to ask for here, to avoid ending up with too many points. Perhaps we can have an internal setting for the target number of points that's used to calculate the window size?

The heatmap also needs a time control on it, but I suspect it'll end working very differently to the device graph.

• Re-investigate leaflet-timedimension
• Get bounds from PHP API
• Fetch new heatmap data and display it based on leaflet-timedimension event

When click on the number in Robin's heat map in Cottingham the screen zooms in, massively to street level, and is all green.

If I zoom out slightly it sorts itself out.

I suspect this is due to the close proximity of several sensors in that area - the heat maps overlap.

The default tab displayed when clicking on a node is 'info'. It may make more sense for the default tab shown to be 'data', as 'data' is most users clicking on a node are likely to want to see 'data' rather than 'info'.

Currently the heat map colours rarely change. Could we use only the latest datapoint to set the colour (based on the DEFRA scale for PM2.5) with the objective of seeing much more variation in the colours on the map?

Could the graph display (and other such like) only show buttons for available sensors.

e.g. if it's a PM2.5/10 only setup, then don't show temp/humidity/pressure/..
If it's a temp monitor only, only show temp, not PM buttons etc.

Not anything urgent, but as more sensors are added, some maybe PM only, some may be VOC and no PM
Also if IAQ, VOC, O3, CO2 start being monitored, they may not have the other sensors.

As test subjects, currently
Matt_01 only has PM2.5, PM10 and nothing else.
robin_05 only has temp/press/humid

Maybe tied in with #11 ?

Installed composer ok
Downloaded the repository zip file and unzipped it.
./build setup setup-dev

pi@PiNAT:~/aq_web/Air-Quality-Web-master $ ./build setup setup-dev
fatal: Not a git repository (or any of the parent directories): .git
./build: line 29: ./lantern-build-engine/lantern.sh: No such file or directory
./build: line 164: tasks_run: command not found
pi@PiNAT:~/aq_web/Air-Quality-Web-master $

I'm following your steps. I believer all dependencies are installed

perhaps a calll to scroll/zoom with no change would cause the display to change as the blob slider is moved.

Add an option to overlay the actualy sensor reading for the device in numbers, much like you get on a weather map.
Possibly replace the inverted-drop.

This would save going to each device to see what the reading is.
It would help colour blind people also (as an added bonus)

(also see old random dodgy drawings in mattermost for examples)

Currently, the heading looks rather out-of-place - and it's been bugging me for a while. It would be great if it looked a bit more like the main connected humber website.

Changing it is easy - it's deciding what to change it to that's the hard part. What should it look like?

/cc @BNNorman, @robinharris, @bsimmo

I'm looking at the documentation under docs for api.

I cannot see a way to ask for a specific reading from a specific device (id) for time now (or past 10 mins, or past hour)
or for all sensor data for 'now' for that device;
I can either get all PM25 data for all sensors 'now'
or the history of one sensor type for a device

e.g.
I would like (I think)
https://aq.connectedhumber.org/api.php?action=fetch-data&device-id=38&datetime=now&reading_type=PM25
to return just the current PM25 reading for device 38

https://aq.connectedhumber.org/api.php?action=fetch-data&device-id=38&datetime=now&reading_type=PM25&average-seconds=600
same but returning the average over that 600 seconds

reading_type=ALL (return all available sensor reading)
or
reading_type=PM25,PM10
reading_type=Temperature,Humidity,Pressure
etc
to return the specified readings.

At the moment it seems I needs multiple requests to cover all the sensors types, that's a lot of data to shift through and must be more sever load for each request ?

In #33, we implemented a new action, device-data-recent. While good, using ST_DISTANCE() results in an inaccurate sort - even over the small distances that we're handling here.

The solution to this is to take into account the curvature of the earth when sorting the devices by distance.

Doing so isn't trivial, but thankfully this StackOverflow answer provides an implementation that does broadly what we're after:

CREATE FUNCTION `ST_DISTANCE_SPHERE`(`pt1` POINT, `pt2` POINT) RETURNS decimal(10,2)
BEGIN
	return 6371000 * 2 * ASIN(SQRT(
	POWER(SIN((ST_Y(pt2) - ST_Y(pt1)) * pi()/180 / 2),
	2) + COS(ST_Y(pt1) * pi()/180 ) * COS(ST_Y(pt2) *
	pi()/180) * POWER(SIN((ST_X(pt2) - ST_X(pt1)) *
	pi()/180 / 2), 2) ));
END
//
DELIMITER ;

Unfortunately, this is a stored function. While this will keep our code neater, we don't currently have permission to add it to the database (stored functions are stored at database-level), so we'll need @BNNorman's assistance here.

If we can add the above stored function to the database, we'll be able to swap out the ST_DISTANCE() call for ST_DISTANCE_SPHERE() instead.

CC @bsimmo.

I'm seeing this clicking on sensors marker and it appears intermittent or has a mind of it's own.

It's almost like the API thinks I've double-clicked the background.

Perhaps this is a default action or an unintended fall through in a case statement? The default should be do nothing if within a given radius of a marker since the user intended to click the marker but may have missed by a pixel or two.

I think it would be useful to add a message to our site that makes it clear where or readings come from. How about:

"The data displayed has been produced by low-cost devices developed by Connected Humber members as part of a community driven effort to build a network of smart sensors in the Humber Region"

We could add this to the changes pop-up that people see when the first hit the site.

Looking at the FTW sensor.

Data for this sensor is delayed (phone defra) so the chart always shows X hours behind. The cron job which gathers the data runs once per hour.

The odd thing is that the defra data for 12:00 today is on the system but it is 16:00 hours now. So the "time ago" ought to show 4 hours (or 5 ; DST/BST) but not 6.

When PM2.5 levels on several devices are in the 30 - 40 range the heat blows remain green but the DEFRA scale suggests they should be yellow. I don't recall every having seen anything other than green (apart from Brian's 3 devices in one location). I know that PM2.5 levels have been way above the green band many times so it now strikes me as 'wrong' that the map only ever displays green. Even if I increase the size of the blob, it remains green.

I can see yellow occasionally if two blobs start to merge but this is a different matter.

Appears when I open a device. If I click ok I can then select a tab to see data. Obviously a teething problem from the recent changes.

The default heat blob radius is rather large and (to a user without knowledge of the project) could indicate that the air quality of an extremely large area is being measured. When other sensors are added to the map, it may also create lots of clutter and overlap. Is it worth reducing the default heat blob radius to something smaller like 0.002?

Brian02, brian02 and brian04 have changed their names to the new standard but the old names are appearing in the pop up boxes.

I'm using the API to detect new devices and add them to my dev copy of the database.
I can get a list of all devices and the device_types info as a combined json BUT to be able to update my local database I need two separate calls:-

list-devices - add an option to return the raw devices table
device-id - add an option to return the raw device_types

I don't care if this is, instead, done with new actions like device-table,device-types-table

Having the raw tables means I can compare local to server and add new entries/remove old thus keeping our copies in sync (we don't care about the readings etc - we run subscriber scripts to capture that but it rejects unknown devices (because the database doesn't have .the requied information.

I'd have a go at writing new actions myself but I haven't gotten my head around your code as it is just yet. Also, thought you'd be able to do it a lot quicker. :-)

Just tried to access the AQ map using Internet Explorer - all I got was the heading - no map.

If a browser is not supported I think we should display a message to the user.

This happens when there is no recent data available - obviously.

Could your code be made to expand the timeframe. Eg. when data not seen today switch to past week or month etc. Chances are some data will then be present otherwise the device needs removing from the table.

When opening the data page buttons appear when there is no data. If it is the default view it gives the impression that there's no data at all.

Buttons with no data should either not be there or greyed out.

When the panel opens it should always default to one view where data is available.

We don't yet support mobile devices, but we've received repeated questions (exhibit a: #43) about mobile support.

We should investigate adding support for mobile devices to the web interface.

This is likely to be a significantly complicated undertaking though. The first order of business is going to be to test all the existing functionality and make a list of things that don't work or need polishing.

The map has a purple colored section but the highest reading is from freetown way at 30 something. Purple should kick in around 70.

Although a single line is ok for a short list does it become scrollable if the list is longer?

Would it be possible to filter the map by device name?
This would allow is to show just HCCxxxxxx sensors. Also, if we include pax counters we would then be able to just show those.
This needs more discussion , perhaps layers might be the way forward?

I've noticed that the x-axis in not locked to be linear, even though time tends to be.
So if data is missed, the axis can become compressed at that particular point (look at ben_pi01 for e.g. 24/02/19 before 12:00hr - 25/02/19 after 15:30hrs)

with MQTT data points can be missed, so these have to be accounted for as 'missed' and either left blank, or just let the line of fit do it's magic without it. The axis should remain linear throughout the time period specified.

Hope that helps.

as noticed and talk about on Slack, the count is increasing on merge, it should be an average (and whatever drop off) at it's simplest

URGENT: I tried systemctl stop mosquitto and restart but always comes up address in use

A suggestion that the default setting should be pm2.5 or pm10 as they are mandatory.

The current seeing of humidity means that any (e.g. Matt's) that do not send that data will fail.

👍

When clicking on a node it should default to a sensor reading - usually 2.5ppm but at the moment it seems there's no default so showing a pop up - no message is shown in the pop up, it's just blank.

Click ok to the pop up and then choose 2.5, 10, temp etc and the system works as normal..
:)

Now that the HTTP API supports CSV, we should add a button tot he device graph UI to export a CSV of the currently displayed data.

Perhaps we could also add an option for a 'high-quality' export too, where it doesn't average the data over such long time periods.

We'd need to ensure that we stream large datasets appropriately though if we encounter issues.

If you click on the top device in my cluster it opens on a no-data page.

If you look at brian02 its is a dust sensor giving just PM10 and PM25 values - yet the map shows Humidity, Temperature and pressure (HT&P). This has solved the problem of 'no data' but it's telling porkies. In fact the three sensors, 02,03 and 04, which don't report HT&P all have the same HT&P readings. I suspect your SQL has gone wonky.

Now that we're acquiring a number of devices (and the naming scheme, while great at defining a device's characteristics, is terrible at making it easy to tell devices apart quickly), we should probably look into implementing a 'last seen' feature, so that it becomes obvious which devices are currently offline.

• Return last-seen information in the HTTP API
• Display the last time a device was seen in the popup
• Change the colour of the marker for offline devices

We should setup a beta testing version of this web interface that runs the very latest from the dev branch.

Much of the work to do this has already been completed (you can see the WIP here, but currently we're getting an error in the logs:

[Thu Jul 18 23:58:43.107039 2019] [php7:error] [pid 3052] PHP Fatal error:  Uncaught Yosymfony\\Toml\\Exception\\ParseException: File "data/settings.toml" cannot be read. in /home/ci/sftp-root/Air-Quality-Web-Beta/www/vendor/yosymfony/toml/src/Toml.php:80\nStack trace:\n#0 /home/ci/sftp-root/Air-Quality-Web-Beta/www/lib/SBRL/TomlConfig.php(62): Yosymfony\\Toml\\Toml::parseFile('data/settings.t...', true)\n#1 /home/ci/sftp-root/Air-Quality-Web-Beta/www/api.php(23): SBRL\\TomlConfig->__construct('data/settings.t...', 'settings.defaul...')\n#2 {main}\n  thrown in /home/ci/sftp-root/Air-Quality-Web-Beta/www/vendor/yosymfony/toml/src/Toml.php on line 80

I've had this before when setting up the production version, but I can't for the life of me remember what the solution was right now. I'll have to think on it and come back to it.

It probably doesn't help that we're dealing with multiple levels of symbolic links, which are probably complicating matters somewhat here.

"Error:No data has been recorded from the device Id or it doesn't exist"

I suggest this is reworded slightly to

"Error:No data has been recorded from the device Id for that parameter in the selected time scale or it doesn't exist"

This would alert people to try clicking the buttons below.

When I click on a device the chart defaults to trying to display Humidity which causes an error mesage to appear since most devices don't record Humidity. Clicking on a device which does record humidity is fine,

The dialog message, should be reworded to something like "No data has been recorded for this chart tab. Try clicking on another tab".

It isn't really an error message anyway - it's more of an information thing.

Just added a new device - should sort itself out in 6 minutes - but when I opened the chart to check it all I got was a spinning thingy. It didn't display a 'no data' message.

the actions list-devices and device-info do not return the recently added altitude information.

Please can this be added. Thanks.

I'm using an Android device with stock Google Chrome.
Click on any node and the pop up opens up then close straight away. No data is shown.

When choosing 'desktop' mode the pop ups open and display data normally.

This has also been reported to me by another user using Android. He also tried desktop mode with similar positive results.

Give me a shout on mattermost if you want more info.
:-)

Made some progress. Built the code in the home folder first then copied it to /var/www/html.

chown -R www-data:www-data is best done AFTER running the build commands. I found doing that created the required data folders (albeit with some warning about circular links - and something about a leaflet dependency that should be manually installed - no I don't know what it was.

The notes on denying access for nginx and apache need to be clarified. In apache the file should be called .htaccess. In nginx you have to edit /etc/nginx/sites-available/default (or if using a virtual site edit the file belonging to that site).

I will go through all this one more time from scratch.