openenergymonitor / emoncms3 Goto Github PK

My local brewer would like to be able to enter the number of hectolitres of beer done each month, manually, and then have the €/hL cost show up in the dashboard, over time. This is where emoncms gets into delivering kpi's to business. Right now, business don't really know this key performance indicator and it's something that is vital to them.

Implementing modules in emoncms

The idea is to make it possible to extend emoncms by installing modules as you would do in many other web applications.

It would be great if a well implemented module system could allow a user to build the application up from the core framework by adding modules one by one.

A modules approach could allow for a more lightweight initial install approach, aiming for a core installation of maybe around 2mb? with options to expand as the user desires?
An example of a module that is not installed as default would be the energy auditing module for creating energy stacks http://openenergymonitor.org/emon/sustainable-energy/energy-stack-maker

It allows for separate version control:

It would be good if the module directory structure worked with git submodules which requires that a modules files, its controller/s, model/s and view/s need to be contained in one folder rather than split across multiple folders as is currently the case.

For example it could be:

Emoncms

• Modules
  • feed
    • feed_controller.php
    • feed_model.php
    • feed_view.php
    • feed_list_view.php

or it could be:

Emoncms

• Modules
  • feed
    • Controllers
      • feed_controller.php
    • Models
      • feed_model.php
    • Views
      • feed_view.php
      • feed_list_view.php

Module definition

One way to define a module is that each module has a unique controller but can have multiple models and multiple views. It can also depend on the models of other modules: a module can have dependencies.

If a module can have multiple models and multiple views should the models and views be in subfolders?

A modules operation can depend on other modules, however would it be beneficial to have an order of dependency:

Ie: statistics depends on user and feed
however user does not depend on statistics.
If both modules depended on each would it make sense to combine them?

Libraries

Modules can also depend on libraries ie: flot, ckeditor, jquery, jqui... Libraries seem to be usually javascript and are included by a modules views. Libraries could be stored in a separate folder to modules – the key distinction is that they have no main application controllers or models.

What would emoncms modules looks like?

Taking the approach that each module has a unique controller, the list below shows a list of modules reflecting the controllers that we currently have in emoncms:

Admin
api
confirm
dash
dashboard
dashboards
feed
input
multigraph
notify
process
statistics
theme
user
vis
..menu – does not currently have a controller but maybe it could be possible to configure the menu from the GUI in future and so would need a controller.

Dependencies

Core?
user
theme
confirm
menu
admin

Data core
input (user)
feed (user)
process (user, feed)
api (user, input, feed, process )

Visualisations
vis (user, feed)
multigraph (user, feed)

Dashboards
dash (user, feed)
dashboard (user, feed)
dashboards (user, feed)

Other
statistics (user, feed)
notify (user, feed)

A todo feature request, copied from feature plans and discussion on the openenergymonitor forums: http://openenergymonitor.org/emon/node/296

A todo feature request, copied from feature plans and discussion on the openenergymonitor forums: http://openenergymonitor.org/emon/node/296

Feature request, see: http://openenergymonitor.org/emon/node/296#comment-4321

One of the things I miss is the ability to post / display info/error messages.

I've cloned the repo and will see about a patch in the next few days.I'll send a PR when the posting part is ready. let me know if this is of interest to anyone else...

About me, a bit of a newby at openenergy.org, using JeeNodes since 3 years to handle my underfloor heating, solar water heating and various other things in our solar powered house. No grid connection here other than internet.

As seen in the screenshot below, positive values appear to be rounded to the nearest whole number once they are greater than 10. But the same formatting seems to be absent for negative values.

Bug or not? I'll let you decide.

Great software, by the way!

Table 'emoncms.users' doesn't existto run script uncomment runnable

the login-data to the database are fine, it seems the php script cant create any tables. What to do?

//UPDATE: i have to read the notes.

if (!$runnable)
{
  echo _("to run script uncomment runnable");
  die ;
}

all fine.

A todo feature request, copied from feature plans and discussion on the openenergymonitor forums: http://openenergymonitor.org/emon/node/296

Add option for disabling further user creation after first (admin) user is created. This could be disabling of public register feature while still allowing the admin user to create and invite users.

Hi, I'm trying to setup a new emoncms3 install.

I've got passed having to un-comment "runnable" from setup.php.

Now I can see the login model, but typing in a new username and password, and clicking "register", gets me:

Not Found

The requested URL /emoncms3/user/create was not found on this server.

regards,
Murray

A todo feature request, copied from feature plans and discussion on the openenergymonitor forums: http://openenergymonitor.org/emon/node/296

What's the best way to do this with dashboards?. Maybe it would be good to have two types of dashboards mobile optimised dashboards and dashboards for larger screens.

Line 30 index.php $q = preg_replace('/[^.\/a-z]/','',$_GET['q']); // filter out all except a-z / .

should be $q = preg_replace('/[^.\/A-z]/','',$_GET['q']); // filter out all except a-z / .

Background:
Creating a db query are more costly then executing a complex query.
The strategy in emoncms seams to be creating db queries instead of doing complex queries and that kills performance, on a fast system it may not be noticeable but on slow boxes and i guess on an Raspberry PI its very noticeable.
Performance is a must and to keep emoncms in the pole position this has to be done right...

function get_user_feeds creates 1 + nr_of_feeds * 2 queries and can be done in 1, this function seams to be hitted every 10th second per opened page

function get_user_feeds($userid,$status)
{
    $result = db_query("SELECT id, name, tag, time, value, DATA_LENGTH + INDEX_LENGTH size, datatype, status ".
    "FROM feeds f, feed_relation r, information_schema.tables i ".
    "WHERE userid = '$userid' and r.feedid = f.id and status = $status ".
    "and table_name = concat('feed_' , f.id) and table_schema = DATABASE()");

    $feeds = array();

    if ($result) {
        $qfeeds = "";
        $feedids = array();
        while ($row = db_fetch_array($result)) {
            $feeds[] = array($row['id'],$row['name'],$row['tag'],strtotime($row['time'])*1000,$row['value'],$row['size'], $row['datatype'], $row['status']);
            $feedids[] = $row['id'];

        }
    }

    usort($feeds, 'compare');       // Sort feeds by tag's
    return $feeds;
}

Its the same with function get_user_feeds_size and it can be taken down to 2 queries.

function get_user_feeds_size($userid)
{
    $result = db_query("SELECT * FROM feed_relation WHERE userid = '$userid'");
    $total = 0;
    if ($result) {
        $feedids = array();
        while ($row = db_fetch_array($result)) {
            $feedids[] = $row['feedid'];
        }
        $feeds = "";
        foreach ($feedids as $feedid) {
            //          $total += get_feedtable_size($row['feedid']);
            if(!empty($feeds)) {
                $feeds .= " or";
            }
            $feeds .= " table_name = 'feed_".$feedid."'";
        }
        $result = db_query("SELECT SUM(DATA_LENGTH + INDEX_LENGTH) TOTAL_LENGTH FROM information_schema.tables WHERE (".$feeds.") and table_schema = DATABASE()");
        $row = db_fetch_array($result);
        $total = $row['TOTAL_LENGTH'];
    }

    return $total;
}

Function get_feed_data are creating 1000 queries per feed if you display a multi-graph chart!
And it can be reduced to 1 query, its not doing 100% the same but in my opinion its more correct to get an average value between the time slots.

function get_feed_data($feedid,$start,$end,$dp)
{
    if ($end == 0) $end = time()*1000;

    $feedname = "feed_".trim($feedid)."";
    $start = $start/1000; $end = $end/1000;

    $data = array();
    if ($dp <= 0)   {
        dp = 1;
    }

    $range = $end - $start;
    $td = $range / $dp;

    $result = db_query("SELECT convert(avg(time), UNSIGNED) time, avg(data) data FROM $feedname WHERE `time` > $start AND `time` < $end group by convert(time / $td, UNSIGNED) order by time Asc");
    while($row = db_fetch_array($result)) {
        $dataValue = $row['data'];
        $time = $row['time'] * 1000;
        $data[] = array($time , $dataValue);
    }
    return $data;
}

There are much more to be done and I will create issues when I find some show stoppers.

Feature idea: a nice URL system where a user can access their account via:

emoncms/username

If a user shared a dashboard publicly anyone can access the users dashboard via their username otherwise a login form is supplied.

To see how this might look its easy to add a few lines of code. In index.php add:

if ($output == null)
{
$userid = get_user_id($controller);
if ($userid) {
$session['userid'] = $userid;
$session['read'] = 1;
$action = "run";
$output = controller("dashboard");
}
}

just below the line $output = controller($controller);

The next step would be to have dashboard pages linked through to the url ie: emoncms/username/solar

Can they be merged into one?

Would be useful to allow alphanumeric or 64bit hex string node names (node parameter in post URL) to simplify commissioning of remote nodes using IEEE802.15.4 radios with EUI64 MAC address as node Id. I have implemented this in a local copy with only some minor changes to master branch code and it seems to work ok.

Hello everyone

For a few days in my emoncms of my raspi appeared a duplicate Node-Key input
from another Raspi.

The data received was associated with the new Node-Key input, so they were not saved in the feed (attach screenshot). This problem started after a scheduled reboot.

After studying the case, I think it's because the apache server starts faster than the mysql server, so emoncms believed that there was no such input and created a new one while start mysql.

I fixed it in the following way:
1.- In MySQL and deleted the new inputs from the 'input' table
2.- Reboot Raspi

Anyway, I think the following changes should be made:

• Modify the emoncms.input table: Do not allow duplicates of nodeid-name

View Server Information after the image.

Emoncms
Version
low-write 9.8.30 : 2018.05.08



Modules
Administration : App v1.1.0 : Backup v1.1.2 : EmonHub Config v1.0.0 : Dashboard v1.2.0 : EventProcesses : Feed : Graph v1.2.0 : Input : notify : postprocess : CoreProcess : Schedule : setup : sync : Time : User : Visualisation : WiFi v1.0.0



Buffer
loading...



Writer
Daemon is running with sleep 60s


Server
OS
Linux 4.9.35+



Host
*********** (127.0.1.1)



Date
2018-09-16 21:29:28 UTC



Uptime
21:29:28 up 38 min,  1 user,  load average: 1.37, 0.79, 0.80


HTTP
Server
Apache/2.4.10 (Raspbian) HTTP/1.1 CGI/1.1 443


MySQL
Version
5.5.57-0+deb8u1



Host
localhost (127.0.0.1)



Date
2018-09-16 21:29:28 (UTC 00:00‌​)



Stats
Uptime: 2238  Threads: 3  Questions: 4411  Slow queries: 0  Opens: 60  Flush tables: 1  Open tables: 53  Queries per second avg: 1.970


Redis
Version
2.8.17



Host
localhost:6379 (127.0.0.1)



Size
395 keys  (548.87K)



Uptime
0 days


MQTT
Version
1.4.14



Host
localhost:1883 (127.0.0.1)


Pi
CPU Temp
55.15°C



Release
emonSD-26Oct17



File-system
Set root file-system temporarily to read-write, (default read-only)


Memory
RAM
Used: 49.34% Total: 481.7 MB Used: 237.65 MB Free: 244.05 MB


Disk
Mount
Stats



/
Used: 64.48% Total: 3.33 GB Used: 2.15 GB Free: 1.02 GB



/boot
Used: 36.27% Total: 59.95 MB Used: 21.74 MB Free: 38.2 MB



/home/pi/data
Used: 0.44% Total: 25.37 GB Used: 113.92 MB Free: 23.95 GB


PHP
Version
5.6.33-0+deb8u1 (Zend Version 2.6.0)



Modules
apache2handler : bcmath : bz2 : calendar : Core v5.6.33-0+deb8u1 : ctype : curl : date v5.6.33-0+deb8u1 : dba : dio v0.0.4RC4 : dom v20031129 : ereg : exif v1.4 : fileinfo v1.0.5 : filter v0.11.0 : ftp : gettext : hash v1.0 : iconv : json v1.3.6 : libxml : mbstring : mcrypt : mhash : mosquitto v0.3.0 : mysql v1.0 : mysqli v0.1 : openssl : pcre : PDO v1.0.4dev : pdo_mysql v1.0.2 : Phar v2.0.2 : posix : readline v5.6.33-0+deb8u1 : redis v2.2.7 : Reflection : session : shmop : SimpleXML v0.1 : soap : sockets : SPL v0.2 : standard v5.6.33-0+deb8u1 : sysvmsg : sysvsem : sysvshm : tokenizer v0.1 : wddx : xml : xmlreader v0.1 : xmlwriter v0.1 : Zend OPcache v7.0.6-devFE : zip v1.12.5 : zlib v2.0 :

To be able to use various custom CSS style sheets you have to get the emoncms theme from the user preferences. Now it's hard-coded in the index.php file with this line of code:

// Set emoncms theme TODO: get from user preferences
$GLOBALS['theme'] = 'basic';

What I want is to track what I've imported and exported, perhaps a kWh/d process that only increments on negative values and another kWh/d that only increments on positive values.
eg:
If I generate 15kWh from solar PV, and use 13kWh over a day, the standard KWh/d input processor will report my consumption as -2kWh, This doesn't account for anything I've imported at night, when the solar PV isn't generating.

Original discussion here:-
http://openenergymonitor.org/emon/node/737

Hi All,

I'm just throwing this out there, to see if someone can help steer me in right direction.

I've got three years of historical timestamped kWh data, for 36 meters, at 15min intervals. I want to get it into emonCMS.

I can transform it into the format expected in emonCMS database: "unixtimestamp", "value".

I can make 36 tables, named feed_1 to feed_36, and insert the data.

I can insert 36 entries into the table "feeds", with a datatype of 3.

What next? I'm not sure how the other feed types get generated, like powerkwhd.

I'll give it a bash tonight and see what happens.... any info appreciated. I'll fork the project and contribute anything useful I come across.

regards,
Murray

A start has been made on this, but it may be better to implement it differently using gettext: http://mel.melaxis.com/devblog/2005/08/06/localizing-php-web-sites-using-gettext/

I'd suggest an export wizard ; though there's a decent amount of work in this. Alternatively, you will need to give people using the emoncms.org site the option to get their data.... Don't think you want to let them logon to the db so the export option is the way to go. Phase 1 might just be one years data, csv format [one click export] ; phase 2 might be the whole wizard?

This would be a section of the documentation that would explore common energy monitoring web app dev challenges in general such as:

• API Design ( REST interface)
• Data Processing: Power to kWh/d, histogram analysis
• Data Storage: MYSQL, table per feed, unix time indexed data
• Visualisation: Multigraph, Zooming through big data, LOD level of detail, AJAX realtime.

This would give some background as to why various elements in emoncms have been coded the way they have and also hopefully help others interested in developing other open source web applications from scratch how to solve similar challenges.

A todo feature request, copied from feature plans and discussion on the openenergymonitor forums: http://openenergymonitor.org/emon/node/296

A todo feature request, copied from feature plans and discussion on the openenergymonitor forums: http://openenergymonitor.org/emon/node/296

This is a widget that I designed for seeing the temperature distribution in a hot water cylinder. Need to add it in.

My publicly available graphs are not displaying when I try to access them from the internet because the source location for the iframe that displays them is pointing to my LAN IP instead of my external URL.

It is interesting to note that if I login, I can view my graphs from the dashboard page just fine when I am accessing the internet.

I hope this screenshot is useful...

Hi there,

Since 5831f49 there is a little problem with Views/feed/list_view.php and Views/input/list_view.php:

This peace of code is not working :
var update = (new Date(feeds[z][3])).getTime();
since Date() can not take MySQL raw date as an arg.

A a small part of the answer here : http://stackoverflow.com/questions/1765803/convert-a-mysql-date-to-javascript-date

Thanks a lot for your amazing work !

I followed the installation manual on the website
1.Created the databse
2. Uncommented $runnable = TRUE;
3. Run http://localhost/openenergymonitor/setup.php
and got the followoing

Fatal error: Call to undefined function _() in C:\wamp_Sites\openenergymonitor\setup.php on line 22

Class missing ?

Cheers Guby

A todo feature request, copied from feature plans and discussion on the openenergymonitor forums: http://openenergymonitor.org/emon/node/296

These would be buttons in the dashboard editor that would allow for easy insertion of a multigraph (maybe the saved version - as in the most recent multi-graph implementation) and kWhdZoomer