brown-ssw / brown-solar-charger Goto Github PK

in the meantime, I've made an error with that function not trigger the alert email for firebase issues

Don't think this exists so here it is.

To create the interactive, live portions of this website necessitates 3 actions on the client-side:

• retrieving the dataset (in .json) from an external database (in this case firebase)
• transforming that data into a format that can be used by JS functions
• passing this newly formatted data into function for gauges, plots, and more

The latter two are not too difficult with some experimentation; its this first point that has proved the most troublesome, especially as we would need to handle multiple jsons (monthly data, daily data, live data).
Some cases to consider are:

• firebase stops providing the json (json is inaccessible)
• handling multiple jsons; each json would necessitate its own xmlhttprequest. The benefit of this is that we can set custom intervals for each json to refresh. However, it is essential to ensure that we are not making more requests than needed.
  • Important to note that these custom intervals can vary wildly (refresh on the hour for daily data and every x minutes/second for live data)
• Station shuts down or drops below the minimum battery threshold (50% of max capacity). This shouldn't be a problem provided the station is able to report if is going to shut down or drop below minimum battery threshold.
• We take the station down for any reason. Firebase should be up 24/7, so we would need to indicate station availability another way. Previously, we discussed adding parameters to the settings part of our json that can let us manually set station availability, and even a custom message to display while the station is down.

function loadJSON2(sub, cb, time) {
    /* where,
    sub is specific portion of json to load
    cb is callback function to run
    time is an object containing ideal and alt times to wait in ms
    */
    //http request to get the json
    var linky = base_url + sub + end_bit;
    var http = new XMLHttpRequest();
    http.open('get', linky); // get the json
    http.onreadystatechange = function() {
        if (this.readyState == 4) { //was I able to access the json?
            if (this.status == 200) {
                var json = JSON.parse(http.responseText);
                cb(json);
                setTimeout(loadJSON2, time.up, sub, cb, time);
            } else {
                isDown2();
                setTimeout(loadJSON2, time.down, sub, cb, time);
            }
        }
    };
    http.send();
}

I've been giving quite a bit of thought to resolving that last bullet point. Currently, the website retrieves the dataset on an interval of our choosing, but this is done in a recursive manner, so we change the interval dynamically and avoid repetitive calls. Even the settings json must refresh on an interval
There are 3 cases to acknowledge in regards to manually taking down the station:

• we shut down the station (true -> false): The program must halt all current timeouts (as no other station , display our custom message, and disable any plots, gauges, etc. as needed. This poses an interesting predicament as timeouts are defined recursively, so attempting to disable them all proves a challenge. It is also essential to make sure that we do not halt the timeouts for loading the settings json. Adding another setTimeout() call in this case may prove useful
• We turn the station back on (false -> true): The program must resume all timeouts for live, monthly, etc. data. This can be done by refreshing the page upon detection of the station turning back on or by calling setTimeout() once more for these datasets.
• the station is still in the same state as before, in which case we should do nothing in regards to other timeouts.

We can define global variables in js to track the status in this way, and assume "true" by default (on initial page load/refreshes) to handle the edge case where someone visits the dashboard while the station has been taken down. Ultimately, the challenge here seems to be defining a comprehensive callback function for the settings json and ensuring that we can dynamically clear timeouts.

I apologize if this seems convoluted; the above were some of my thoughts I had while brainstorming. Please reply if you have any questions or suggestions!

over MQTT
what needs to be sent, and how often

This should probably be a javascript function.
You can use the drawGraph1 function that's already in the code as a starting point if you want. It displays a simple bar graph of json data, and can be resized, but isn't easily reusable for other graphs and is missing many things like labels and axes.
I'm likely missing something, and feel free to go far outside this structure if you want to or for any reason, but here's some ideas to start with:
some parameters you might want the function to expect
name of the canvas to draw on
name of the div that canvas is inside of (for resizing)
data to display (for drawGraph1 I think you'd want to get json_obj.graph1)
color and style options
scale for y axis (or do auto scaling) You may want to have scaling or unit data somewhere in the json data-just make an issue asking for it
labels for axes, and maybe for numerical value of bar graph
maybe have a line or scatter plot option?

And you can ignore all of this if you'd rather figure out how to use someone else's library to make plots!

I think the script was either finished or very close, but this issue is just a reminder to come back and check it over once the esp32 is sending data to make sure everything matches up

use SPIFFS?

hopefully these settings would last between code uploads, and would be set from the firebase database

some variable names are confusing, and maybe some of the logic could be commented

Currently, the esp32 waits until it is connected to a wifi network before continuing to start, doing a full reboot if there's no connection after 10 seconds. That's fine for testing, but isn't a particularly good solution since not having a wifi network available could make the station completely non functional (a power outage shouldn't shut down a solar station!)

Test restarting just the wifi connection (calling Wifi.begin() again?) instead of a hard restart.

This does mean that every other piece of code that expects to have internet connection needs to be able to handle not actually having a connection. Make more issues as needed.

In general, since the esp32 may be responsible for keeping some parts of the station active (turning the inverter on and off, solar tracking) we should always try to keep the code running even if some components have issues.

Doug Wilkinson suggested that we think of a way to automatically reset the esp32 if it locks up.

ESP32s seem to have some internal watchdog timer features-look into enabling that

A second microcontroller could reset the esp32 if the esp32 stops telling the microcontroller that it is functioning. The second microcontroller could be a much simpler device, or maybe a second esp32 that could actually communicate with us or even take over.

"daily" data should be removed after 24 hours, "month data" should be removed after 30 days

this will keep the database from growing too large

Could be done by firebase functions (requires blaze plan), by esp32 on station possibly, maybe by script running from spreadsheet, google cloud platform may give a free low power virtual machine?
Another way to have the esp32 do this is have it store the data locally and overwrite each array completely each update (bonus, could write data as proper json array)
Experiments are needed

It could look good to display some values with a speedometer style dial indicator.

Similar to the graphs, this could be a javascript function that gets the data to display, what range to show, and some formatting information

following the JSON format we chose, the esp32 needs to be able to add data to the database, and calculate values for day and month long intervals either from internally stored data or by pulling the database data down and processing it

Currently if there is an error while starting the file system that holds the html file the esp32 serial prints an error than freezes itself.
That's fine for now, but a better response to the error would be to still boot, just show a simple error page instead of the full html file.
In general, since the esp32 may be responsible for keeping some parts of the station active (turning the inverter on and off, solar tracking) we should always try to keep the code running even if some components have issues.

cumulative wh calcs are started and there may be others (like for monthData)

does it need to do something to reconnect if the wifi goes down?

The website should be able to get relatively up to date data (maybe even up to the minute), along with graphs over the past day or two, and longer term graphs over weeks and months.
It would be useful to log quite frequent data somewhere that we could analyze (this issue is related to issues #8 and #9 ).
Long term data shouldn't be only in RAM in case power is lost.

If we can get an sd card working, that would be great since it would provide lots of read and writable space that lasts even if the esp32 is reset.
Sending data to a google sheets would be a great way to log data in a way that's more accessible to us than a card inside the station, but might not be as reliable of a place to keep data the esp32 might want to read back and work with.

• on boot (maybe be able to tell how long it was off for?)
• solar charger errors
• circuit breakers

To do for basic website design:

• Create Figma page
• Implement HTML and CSS for page
• Show to group

after planning, make issues for website components that need to be made, and an issue for exactly what is expected to be in the json data. Make sure to say how the json data should define the scale or units of it's data.
Think about what data should be showed in graphs, and what intervals. Should graphs be over constant lengths of time, or should a daily graph extend as the day goes by?

loop() should be able to run quickly, so use of delay() should be avoided

see this forum post for a great example of a method to do this

@vskbellala designed a format, hopefully it can be made by an esp32 easily

I know that isn't a specific issue, this is just to remind us that we need to figure out what needs to be done.

I'm pretty sure that the current libraries that the esp32 has for handling timestamps have not been updated to be able to handle dates past 2038 since a larger variable is needed to hold Unix epoch time. The station might not last for 17 years, and the batteries certainly won't. Before 2038 I expect the time libraries will be updated and the code in the station could be updated if needed. Just something fun to think about.

Not really on the same topic, but the code running on the esp32 might sometimes crash. One thing to look into is whether the esp32 can detect that and reset itself. Something else to think about would be having a second simpler processor monitoring the esp32 (maybe listening for a periodic signal from the esp32?) and having the capability to reset the esp32 and maybe also have some control over turning off parts of the station like the inverter if the esp32 is usually doing the overcurrent protection (but there are also fuses).

esp32s can be reprogrammed over wifi which would be really convenient to have for the station so we don't have to open it up to update the code.
things to look into:

• turn OTA on and off through database (could help with security)
• integrate OTA with current code

3 ways to connect for OTA:

• esp32 makes an AP
• connect to esp32 through brown's wifi network
• this article is neat, I think it might let us reprogram the esp32 even if we don't have a direct wifi connection to it

we can discuss changes to this, but I'm thinking something like monthly usage summaries with a dump of that month's detailed logs along with immediate error reports if the station goes offline.

What data and status information do we want to show?
What types of displays are the best for showing that

Find a factor to convert watt hours to amount of co2 saved for RI / Brown's form of energy generation

if the esp32 could put data into a google sheet, that could be a great way to log data

Maybe a google script can ask the server for its data and load that into the sheet?

get some constant values, and maybe other settings too

I don't understand the Json library particularly well so maybe I'm wrong that this would help, but here's something that might give the esp32 a bit less work to do.

convert the json document (dataDoc) to a globally declared String right after updating the data in the json document. Then when the data is requested the String can be sent right away without doing the string conversion each time

firebaseAvailable gets flipped true and false if some things (like reads) can be done but writes can't be.

perhaps the best solution would be to start a timer or something and only allow one alert per day or so

in theory it would be cool to not reset the alert until every firebase function starts working again, but that could be difficult

an sd card would give lots of space for keeping logs of data to analyze later to find out how the station is getting used

an sd card could also be a way for the esp32 to "remember" even if it gets reset

I think there are example programs that attach files on an sd card to an email

It would be good for it to know if the email went through or if it should try resending.

After that, it should be made to print less stuff

Even if the website isn't hosted by the station, I think the station still needs to be set up as a server so it can be asked for the json data.
Besides internet connection that we can probably do ourselves following this guide.
That would mean making the esp32 publicly accessible and giving it at least a constant address if not a nice url. I think we want to be put in the DDMZ so we can be "accessed from the internet". I don't know how hard it is to get an esp32 to pass their security scans. In theory you can get a url by going to the help desk and requesting a name though Luke's experience says we shouldn't expect this.

Alternatives not requiring anything from Brown IT:
Have the station call the website instead of the website contacting the station. (sending a POST request with the data to the server?) This would probably require more of a back end for the website.

Have a second esp32 or a computer or server somewhere to receive data from the station and act as the backend for a static website. I'd rather not have the station relying on bouncing data off an esp32 at my parents' house, but in theory that would work. Another idea I don't really like but could theoretically work would be using google sheets as the backend database.

• Layouts
  • carousel? multiple subpages of information? where do I put the blurbs? etc.
• Visualizations
• Humanize the data; make it more understandable
• You can charge 20 iphones with this station
• At the current power state, it will take x hours to charge your laptop to 100%

Just some interesting ideas of what we could add!

I don't expect to be able to write much code until we have access to the solar charger. I'm just making this issue now as a place to put research and documentation.

Renogy is surprisingly supportive on their support forum of people making projects that interface with their chargers
(i’ve never heard these terms before, so don’t trust this, but it seems that) the charge controller has an rj45 (ethernet cable style) plug and uses the RS-485 electrical interface with the Modbus communication protocol so you can ask it for data or change settings according to this reference from Renogy.
Just some of the information supposedly available is current and voltage from solar panels, current and voltage to battery, battery charge state, even some longer term data
(power to inverter is the only thing that would have to be measured separately)
charger