Data appears incomplete when compared to other DCC fed sources. about ha-hildebrandglow-dcc HOT 58 OPEN

Hey all, as correctly deduced by some of you, this issue lies with the DCC/Bright providing the data later than the 30 minute delay, causing Home Assistant to be further behind and miss out the end of the day.

This seems to have deteriorated as 2023 progressed, and so I have contacted Hildebrand to see if they have any pointers.

I do have an idea on how to resolve this, but I'm not 100% sure it'll work - watch this space!

from ha-hildebrandglow-dcc.

I emailed Hildebrand support yesterday to ask if it was a widespread issue

I had a detailed and friendly reply, it seems that the data gaps are a capacity issue due to the popularity of the service, they are upgrading the infrastructure as soon as they can.

from ha-hildebrandglow-dcc.

The IHD is the way to go.
Great youtube video here on how to set it up.
https://youtu.be/r6M9czhUmtA

from ha-hildebrandglow-dcc.

I recently added this integration to HA and like others I am getting incomplete data. The data I am getting through the Hildebrand app is also incomplete.

I'm frustrated that it is so difficult / costly to get access to smart meter data. My smart meter data should be freely accessible to me on my phone at all times (not on a stand-alone smart meter display). This discussion has inspired me to start a petition.

from ha-hildebrandglow-dcc.

Looking at my own history charts & those posted by others in issue #292 , it seems to be the kWh consumed anywhere between 12am & around 8am (or possibly all the way up to 12pm) are lost/reset. So if you are charging your car (or other significant consumption) overnight, then that would be why you see a significant discrepancy.

Hopefully @HandyHat is on the case.

from ha-hildebrandglow-dcc.

I am seeing this too. The new algorithm is considerably under reporting compared with the bright app.
I think the issue may be the new algorithm does not “catch up” if there is a missed reading. The original algorithm caught up so was accurate, albeit putting the consumption in the wrong time slot.

from ha-hildebrandglow-dcc.

Looking at my own history charts & those posted by others in issue #292 , it seems to be the kWh consumed anywhere between 12am & around 8am (or possibly all the way up to 12pm) are lost/reset. So if you are charging your car (or other significant consumption) overnight, then that would be why you see a significant discrepancy.

Hopefully @HandyHat is on the case.

I have similar issues. Big gaps in the data and the daily totals are way out compared to actual usage.

This was yesterday, bearing in mind we have storage heaters and most of our energy use is between 00.30 and 07.30!

from ha-hildebrandglow-dcc.

I came across this update from Bright about their API. it sheds some light onto the issue and perhaps offers a way forward?

https://forum.glowmarkt.com/index.php?p=/discussion/347/dcc-data-only-changes

from ha-hildebrandglow-dcc.

Same for me. Data is not usable now.

from ha-hildebrandglow-dcc.

My Bright app isn’t much better anymore! This is today!

from ha-hildebrandglow-dcc.

I also contacted Hildebrand and they were talking about a couple of weeks to upgrade their infra, so this issue is likely to be around for a while. They did mention that if you have the CAD then you shouldn't have these issues so I can test that when mine arrives in a few days. It should connect directly with the meter.

from ha-hildebrandglow-dcc.

Interesting, so is this an issue at Hildebrand with their infrastructure rather than with this integration?

I am keen to get my hands on one of their SMETS CAD's without display, but been on the waiting list for ages now and still no sign. Get the impression it's in high demand so when their new production run arrives it'll be sold out almost immediately.

from ha-hildebrandglow-dcc.

So it looks that this integration is useless now...
Is it only way now to get actual data in real time is using Hildebrand IHD?

from ha-hildebrandglow-dcc.

This is disappointing as it seemed like a good way to get my gas usage - I can get my electricity from my inverter. I'll have to look at other options instead.

from ha-hildebrandglow-dcc.

Will the CAD be able to bring in historical data. Thinking of taking the £70 hit

Based on info sent to me from Glow Support:-
The CAD provides (almost) realtime power and the meter reading at that point in time, every ten seconds. It does the same for gas and it gives tariff information/standing charge so HA could calculate the cost if you want. For electric, it can differentiate between Import and Export but the Export figure may be limited to the meter reading only - depends on firmware version and meter manufacturer. As well as sending to HA, it also sends this data in realtime to the Bright servers so the Bright App on your phone will also show realtime power.

It does not send any historical data to HA and if it did, I can't see how HA would use that data. Once the required sensors are created and being used, HA will build it's own history going forward but it does not go back in time prior to the date/time when the sensors were created.

The CAD will also calculate the usage for the half-hourly periods we are familiar with in the App and hold this locally in the CAD. Overnight (or some time the next day if the Bright App is anything to go by), the 'official' half -hourly figures held in the DCC are downloaded to the CAD and overwrite the half-hourly figures created by the CAD for the previous day. In this way, the CAD and Bright App will having matching figures and history.

from ha-hildebrandglow-dcc.

I recently added this integration to HA and like others I am getting incomplete data. The data I am getting through the Hildebrand app is also incomplete.

I'm frustrated that it is so difficult / costly to get access to smart meter data. My smart meter data should be freely accessible to me on my phone at all times (not on a stand-alone smart meter display). This discussion has inspired me to start a petition.

Signed!

from ha-hildebrandglow-dcc.

I recently added this integration to HA and like others I am getting incomplete data. The data I am getting through the Hildebrand app is also incomplete.

I'm frustrated that it is so difficult / costly to get access to smart meter data. My smart meter data should be freely accessible to me on my phone at all times (not on a stand-alone smart meter display). This discussion has inspired me to start a petition.

Signed

from ha-hildebrandglow-dcc.

I bit the bullet and bought the CAD with display (I have a SMETS2 meter) a couple of weeks ago.
It was £73.34 including shipping, it arrived 2 days later. I plugged it in, and was fully operational by the next day.
It is an amazing experience to see near real time data going into Home Assistant. I've set up alerts for when we're feeding back to the grid as our contract means we don't get anything for it, so the investment will pay off for our circumstances.
Plus, I get the fun of creating "near real time" dashboards showing energy consumed / produced, how our bill is mounting up over the month, and how much we're saving with our solar panels now that we can better manage our usage on the sunny days.
Set up was pretty clear.
I don't think you'll ever get near real time data over the internet, there's just too many points of failure between the 4G SIM cards in the "Smart" meters, the servers at the electric companies moving massive amounts of data, then in this case an extra hop to Hildebrand servers, and then back to your HA. Just sayin'.

from ha-hildebrandglow-dcc.

I recently added this integration to HA and like others I am getting incomplete data. The data I am getting through the Hildebrand app is also incomplete.

I'm frustrated that it is so difficult / costly to get access to smart meter data. My smart meter data should be freely accessible to me on my phone at all times (not on a stand-alone smart meter display). This discussion has inspired me to start a petition.

Signed

from ha-hildebrandglow-dcc.

I seems that integration is intentionally broken for us to buy them IHD...

from ha-hildebrandglow-dcc.

I bit the bullet and bought the CAD with display (I have a SMETS2 meter) a couple of weeks ago. It was £73.34 including shipping, it arrived 2 days later. I plugged it in, and was fully operational by the next day. It is an amazing experience to see near real time data going into Home Assistant. I've set up alerts for when we're feeding back to the grid as our contract means we don't get anything for it, so the investment will pay off for our circumstances. Plus, I get the fun of creating "near real time" dashboards showing energy consumed / produced, how our bill is mounting up over the month, and how much we're saving with our solar panels now that we can better manage our usage on the sunny days. Set up was pretty clear. I don't think you'll ever get near real time data over the internet, there's just too many points of failure between the 4G SIM cards in the "Smart" meters, the servers at the electric companies moving massive amounts of data, then in this case an extra hop to Hildebrand servers, and then back to your HA. Just sayin'.

This sounds great.

I also have some solar panels and do not want to export to the grid. My current methods for measuring grid instant usage is an LDR on the pulsing LED on the actual main unit. This works and lines up with what is on my meter display. But, it is not aware of the direction of flow. So if I suddenly start shifting 300w into the grid, it just shows me 300w grid usage in home assistant. Not 0 or -300w. I have set up some automation to act as power dumps to stop exporting, but it's not perfect.

Does their CAD show instant data in HASS and, does it differentiate between grid import/export? Or even just show relative positive/negative figure? Buying it now if it does. (Edit: Yes it does)

I did raise a support case before coming here. Pretty much a copy paste of some of the responses above. Pushing the CAD/IHD.

EDIT: Didn't see the other responses above. It does exactly that. :) Also signed the petition.

from ha-hildebrandglow-dcc.

So if I've understood everything so far, the issue we have been seeing in Home Assistant with this integration, is not in fact an issue with the integration itself? It's down to some changes made by GlowMarkt on their existing API and that they are awaiting an upgrade to their existing capacity to the DCC (weeks) and until that happens, data can only be retrieved on a 2 hourly basis (will that work for this integration?)

As a result of this issue, I started looking at alternative options (bar paying £70 for a GlowMarkt CAD). My energy provider in the UK is Octopus Energy - they have developed a Home Assistant integration (see https://github.com/BottlecapDave/HomeAssistant-OctopusEnergy) but currently this is very limited providing only ONE daily electric & gas reading for the previous 24hrs. However, for their own customers, they are developing 'Octopus Home Mini' (see https://octopus.energy/octopus-home-mini-faq/) which is a ZigBee device which connects directly with your SmartMeters and provides 'live' meter data. Using the integration, the 'Octopus Home Mini' can provide data to Home Assistant. Currently the 'Octopus Home Mini' is not generally available (it's in Beta), but customers can apply to go on the waiting list to receive one (I've no idea how long the wait is).

from ha-hildebrandglow-dcc.

I think this is the same as issue #292 where the accumulating cost appears to go down/reset at approximately 1.30am, resulting in the days total cost being lower than it otherwise should be.

from ha-hildebrandglow-dcc.

I think this is the same as issue #292 where the accumulating cost appears to go down/reset at approximately 1.30am, resulting in the days total cost being lower than it otherwise should be.

Homes assistant is reporting 18.1kWh usage yesterday Loop energy and iHD are reporting 26.3kWh that's a significant difference, and my EV charger is reporting 17kWh usage yesterday just to charge my car.

from ha-hildebrandglow-dcc.

Do you charge your car overnight? If HomeAssistant is losing/resetting it's energy accumulation around 1.30am, then perhaps that would account for the significant discrepancy lost while charging your car. Presumably if you don't charge your car (or avoid using energy 12am-2am) then the discrepancy would be much smaller.

I think the developer is investigating (hopefully).

Is the integration losing/resetting the kWh consumed? Or is it just the cost of the kWh consumed?

from ha-hildebrandglow-dcc.

Whether this issue is linked to the resetting of data of not, my DCC data seems to begin the day increasing normally until between 0102-0104 hours and then stops with various issues (readings going up/down and remaining flat) until resuming at a later point in the day when it fails to catch up with the DCC recorded totals.

I am also using a template sensor to sum the majority of my home's usage through kasa and tapo wifi plugs combined with powercalc integration to do my lights and appliances with stable/constant usage. When I map both and compare with bright app readings both come up short but the DCC via HA is dreadful. Bright app for the 6th is 8.98, 7th 6.804, 8th 6.735
The system has gone until 1733 hours before starting to increase normally but never catch up.

from ha-hildebrandglow-dcc.

Also seeing large gaps in my data. Last time I had a full day of data was 23rd Jan. Mostly seems to be missing between 2am to around 8am ±1hr. Today is particularly bad as I only have 4 data points so far (00:00, 01:00, 12:00 and 18:00).

from ha-hildebrandglow-dcc.

I too have gaps in the data, just as @Markbez graph above and others.
In my case at least, during the day I also have gaps in the data on the phone app as well, so I think for me it is possibly a server side issue (I emailed Hildebrand support yesterday to ask if it was a widespread issue, awaiting their reply - although I appreciate API access to their servers might not be a priority for them).
Interestingly, sometime later the missing data in the app seems to get backfilled, but this doesn't also backfill into Home Assistant.

from ha-hildebrandglow-dcc.

Looking at my own history charts & those posted by others in issue #292 , it seems to be the kWh consumed anywhere between 12am & around 8am (or possibly all the way up to 12pm) are lost/reset. So if you are charging your car (or other significant consumption) overnight, then that would be why you see a significant discrepancy.
Hopefully @HandyHat is on the case.

I have similar issues. Big gaps in the data and the daily totals are way out compared to actual usage.

This was yesterday, bearing in mind we have storage heaters and most of our energy use is between 00.30 and 07.30!

My graphs look just like yours and my HA is only showing about 15~20% of my genuine consumption. This situation is getting worse every day and yesterday I kept watching the Bright App to see what happened. At 09:00, the App showed the first two half hour periods. It did not change state until 18:30 when it started adding in the data for the early hours. HA mirrored this with a flatline from 01:00 to 18:00 and then slowly increasing each half hour to midnight. At midnight, the HA integration did it's reset but the Bright App was only showing data to 08:00, around 16hours of lag. As I write this, the Bright App is showing the first two half hours of today, but yesterday is incomplete with data from midnight to 11:30. I am still waiting for 12hours worth of data from yesterday. I know the data is there in the DCC as my suppliers website shows it.

from ha-hildebrandglow-dcc.

Similar issues here. It's making the integration very unhelpful. Note that both the Bright app and my supplier are showing the right data for yesterday now, but HA isn't and the Bright app is only showing a gas figure for 1am at present (4:30 pm).

from ha-hildebrandglow-dcc.

Similar issue here too, the integration only seem to work if Bright App/DCC feed is timely. For the past week or so my feed is no where near real-time, it seems that it catches up todays readings in the early hours of the next day, and the integration does not do any catch up and I always get incorrect figures now.

from ha-hildebrandglow-dcc.

exactly the same issue I am experiencing too. Data is incomplete between the exact time frames as above

from ha-hildebrandglow-dcc.

Just been directed to this bug from the main discussion. I unfortunately am seeing the exact same issue. The Bright App and my SMETS2 meter are correct but what's reported in HA is missing data: #310

Is there any additional debug logging or similar I can provide from my instance that may help solve it?

Thanks

from ha-hildebrandglow-dcc.

I am also experiencing something similar. Difference in readings between Bright and HA over longer periods. See below at images.


PS also notice Bright treats costs as 16bit signed number meaning 32.768 p/kWhr is max and higher than that as with today's exorbitant prices means negative Values. I'll use a template to correct for this 65.536 - (value).

from ha-hildebrandglow-dcc.

I had an update this week from Jane at Hildebrand "This implies your software is making a different api call to get the consumption data than bright does.
If someone tells us exactly what call is not working compared to bright we will have a look.
Ideally please raise a ticket on this specific issue by emailing [email protected]
The reality is if you want realtime up to date data buy the glowmarkt IHD and get a local mqtt client - everything else is something of a fudge."

Its a free service so we should not really have high expectations of them but whoever wrote the API probably needs to raise a support ticket and be patient.

from ha-hildebrandglow-dcc.

from ha-hildebrandglow-dcc.

So it looks that this integration is useless now... Is it only way now to get actual data in real time is using Hildebrand IHD?

Honestly, this was my thought too. Not sure if Hildebrand are essentially deprecating the HA/community use of the API.

Possibly related: I was aware people in the past could request for Cloud MQTT access for HA integration, I asked this question (ironically a couple of day before the troubles started) and got this response from Jane Wilson kinda sidestepping the request saying that this could only (now?) be done with their CAD:

As you realise, you will need our CAD for MQTT as it is intended to be used - MQTT is a publish, in near real-time, of data collected on the HAN from your meters by our devices. It is formatted into some json objects and then published for the user to consume.

There is no (and cannot be an) equivalent of the half hour data for MQTT. If you absolutely need the 30 minute data in MQTT you will need to code against our published API and then inject it into whatever broker you wish.

API access is automatic; you use your Bright credentials. Further information here : Repository URL - https://bitbucket.org/ijosh/brightglowmarkt/src/master/

The Bright Application ID is b0f1b774-a586-4f72-9edd-27ead8aa7a8d

This is our Display CAD: https://shop.glowmarkt.com/products/display-and-cad-combined-for-smart-meter-customers - that comes with local MQTT and cloud MQTT on request.

All said, this is kinda fair enough honestly I was surprised there was any solution to getting detailed (near) real-time access to your own energy data, let alone a free one. While personally I think the price tag of the CAD/IHD is a little steep @ £70 you will end up with a truly local solution and that nicely fit the sprit of Home Assistant, so yeah, I'll probably end up going this route.

from ha-hildebrandglow-dcc.

would think about the smet2 stick that they have been talking about for a while but never seems to have appeared

from ha-hildebrandglow-dcc.

Yeah local connection sounds really good, but £70 is a big price tag. I have Chameleon IHD from Bulb that used to work with ST. Now that integration is totally messed up as well...
I suppose since there is no other options to read our meters I might go for CAD/IHD as well.

from ha-hildebrandglow-dcc.

I thought it was something I did! I have the same issue :/

May also go via the CAD/IHD route. £70 though..? hmm. What realtime data will that provide? Obviously consumption as you go through the day would be good. But, will it give you realtime instance usage? Anyone got one that can provide some info here?

from ha-hildebrandglow-dcc.

Previously I was using a Shelly EM (with a CT clamp on the live feed at the meter). But after I got a smart meter fitted I switched to this - I thought that the readings would be more accurate if they were read from the meter, and I could also do away with an extra device (the Shelly).

I understand completely that this integration and the API from Hildebrand are all in "best endeavors" and good will, so no complaints from me - but after trying both I've now decided to go back to the Shelly. I was still using it anyway to measure "solar power back into the grid" - a function which Hildebrand say is still in the pipeline as a function for the CAD, if I were to go that route.

The Shelly + clamp were approximately the same cost that the CAD would be, provides near-realtime readings which I think are close enough to the readings from the meter to be good enough for me. It has MQTT and a web interface and integrates into HA. Minor point, but it is also not tied to the meter so can be taken if I move house. It does require 240V power (for power and reference) and the clamp might be tight to fit in some meter installations. It won't suit everyone.

YMMV, but the Shelly works well (enough) for me. Best of luck to this integration and to Hildebrand though.

Apologies for straying from a bug report, but it might help others here to have another option to consider.

from ha-hildebrandglow-dcc.

Yeah, this was only way to get gas usage...
I was just about to place an order for CAD IHD and guess what, PayPal payment option is messed up as well...

from ha-hildebrandglow-dcc.

I have got a IHD unit at home and it says ZigBee. Anyone can lead me with a tutorial on how to get it into HA?

from ha-hildebrandglow-dcc.

Most of IHD operate on zigbee but it is locked for public, so no use for us...

from ha-hildebrandglow-dcc.

@Prash1407 I don't think you can use a normal IHD in HA. Zigbee is just the communication protocol in use and from the research I have done its a secure network which you cannot link to.

from ha-hildebrandglow-dcc.

@Prash1407 I don't think you can use a normal IHD in HA. Zigbee is just the communication protocol in use and from the research I have done its a secure network which you cannot link to.

😥 Had a little hope I could get it back to how it used to be, so only way to go this CAD IHD way and then somehow implement?

from ha-hildebrandglow-dcc.

So I was going to buy a CAD but after getting solar panels and an inverter there was the realisation that actually the Growatt inveter has all my electricity stats in detail so after a bit of research I came across Solar Assistant which is basically a 2nd Pi running a program and using a custom lead to connect to the inverter which took me all of 5 minutes to make. It creates MQTT communication to HA so now I get detailed grid electricity, export to grid, battery stats and solar production and many other stats.
Sadly the CAD is the only solution for me to get my gas stats but to be honest for 4-5 months a year I might just use the E.On IHD and put the meter readings into a database manually for the £70 the CAD costs and of course there is never a guarantee that the technology we buy today will work tomorrow. I've sadly seen the demise of so many cloud providers and services and when the tech becomes a door wedge or paperweight.
My view is that energy providers should use something opensource that people like us can access free of charge without the need for additional hardware or software and just a simple intergration.

from ha-hildebrandglow-dcc.

@Prash1407 I don't think you can use a normal IHD in HA. Zigbee is just the communication protocol in use and from the research I have done its a secure network which you cannot link to.

😥 Had a little hope I could get it back to how it used to be, so only way to go this CAD IHD way and then somehow implement?

Yes the CAD is at the moment the ONLY soution in the marketplace at this moment in time which gives you regular snapshots.

I have seen projects online using optical sensors, magnetic reed switches and even camera technology coupled with OCR technology to get the meter reads (saw this for water meters https://community.home-assistant.io/t/ocr-on-camera-image-for-water-electricity-meter/124548) or (https://github.com/jomjol/AI-on-the-edge-device) but I guess it depends on how far you want to go down the rabbit hole. CAD links to HA through MQTT which to be fair makes it reasonably simple to set up.

from ha-hildebrandglow-dcc.

Just realized, mine is actually a IHD-CAD - just went to check it.

from ha-hildebrandglow-dcc.

Consumer access device that means nothing.
Hildebrand are the only company offering a CAD that does MQTT communication so without that feature at the moment its not possible to link just any CAD to HA
Of course if its just electricity monitoring you want you could look at one of the many compatible devices for HA I have read that the Shelley EM is an option at about £50 but then the CAD that Hildebrand offer is just £20 more for gas readings every 30 mins

from ha-hildebrandglow-dcc.

Will the CAD be able to bring in historical data. Thinking of taking the £70 hit

from ha-hildebrandglow-dcc.

Will the CAD be able to bring in historical data. Thinking of taking the £70 hit

Real time usage only.

from ha-hildebrandglow-dcc.

Will the CAD be able to bring in historical data. Thinking of taking the £70 hit

If you don't need the display, and you have SMETS1 meters, the £50 https://shop.glowmarkt.com/products/glow-stick device might be sufficient

from ha-hildebrandglow-dcc.

SMETS2 :( shame there is no USB for that

from ha-hildebrandglow-dcc.

I recently added this integration to HA and like others I am getting incomplete data. The data I am getting through the Hildebrand app is also incomplete.

I'm frustrated that it is so difficult / costly to get access to smart meter data. My smart meter data should be freely accessible to me on my phone at all times (not on a stand-alone smart meter display). This discussion has inspired me to start a petition.

Also signed. Totally agree: It's fundamentally our data that us techy-savvy energy conscience people should be able to access far easier. Hopefully with this grass roots adoption in the future our property's energy data feed will be more open and easier to use for the common person.

from ha-hildebrandglow-dcc.

It seems that readings for current day is back now in integration and more or less same as from local IHD. Current day cost seems to be correct as well. Or maybe it is because I have installed Hildebrand display...
I did get them IHD, maybe I rushed but local information is always appreciated, less chance to break...

from ha-hildebrandglow-dcc.

I am seeing this too. The new algorithm is considerably under reporting compared with the bright app.
I think the issue may be the new algorithm does not “catch up” if there is a missed reading. The original algorithm caught up so was accurate, albeit putting the consumption in the wrong time slot.

I am seeing exactly the same with Bright App

from ha-hildebrandglow-dcc.

Having the same issue, found out the hard way

this is a graph comparing DCC data to Home Assistant for the 23rd of Feb

from ha-hildebrandglow-dcc.