It looks like while Installing on Rocky Linux docker compose does not get found in the repos.

Nice work around is the following:

1. $ sudo curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
2. $ sudo chmod +x /usr/local/bin/docker-compose
3. $ sudo ln -s /usr/local/bin/docker-compose /usr/bin/docker-compose

Source: https://tastethelinux.com/install-docker-and-docker-compose-on-rocky-linux-8/

is this code still running with the Solana Blockchain?

Only options were Bobcat, LongAP, Nebra, and SenseCap. I did not see a RAK option. Is it there and I'm just not seeing it?

Prometheus limitations make monitoring multiple hosts cumbersome. Instead of allowing multiple miners to push the same metric with different tags, you have to create a new metric for each miner. Statsd will allow a single metric from multiple sources as long as the tags are different

Raspberry Pi 4 with newest OS.

Error Message:
Complete!
Loaded plugins: fastestmirror, ovl
Loading mirror speeds from cached hostfile

• base: ftp.osuosl.org
• centos-kernel: mirror.12z.eu
• extras: ftp.osuosl.org
• updates: ftp.osuosl.org
  No package epel-release available.
  Error: Nothing to do
  ERROR: Service 'hnt_monitor' failed to build: The command '/bin/sh -c yum update -y && yum -y install epel-release && yum update -y && yum install -y jq && mkdir -p /opt/hnt_monitor && rm -rf /tmp/* && yum clean all -y' returned a non-zero code: 1
  root@raspberrypi:/home/pi/hnt_monitor#

hope anyone can help me why reward values are wrong for me
hnt_miner_reward_earned_15m to 30 days is 0 or either 1

hnt_miner_reward_earned_1d is 0.0181 same as 7d and 30d... all 3 values are wrong

other stuff is correct like CPU temp, usage, reward scale and so on but rewards are very wrong

The docker.io needs to be installed as root but if the parts that don't need root could be installed as none root it would make things more secure.

I have the Docker version running on my synology diskstation, but unfortunately I can't get to the panel
Application endpoint
grafana http: // localhost: 3000

prometheus http: // localhost: 9090

prometheus pushgateway http: // localhost: 9091

nhatfield-hnt_monitor1

Block height for Sensecap seems to show the blockchain height, not the synced height of a hotspot.

In the API Docs (https://status-api-doc.sensecapmx.cloud/) in the device detail request there's a separate field for Block height of device (called 'height') and another for the height of the blockchain (called 'totalHeight'), looks like the monitoring might be reading the latter field.

Need test for the build confidence

Add the reward earnings metrics as a new collector

Need a build pipeline to test updates

Create a log function so it is easier to send logged data. Right now each service has to tailor the full log message multiple times.

Successfully tagged hnt_monitor:latest
Creating grafana                ... error
Creating prometheus ... 
Creating prometheus_pushgateway ... 

Creating prometheus_pushgateway ... error
Creating prometheus             ... error
ERROR: for prometheus_pushgateway  Cannot create container for service prometheus_pushgateway: invalid IPv4 address: 10.30.0.04

ERROR: for prometheus  Cannot create container for service prometheus: invalid IPv4 address: 10.30.0.03

ERROR: for grafana  Cannot create container for service grafana: invalid IPv4 address: 10.30.0.02

ERROR: for prometheus_pushgateway  Cannot create container for service prometheus_pushgateway: invalid IPv4 address: 10.30.0.04

ERROR: for prometheus  Cannot create container for service prometheus: invalid IPv4 address: 10.30.0.03

Fix: remove the zero from the last ips like 04 -> 4 and docker-compose -f hnt_monitor.yml up -d --build

Pls fix in next version.

i saw there is a closed issue with the same error:

Step 9/9 : ENTRYPOINT ["./entrypoint"]
 ---> Using cache
 ---> e05d02cdda97
Successfully built e05d02cdda97
Successfully tagged hnt_monitor:latest
Recreating 992ae87577fa_prometheus ...
Recreating 6c8e55c4bf64_grafana                ... error
Recreating 5b1c29c87d20_prometheus_pushgateway ...

Recreating 992ae87577fa_prometheus             ... error

ERROR: for 5b1c29c87d20_prometheus_pushgateway  Cannot create container for service prometheus_pushgateway: invalid IPv4 address: 10.30.0.04

ERROR: for 992ae87577fa_prometheus  Cannot create container for service prometheus: invalid IPv4 address: 10.30.0.03

ERROR: for grafana  Cannot create container for service grafana: invalid IPv4 address: 10.30.0.02

ERROR: for prometheus_pushgateway  Cannot create container for service prometheus_pushgateway: invalid IPv4 address: 10.30.0.04

ERROR: for prometheus  Cannot create container for service prometheus: invalid IPv4 address: 10.30.0.03
ERROR: Encountered errors while bringing up the project.
root@store:/home/steam/hnt_monitor-7.10.0#

in the closed issue someone say it caused by version of docker but i use the latest:

root@store:/home/steam/hnt_monitor-7.10.0# docker --version
Docker version 20.10.16, build aa7e414

Add prometheus, prometheus push gateway, grafana, and the hnt_monitor as a single docker-compose file.

Hi,
I don't have access to my miners because they are in a different location. But I would like to grab some data via the API. I came across your project and it seems awesome. I was able to set it up in docker and I define my two hotspots but the beacon and challenges are reporting like zero. Are those required for local IP or API?

So my witnesses are reporting back zero. Kind of odd.

I basically just want to use your project to scrape API data and show it in grafana.

Great project! Already using it for some days. :-)

Would be nice if all the SenseCAP API provided information would be pulled.
E.g. FW version, Fan-Status, ...

And if we could have a helium wallet overview, e.g. total balance of wallet, daily income

Add the block height to collected metrics

Possibly others might need this same treatment. Currently when the api does not respond, it produces 0's in the UI. We should retry the api query before sending null values

I have 4 hotspot addresses in HNT_HOTSPOT_ADDRESSES. 3 Rak MNTD and 1 Sensecap. I also have the Sensecap API setup. This morning I went to check my dashboard and now see two additional miners in my metrics: shambolic-walnut-chimpanzee and zealous-marigold-mouse. This is day 5 with the stack running and the first time I've seen them.

I checked these out on Helium Explorer just to cross references their addresses to my config and verify their wallet addresses weren't similar.

The instance name is hard coded to custom-rusty-porcupine

Centos does not have official support for the epel repo, so this workaround has to be done in order to build the docker image for ARM platforms.

This is not related to the grafana but the prometheus unless the API is giving back false data.

First of:

hnt_miner_reward_earned_1d is correct

however the 7d and 30d showing the exact same value which would be only true if I would just setup the miner not long ago.

How are these windows working then? Like my current daily reward is 0.06 HNT, according to the explorer my 7d is 0.5 HNT and the 30D is 1.8 HNT whilst with your metrics I get:

hnt_miner_reward_earned_1d 0.06

hnt_miner_reward_earned_7d 0.06

hnt_miner_reward_earned_30d 0.06

Now things gets more odd with the hnt_miner_reward_1d 7d 30d, I don't know what this supposed to display but I get 5 for all, even my overall HNT earnings are not 5.

Would be nice to add pisces also. Did not research if they have a public api.

When first launching the services they should backfill 14 days worth of data. This should only happen on initial setups that have no existing data

We should create a database for this and store the data locally. That would allow us to not tax the API as often, and also return small data sets of only 15 minutes

Thinking about a backend like cassandra with modification to the scripts. Then we can post the same data while also not hitting the API as hard

How do you get the block height and cpu core temperatures per miner.

The block height only available as a HNT metric coming from their system not the nodes. CPU is not available at all
Mine is nebra I have added this to the hnt_monitor.yml

    environment:
      DO_NOT_REMOVE: "setup"
      HNT_HOTSPOT_MONITOR: "true"
      HNT_NEBRA_MONITOR: "true"

Also what is the point of having the internal LAN ips defined for Nebras?

Getting the following error after installing the stack.

2021-12-01T02:52:34 [INFO] [collector.info] [local] - getting hotspot info data for [local (115....k21)]
2021-12-01T02:52:34 [INFO] [collector.rewards] [local] - getting hotspot rewards data for [local (115....k21)]
2021-12-01T02:52:34 [INFO] [collector.bobcat.miner] [local] - getting bobcat miner data for [local (192.168.88.67)]
2021-12-01T02:52:34 [WARN] [collector.bobcat.status] [local] - bad response from the api gateway while retrieving status data for 192.168.88.67. Retrying in 5 seconds...
2021-12-01T02:52:50 [WARN] [collector.rewards] [local] - bad response from the api gateway while retrieving rewards data for 115....k21. Retrying in 5 seconds...

The bobcat miner dashboards have data but anything with the helium network is not being updated because of the api gateway error. I truncated the key but it is the full key of course in the setup. Unable to find any help on this on the web. I'm sure its something simple.

System is ubuntu 18.04. thanks.

Right now the data grows indefinitely, until manually purged. The system should handle this with some user adjustable thresholds