Provide a reliable lightweight monitor with notifications about projects HOT 7 OPEN

Rough estimate $500.

from projects.

As per the Bisq Project management guidelines, this project is approved for ops budget allocation and the project has been assigned to @jmacxx

from projects.

Progress:

• JSON feed of active alerts, & alerts file documentation.
• JSON feed of alert history
• A user-friendly app.
• Chimp's detailed app.
• Keybase alerts.
• Documented setup procedure for server.

from projects.

Replaces bisq-network/bisq#4665

from projects.

FYI this is what I am working on. Please let me know if anything is wrong..

For Wiz' alerting infrastructure:

1. Provide a JSON file that contains details only of CURRENT ACTIVE ALERTS. If there are no alerts it should be empty. Format should include timestamp, seednode id, field triggering the alert, value of field, rule name that triggered the alert.

For use by a web client UI:

2. Provide JSON files of the latest raw data queried from seed nodes. This is already available as a WIP at http://46.101.179.224:8082/seednode_json and is essentially the same as described in "Example json" for all seednodes.
3. Provide a JSON file of data flagged by the serverside Java analytics: ERROR, WARN, or INFO. Should include the tag names (same as in [2]), alert level and timestamp. This will be used by the UI to highlight relevant data elements.
4. Provide an aggregated JSON view of the current seednode data, i.e. values averaged across all seednodes. I originally implemented this in the prototype GUI but it would make more sense to do it at the server and leave the GUI just to display, no logic. Identical tag names as [2].
5. Provide a JSON file listing alerts that happened in the past (with timestamps). A historical log for display in GUI. Same format as [1] but historical.

Alerts

Highest priority according to Wiz is there should be no instances of false alerts. To achieve that we need to define upfront exactly what the alerting criteria are (clarify the list posted OP). Here follows a proposed list based on the original spec:

Hopefully @wiz will have some suggestions as to which in this list are important, and which are not necessarily so important.

• A seednode is unreachable for > 3 minutes
• A seednode has uptime of > 26 hours
• A seednode's memory usage is over 1.5Gb
• A seednode's numConnections > 2x the seednode's maxConnections
• A seednode's Offer count is < 50% of the 24 hour moving average
• A seednode's Mediator count < 1
• A seednode's RefundAgent count < 1
• Across all seednodes, if there is any deviation of +/- 3 in numBsqBlocks
• Across all seednodes, if there is any deviation of +/- 3 in daoStateChainHeight
• Across all seednodes, if there is any deviation > 30% in SignedWitness
• Across all seednodes, if there is any deviation > 30% in AccountAgeWitness
• Across all seednodes, if there is any deviation > 30% in MailboxStoragePayload
• Across all seednodes, if there is any deviation > 30% in TradeStatistics3
• Across all seednodes, if daoStateHash is not consistent if and only if daoStateChainHeight is the same. This seems a bit tricky any may need some thought. [see below] difficult
• Across all seednodes, Mediator count should be consistent [see below] difficult
• Across all seednodes, RefundAgent count should be consistent [see below] difficult
• Across all seednodes, Filter count should be consistent [see below] difficult
• Across all seednodes, Alert count should be consistent [see below] difficult

[NB] difficult due to timing of differences between seednodes being updated/queried. The operations are not atomic so values can differ for a while. Need some thought on how to implement these cases without generating false alerts.

from projects.

@jmacxx
While trying to fine-tune http://46.101.179.224/ I saw that it is not trivial to avoid false positives.

One problem is that when for instance a new mediator gets published some nodes might get it earlier then other if the request event happends just around the publishing time. But we cannot lower the criteria as normally a diff of 1 is an alert case.

One option is to do repeated requests if an alert/warning is triggered. But that need to be done on all nodes as it can be that the node which caused the alert has the fresh data and the others have been behind (e.g. a new block arrived), so only if we repeat all requests we will find out if it was a false positive.

Maybe easiest to find out if that really helps it to decrease request interval and add some more tolerant thresholds to number of alerts/warnings. But that need to be done per param as for instance a new mediator is a very rare event but a new block happens every 10 minutes so we can expect more alerts for blocks.

from projects.

After more thought about the problem I think we need a secondary data set as kind of overlay to interpret an alert/warning correctly. E.g. If we know the time of new blocks being publihsed we can filter out all the alerts/warnings which happened around that event. For other data like number of offers its more difficult as we don't have a primary more reliable source then the seed nodes themselfes. We still could run the monitor as full p2p node and thus receiving the data from the network independently and using that as reference. It only can be applied on past data, as at the most recent data we still don't know if in the next moments an event happens which would let the alert be seen as false positive (e.g. num offer spiked from 350 to 500 offers in a few minutes). Not trivial how to deal with all that ;-(

from projects.