Add an option to enforce maxAutoPpm to be respected, as with some nodes, the rebalancing can get very expensive after the fees have been adjusted according to the suggestions from jet analyze-fees

This issue seems to be specific to RaspiBlitz, observed on two instances.

Workaround: lncli listchannels | jq -r '.channels[] | select(.num_updates) | "(.chan_id) (.num_updates)"' | sort -rgk 2 | head -10

After stopping auto-rebalance no active rebalances where shown as expected...

Then I changed the maxAutoPpm to 300 in config.json

After the restart of auto-rebalance the newly set fees where not picked up:

So I stopped auto-rebalance again but 16 processes where still showing up when checking using:

ps -aux | grep 'bos rebalance'

So I think there might be an issue :-)

It would be nice to have the max ppm fees take profitability margin into account and adjust itself accordingly - instead of being fixed. While setting a overall max fee works for most channels, it fails in cases where the fee rate is too high. As it should in order to limit the total cost.

LOOP is one such example where my fee (lc_rate) is high enough that it would make sense(profitable) to re-balance it with even a higher fee than the current 500ppm - it should start with lowest fee possible and in case it fails because of low fees, should try with a higher fee rate as long as it is profitable to do so, taking rebalance fee & fee earning opportunity for the amount being rebalanced.

When jet tries to rebalance and fails due to high fee, it sees the min fee that would be required to rebalance. So would be great if it could save this data and provide it as a recomendation per peer, like rebalance will likely to go through with X ppm

lnd creates a cache of the graph for forwards (optimization). in case of graph errors, cache records may cause failedtofindpath errors to linger. calling deleteforwardingreputations (bos call deleteforwardingreputations) clears out the cache.

Have jet telegram warn about an excessive number of failedtofindpath errors, and consider calling deleteforwardingreputations automatically when the number of errors exceeds a threshold.

I think there might be an error when restarting all services after I pulled the changes.

It seems that telegram is stared and then stopped instead of vice versa!

Telegram reports channel.db size 4,953 gb exceeds 5 gb but it's not.

du -h ~/umbrel/lnd/data/graph/mainnet/channel.db reports a size of 4.7 GB

for autorebalance.js if the option --dryrun is specified, the loop should stop after the first iteration.

This will simplify jet install by eliminating the BOS install dependency by calling the BOS API directly.

A node opened a channel to me, but it seems like their node is new and not known to the network. Jet monitor exited with an error. It seems that I won't be able to use monitor until this node is known.

node: 021358c92d60fa9fc4abbdd0645de3ee581d27ee9366d897317aa344a8f3768b26, error: Error: 5 NOT_FOUND: unable to find node
(node:18087) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'node' of null
    at /home/bos/lightning-jet/api/utils.js:163:20
    at Array.forEach (<anonymous>)
    at listForcedClosingFormattedSync (/home/bos/lightning-jet/api/utils.js:162:17)
    at runMonitorLoop (/home/bos/lightning-jet/jet:377:16)
    at /home/bos/lightning-jet/jet:315:7
    at new Promise (<anonymous>)
    at Command._action (/home/bos/lightning-jet/jet:313:12)
    at Command._run (/home/bos/lightning-jet/node_modules/caporal/lib/command.js:483:42)
    at Program._run (/home/bos/lightning-jet/node_modules/caporal/lib/program.js:183:16)
    at Program.parse (/home/bos/lightning-jet/node_modules/caporal/lib/program.js:289:17)
    at Object.<anonymous> (/home/bos/lightning-jet/jet:342:5)
    at Module._compile (internal/modules/cjs/loader.js:1072:14)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1101:10)
    at Module.load (internal/modules/cjs/loader.js:937:32)
    at Function.Module._load (internal/modules/cjs/loader.js:778:12)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:76:12)
(Use `node --trace-warnings ...` to show where the warning was created)
(node:18087) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:18087) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

While a single circular rebalance was in progress, I did a dry run using autorebalance.js

node service/autorebalance.js --dryrun

The circular rebalance finished successfully...

... but the following error was reported:

Output of autorebalance:
autorebalance.log

Raspiblitz 1.7.1
lightning-jet v 1.3.0 (installed under bos user)

Observed more rebalance instances than the limit set in the config.json.

• Config.json contains: "maxInstances": 10,
• Counted (18) node /home/bos/lightning-jet/jet rebalance ... processes in htop
• jet monitor displays 14 rebalances in progress

Add a docker-compose.yml so it's easier to bring up a docker container using Docker compose.

git pull
Updating 7f23daa..cad71b1
error: Your local changes to the following files would be overwritten by merge:
 package-lock.json
Please commit your changes or stash them before you merge.
Aborting

Workaround: git checkout -- package-lock.json, then retry git pull

Make channel capacity for autorebalancer configurable or reduce it to 990,000 sats so channels with exactly 1mil sats before fees could be also rebalanced automatically

Per discussion, current implementation of exclude in the config.json: "excludes nodes classified as outbound or balanced, it does not exclude inbound nodes."

In my experience I have had a couple of special cases where I wanted to, e.g. (1) exclude outbound rebalances from an inbound peer
or (2) exclude outbound rebalances on an outbound peer (new channel, 100% local). From these experiences I was thinking there might be value in having finer control of how peers are excluded: either entirely, in only the outbound direction, or only in the inbound direction.

Neski, [11/20/2021 21:38]
[In reply to Neski]
@NanIthilHODL for example:

"exclude": [
"pub id1:all" ,
"pub id2:inbound"
"pub id3:outbound"
]

will exlcude pub id1 from inbound and outbound rebalance peers, pub id2 from inbound, and pub id3 from outbound peers.

Scenario: a peer, at the top of the rebalancing list, keeps on erroring out (fee too high, failed to find path, etc). This may prevent other [outbound] peers from being rebalanced. In the worst cases, a "bad" peer can completely stall rebalancer, even though it is unlikely.

I find the naming "stuck htlcs" a kind of confusing, may I suggest to rename it to "active htlc's" :-)

I've changes the local fees from 199 to 299 but jet telegram reported

But Thunderhub shows the following

Jet monitors the size of channel.db and notifies via telegram when the size goes over a limit. On RaspiBlitz, channel.db runs from the bitcoin:bitcoin account, whereas jet runs from the bos (BalanceOfSatoshis) account. Subsequently, jet does not have read access to the channel.db and can not notify via telegram when the channel.db size goes over the limit.

Log:

[cameri@umbrel lightning-jet-config]$ sudo docker exec -it lightning-jet cat /tmp/autorebalance.log
max commands: 10
max ppm: 500
max pending htlcs: 4
classifying peers...
no events found
classify: TypeError: Cannot read properties of undefined (reading 'peers')
    at classifyPeersAnalyzer (/app/api/htlc-analyzer.js:20:16)
    at classify (/app/service/autorebalance.js:125:18)
    at Object.<anonymous> (/app/service/autorebalance.js:100:1)
    at Module._compile (node:internal/modules/cjs/loader:1101:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1153:10)
    at Module.load (node:internal/modules/cjs/loader:981:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:81:12)
    at node:internal/main/run_main_module:17:47
finished classifying
12/31 05:16 AM
generating htlc history...

--------------------------------
12/31 05:16 AM
runLoop error: TypeError: Cannot read properties of undefined (reading 'inbound')
TypeError: Cannot read properties of undefined (reading 'inbound')
    at runLoopExec (/app/service/autorebalance.js:157:14)
    at runLoop (/app/service/autorebalance.js:139:5)
    at Object.<anonymous> (/app/service/autorebalance.js:106:1)
    at Module._compile (node:internal/modules/cjs/loader:1101:14)
    at Object.Module._extensions..js (node:internal/modules/cjs/loader:1153:10)
    at Module.load (node:internal/modules/cjs/loader:981:32)
    at Function.Module._load (node:internal/modules/cjs/loader:822:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:81:12)
    at node:internal/main/run_main_module:17:47

Currently there's no documentation for docker on:

• how to build the docker image from source
• how to publish (git tags?)
• how to run with mynode
• how to run with raspiblitz
• how to run with umbrel
• how to run with native setup
• how to run commands on docker container

[based on alpha testing feedback] Add an option to see individual htlcs in jet htlc-analyzer along with the dates.

[based on alpha testing feedback] some platforms, like raspiblitz, require extra file permissions to access local files. Add base64 encoded TLC and macaroon options to config.json for flexibility.

This is a suggestion only :-)

Unfortunately the JSON file format doesn't allow any comments and I would like to suggest the use of YAML

WHY?
I find it laboriously as a noob to keep separate notes and would appreciate it very much to have the possibility to comment things I change in config.json, as my old brain is not so fit anymore :-)

Especially the entries for "avoid" and "exclude" which are pubkeys.

"To create jet bot: initiate a converation with BotFarther on your Telegram app. Select bot's name (e.g. JET bot) and bot's username (e.g. jet_bot)."

Should read: "To create jet bot: initiate a conversation with BotFather on your Telegram app. Select bot's name (e.g. JET bot) and bot's username (e.g. jet_bot)."

I recently updated my umbrel to the latest version. I updated using the UI dashboard of umbrel. After a long while, realised the jet services are not running. Here are the logs that may help investigate further:

11/15 06:23 PM
error running the loop: Error: Error: 2 UNKNOWN: waiting to start, RPC services not available
Error: async hook stack has become corrupted (actual: 148726, expected: 1150)
 1: 0xa5834 node::AsyncWrap::PopAsyncContext(v8::FunctionCallbackInfo<v8::Value> const&) [node]
 2: 0xd2a7c  [node]
 3: 0xd9a4  [node]
 4: 0xd3f20 v8::internal::Builtin_HandleApiCall(int, unsigned long*, v8::internal::Isolate*) [node]
 5: 0x1916c  [node]

the timing of the error is close to when I updated my node, leading to the suspicion that it was due to the restart of the node that some file got corrupted and led to the services conking up.

After running 1.3.0 rebalancer gets "stuck." I.e., jet monitor thinks rebalances are in process and in fact associated processes are running, but no htlcs appear, rebalances never countdown, etc.

Runining on Raspiblitz 1.7.1, lightning-jet installed under the "bos" user.

Example error from rebalance log follows:

* rebalancing from BCash_Is_Trash to SeasickDiver
* remaining amount: 1,000,000
* time left: 30 mins
(node:9153) UnhandledPromiseRejectionWarning: Error: Cannot find module 'balanceofsatoshis/lnd'
Require stack:
- /home/bos/lightning-jet/bos/rebalance.js
- /home/bos/lightning-jet/api/rebalance.js
- /home/bos/lightning-jet/jet
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:889:15)
    at Function.Module._load (internal/modules/cjs/loader.js:745:27)
    at Module.require (internal/modules/cjs/loader.js:961:19)
    at require (internal/modules/cjs/helpers.js:92:18)
    at lazy (/home/bos/lightning-jet/node_modules/import-lazy/index.js:3:33)
    at Object.get (/home/bos/lightning-jet/node_modules/import-lazy/index.js:11:22)
    at lndForNode (/home/bos/lightning-jet/bos/rebalance.js:5:42)
    at Object.rebalance (/home/bos/lightning-jet/bos/rebalance.js:67:29)
    at rebalanceSync (/home/bos/lightning-jet/bos/rebalance.js:23:20)
    at module.exports (/home/bos/lightning-jet/api/rebalance.js:232:20)
    at Object.apply (/home/bos/lightning-jet/node_modules/import-lazy/index.js:16:20)
    at /home/bos/lightning-jet/jet:95:7
    at new Promise (<anonymous>)
    at Command._action (/home/bos/lightning-jet/jet:92:12)
    at Command._run (/home/bos/lightning-jet/node_modules/caporal/lib/command.js:483:42)
    at Program._run (/home/bos/lightning-jet/node_modules/caporal/lib/program.js:183:16)
(Use `node --trace-warnings ...` to show where the warning was created)
(node:9153) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:9153) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

It's hard to set the minutes correctly when trying to retrieve all the past rebalances using jet rebalance-history [mins]

At the moment I have to approach it and increase the [mins] value until the following error is thrown :-)

May I suggest to modify the code, so that the default without [mins] value shows all rebalances

Also a filtering possibility would be great, although it can be done with grep, e.g.

jet rebalance-history 13000 | grep 'success'

Rebalance generated htlc failures are being recorded in the htlc log file from jet.db

It would be ideal if those records were not written to jet.db in the first place, as htlc-analyzer is skipping them anyway because they are not a result of "organic" routing events.

I've edited /api/config.json

... but unfortunately "maxAutoPpm": 300 is not taken into account when starting a new autorebalance

May I recommend to change the following timestamps to 24 hour format since it is not evident whether it's AM or PM

rebalances in progress:

successful rebalances over the past 4 hour(s) (10 max lines):

The summary shown on index 12 + 13 is:

'remote ppm exceeds local. revisit local ppm and / or max ppm based on suggested range'

But local ppm is 200 ppm higher than remote ppm and maxAutoPpm is set to 550 in config.json.

Somehow a strange display behaviour in jet monitor as pending htlcs are lower than rebalances in progress

zombie@zombie:~$ uname -a
Linux zombie 5.11.0-41-generic #45-Ubuntu SMP Fri Nov 5 11:37:01 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

zombie:~$ jet channeldb
channel.db is located at /home/zombie/.lnd/data/graph/mainnet/channel.db
/home/zombie/.lnd/data/graph/signet/channel.db
checkChannelDb: Error: ENOENT: no such file or directory, stat '/home/zombie/.lnd/data/graph/mainnet/channel.db
/home/zombie/.lnd/data/graph/signet/channel.db'
/home/zombie/lightning-jet/api/channeldb.js:30
if (res.priority === priority.urgent) {

I think it's very annoying to receive a reminder every 5 minutes and it spams telegram unnessecarily!

Personally an hourly one would be sufficient and a symbol ⚠️ could be added to give it more expression!

Maybe also an option in config.json could be made to set the interval manually...

I think a version/change history would be great, so we could see easily what has been changed since the last update.

zombie:~$ uname -a
Linux zombie 5.11.0-41-generic #45-Ubuntu SMP Fri Nov 5 11:37:01 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

zombie:~$ jet rebalance-history c-otto.de
rebalance history for c-otto.de
┌─────────┐
│ (index) │
├─────────┤
└─────────┘

I did the following manual rebalance using the following command:

jet rebalance 'BTC Sessions' 024706bbb0ef2a7dda56 500000

... and it seemed to have started successfully!

But unfortunately the display in jet monitor` is some how wrong, as it doesn't show the peer balanced into, instead it shows a part of the peer name balancing from, it seems that the white space in the peer alias is causing the issue.

Error: Error: 2 UNKNOWN: waiting to start, RPC services not available from /tmp/autorebalance.log.

The service is running based on jet status, but rebalances arent being kicked off.

Noticed during alpha testing (at least two instances).

workaround: jet restart rebalancer

[based on the alpha testing feedback] E.g. Wallet of Satoshi is easy to rebalance manually by manually sending sats to the wallet, and then back to the channel.

I've edited /api/config.json

... but unfortunately "maxInstances": 5 is not taken into account when start a new autorebalance

The exception occurred due to the lnd API connection being terminated by lnd. The exact root cause for why the connection is terminated is unknown, but it is likely due to lnd being overloaded.

When an exception occurs, it causes the stream of htlc events to terminate, subsequently resulting in missed hltcs not being recorded in the database.

logging events...
node:internal/process/promises:246
triggerUncaughtException(err, true /* fromPromise */);
^

Error: 2 UNKNOWN: htlc event subscription terminated
at Object.callErrorFromStatus (/home/jeremy/lightning-jet/node_modules/@grpc/grpc-js/build/src/call.js:31:26)
at Object.onReceiveStatus (/home/jeremy/lightning-jet/node_modules/@grpc/grpc-js/build/src/client.js:330:49)
at Object.onReceiveStatus (/home/jeremy/lightning-jet/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:299:181)
at /home/jeremy/lightning-jet/node_modules/@grpc/grpc-js/build/src/call-stream.js:145:78
at processTicksAndRejections (node:internal/process/task_queues:78:11) {
code: 2,
details: 'htlc event subscription terminated',
metadata: Metadata { internalRepr: Map(0) {}, options: {} }
}

More than an issue, this is an observation. I have an outbound peer (as per jet peers) with which the fee structure was very low - 80(fees set by peer)/87(fees set by me); this was the case until recently, my peer increased their fee to 150 and analyze-fees suggested I increase mine to 201; after some time it suggested me to increase it to 501 (as the peer had increased to 450); so it went and now we are at 1050/1201ppm from 80/87ppm. Incidentally, the peer also happens to use jet and likely increased the fees based on their results/suggestions of analyze-fees

This primarily is due to the fact that the peer increased their fee (manually or because of some tool which may have done this). How should the analyze-fees work in such cases to prevent such a situation?

As I was not sure where to add the new option rebalancer.maxPendingHtlcs it could be very helpful to add an example of the config.json for better understanding.

Hi,

first I wan't to thank you for this fantastic tool and I made already a few successful circular rebalances with it.

Please find below some details about a rebalance that seems to have failed.

This rebalance was started yesterday afternoon and was still running when I checked it this morning, so it was running roughly 14 hours and so I decided to abort it using CTRL-C

Please find below a screenshoot and a log file:

htlc-logger.log

If some further information is needed don't hesitate to contact me :-)

jet monitor crashed with the following result: