re: #22

SLPDB is relatively straightforward

Need to run these from a machine/container that has mongodb installed, and can talk to mongodb-slpdb.

We should stop slpdb and slpserve containers prior to doing a restore. Not sure if there is any need to stop these while dumping.

Dump

mongodump -d slpdb_test --archive=/tmp/slpdb_test.mongo.gz --authenticationDatabase admin --gzip mongodb://mongouser:mongopass@mongodb-slpdb:27017

Restore

mongorestore  --drop  --authenticationDatabase admin --archive=/tmp/slpdb_test.mongo.gz --gzip   mongodb://mongouser:mongopass@mongodb-slpdb:27017

Many users report issues with syncing smartbch on fresh install, someone needs to reproduce the issue and provide instructions.

There is 2 package that needs to be upgraded.

• 1. markdown-it
• 2. nanoid

Currently when I run bchn node, the wallet functions are enabled by default. This might lead me to expect that I can use this deployment for storing my BCH keys. If this is the intended usage then we should have ways for user to backup and preserve and provide their wallet file, if not then we should run the node with -disablewallet argument.

Instead of using bash script, use Python with Jinja2 for templating and Typer (https://typer.tiangolo.com/) for command line interface builder. This changes should be compatibled with ./services start.

Proposed command:

Example

❯ ./bch-devsuite init
[?] Which components do you want to install?: 
 > X Local node (Bitcoin Unlimited node or Bitcoin Cash Node)
   X REST service (A rest.bitcoin.com-compatible or Bch-API)
   X SLP services (SLPDB, slpserve)

[?] Local node?: Bitcoin Unlimited Node
 > Bitcoin Unlimited Node
   Bitcoin Cash Node

[?] Network?: regtest
 > regtest
   testnet
   mainnet

[?] Please enter a username for the node RPC connection: tle
[?] Please enter a password for the node RPC connection: alpine
[?] Please enter number of initial blocks for regtest: 200
[?] REST Service?: bchrest
 > bchrest
   bchapi

SLPDB uses mongodb, a username and password is recommended.
[?] Please enter a username for mongodb: tle
[?] Please enter a password for mongodb: alpine
[?] docker network: bch-devsuite-network
[?] Exposed port? (Y/n): 
{
  "local_node": {
    "node": "bu",
    "network": "regtest",
    "rpc_conf": {
      "username": "tle",
      "password": "alpine"
    },
    "regtest_blocks": "200",
    "wait_time": 6
  },
  "rest_service": "bchrest",
  "slp": {
    "username": "tle",
    "password": "alpine"
  },
  "exposed_ports": true,
  "docker_network": "bch-devsuite-network"
}
Confirm? [y/N]: y

#########################
# VALIDATE DEPENDENCIES #
#########################
docker...✓✓
docker-compose...✓

##############
# SETTING UP #
##############
Creating docker network...ALREADY CREATED, SKIPPED
Setting up Bitcoin Unlimited node...✓
Generate docker-compose file...✓
Pull the Docker images...
Pulling bch-node      ... done
Pulling bchrest       ... done
Pulling opensight     ... done
Pulling mongodb-slpdb ... done
Pulling slpdb         ... done
Pulling slpserve      ... done
✓
Generating 200 initial blocks for regtest...
Creating bch-node ... done
[
  "20662918c049525e62ff72f84709981e6ea25ab6d4197256f94dfceb798ac2c2",
  ...,
  "1b52bf703cbdce2a8bcfbaabf0bfedf8599fff775143c51c9124337c882cb714"
]
✓
Stopping local node...Bitcoin server stopping
✓


Finished. Please use './bch-devsuite start' to run your new regtest node.

This version will be running on the local machine, not in Docker and we will improve by put it in a docker image later on.

It would be quite useful to add argument parsing capabilities to the ./clean script so it can preform only specific operations that the users wishes. For example:

./clean wallet  # removes only user's wallet files
./clean node  # removes only the current bitcoind+nginx services
./clean all  # preforms every cleaning operation

Running the script without any arguments should display a usage message of possible arguments

Upgrade bitcoincash node to v26.0.0

re: #22

To make a dump (make sure the donor smartbchd is stopped)

sudo tar --exclude=priv_validator_state.json --use-compress-program=pigz -cvf /tmp/smartbch.tar.gz -C data/smartbchd data

To restore (make sure the recipient smartbchd is stopped and clean)

sudo tar --use-compress-program=pigz -xvf  /tmp/smartbch.tar.gz -C data/smartbchd/

To make it easier for users to adopt our devsuite, instead of having them install python3-venv, and pip install libraries, we ship everything in a docker image instead.

User can supply a config file instead of having to go through init wizard, this will make deployment easier on their end.

Port 26656 for smartbch needs to be externally exposed, without this the running node can only have outbound connections. With this port exposed the node might be able to serve other users behind NAT.

re: #22

Provide an automated way to bootstrap testnet and mainnet Bitcoin Cash Node from a block file instead of waiting on network.

Bitcoin Cash Node enables us to provide it with a block file bootstrap.dat which it uses for the initial sync instead of fetching the blocks over the network.

How to generate bootstrap.dat?

1. Prepare python scripts, specifically linearize-data.py and linearize-hashes.py from https://gitlab.com/bitcoin-cash-node/bitcoin-cash-node/-/tree/master/contrib/linearize
2. Patch linearize-data.py with this upstream commit
3. create linearize.cfg

# bitcoind RPC settings (linearize-hashes)
rpcuser=FIXME
rpcpassword=FIXME
host=<ip of bchd container>
port=<port>

max_height=<current height of blockchain, optionally round it to something nice>

netmagic=0b110907
genesis=000000000933ea01ad0ee984209779baaec3ced90fa3f408719526f8d77f4943
input=/path/to/bch-toolkit/data/bitcoind/testnet3/blocks/
output_file=/tmp/bootstrap.dat
hashlist=/tmp/hashlist.txt

out_of_order_cache_sz = 100000000
rev_hash_bytes = False
file_timestamp = 0
split_timestamp = 0
debug_output = False

4. Generate hashlist

./linearize-hashes.py linearize.cfg > /tmp/hashlist.txt

5. Generate bootstrap.dat

./linearize-data.py linearize.cfg

This results in a 40GB file /tmp/bootstrap.dat

How to restore bootstrap.dat using?

1. Copy the file into correct place before starting bchd container

cp /tmp/bootstrap.dat data/bitcoind/testnet3/

2. Start bchd container
3. Once sync is complete, bchd renames the file to bootstrap.dat.old. This file is safe to delete.

It should take about 40 minutes(on my laptop) to sync testnet.

Action items.

• Figure out how and where do we store the dumps
• Figure out automation for restore process
• Figure out cleanup step to delete bootstrap.dat.old

Right now we hard code blockchain data storage directory.

Smartbch recently released v0.4.2, bch-devsuite should update to use that version

After setup is complete should have a message to show what endpoint to connect, summarize all the port.
Inspire by create-react-app

re: #22

Commands below require pigz to be installed. If its not then remove --use-compress-program=pigz from the tar commands and add z argument.

Generate dump

BU node must be stopped

sudo tar --exclude=debug.log --exclude=mempool.dat --exclude=wallet.dat --exclude=.lock --exclude=peers.dat --exclude=orphanpool.dat --use-compress-program=pigz -cvf /tmp/bu.tar.gz -C data/bitcoind testnet3

Restore dump

Put the files in the directory before creating the container

mkdir -p data/bitcoind/testnet3
tar --use-compress-program=pigz -xvf  /tmp/bu.tar.gz -C data/bitcoind/

Current versions of BCH nodes might stop working from 15th May onwards. Update the versions to

• BCH Unlimited release 1.10.0 with ElectrsCash 3.1
• v24.0.0

Also provide instructions for user to upgrade.

When running Bitcoin Cash Node, the config file configs/bitcoincashnode.conf contains reindex=1

We should remove it, since this makes container restart incredibly slow. A reindex should only be used manually if the user knows the data got corrupted.

use-case: Run devsuite without smartbch

In current master, if I build-config without selecting smartbch, I get following error

  File "/home/biggestfan/bch-devsuite/./bch-devsuite", line 349, in setup_docker_compose
    if config.smartbch.enabled:
AttributeError: 'NoneType' object has no attribute 'enabled'

Work-around is to select smartbch in wizard, but hand-edit the generated toml to say enabled=false in [smartbch] section.

The wizard should have generated a valid config in the first place.

Currently the included scripts expect that the virtualenv python package should be available. Anaconda manages its virtual environments in a slightly different way, and support can be added to the scripts so that users who prefer Anaconda can still run them.

bch-devsuite can support deployment of smartbch

Adding documentation on how what resource they should expect and also the load test result.

Currently, the setup script does not support deploying both rest.bitcoin.com and bch-api at the same time.
Time estimate : 2 day

Currently users of bch-toolkit need to wait long times (often several days) to fully sync node+slpdb to mainnet. We should provide dumps so that the bootstrap process can be made faster.

Large transactions (tried with 43 and 47 UTXOs) are not being broadcasted by https://rest.bch.actorforth.org/#/rawtransactions/sendRawTransactionSingle.

The same transaction was broadcasted fine by this API: https://insomnia.fountainhead.cash/v1/tx/broadcast. See docs here.

Upgrade bitcoin unlimited to 2.0.0.0

Create and deploy docs website for bch-toolkit

• generate from readme
• auto build and deploy using CI

Any RPC or CLI methods doesn't take into account what is in the mempool. It affects:
The node has spendzeroconfchange=1

The release version 1.9.2 has bug relating to checkpoints, but upstream dev branch has a fix.

re: #smartbch/smartbch#60

• Wait for official fix/release
• Apply the fix to devsuite

I propose the following structure:

• Name of project (rebrand to DevSuite)
• Badges (Tests, container deployment, coding style and so on..)
• Getting started/quickstart
• Advanced usage (or just link to the docs after QuickStart)
• Architecture
• Contribution guide (maybe just link to CONTRIBUTING.md)

Currently the devsuite need local BCH RPC deployed via devsuite script, it should be configurable to connect to any remote BCH node.