hyperledger / firefly-cli Goto Github PK
View Code? Open in Web Editor NEWFireFly Command Line Interface (CLI)
License: Apache License 2.0
FireFly Command Line Interface (CLI)
License: Apache License 2.0
To keep consistency across codebases, we should add the linters to the Makefile
to this repo from https://github.com/hyperledger-labs/firefly/blob/main/Makefile
firefly logs command is broken. Please find the detailed error below.
ff logs dev1
getting logs...
Define and run multi-container applications with Docker.
Usage:
docker-compose [-f <arg>...] [options] [COMMAND] [ARGS...]
docker-compose -h|--help
For developers working on core, it would be handy to have a mode that generates the YAML files for running core outside of docker-compose using the exposed ports.
This is a manual activity today, that has to be re-done each time on the ff remove
/ ff init
cycle:
ff init
/ ff start -n
docker-compose.yml
to comment out one or more firefly_core_0
servicesfirefly_1.core
etc. to change all the docker link hostnames, into localhost:510x
hostnamesff stop
/ ff start
launch.json
etc)When I execute:
ff init -b besu dev1
and then:
ff start dev1
I get:
reading stack config... done
this will take a few seconds longer since this is the first time you're running this stack...
⡿ Error: an error occurred - rolling back changes... Error: docker-compose up -d
Failed [1] Creating network "dev1_default" with the default driver
Creating volume "dev1_ethconnect_abis_0" with default driver
Creating volume "dev1_ethconnect_abis_1" with default driver
Creating volume "dev1_ethconnect_events_0" with default driver
Creating volume "dev1_ethconnect_events_1" with default driver
Creating volume "dev1_ipfs_data_0" with default driver
Creating volume "dev1_ipfs_data_1" with default driver
Creating volume "dev1_ipfs_staging_0" with default driver
Creating volume "dev1_ipfs_staging_1" with default driver
Creating volume "dev1_member1tessera" with default driver
Creating volume "dev1_public-keys" with default driver
Creating dev1_validator1_1 ...
Creating dev1_ipfs_0 ...
Creating dev1_member1tessera_1 ...
Creating dev1_ipfs_1 ...
Creating dev1_dataexchange_0 ...
Creating dev1_dataexchange_1 ...
Creating dev1_ipfs_1 ... done
Creating dev1_member1tessera_1 ... done
Creating dev1_ipfs_0 ... done
Creating dev1_validator1_1 ... done
Creating dev1_rpcnode_1 ...
Creating dev1_dataexchange_0 ... done
Creating dev1_dataexchange_1 ... done
Creating dev1_rpcnode_1 ... done
ERROR: for firefly_core_0 Container "3d03ad4e2b1a" is unhealthy.
ERROR: for ethsigner Container "3d03ad4e2b1a" is unhealthy.
ERROR: for member1besu Container "3d03ad4e2b1a" is unhealthy.
Encountered errors while bringing up the project.
- all changes rolled back
Usage:
ff start <stack_name> [flags]
Flags:
-h, --help help for start
-b, --no-rollback Do not automatically rollback changes if first time setup fails
Global Flags:
--ansi string control when to print ANSI control characters ("never"|"always"|"auto") (default "auto")
-v, --verbose verbose log output
Error: docker-compose up -d
Failed [1] Creating network "dev1_default" with the default driver
Creating volume "dev1_ethconnect_abis_0" with default driver
Creating volume "dev1_ethconnect_abis_1" with default driver
Creating volume "dev1_ethconnect_events_0" with default driver
Creating volume "dev1_ethconnect_events_1" with default driver
Creating volume "dev1_ipfs_data_0" with default driver
Creating volume "dev1_ipfs_data_1" with default driver
Creating volume "dev1_ipfs_staging_0" with default driver
Creating volume "dev1_ipfs_staging_1" with default driver
Creating volume "dev1_member1tessera" with default driver
Creating volume "dev1_public-keys" with default driver
Creating dev1_validator1_1 ...
Creating dev1_ipfs_0 ...
Creating dev1_member1tessera_1 ...
Creating dev1_ipfs_1 ...
Creating dev1_dataexchange_0 ...
Creating dev1_dataexchange_1 ...
Creating dev1_ipfs_1 ... done
Creating dev1_member1tessera_1 ... done
Creating dev1_ipfs_0 ... done
Creating dev1_validator1_1 ... done
Creating dev1_rpcnode_1 ...
Creating dev1_dataexchange_0 ... done
Creating dev1_dataexchange_1 ... done
Creating dev1_rpcnode_1 ... done
ERROR: for firefly_core_0 Container "3d03ad4e2b1a" is unhealthy.
ERROR: for ethsigner Container "3d03ad4e2b1a" is unhealthy.
ERROR: for member1besu Container "3d03ad4e2b1a" is unhealthy.
Encountered errors while bringing up the project.
- all changes rolled back
If you point the CLI to a local manifest with -m manifest.json
, and that manifest contains a local Docker definitions such as:
"tokens-erc1155": {
"image": "ghcr.io/hyperledger/firefly-tokens-erc1155",
"local": true
}
When running ff start
, the CLI still pulls the latest image from the server, overwriting the local image.
This seems to be due to a duplication in the PullStack
method here. It first iterates the images defined in the manifest, and does properly skip those tagged as local. But then it goes on to pull the latest for all images defined by the blockchain and token plugins, defeating the purpose of the earlier loop.
Side node: because it first pulls all the images defined in the manifest, it will pull images even if you're ultimately not using them (ie it pulls fabconnect even if your blockchain provider is Ethereum).
Conclusion from all this: we need better "pull" logic to intersect the info from these two places - it should only pull the Dockers actually used in the stack, but it should also honor the versions and "local" flag defined for those images in the manifest.
CLI README should have advanced usage examples:
like in "https://hyperledger.github.io/firefly/contributors/advanced_cli_usage.html" link
The "usage" info for the active CLI command is always dumped when an error is encountered. This should only be dumped for errors that stem from invalid syntax, not from functional errors executing the command.
Some users are brand new to Go, and struggle with the go install
1.16 route to getting the CLI up.
It would be good to have binary distributions published of the CLI for download, cross compiled for a number of platforms.
Hoping there's a git action off the shelf for this, that can be wired to creation of tags (similar to how the UI builds today).
Note this is a small step towards integration with binary package managers like
brew
andapt
Would mean syntax highlighting and other helpful features in editors.
There is no requirement for the messaging blockchain and the tokens blockchain to be the same - FireFly can fully support using Fabric as the primary blockchain but using Ethereum ERC1155 for tokens. Currently the CLI blocks this combination.
The setup logic should be adjusted to properly handle this combination - which means spinning up containers for both ethconnect and fabconnect.
This has lead to sporadic timeout failures in FireFly E2E tests such as this one: https://github.com/hyperledger/firefly/runs/4026832368?check_suite_focus=true
The pull
command should pull all images required to run the stack.
When I try to get /webui/
on the IPFS container, it redirects me to:
/ipfs/bafybeif4zkmu7qdhkpf3pnhwxipylqleof7rl6ojbe7mq3fzogz6m4xk3i/
However, it then doesn't return, and I assume that's because that needs to be published to the IPFS network as part of the CLI initialization ❔
ff start [stack_name]
should start up stack completely per user prompted information
Upon initializing the stack and issuing the ff start [stack_name]
command, containers are spinning up but the postgres containers are failing with apparent permissions errors which cause other containers to fail due to inability to connect to the database container.
I have executed chmod -R 0755 ~/.firefly/stacks/[stack_name]/data/postgres_[N]
on each of the indicated node folders.
The ff version
command is currently not printing the correct information. This is likely a build time issue with some flags not being set correctly.
It would be nice to have a cli command to generate the firefly.core
skeleton.
We should add an ff info
command to the CLI that will show which version of each component is in a stack
Windows 10 19041.vb_release.191206-1406
Docker 20.10.7
Ubuntu 20.04.2 LTS
WSL2
user@WinDev2104Eval:/mnt/c/Users/User$ ff start dev
reading stack config... done
starting FireFly stack 'dev'...
this will take a few seconds longer since this is the first time you're running this stack...
doneError: docker-compose up -d
Failed [1] Starting dev_dataexchange_1_1 ...
Recreating dev_ganache_1 ...
Recreating dev_ipfs_1_1 ...
Starting dev_dataexchange_0_1 ...
Recreating dev_ipfs_0_1 ...
Starting dev_postgres_0_1 ...
Starting dev_postgres_1_1 ...
Starting dev_postgres_1_1 ... error
ERROR: for dev_postgres_1_1 Cannot start service postgres_1: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:76: mounting "/run/desktop/mnt/host/wsl/docker-desktop-bind-mounts/Ubuntu/b897274b7ee0a84e4b7a08f01135f7abf252252387a6bfd4a83f4d48be882275" to rootfs at "/var/lib/postgresql/data" caused: mount through procfd: no such file or directory: unknown
Starting dev_dataexchange_0_1 ... error
ERROR: for dev_dataexchange_0_1 Cannot start service dataexchange_0: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:76: mounting "/run/desktop/mnt/host/wsl/docker-desktop-bind-mounts/Ubuntu/f6af0a37e04f2cabccc680a8ec01fefbb2e6b72f8af9fc12ac2b418d1b6f11ab" to rootfs at "/data" caused: mount through procfd: no such file or directory: unknown
Starting dev_postgres_0_1 ... error
ERROR: for dev_postgres_0_1 Cannot start service postgres_0: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:76: mounting "/run/desktop/mnt/host/wsl/docker-desktop-bind-mounts/Ubuntu/6ff0531043d38f4b8ce90f6bbcf311b1211f9b53d3de8d3fc725d6472b2bc693" to rootfs at "/var/lib/postgresql/data" caused: mount through procfd: no such file or directory: unknown
Starting dev_dataexchange_1_1 ... error
ERROR: for dev_dataexchange_1_1 Cannot start service dataexchange_1: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:76: mounting "/run/desktop/mnt/host/wsl/docker-desktop-bind-mounts/Ubuntu/aafe84939634b8d96987e93acabc4150d998cb9829da0f60274d65768945ca13" to rootfs at "/data" caused: mount through procfd: no such file or directory: unknown
Recreating dev_ipfs_1_1 ... done
Recreating dev_ipfs_0_1 ... done
Recreating dev_ganache_1 ... done
ERROR: for postgres_1 Cannot start service postgres_1: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:76: mounting "/run/desktop/mnt/host/wsl/docker-desktop-bind-mounts/Ubuntu/b897274b7ee0a84e4b7a08f01135f7abf252252387a6bfd4a83f4d48be882275" to rootfs at "/var/lib/postgresql/data" caused: mount through procfd: no such file or directory: unknown
ERROR: for dataexchange_0 Cannot start service dataexchange_0: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:76: mounting "/run/desktop/mnt/host/wsl/docker-desktop-bind-mounts/Ubuntu/f6af0a37e04f2cabccc680a8ec01fefbb2e6b72f8af9fc12ac2b418d1b6f11ab" to rootfs at "/data" caused: mount through procfd: no such file or directory: unknown
ERROR: for postgres_0 Cannot start service postgres_0: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:76: mounting "/run/desktop/mnt/host/wsl/docker-desktop-bind-mounts/Ubuntu/6ff0531043d38f4b8ce90f6bbcf311b1211f9b53d3de8d3fc725d6472b2bc693" to rootfs at "/var/lib/postgresql/data" caused: mount through procfd: no such file or directory: unknown
ERROR: for dataexchange_1 Cannot start service dataexchange_1: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: rootfs_linux.go:76: mounting "/run/desktop/mnt/host/wsl/docker-desktop-bind-mounts/Ubuntu/aafe84939634b8d96987e93acabc4150d998cb9829da0f60274d65768945ca13" to rootfs at "/data" caused: mount through procfd: no such file or directory: unknown
ERROR: for ethconnect_0 Container "d2b2ab1a1280" is unhealthy.
ERROR: for ethconnect_1 Container "d2b2ab1a1280" is unhealthy.
Encountered errors while bringing up the project.
Usage:
ff start <stack_name> [flags]
Flags:
-h, --help help for start
-n, --no-pull Do not pull latest images when starting
Global Flags:
--ansi string control when to print ANSI control characters ("never"|"always"|"auto") (default "auto") (default "auto")
-v, --verbose verbose log output
Error: docker-compose up -d
OS: macOS Version 11.3
commands run:
Snippet from the terminal:
firefly % ff start trial --verbose
reading stack config... done
this will take a few seconds longer since this is the first time you're running this stack...
starting FireFly stack 'trial'... initializing blockchain node
/usr/local/bin/docker-compose down --volumes
trial Warning: No resource found to remove
Error: docker run --rm -v trial_geth:/data ethereum/client-go:release-1.9 --datadir /data --nousb init /data/genesis.json
Failed [1] INFO [10-01|19:14:55.834] Maximum peer count ETH=50 LES=0 total=50
INFO [10-01|19:14:55.834] Smartcard socket not found, disabling err="stat /run/pcscd/pcscd.comm: no such file or directory"
INFO [10-01|19:14:55.835] Set global gas cap cap=25000000
INFO [10-01|19:14:55.835] Allocated cache and file handles database=/data/geth/chaindata cache=16.00MiB handles=16
Fatal: Failed to write genesis block: database contains incompatible genesis (have 4e89bfe2e09d6140e33bd6b48124418bc37dcb8ebaa2f3c9cda65c4e47da5a67, new fa0217c58f30bd25803991de9690f7b39678582443e358819638b4d162a8bf9f)
INFO [10-01|19:14:55.844] Persisted trie from memory database nodes=3 size=455.00B time="19.156µs" gcnodes=0 gcsize=0.00B gctime=0s livenodes=1 livesize=0.00B
Fatal: Failed to write genesis block: database contains incompatible genesis (have 4e89bfe2e09d6140e33bd6b48124418bc37dcb8ebaa2f3c9cda65c4e47da5a67, new fa0217c58f30bd25803991de9690f7b39678582443e358819638b4d162a8bf9f)
Flags:
-h, --help help for start
-n, --no-pull Do not pull latest images when starting
-b, --no-rollback Do not automatically rollback changes if first time setup fails
Global Flags:
--ansi string control when to print ANSI control characters ("never"|"always"|"auto") (default "auto") (default "auto")
-v, --verbose verbose log output
Error: docker run --rm -v trial_geth:/data ethereum/client-go:release-1.9 --datadir /data --nousb init /data/genesis.json
Failed [1] INFO [10-01|19:14:55.834] Maximum peer count ETH=50 LES=0 total=50
INFO [10-01|19:14:55.834] Smartcard socket not found, disabling err="stat /run/pcscd/pcscd.comm: no such file or directory"
INFO [10-01|19:14:55.835] Set global gas cap cap=25000000
INFO [10-01|19:14:55.835] Allocated cache and file handles database=/data/geth/chaindata cache=16.00MiB handles=16
Fatal: Failed to write genesis block: database contains incompatible genesis (have 4e89bfe2e09d6140e33bd6b48124418bc37dcb8ebaa2f3c9cda65c4e47da5a67, new fa0217c58f30bd25803991de9690f7b39678582443e358819638b4d162a8bf9f)
INFO [10-01|19:14:55.844] Persisted trie from memory database nodes=3 size=455.00B time="19.156µs" gcnodes=0 gcsize=0.00B gctime=0s livenodes=1 livesize=0.00B
Fatal: Failed to write genesis block: database contains incompatible genesis (have 4e89bfe2e09d6140e33bd6b48124418bc37dcb8ebaa2f3c9cda65c4e47da5a67, new fa0217c58f30bd25803991de9690f7b39678582443e358819638b4d162a8bf9f)
ipfs containers fail on startup after using ff cli with ipfs/go-ipfs:latest image as referenced in this issue. The temporary fix mentioned in the issue is to revert to v0.10.0.
ff cli workaround is manually editing the docker-compose file to v0.10.0 in order for the stack to start. Local firefly stack running with workaround appears to function but runs into 2 E2E test failures TestE2EBroadcastBlob and TestE2EPrivateBlobDatatypeTagged.
E2E Test Output below
ipfs_issue_e2e_logs.txt
We should add unit tests to verify CLI functionality. We should probably also move some of the logic out of the cmd
module and into the internal
module.
I just generated a 2 party stack (as part of running the firefly E2E tests), and some of the URLs in the config are mismatched. Notice in the file below, it is for org_0
and points to ipfs_0
and dataexchange_0
, but also points to ethconnect_1
and tokens_1
. This causes some strange failures since certain actions go to the wrong container.
firefly_core_0.yaml
log:
level: debug
debug:
port: 6060
http:
port: 5000
address: 0.0.0.0
publicURL: http://127.0.0.1:5000
admin:
port: 5101
address: 0.0.0.0
enabled: true
preinit: true
publicURL: http://127.0.0.1:5101
ui:
path: ./frontend
node:
name: node_0
org:
name: org_0
identity: 0xee08e985af4544e46dd735cf13080a39a8761146
blockchain:
type: ethereum
ethereum:
ethconnect:
url: http://ethconnect_1:8080
instance: /contracts/firefly
topic: "1"
database:
type: sqlite3
sqlite3:
url: /etc/firefly/db?_busy_timeout=5000
migrations:
auto: true
publicstorage:
type: ipfs
ipfs:
api:
url: http://ipfs_0:5001
gateway:
url: http://ipfs_0:8080
dataexchange:
https:
url: http://dataexchange_0:3000
tokens:
- connector: https
name: erc1155
url: http://tokens_1:3000
Due to licensing issues, we should remove dependencies on go-ethereum packages. These should be fairly easy to remove as the only thing they are used for right now is generating key pairs and public addresses.
When running ff start
, nodes intermittently fail to register with the following error:
Post "http://localhost:5000/api/v1/network/register/node/organization": dial tcp 127.0.0.1:5000: connect: connection refused
Furthermore, the registration is not re-attempted on subsequent starts, so you generally have to remove and recreate the stack.
There are a lot of things that need to happen the first time a stack starts up.
A failure at any one of these steps could result in a stack that's just not usable. I'm wondering if we want to catch errors that are thrown, and automatically run a resetStack
so the user can just try the start
command again?
There are a couple of potential issues with the init
command right now:
" "
which causes all kinds of problems, including files being written to the wrong directoryfirefly % ff start trial --verbose
reading stack config... done
this will take a few seconds longer since this is the first time you're running this stack...
starting FireFly stack 'trial'... initializing blockchain node
/usr/local/bin/docker-compose down --volumes
trial Warning: No resource found to remove
Error: docker run --rm -v /Users/kaulgudramer/.firefly/stacks/trial/blockchain:/geth -v trial_geth:/data ethereum/client-go:release-1.9 --nousb account import --password /geth/password --keystore /data/keystore /geth/0/keyfile
Failed [1] INFO [10-01|19:15:54.069] Maximum peer count ETH=50 LES=0 total=50
INFO [10-01|19:15:54.069] Smartcard socket not found, disabling err="stat /run/pcscd/pcscd.comm: no such file or directory"
INFO [10-01|19:15:54.070] Set global gas cap cap=25000000
Fatal: Could not create the account: account already exists
Fatal: Could not create the account: account already exists
Flags:
-h, --help help for start
-n, --no-pull Do not pull latest images when starting
-b, --no-rollback Do not automatically rollback changes if first time setup fails
Global Flags:
--ansi string control when to print ANSI control characters ("never"|"always"|"auto") (default "auto") (default "auto")
-v, --verbose verbose log output
Error: docker run --rm -v /Users/kaulgudramer/.firefly/stacks/trial/blockchain:/geth -v trial_geth:/data ethereum/client-go:release-1.9 --nousb account import --password /geth/password --keystore /data/keystore /geth/0/keyfile
Failed [1] INFO [10-01|19:15:54.069] Maximum peer count ETH=50 LES=0 total=50
INFO [10-01|19:15:54.069] Smartcard socket not found, disabling err="stat /run/pcscd/pcscd.comm: no such file or directory"
INFO [10-01|19:15:54.070] Set global gas cap cap=25000000
Fatal: Could not create the account: account already exists
Fatal: Could not create the account: account already exists
On setting up a new dev environment it's easy to miss steps, especially if you're in a hurry, or if you're like me and think you know what you're doing without reading the instructions. For example, if you install docker
but not docker-compose
(on Linux they're two different packages), some things will work, but other things will not. It would be great to have some checks to make sure prerequisites are in place before starting to try to use docker or docker-compose. If things are not set up correctly, it would be nice for the CLI to print out a helpful error message. Right now, if docker-compose is not installed, usually the error that gets printed is very cryptic.
The docker daemon not running in the background is another common issue that people sometimes run into.
Lastly, the current user might not have permission to run docker commands without sudo
if they're not in the right group. It would be great if the CLI could run quick pre-check to make sure all these things are set up correctly. I think this would really help new devs setting up their environment for the first time.
It would be awesome to have the cli distributed & managed through homebrew (and other package managers), so users do not need to manually upgrade versions, etc.
Using Git Bash on Windows, the promptui
functions seem to get skipped over, including validation which leads to problems. For instance, it's possible to create a stack with no name, or with an invalid number of members. More testing is needed on this to determine why this shell behaves differently.
This needs a bit more investigation. It seems like sometimes FireFly may be closing the HTTP connection before responding when the /config/reset
endpoint is called, but FireFly has code to specifically make sure it finishes the response before restarting the HTTP server. It may be easier to gracefully handle the EOF in the CLI, but I'm not sure that's the "right" answer here.
This behavior ends up resulting in the CLI rolling back the stack. If the user simply tries again it usually works, but this happens often enough that it's really annoying.
ff start <stack_name>
reading stack config... done
this will take a few seconds longer since this is the first time you're running this stack...
⣟ Error: an error occurred - rolling back changes... Error: Post "http://localhost:5201/admin/api/v1/config/reset": EOF - all changes rolled back
Usage:
ff start <stack_name> [flags]
Flags:
-h, --help help for start
-n, --no-pull Do not pull latest images when starting
-b, --no-rollback Do not automatically rollback changes if first time setup fails
Global Flags:
--ansi string control when to print ANSI control characters ("never"|"always"|"auto") (default "auto") (default "auto")
-v, --verbose verbose log output
Error: Post "http://localhost:5201/admin/api/v1/config/reset": EOF - all changes rolled back
I'm seeing the blow error with increasing frequency when running make e2e
in FireFly Core.
I believe the problem is the CLI is terminating the FireFly core process while it's competing the automigrations.
Few potential enhancements I can see that might resolve this:
admin.preinit
feature, so the restart is not necessarydocker-compose.yml
until laterSymptom in CLI (after long delay)
...
registering org_0 and node_0
Error: Post "http://localhost:5000/api/v1/network/register/node/organization": dial tcp [::1]:5000: connect: connection refused
Usage:
ff start <stack_name> [flags]
Flags:
-h, --help help for start
-n, --no-pull Do not pull latest images when starting
Global Flags:
--ansi string control when to print ANSI control characters ("never"|"always"|"auto") (default "auto") (default "auto")
-v, --verbose verbose log output
Error: Post "http://localhost:5000/api/v1/network/register/node/organization": dial tcp [::1]:5000: connect: connection refused
make: *** [e2e] Error 1
Underlying logs in core:
docker compose logs -f firefly_core_0
firefly_core_0_1 | [2021-06-21T02:52:37.294Z] DEBUG Log level: debug pid=1
firefly_core_0_1 | [2021-06-21T02:52:37.295Z] INFO Project Firefly pid=1
firefly_core_0_1 | [2021-06-21T02:52:37.295Z] INFO © Copyright 2021 Kaleido, Inc. pid=1
firefly_core_0_1 | [2021-06-21T02:52:37.295Z] DEBUG Debug HTTP endpoint listening on localhost:6060 pid=1
firefly_core_0_1 | [2021-06-21T02:52:39.525Z] INFO Shutting down due to terminated pid=1
firefly_core_0_1 | [2021-06-21T02:52:44.156Z] DEBUG Log level: debug pid=1
firefly_core_0_1 | [2021-06-21T02:52:44.156Z] INFO Project Firefly pid=1
firefly_core_0_1 | [2021-06-21T02:52:44.156Z] INFO © Copyright 2021 Kaleido, Inc. pid=1
firefly_core_0_1 | [2021-06-21T02:52:44.157Z] DEBUG Debug HTTP endpoint listening on localhost:6060 pid=1
firefly_core_0_1 | Error: FF10163: Database migration failed: FF10163: Database migration failed: Dirty database version 19. Fix and force version.
firefly_core_0_1 | Usage:
firefly_core_0_1 | firefly [flags]
firefly_core_0_1 | firefly [command]
firefly_core_0_1 |
firefly_core_0_1 | Available Commands:
firefly_core_0_1 | help Help about any command
firefly_core_0_1 | showconfig List out the configuration options
firefly_core_0_1 |
firefly_core_0_1 | Flags:
firefly_core_0_1 | -f, --config string config file
firefly_core_0_1 | -h, --help help for firefly
firefly_core_0_1 |
firefly_core_0_1 | Use "firefly [command] --help" for more information about a command.
firefly_core_0_1 |
firefly_core_0_1 | FF10163: Database migration failed: FF10163: Database migration failed: Dirty database version 19. Fix and force version.
In order to make sure PRs are acceptable, we should be testing that the code compiles and passes all the linting rules added in #42
A FireFly stack uses Docker to expose quite a few ports. Unfortunately when docker-compose brings up a stack, if there is something already listening on a port that it is exposing, it silently fails to forward the port to the container. The stack appears to come up correctly, but some services likely will not work correctly.
The CLI should check all the ports required by the stack before starting it to make sure they are free. If the port is not free, the CLI should print an error explaining which port needs to be freed and then exit.
The IPFS nodes that FireFly CLI creates should be part of their own private network, and not be connected to public IPFS. We do not want data uploaded to a local FireFly to be accessible to the public internet. This will require some work on the generated docker-compose.yaml
file.
If there is a new version of firefly
or other components of the stack available, I want to be able to upgrade my local dev stack to use the new version, without losing any of my data.
It's important that I be able to upgrade without data loss, and test my app against the new version because this is what developers will need to do in a production environment. Verifying that the new version is backwards compatible with existing data is very important.
A few releases back (can't remember exactly when) this config was deprecated:
It's now org.key
.
Would be good for the CLI to switch over, as it will print a warning on every start up that the old config is used.
Note that the same is now true in v0.13.0 for dataexchange.https
here (now dataexchange.ffdx
):
firefly-cli/internal/core/firefly_config.go
Line 105 in 03797a8
We use a string for member identifiers throughout the codebase. However, right now the strings are generated by simply incrementing a counter as each member's config is generated. Users should be able to customize the name of each member both interactively and by passing flags.
Steps to Reproduce:
ff init stackone 2 --prometheus-enabled; ff init stacktwo 2 --prometheus-enabled
ff start stackone; ff stop stackone
ff start stacktwo
ERROR: for prometheus Cannot create container for service prometheus: Conflict. The container name "/prometheus" is already in use by container <container_name>. You have to remove (or rename) that container to be able to reuse that name.
macOS
zsh: bad CPU type in executable: ff
With hyperledger/firefly#321, firefly
will now have a metrics
section in its config file for configuring the Prometheus metrics server it comes with.
ff
needs to be updated to include templating / generating this section of config. And then as a potential enhancement to the local dev experience, I think it would be great if we had a CLI option to also standup a Prometheus and Grafana via docker-compose
when start
ing a stack. These would automatically know to scrape all FireFlys w/in a network for metrics.
If the docker compose version is v2.0.0, the name of the container is prefixed with a '-' instead of '_' causing the below error-
Error: docker cp trial2_firefly_core_0_1:/firefly/contracts /Users/isriniva/.firefly/stacks/trial2
Failed [1] Error: No such container:path: trial2_firefly_core_0_1:/firefly/contracts
Actual container created -
docker container ls
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
96d2ade0afc4 ghcr.io/hyperledger/firefly:latest "firefly" 5 minutes ago Up 4 minutes 0.0.0.0:5000->5000/tcp, 0.0.0.0:5101->5101/tcp trial2-firefly_core_0-1
04b5732a640d ghcr.io/hyperledger/firefly:latest "firefly" 5 minutes ago Up 4 minutes 0.0.0.0:5001->5001/tcp, 0.0.0.0:5201->5201/tcp trial2-firefly_core_1-1
➜ firefly-cli git:(main) docker --version
Docker version 20.10.8, build 3967b7d
➜ firefly-cli git:(main) docker-compose --version
Docker Compose version v2.0.0
➜ firefly-cli git:(main)
Error saying volume already exists but was not created by Docker Compose. Upgrading to docker compose v2.2.2 solves the issue.
Previously, if a named volume was created before running docker-compose up
it would still be removed when running docker-compose down --volumes
. In docker-compose
2.0.x this no longer appears to be the case. This causes all kinds of problems and confusion when the CLI reports that it has rolled changes back, but leftover volumes still have old data on them, causing subsequent runs to fail.
The CLI needs to perform special one-time initialization the first time a stack is started. To detect if the stack has been run before, currently it checks for the existence of a cert.pem
file used by the dataexchange service. We should not really rely on that, but should instead define a specific config value or touch a file to note that first-time setup is complete. The value must also be reset during an ff reset
command.
We should add an ff ls
command to list the stacks that currently exist. It would also be nice to indicate if any of them are running or not.
Would be nice to:
Could also discuss if running multiple stacks simultaneously should be supported - right now every stack picks the same port numbers so they will always conflict.
CLI version: 0.0.44
Running into an issue unlocking accounts when creating a stack with 3 members, geth seems to be crashing
INFO [02-23|21:36:24.190] Maximum peer count ETH=50 LES=0 total=50
INFO [02-23|21:36:24.191] Smartcard socket not found, disabling err="stat /run/pcscd/pcscd.comm: no such file or directory"
WARN [02-23|21:36:24.191] Sanitizing cache to Go's GC limits provided=1024 updated=662
INFO [02-23|21:36:24.192] Set global gas cap cap=50,000,000
WARN [02-23|21:36:24.192] Sanitizing invalid miner gas price provided=0 updated=0
INFO [02-23|21:36:24.193] Allocated trie memory caches clean=99.00MiB dirty=165.00MiB
INFO [02-23|21:36:24.193] Allocated cache and file handles database=/data/geth/chaindata cache=329.00MiB handles=524,288
INFO [02-23|21:36:24.247] Opened ancient database database=/data/geth/chaindata/ancient readonly=false
INFO [02-23|21:36:24.248] Initialised chain configuration config="{ChainID: 2021 Homestead: 0 DAO: <nil> DAOSupport: false EIP150: 0 EIP155: 0 EIP158: 0 Byzantium: 0 Constantinople: 0 Petersburg: 0 Istanbul: 0, Muir Glacier: <nil>, Berlin: <nil>, London: <nil>, Arrow Glacier: <nil>, MergeFork: <nil>, Engine: clique}"
INFO [02-23|21:36:24.248] Initialising Ethereum protocol network=2021 dbversion=<nil>
INFO [02-23|21:36:24.249] Loaded most recent local header number=0 hash=c408a8..dce2ac td=1 age=7mo2w1d
INFO [02-23|21:36:24.249] Loaded most recent local full block number=0 hash=c408a8..dce2ac td=1 age=7mo2w1d
INFO [02-23|21:36:24.249] Loaded most recent local fast block number=0 hash=c408a8..dce2ac td=1 age=7mo2w1d
WARN [02-23|21:36:24.249] Failed to load snapshot, regenerating err="missing or corrupted snapshot"
INFO [02-23|21:36:24.250] Rebuilding state snapshot
INFO [02-23|21:36:24.250] Resuming state snapshot generation root=0f51ed..0d5486 accounts=0 slots=0 storage=0.00B elapsed="394.166µs"
INFO [02-23|21:36:24.251] Generated state snapshot accounts=3 slots=0 storage=210.00B elapsed=1.102ms
INFO [02-23|21:36:24.252] Regenerated local transaction journal transactions=0 accounts=0
INFO [02-23|21:36:24.253] Stored checkpoint snapshot to disk number=0 hash=c408a8..dce2ac
INFO [02-23|21:36:24.253] Gasprice oracle is ignoring threshold set threshold=2
WARN [02-23|21:36:24.254] Error reading unclean shutdown markers error="leveldb: not found"
INFO [02-23|21:36:24.254] Starting peer-to-peer node instance=Geth/v1.10.16-stable-20356e57/linux-arm64/go1.17.7
INFO [02-23|21:36:24.265] New local node record seq=1,645,652,184,264 id=1b6f59cf886b6815 ip=127.0.0.1 udp=0 tcp=30311
INFO [02-23|21:36:24.265] Started P2P networking self="enode://b910ef363120bcf2b33df7baa44078acdce7c2c0385974ae6cc692fc45629a942cde2edfbdc9a7de73673570d33b847c9ad725525cd82fadbbc6c08dc311bdbf@127.0.0.1:30311?discport=0"
INFO [02-23|21:36:24.267] IPC endpoint opened url=/data/geth.ipc
INFO [02-23|21:36:24.267] HTTP server started endpoint=[::]:8545 prefix= cors= vhosts=*
INFO [02-23|21:36:24.267] Transaction pool price threshold updated price=0
INFO [02-23|21:36:24.267] Updated mining threads threads=0
INFO [02-23|21:36:24.267] Transaction pool price threshold updated price=0
INFO [02-23|21:36:24.267] Etherbase automatically configured address=0xa24220382c330594Cce66469eD33C913a3A6d991
INFO [02-23|21:36:24.267] Commit new sealing work number=1 sealhash=4a0634..06b743 uncles=0 txs=0 gas=0 fees=0 elapsed="214.458µs"
WARN [02-23|21:36:24.268] Block sealing failed err="sealing paused while waiting for transactions"
INFO [02-23|21:36:24.268] Commit new sealing work number=1 sealhash=4a0634..06b743 uncles=0 txs=0 gas=0 fees=0 elapsed="339.291µs"
If any of the docker/docker-compose commands wrapped by ff
exit with an error (non-zero) status, this should be treated as an error and surfaced by the CLI. Currently it checks for errors at the Go level but does not check exit codes.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.