Hello there,

Currently you can make a peer join or configure only one channel with the chart. It would be nice, like it's done for the chaincodes, to be able to configure multiple.

The 9443 port is used by Kuberenetes in docker for mac (precisely kube-compose-api-server)

https://github.com/SubstraFoundation/hlf-k8s/blob/11cd70b744348e33778b3ee99157bc4137ad8df0/python-scripts/conf/owkin/peers/peer1.py#L32

So in a docker-compose setup, if kuberenetes is enabled, starting hlf-k8s will fail.
Is it possible to bind another port or to make prometheus optional by default ?

Hello do we have an example for this?

Update [02-18-2022]
I've been experimenting with deploying multiple peers for an org in the same namespace. Using the existing configuration with external chaincode, im able to deploy a peer 2 to the same namespace and use the same chaincode pod that was deployed as part of the peer 1 release.

There's a modification that is required here. For chaincode operator, each release will deploy a chaincode deployment and operator if there's a chaincode object from values.yaml. The current implementation packages a new cc package(for each release) and this creation of package causes a potential conflict of the same intended chaincode to be used across the org.

I have a working example that will allow peer 2 to use the same chaincode package from a secret that was published from peer1's release.

I think this design is optimal as you can export the secret to import it across clusters if you have peers in other clusters to use.

Will add a PR when I get a chance but any feedback on this approach is appreciated OR if there's already an existing method from current implementation that I had missed would save me some effort.

Hi there,

hlf-ca.orderer.host variable seems unused am I right?

There are currently no instructions on how to upgrade the chaincode version on a running installation.

Hello, if I use these helm charts to deploy my network, how can I extend my existed network? For example If I need to add a second channel (which will be joined by all the existing orgs) and deploy a new chaincode to this channel (or update the chaincode of an old channel). Can I do that somehow?

Finally, the gateway client functionality can also be deployed to k8s?

Hi!

I trying to start hlf on my kubernetes cluster on OpenStack with csi-sc-cinderplugin for PVC.
Some pods succesfully started (like couchdb, ca).
Other pods is stuck in ContainerCreating status. I've got typical errors from these pods like:

Events:
  Type     Reason       Age                    From               Message
  ----     ------       ----                   ----               -------
  Normal   Scheduled    48m                    default-scheduler  Successfully assigned hlf-dev/hlf-hlf-k8s-toolbox-775bd96b6f-xph7m to k8s5
  Warning  FailedMount  48m (x2 over 48m)      kubelet            MountVolume.SetUp failed for volume "ord-tls-rootcert" : secret "ord-tls-rootcert" not found
  Warning  FailedMount  48m (x2 over 48m)      kubelet            MountVolume.SetUp failed for volume "id-key" : secret "hlf-msp-key-user" not found
  Warning  FailedMount  48m (x2 over 48m)      kubelet            MountVolume.SetUp failed for volume "cacert" : secret "hlf-cacert" not found
  Warning  FailedMount  48m (x2 over 48m)      kubelet            MountVolume.SetUp failed for volume "admin-key" : secret "hlf-msp-key-admin" not found
  Warning  FailedMount  48m (x2 over 48m)      kubelet            MountVolume.SetUp failed for volume "tls-client" : secret "hlf-tls-user" not found
  Warning  FailedMount  48m (x3 over 48m)      kubelet            MountVolume.SetUp failed for volume "tls-rootcert" : secret "hlf-cacert" not found
  Warning  FailedMount  48m (x2 over 48m)      kubelet            MountVolume.SetUp failed for volume "admin-cert" : secret "hlf-msp-cert-admin" not found
  Warning  FailedMount  42m (x10 over 48m)     kubelet            MountVolume.SetUp failed for volume "id-cert" : secret "hlf-msp-cert-user" not found
  Warning  FailedMount  38m (x8 over 48m)      kubelet            MountVolume.SetUp failed for volume "tls-clientrootcert" : secret "hlf-cacert" not found
  Warning  FailedMount  22m (x19 over 48m)     kubelet            MountVolume.SetUp failed for volume "tls" : secret "hlf-tls-admin" not found
  Warning  FailedMount  3m31s (x109 over 48m)  kubelet            (combined from similar events): Unable to attach or mount volumes: unmounted volumes=[id-key cacert tls-clientrootcert tls-rootcert tls-client id-cert ord-tls-rootcert admin-key tls admin-cert], unattached volumes=[id-key cacert tls-clientrootcert kube-api-access-x4kqv tls-rootcert tls-client id-cert ord-tls-rootcert admin-key fabric-config tls admin-cert]: timed out waiting for the condition

Please help me to debug & resolve.

I've tried changing the yaml configuration files to run the cluster with more then one peer. I've changed the skaffold.yaml file as follows:

• I changed the policy in the network-org-1-peer-1:

chaincodes[0].policy: OR("MyOrg1MSP.member"\,"MyOrg2MSP.member"\,"MyOrg3MSP.member")

• Added some lines to the network-org-1-peer-1 config:

channels[0].extraOrgs[1].name: MyOrg3
channels[0].extraOrgs[1].secret: org-3-org-config-anchor
fetchSecrets[2].from: org-config-anchor
fetchSecrets[2].to: org-3-org-config-anchor
fetchSecrets[2].filename: configOrgWithAnchors.json
fetchSecrets[2].namespace: org-3

• Added a third organization with the following config (copied from the 2nd):

- name: network-org-3-peer-1
  chartPath: charts/hlf-k8s
  namespace: org-3
  imageStrategy:
    helm: {}
  values:
    image: substrafoundation/hlf-k8s
  setValues:
    nginx-ingress.enabled: true
    nginx-ingress.controller.scope.enabled: true
    ca.caName: rcaOrg3
    peer.host: network-org-3-peer-1.org-3  # {name}.{namespace}
    organization.id: MyOrg3MSP
    organization.name: MyOrg3
    peer.peer.mspID: MyOrg3MSP
    chaincodes[0].name: mycc
    chaincodes[0].version: "1.0"
    # Note: Instead of an URL, you can use an absolute path, e.g. /home/johndoe/code/substra-chaincode
    #       This path folder must be accessible to kubernetes. See README for details.
    chaincodes[0].src: https://github.com/SubstraFoundation/substra-chaincode/archive/master.tar.gz
    channels[0].name: mychannel
    channels[0].join: true
    orderer.host: network-orderer.orderer
    peer.peer.gossip.externalEndpoint: network-org-3-peer-1.org-3:7051  # {name}.{namespace}:{port}
    fetchSecrets[0].from: ord-tls-rootcert
    fetchSecrets[0].to: ord-tls-rootcert
    fetchSecrets[0].filename: cacert.pem
    fetchSecrets[0].namespace: orderer

What happens is that org-1 creates the channel, then the first peer to finish its setup between org-2 and org-3 manages to join it, but then the last peer can't because the handshake fails.

Here's the error I get from the network-org-1-peer-1-hlf-k8s-add-org-mychannel-myorg2 job:

Error: got unexpected status: BAD_REQUEST -- error applying config update to existing channel 'mychannel': error authorizing update: error validating DeltaSet: policy for [Group]  /Channel/Application not satisfied: implicit policy ev aluation failed - 1 sub-policies were satisfied, but this policy requires 2 of the 'Admins' sub-policies to be satisfied
Testing the connection with the orderer:Testing the connection with the orderer:Testing the connection with the orderer:

And here's the error I get from the network-org-2-peer-1-hlf-k8s-channel-join-0 job:

Error: failed to create deliver client: orderer client failed to connect to network-orderer.orderer:7050: failed to create new connection: connection error: desc = "transport: authentication handshake failed: remote error: tls: bad certificate"

I'm trying to run hlf-k8s using the latest line on the compatibility table. I'm running Ubuntu Server 18.05.4, with the following software versions:

• minikube 1.9.2
• docker 19.03.13
• kubernetes 1.15.11
• hlf-k8s 0.0.13
• substra-chaincode 0.0.11
• skaffold 1.8.0
• helm 3.4.1
• kubectl 1.18.0

I've tried using a local (by removing the chaincode[0].src entry in the skaffold.yaml file, and setting chaincodes[0].hostPath to the absolute path to the chaincode repository) and a remote (by setting chaincode[0].src to https://github.com/SubstraFoundation/substra-chaincode/archive/0.0.11.tar.gz in the skaffold.yaml file) version of the chaincode repository.
I've changed the hlf-k8s chart's requirements.yaml file to replace https://kubernetes-charts.storage.googleapis.com with https://charts.helm.sh/stable, and then ran a

helm dependency update $absolute_path_to_chart

I then run the skaffold run command and here's the error I get from the org-1/network-org-1-peer-1-hlf-peer pod:

[endorser] callChaincode -> INFO 223 [yourchannel][4ba51601] Entry chaincode: name:"lscc"
[lscc] executeDeployOrUpgrade -> ERRO 224 cannot get package for chaincode (mycc:1.0)-err:open /var/hyperledger/production/chaincodes/mycc.1.0: no such file or directory
[endorser] callChaincode -> INFO 225 [yourchannel][4ba51601] Exit chaincode: name:"lscc"  (1ms)
[endorser] ProcessProposal -> ERRO 226 [yourchannel][4ba51601] simulateProposal() resulted in chaincode name:"lscc"  response status 500 for txid: 4ba516018a438e897a69671558f35ea65f6fa339ee943a7eee0ed930458325fe
[comm.grpc.server] 1 -> INFO 227 unary call completed grpc.service=protos.Endorser grpc.method=ProcessProposal grpc.peer_address=172.17.0.26:42370 grpc.peer_subject="CN=user,OU=peer,O=Hyperledger,ST=North Carolina,C=US" grpc.code=OK grpc.call_duration=1.501797ms

With https://helm.sh/docs/topics/charts/#schema-files, we need to warn user if they deploy hlf-k8s without persistence over peer, orderer, ca, couchdb !

For stability purposes I'm trying to run the substra platform on microk8s instead of minikube (on ubuntu 18.04.5). Using the configuration that was working with the latter:

• docker 19.03.12
• kubernetes 1.15.11
• hlf-k8s 0.0.12
• substra-backend 0.0.19
• skaffold 1.8.0
  I've started microk8s, enabled dns, ingress, rbac and helm 2.14.3. I've added the bitnami repository to helm, and for tiller to be successfuly setup on the cluster I've setup a service account and a role binding and applied it to the cluster using microk8s.kubectl apply -f my-setup.yaml.
  But when I run skaffold run from my hlf-k8s folder I end up with the following error:

FATA[0001] failed to build: build failed: building [substrafoundation/hlf-k8s]: build artifact: denied: requested access to the resource is denied

I've tried to docker logout && docker login, and also to do a helm repo update with no success. And when I tried to install the chart manualy using microk8s.helm install --name orderer --namespace orderer substra/hlf-k8s --version 1.3.0 -f ./charts/hlf-k8s/values.yaml I get the following error:

Error: failed to download "substra/hlf-k8s" (hint: running `helm repo update` may help)

When owkin/charts#23 is merged we need to update the helm dep :)

This condition returns true if the channel has joined a channel name which includes the target channel name.

For instance, if the peer has joined demochannel2, then the condition will return true for demochannel even though the peer hasn't joined demochannel

peer channel fetch config writes its output inside / for some operator.

Hello there 👋

I tried today to make hlf-k8s works behind a proxy but it seems like I need to do a lot of changes in order to make it work.

First, configuring a forward proxy usually means setting a set of environment variables (HTTP_PROXY / NO_PROXY / HTTPS_PROXY etc.) but the chart doesn't allow to put extraEnv on the pods.

Then, some commands won't use the HTTP_PROXY configuration, like netcat (it doesn't use http protocol) so that means that this for example won't work: https://github.com/SubstraFoundation/hlf-k8s/blob/master/charts/hlf-k8s/templates/deployment-appchannel-operator.yaml#L52

Finally, after some digging it seems that the Hyperledger Fabric community discourage this setup and suggest using NAT / vpn instead:

• https://stackoverflow.com/questions/62417442/hyperledger-fabric-behind-firewall-proxy
• https://lists.hyperledger.org/g/fabric/topic/17549689

I agree that it doesn't make sense for HLF to be behind a proxy (as introducing a proxy kind of break the trust in the network anyway), but, is it even doable ? What are your thoughts on that ?

Is it possible to add new org to an already deployed network?

Hi, how can I solve this problem?
I set up my environment using docker. However, when I run the python3 start.py, I got the following errors:

Loading configuration
Loaded configuration: /substra/data/orgs/chu-nantes/configtx.yaml
[common.tools.configtxgen.encoder] NewConsortiumOrgGroup
WARN Default policy emission is deprecated, please include policy specifications for the orderer org group chu-nantes in configtx.yaml
/substra/conf/config/conf-chu-nantes.json
Will getChannelConfigBlockWithOrderer
got ChannelConfigBlockWithOrderer
signAndPushSystemUpdateProposal
Generating channel configuration transaction at /substra/data/channel/substrachannel.tx
....
channel creation: True
Wait For Peers to join channel
Join channel substrachannel with peers ['peer1-chu-nantes', 'peer2-chu-nantes'] ...
fail to get genesis block
Will retry to make peers join channel
fail to get genesis block
Will retry to make peers join channel
Peers ['peer1-chu-nantes', 'peer2-chu-nantes'] successfully joined channel substrachannel
Installing chaincode on ['peer1-chu-nantes', 'peer2-chu-nantes'] ...
policy: OR('chu-nantesMSP.member')
Instantiated chaincode with policy: {'identities': [{'role': {'name': 'member', 'mspId': 'chu-nantesMSP'}}], 'policy': {'1-of': [{'signed-by': 0}]}} and result: "timeout expired while starting chaincode substracc:1.0 for transaction"
Try to query chaincode from peer ['peer1-chu-nantes', 'peer2-chu-nantes'] on org chu-nantes
Traceback (most recent call last):
File "/scripts/run.py", line 135, in
add_org()
File "/scripts/run.py", line 116, in add_org
if client.queryChaincodeFromPeers() == '[]':
File "/scripts/utils/run_utils.py", line 283, in queryChaincodeFromPeers
cc_name=self.chaincode_name,
File "/usr/lib/python3.6/asyncio/base_events.py", line 484, in run_until_complete
return future.result()
File "/usr/local/lib/python3.6/dist-packages/hfc/fabric/client.py", line 1662, in chaincode_query
raise Exception(res)
Exception: [response {
status: 500
message: "make sure the chaincode substracc has been successfully instantiated and try again: chaincode substracc not found"
}
, response {
status: 500
message: "make sure the chaincode substracc has been successfully instantiated and try again: chaincode substracc not found"
}
]