Fresh installation of core on IBM kubernetes. I tried on 2 different clusters ver 1.13.10_1534 and 1.14.6_1531. Result is same:

Error: failed to create containerd task: OCI runtime create failed: container_linux.go:345: starting container process caused "exec: "sh": executable file not found in $PATH": unknown

kubectl -n stellar-testnet get pods

NAME READY STATUS RESTARTS AGE
stellar-core-dcfccfd9b-bbf5j 0/1 CrashLoopBackOff 7 25m
stellar-core-postgresql-0 0/1 Init:CrashLoopBackOff 7 24m

Looking at the pod details, there are these commands specified:

Command:
sh
-c
mkdir -p /bitnami/postgresql/data
chmod 700 /bitnami/postgresql/data
find /bitnami/postgresql -mindepth 1 -maxdepth 1 -not -name ".snapshot" -not -name "lost+found" |
xargs chown -R 1001:1001

The first command is sh
In the pod/deployment yaml, this command should to be qualified to include the path eg. /bin/sh instead of just sh

IBM support said that the issue happens is due to a containerd issue:
containerd/containerd#3597
IKS will pick up the containerd fix for this when they release version 1.2.9.

Not sure (i'm new in kubernetes) but I think that sh is defined in
stellar-helm-charts/stellar-core/requirements.yaml which installing postgresql

Any idea how to solve it?

Hello

I'm trying to install these charts. I'm following the instructions on the readme.

I do the following:

1. helm repo update
2. helm dependency update stellar-core

At this point it fails with:

Error: could not find stellar-core: stat stellar-core: no such file or directory

Currently, migrations are run as a job in the Stellar Horizon chart, see

@todkap Do you think an init container or a job is better suited for this?

Cloning this repository and running helm install/update with the local files is fine for testing – for making people's lives easier we should host a helm repository.

Note: I tried publishing the charts in https://github.com/helm/charts/tree/master/stable/ (and you can actually find an outdated core there) but the response time is terrible and I'd rather move fast with a self-hosted repository.

Currently, initializeDatabase passes down INITIALIZE_DATABASE=true to the Stellar Core container. Instead, we should use an init container for this operation instead of using the Stellar Core container's entrypoint.

Note: I think we should make sure that we don't overwrite an existing database, so the logic from https://github.com/satoshipay/docker-stellar-core/blob/8dac8b3df22f1e1ec92cde6f09caf3b54199b8c6/entry.sh#L5-L25 should be copied.

Proposed Table of Contents

• Background of Stellar @andrenarchy
• Existing ways to deploy Stellar ( Docker Compose and Docker All-in-One Image) - me
• Setting up Helm - me
• Creation of a Service Account - me
• Pointer to GitHub README for Cloud Provider specific set up @andrenarchy (need to add me as contributor)
  • IBM Block Storage will be covered there.
• Generating nodeSeed - me
• Helm install of stellar core (walk through of key overrides) @andrenarchy
• Verifying stellar core has achieved consensus @andrenarchy
• Installing Horizon Helm Chart (walk through of key overrides) @andrenarchy
• Validating Horizon Core is ready to handle requests @andrenarchy
• Demonstrate how to access Horizon APIs via Ingress -me
• Simple test with Friendbot to demonstrate Account Creation works as expected (K8 nodes are synching properly with Testnet and we can see the newly created account) -me

Running

 --namespace stellar-testnet \
 --name stellar-core \
 --set nodeSeed=$NODE_SEED \
 --values stellar-core.testnet.values.yaml \
  stellar-core

it fails with:

Error: validation failed: unable to recognize "": no matches for kind "StatefulSet" in version "apps/v1beta2" 

Running kubectl version:

Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.3", GitCommit:"2e7996e3e2712684bc73f0dec0200d64eec7fe40", GitTreeState:"clean", BuildDate:"2020-05-20T12:43:34Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

Helm version:

Server: &version.Version{SemVer:"v2.16.7", GitCommit:"5f2584fd3d35552c4af26036f0c464191287986b", GitTreeState:"clean"}

minikube version:
minikube version: v1.11.0

For speeding up initialization of a new Stellar Core and Stellar Horizon, we should support creating PVs from snapshots, see https://kubernetes.io/blog/2018/10/09/introducing-volume-snapshot-alpha-for-kubernetes/. This applies to the Core database, Core itself, and the Horizon database.

Currently, initializeHistoryArchives passes down INITIALIZE_HISTORY_ARCHIVES=true to the Stellar Core container. Instead, we should use an init container for this operation instead of using the Stellar Core container's entrypoint.

Note: I think we should make sure that we don't overwrite an existing history archive accidentally, so the logic from https://github.com/satoshipay/docker-stellar-core/blob/8dac8b3df22f1e1ec92cde6f09caf3b54199b8c6/entry.sh#L27-L49 should be copied.

We found in our testing that attempting to start both helm charts (core and horizon) at the same time causes errors such as gaps in the ledger.

ime="2019-05-28T13:37:15.695Z" level=info msg="Ingesting ledgers..." first_ledger=556188 last_ledger=556287 pid=1 service=ingest
time="2019-05-28T13:37:15.699Z" level=error msg="Error ingesting ledgers" err="Gap detected in stellar-core database (ledger=556188). More information: https://www.stellar.org/developers/software/known-issues.html#gaps-detected" first_ledger=556188 last_ledger=556287 pid=1 service=ingest
time="2019-05-28T13:37:16.623Z" level=info msg="Ingesting ledgers..." first_ledger=556188 last_ledger=556293 pid=1 service=ingest
time="2019-05-28T13:37:16.625Z" level=error msg="Error ingesting ledgers" err="Gap detected in stellar-core database (ledger=556188). More information: https://www.stellar.org/developers/software/known-issues.html#gaps-detected" first_ledger=556188 last_ledger=556293 pid=1 service=ingest

Delaying the start of horizon until core has passed the readiness probes seems to resolve the issue.

Install script that handles the check for readiness to be complete

nodeSeed=SCXZBWO7UYZ3TLJFLQG54MICBKRIODW7FV673B4AQINU3VLXOLRISHN7
namespace=stellar-testnet
kubectl config set-context $(kubectl config current-context) --namespace=$namespace

stellarCore=stellar-core
# Delete stellar core and reinstall
helm delete $stellarCore --purge
helm install --namespace $namespace --name $stellarCore --set nodeSeed=$nodeSeed --values stellar-core.testnet.values.yaml stellar-core

echo "Waiting for $stellarCore to be running"
## Verify everything is running before taking next step
statusCheck="NOT_STARTED"
while [ "$statusCheck" != "" ] ; do
	sleep 60
	statusCheck=$(kubectl get pods --namespace $namespace -o json | jq '.items[].status.phase' | grep -v "Running")
	echo "Still starting $stellarCore pods $(date)"
done

stellarHorizon=stellar-horizon
helm repo update
helm dependency update stellar-horizon
helm delete $stellarHorizon --purge
helm install \
  --namespace $namespace \
  --name $stellarHorizon \
  --set ingress.enabled=true \
  --set ingress.hosts[0].name="stellar-satoshipay.us-south.containers.appdomain.cloud" \
  --set ingress.hosts[0].tlsSecret="stellar-satoshipay" \
  --set service.port=8000 \
  --values stellar-horizon.testnet.values.yaml \
  stellar-horizon

Check sync state in Core readiness probe.

For publishing history archives, it would be nice if the chart/images also supports the IBM cloud.

To do: Check whether this is already supported by the aws flavor of the Stellar Core image (via S3 API in IBM's object storage).

Having stellar-core running and validated, when trying to install stellar-horizon get this error:

Initializing Horizon database...
2020/06/09 03:16:55 No migrations applied.
2020/06/09 03:16:55 --history-archive-urls must be set when --ingest is set

Kubernetes version: 1.15.11

NAME                               READY   STATUS             RESTARTS   AGE
stellar-core-799666f9ff-t6d5q      1/1     Running            4          19h
stellar-core-postgresql-0          1/1     Running            0          19h
stellar-horizon-585d6667b9-sf8l5   0/1     CrashLoopBackOff   11         34m
stellar-horizon-postgresql-0       1/1     Running            0          34m

The /info endpoint of Stellar Core's HTTP API is safe to expose and provides meaningful information, e.g., for monitoring. We should expose this endpoint but we need to make sure that no other paths can be reached. This means we need to verify that

1. Kubernetes does not allow access to another path (in any ingress controller)
2. Core does not normalize relative URLs

Hello

Do you have - and are you able to share - the pubnet version of stellar-core.testnet.values.yaml?

Besides the nodeseed is there anything that is sensitive in that file?