This BOSH release and deployment manifest deploy a cluster of kafka/kafka manager/zookeeper, plus a Cloud Foundry service broker

License: Apache License 2.0

Shell 100.00%

kafka kafka-manager cloudfoundry cloud-foundry bosh-release bosh

kafka-service-broker-boshrelease's Introduction

BOSH release for kafka-service-broker

This BOSH release and deployment manifest deploy a cluster of kafka/kafka manager/zookeeper, plus a Cloud Foundry service broker.

Install

export BOSH_ENVIRONMENT=<bosh-alias>
export BOSH_DEPLOYMENT=kafka-service-broker

# name of cf deployment in BOSH, e.g. 'cf'
cf=cf
system_domain=$(bosh -d $cf manifest | bosh int - --path /instance_groups/name=api/jobs/name=cloud_controller_ng/properties/system_domain)
skip_verify=$(bosh -d $cf manifest | bosh int - --path /instance_groups/name=api/jobs/name=cloud_controller_ng/properties/ssl/skip_cert_verify)

cf_admin_password=$(credhub get -n $(credhub find -n cf/cf_admin_password -j | jq -r ".credentials[0].name") -j | jq -r ".value")

bosh deploy manifests/kafka-service-broker.yml \
  --vars-store tmp/creds.yml \
  -o manifests/operators/cf-integration.yml \
  -v cf-api-url=https://api.$system_domain \
  -v cf-skip-ssl-validation=$skip_verify \
  -v cf-admin-username=admin \
  -v "cf-admin-password=$cf_admin_password"

bosh run-errand broker-registrar

The cf-* variables are used to register the broker as a service broker on Cloud Foundry.

Uninstall

To uninstall the service broker and kafka/zookeeper clusters=... \

export BOSH_ENVIRONMENT=<bosh-alias>
export BOSH_DEPLOYMENT=kafka-service-broker
bosh run-errand broker-deregistrar
bosh delete-deployment

kafka-service-broker-boshrelease's People

Contributors

Stargazers

Watchers

Forkers

poblin-orange 95rade myminseok bert-r

kafka-service-broker-boshrelease's Issues

Stemcell with Operating System 'ubuntu-trusty' doesn't exist

Unable to deploy on cf dev, with trusty going out of support imo xenial should be the default.

Task 833 | 13:24:39 | Preparing deployment: Preparing deployment (00:00:00) L Error: Stemcell with Operating System 'ubuntu-trusty' doesn't exist

broker-deregistrar errand failed

Task 1568 | 02:36:09 | Fetching logs for broker-deregistrar/2e7d01ac-aaa6-4e6e-a931-d83ed87b21ba (0): Finding and packing log files (00:00:01)

Task 1568 Started  Sat Dec  9 02:34:29 UTC 2017
Task 1568 Finished Sat Dec  9 02:36:10 UTC 2017
Task 1568 Duration 00:01:41
Task 1568 done

Instance   broker-deregistrar/2e7d01ac-aaa6-4e6e-a931-d83ed87b21ba
Exit Code  1
Stdout     CF_API_URL=https://api.18-220-96-60.sslip.io
           CF_SKIP_SSL_VALIDATION=false
           CF_ADMIN_USERNAME=admin
           BROKER_NAME=starkandwayne-kafka
           Setting api endpoint to https://api.18-220-96-60.sslip.io...
           OK

           api endpoint:   https://api.18-220-96-60.sslip.io
           api version:    2.99.0
           Not logged in. Use 'cf login' to log in.
           API endpoint: https://api.18-220-96-60.sslip.io
           Authenticating...
           OK
           Use 'cf target' to view or set your target org and space.
           /var/vcap/jobs/broker-deregistrar/bin/run: line 62: SERVICE_NAMES[@]: unbound variable

publish on bosh.io

publishing this bosh release on bosh.io would ease concourse integration

Any screenshots for installing Kafka?

Hi
I knew that you have mentioned that installation steps in Readme, however Im unable to understand how to do this.

What Im doing:

I opened terminal on my mac machine and cloned the github repo.
Trying to execute the installation steps on terminal (attached screenshot)

Im getting errors in the 1st command itself. So, it would be good if you provide any screenshots for the installation steps.

Thanks, Manju

fix CI to test broker-deploy

https://ci.starkandwayne.com/teams/main/pipelines/kafka-service-broker-boshrelease/jobs/testflight/builds/61

Task 445 | 09:42:09 | Preparing deployment: Preparing deployment
Task 445 | 09:42:10 | Preparing package compilation: Finding packages to compile (00:00:00)
Task 445 | 09:42:10 | Preparing deployment: Preparing deployment (00:00:01)
Task 445 | 09:42:10 | Creating missing vms: broker-deploy/6e53f06b-4c13-4e25-b943-036582180ce8 (0) (00:00:08)
Task 445 | 09:42:18 | Updating instance broker-deploy: broker-deploy/6e53f06b-4c13-4e25-b943-036582180ce8 (0) (00:00:22)
Task 445 | 09:42:40 | Running errand: broker-deploy/0 (00:00:06)
Task 445 | 09:42:46 | Fetching logs for broker-deploy/6e53f06b-4c13-4e25-b943-036582180ce8 (0): Finding and packing log files (00:00:01)

Task 445 Started  Thu Aug 17 09:42:09 UTC 2017
Task 445 Finished Thu Aug 17 09:42:47 UTC 2017
Task 445 Duration 00:00:38
Task 445 done

Exit Code  1                                                                                                                                                                
Stdout     Setting api endpoint to 10-58-111-49.sslip.io...                                                                                                      
           Request error: Get https://10-58-111-49.sslip.io/v2/info: dial tcp 10.58.111.49:443: i/o timeout                                                                 
           TIP: If you are behind a firewall and require an HTTP proxy, verify the https_proxy environment variable is correctly set. Else, check your network connection.  
           FAILED                                                                                                                                                
                                                                                                                                                                            
Stderr     -

Bump memory size for Kafka-Service-Broker app to 1GB

The current version is failing to start because of the 512MB limit

Cannot calculate JVM memory configuration: There is insufficient memory remaining for heap. Memory limit 512M is less than allocated memory 711821K

Full error:

2017-10-13T16:39:40.78-0700 [STG/0]      OUT Downloading app package...
2017-10-13T16:39:42.37-0700 [STG/0]      OUT Downloaded app package (37.6M)
2017-10-13T16:39:45.67-0700 [STG/0]      OUT -----> Java Buildpack v4.5 (offline) | https://github.com/cloudfoundry/java-buildpack.git#ffeefb9
2017-10-13T16:39:45.83-0700 [STG/0]      OUT -----> Downloading Jvmkill Agent 1.10.0_RELEASE from https://java-buildpack.cloudfoundry.org/jvmkill/trusty/x86_64/jvmkill-1.10.0_RELEASE.so (found in cache)
2017-10-13T16:39:45.84-0700 [STG/0]      OUT -----> Downloading Open Jdk JRE 1.8.0_141 from https://java-buildpack.cloudfoundry.org/openjdk/trusty/x86_64/openjdk-1.8.0_141.tar.gz (found in cache)
2017-10-13T16:39:46.96-0700 [STG/0]      OUT        Expanding Open Jdk JRE to .java-buildpack/open_jdk_jre (1.1s)
2017-10-13T16:39:46.96-0700 [STG/0]      OUT -----> Downloading Open JDK Like Memory Calculator 3.9.0_RELEASE from https://java-buildpack.cloudfoundry.org/memory-calculator/trusty/x86_64/memory-calculator-3.9.0_RELEASE.tar.gz (found in cache)
2017-10-13T16:39:47.83-0700 [STG/0]      OUT        Loaded Classes: 19319, Threads: 300
2017-10-13T16:39:47.85-0700 [STG/0]      OUT -----> Downloading Client Certificate Mapper 1.2.0_RELEASE from https://java-buildpack.cloudfoundry.org/client-certificate-mapper/client-certificate-mapper-1.2.0_RELEASE.jar (found in cache)
2017-10-13T16:39:47.85-0700 [STG/0]      OUT -----> Downloading Container Security Provider 1.8.0_RELEASE from https://java-buildpack.cloudfoundry.org/container-security-provider/container-security-provider-1.8.0_RELEASE.jar (found in cache)
2017-10-13T16:39:47.85-0700 [STG/0]      OUT -----> Downloading Spring Auto Reconfiguration 1.12.0_RELEASE from https://java-buildpack.cloudfoundry.org/auto-reconfiguration/auto-reconfiguration-1.12.0_RELEASE.jar (found in cache)
2017-10-13T16:39:58.17-0700 [STG/0]      OUT Exit status 0
2017-10-13T16:39:58.17-0700 [STG/0]      OUT Uploading droplet, build artifacts cache...
2017-10-13T16:39:58.17-0700 [STG/0]      OUT Uploading build artifacts cache...
2017-10-13T16:39:58.17-0700 [STG/0]      OUT Uploading droplet...
2017-10-13T16:39:58.22-0700 [STG/0]      OUT Uploaded build artifacts cache (131B)
2017-10-13T16:39:59.37-0700 [API/0]      OUT Creating droplet for app with guid 7aeacc5d-416c-4762-a974-e707d63b13dc
2017-10-13T16:40:01.66-0700 [STG/0]      OUT Uploaded droplet (83.9M)
2017-10-13T16:40:01.67-0700 [STG/0]      OUT Uploading complete
2017-10-13T16:40:01.69-0700 [STG/0]      OUT Stopping instance 3e669aa6-3fed-4a72-ad59-151de3639d7b
2017-10-13T16:40:01.69-0700 [STG/0]      OUT Destroying container
2017-10-13T16:40:01.91-0700 [CELL/0]     OUT Creating container
2017-10-13T16:40:02.04-0700 [STG/0]      OUT Successfully destroyed container
2017-10-13T16:40:02.28-0700 [CELL/0]     OUT Successfully created container
2017-10-13T16:40:05.48-0700 [CELL/0]     OUT Starting health monitoring of container
2017-10-13T16:40:05.65-0700 [APP/PROC/WEB/0]ERR Cannot calculate JVM memory configuration: There is insufficient memory remaining for heap. Memory limit 512M is less than allocated memory 711821K (-XX:ReservedCodeCacheSize=240M, -XX:MaxDirectMemorySize=10M, -XX:MaxMetaspaceSize=127117K, -XX:CompressedClassSpaceSize=21M, -Xss1M * 300 threads)
2017-10-13T16:40:05.66-0700 [APP/PROC/WEB/0]OUT Exit status 1
2017-10-13T16:40:05.66-0700 [CELL/0]     OUT Exit status 0
2017-10-13T16:40:05.69-0700 [CELL/0]     OUT Stopping instance 27fdeda6-d260-460f-7263-c20b
2017-10-13T16:40:05.69-0700 [CELL/0]     OUT Destroying container
2017-10-13T16:40:05.71-0700 [API/0]      OUT Process has crashed with type: "web"

Increasing the memory limit to 1GB allows broker to start.

property to configure broker domain

the deploy errand should enable operator to specify broker domain (can set app name, org, missing domain)

btw: very nice work, glad to get cloudfoundry community Kafka solution :-)

The installation of the release was well done:

The kafka-service-broker is avaible on the CF marketplace/services :

As you can see, there are two apps bounded to the kafka broker, and the cf env draw show that kafka VCAP_SERVICES are well mentioned :

however, when I generate application logs, no message is sent to the topic. Can you help me with this please?

Thank you

Errand broker-registrar fails because it can't access system domain API endpoint (connect timeout)

Below is the output from the errand log. Should I setup something to enable access from the kafka-service-broker VM to the system domain API endpoint? I can access the system API endpoint from other deployments but from this particular one it appears to be blocked.

Thanks.

ubuntu@ops-manager-vm:~/kafka-service-broker-boshrelease$ bosh run-errand broker-registrar
Using environment '10.0.0.10' as client 'ops_manager'
Using deployment 'kafka-service-broker'
Task 386
Task 386 | 18:29:31 | Preparing deployment: Preparing deployment
Task 386 | 18:29:32 | Warning: Ambiguous request: the requested errand name 'broker-registrar' matches both a job name and an errand instance group name. Executing errand on all relevant instances with job 'broker
-registrar'.
Task 386 | 18:29:33 | Preparing package compilation: Finding packages to compile (00:00:00)
Task 386 | 18:29:33 | Preparing deployment: Preparing deployment (00:00:02)
Task 386 | 18:29:33 | Creating missing vms: broker-registrar/afa3b5a8-6374-4144-945f-ecd2e5e9eb8b (0) (00:01:16)
Task 386 | 18:30:49 | Updating instance broker-registrar: broker-registrar/afa3b5a8-6374-4144-945f-ecd2e5e9eb8b (0) (00:00:13)
Task 386 | 18:31:02 | Running errand: broker-registrar/afa3b5a8-6374-4144-945f-ecd2e5e9eb8b (0) (00:00:15)
Task 386 | 18:31:17 | Fetching logs for broker-registrar/afa3b5a8-6374-4144-945f-ecd2e5e9eb8b (0): Finding and packing log files (00:00:01)
Task 386 Started  Sun May 26 18:29:31 UTC 2019
Task 386 Finished Sun May 26 18:31:18 UTC 2019
Task 386 Duration 00:01:47
Task 386 done
Instance   broker-registrar/afa3b5a8-6374-4144-945f-ecd2e5e9eb8b  
Exit Code  1  
Stdout     CF_API_URL=https://api.sys.pcf.pvtl.dev  
           CF_SKIP_SSL_VALIDATION=false  
           CF_ADMIN_USERNAME=admin  
           BROKER_NAME=starkandwayne-kafka  
           BROKER_URL=http://10.0.8.18:8080  
           BROKER_USERNAME=broker  
           Setting api endpoint to https://api.sys.pcf.pvtl.dev...  
           Request error: Get https://api.sys.pcf.pvtl.dev/v2/info: dial tcp 35.244.254.11:443: i/o timeout  
           TIP: If you are behind a firewall and require an HTTP proxy, verify the https_proxy environment variable is correctly set. Else, check your network connection.  
           FAILED  
             
Stderr     -  
1 errand(s)
Errand 'broker-registrar' completed with error (exit code 1)
Exit code 1

cloudfoundry-community / kafka-service-broker-boshrelease Goto Github PK