Additional info
Apologies in advance for being so verbose in my explanation. I appreciate your time to review this issue knowing that you must be very busy working on more critical issues.
Describe the bug
Unfortunately, I was unable to get the fullstack swarm to start as expected. (pingidentity-devops-getting-started/12-docker-swarm/fullstack.yaml).
Below is what I observed, and the steps I took to get the fullstack swarm to start as expected. I'm still not sure exactly why my solution worked, but it did, and I am stoked. :)
I discovered the following error in the service logs for the pingaccess, pingfederate-admin, and pingfederate swarm services (docker service logs -f ${service-name})
Example error from fullstack_pingfederate service logs:
fullstack_pingfederate.1.zfaznu425nhx@node1 | ----- Starting hook: ./bootstrap.sh
fullstack_pingfederate.1.zfaznu425nhx@node1 | ### Bootstrap
fullstack_pingfederate.1.zfaznu425nhx@node1 | ----- Starting hook: /opt/entrypoint.sh
fullstack_pingfederate.1.zfaznu425nhx@node1 | Command: sh -c wait-for pingfederate-admin:9999 -t 600 -- entrypoint.sh start-server
fullstack_pingfederate.1.zfaznu425nhx@node1 | ##################################################################################
fullstack_pingfederate.1.zfaznu425nhx@node1 | # Ping Identity DevOps Docker Image
fullstack_pingfederate.1.zfaznu425nhx@node1 | # IMAGE_VERSION: pingfederate-alpine-10.0.0-200126-9f12
fullstack_pingfederate.1.zfaznu425nhx@node1 | # IMAGE_GIT_REV: 9f120fd51f74475c6bf5d6f741a92981be78fc55
fullstack_pingfederate.1.zfaznu425nhx@node1 | # STARTED: Fri Jan 31 10:30:52 UTC 2020
fullstack_pingfederate.1.zfaznu425nhx@node1 | # HOSTNAME: e114761ebc8f
fullstack_pingfederate.1.zfaznu425nhx@node1 | # DOMAINNAME:
fullstack_pingfederate.1.zfaznu425nhx@node1 | ##################################################################################
fullstack_pingfederate.1.zfaznu425nhx@node1 | Running command: sh -c wait-for pingfederate-admin:9999 -t 600 -- entrypoint.sh start-server
fullstack_pingfederate.1.zfaznu425nhx@node1 | Error: you need to provide a host and port to test.
fullstack_pingfederate.1.zfaznu425nhx@node1 | Usage:
fullstack_pingfederate.1.zfaznu425nhx@node1 | host:port [-t timeout] [-- command args]
fullstack_pingfederate.1.zfaznu425nhx@node1 | -q | --quiet Do not output any status messages
fullstack_pingfederate.1.zfaznu425nhx@node1 | -t TIMEOUT | --timeout=timeout Timeout in seconds, zero for no timeout
fullstack_pingfederate.1.zfaznu425nhx@node1 | -- COMMAND ARGS Execute command with args after the test finishes
Troubleshooting
I ended-up commenting line 29 of 'fullstack.yaml' so that I could get the pingfederate-admin continer to simply start. I also added a published port of 22 so that I could "SSH" into the container and test the /opt/wait-for script, which worked as expected from the container interactive shell.
Next, I double-checked the Docker Compose documentation and tested using a different syntax for the service command override, which didn't fix the issue.
https://docs.docker.com/compose/compose-file/#command
command
Override the default command.
command: bundle exec thin -p 3000
The command can also be a list, in a manner similar to dockerfile:
command: ["bundle", "exec", "thin", "-p", "3000"]
Fix
Keeping the alternate command syntax, I next tried modifying the command to use the absolute path to the script, instead of invoking the script with 'sh -c'. With the below modifications to fullstack.yaml, I was able to successfully start the swarm. :)
line 7 original (generates similar error as above):
command: ["sh", "-c", "wait-for pingfederate:9031 -t 700 -- entrypoint.sh start-server"]
line 7 changed (works as expected):
command: /opt/wait-for pingfederate:9031 -t 700 -- entrypoint.sh start-server
line 29 original (generates similar error as above):
command: ["sh", "-c", "wait-for pingdirectory:389 -t 300 -- entrypoint.sh start-server"]
line 29 changed (works as expected):
command: /opt/wait-for pingdirectory:389 -t 300 -- entrypoint.sh start-server
line 48 original (generates similar error as above):
command: ["sh", "-c", "wait-for pingfederate-admin:9999 -t 600 -- entrypoint.sh start-server"]
line 48 changed (works as expected):
command: /opt/wait-for pingfederate-admin:9999 -t 600 -- entrypoint.sh start-server
Example success from fullstack_pingaccess service logs:
~/projects/pingidentity-devops-getting-started/12-docker-swarm$ docker service logs -f fullstack_pingaccess
fullstack_pingaccess.1.l8vcjwx57xb9@node1 | ----- Starting hook: ./bootstrap.sh
fullstack_pingaccess.1.l8vcjwx57xb9@node1 | ### Bootstrap
fullstack_pingaccess.1.l8vcjwx57xb9@node1 | ----- Starting hook: /opt/entrypoint.sh
fullstack_pingaccess.1.l8vcjwx57xb9@node1 | Command: /opt/wait-for pingfederate:9031 -t 700 -- entrypoint.sh start-server
fullstack_pingaccess.1.l8vcjwx57xb9@node1 | ##################################################################################
fullstack_pingaccess.1.l8vcjwx57xb9@node1 | # Ping Identity DevOps Docker Image
fullstack_pingaccess.1.l8vcjwx57xb9@node1 | # IMAGE_VERSION: pingaccess-alpine-6.0.0-200126-9f12
fullstack_pingaccess.1.l8vcjwx57xb9@node1 | # IMAGE_GIT_REV: 9f120fd51f74475c6bf5d6f741a92981be78fc55
fullstack_pingaccess.1.l8vcjwx57xb9@node1 | # STARTED: Fri Jan 31 12:06:22 UTC 2020
fullstack_pingaccess.1.l8vcjwx57xb9@node1 | # HOSTNAME: 66a8c9659a89
fullstack_pingaccess.1.l8vcjwx57xb9@node1 | # DOMAINNAME:
fullstack_pingaccess.1.l8vcjwx57xb9@node1 | ##################################################################################
fullstack_pingaccess.1.l8vcjwx57xb9@node1 | Running command: /opt/wait-for pingfederate:9031 -t 700 -- entrypoint.sh start-server
...
...
fullstack_pingaccess.1.pl2n1k4jphfw@node2 | 2020-01-31T12:15:05,953 INFO [] com.pingidentity.pa.cli.RouterCLI - PingAccess started in 30s:214ms
/projects/pingidentity-devops-getting-started/12-docker-swarm$ docker service ls
ID NAME MODE REPLICAS IMAGE PORTS
h4rb1wl1phrh fullstack_pingaccess replicated 1/1 pingidentity/pingaccess:edge *:443->443/tcp, *:9000->9000/tcp
7f7s7g4yt64v fullstack_pingdataconsole replicated 1/1 pingidentity/pingdataconsole:edge *:8443->8443/tcp
pskuz8cndc4z fullstack_pingdirectory replicated 1/1 pingidentity/pingdirectory:edge *:1443->443/tcp, *:1636->636/tcp
tdl528pb94i7 fullstack_pingfederate replicated 1/1 pingidentity/pingfederate:edge *:9031->9031/tcp
p41ew8jtwl2m fullstack_pingfederate-admin replicated 1/1 pingidentity/pingfederate:edge *:9999->9999/tcp
Environment:
- Github Repo: pingidentity-devops-getting-started/12-docker-swarm/fullstack.yaml
- Docker Container: n/a
- Cloud Environment: 3x CentOs 7.7.1908 Docker Swarm worker node VMs that I manage from my Linux Mint dev VM, which is the Docker Swarm manager.
Additional info
After discovering Ping Identity DevOps, I realized that I wanted/needed to learn Docker. I started learning Docker in December, 2019 by watching a really good Udemy course by Bret Fisher, a Docker Captain (https://www.bretfisher.com/). I am about 50% through the course, which last covered Docker Swarm Mode, and next up is Kubernetes (excited :) ).
I say all that so you will hopefully understand that I may not be using the most correct terms when I describe the issue I experienced, and also to say thank you for creating such awesome documentation! I simply cannot wait until we can hopefully start managing our PingFederate and PingAccess services using DevOps practices, instead of the very manual, error-prone method we use today to manage our considerably-sized clusters.
I've finally learned enough about Docker to start testing with Ping Identity products on my home lab where I have 3x CentOs 7.7.1908 Docker Swarm worker node VMs that I manage from my Linux Mint dev VM, which is the Docker Swarm manager.
Right now I am primarily focused on learning Swarm because, as it was interpreted by me from Bret Fisher's course, a small team that is new to DevOps using Docker will most likely be successful moving to Swarm first, since Swarm will cover most use-cases (80-20 rule), and then, things like Kubernetes, Docker Enterprise UCP, etc. can be added later, as the DevOps practices of the team matures.