Observation. When creating default.conf file via the consul template an upstream is cr

I dropped that shell in a gist: <a href="https://gist.github.com/tgross/2c98898

I have cleaned up my . <a href="https://gist.github.com/donniev/4a006781af544fdd

Ok here is config <div class="snippet-clipboard-content notranslate position-relat

<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/us

Example Nginx config grabs all services,about tritondatacenter/containerpilot

Comments (22)

tgross commented on May 20, 2024

I dropped that shell script in a gist: https://gist.github.com/tgross/2c98898ff5188d2986af so that people who are reading the thread don't have to download locally.

This approach ties us to using links, which I feel like get a little confusing when it comes to having multiple nodes for a single backend application. But I definitely see your point here. There a couple of approaches we could use here:

Similar to yours in spirit would be what I did in another example. The upstreams are wrapped in a {{ if service "appname" }} block so that only the services we want show up.
We could check that {{.Port}} is non-zero.
We could add tags to the Containerbuddy configuration of the upstream services and make sure they get queried by our consul-template.

As you say, it doesn't affect the example programs, but it would probably be a good idea for us to include one of these approaches in the example programs so that it's clear how a user might try it. I'll implement this.

from containerpilot.

donniev commented on May 20, 2024

I have cleaned up my script. https://gist.github.com/donniev/4a006781af544fdda1a5 The original does not do the job completely as it failed to wrap some sections.

from containerpilot.

donniev commented on May 20, 2024

My main goal is that this can be automated so you don't have to create a new template file manually as you add services. Checking for the port will not help if you have a service (like redis) which exposes a port but still should not be proxied by nginx. I like the idea of using tags so that everything is visible to consul.

from containerpilot.

tgross commented on May 20, 2024

My main goal is that this can be automated so you don't have to create a new template file manually as you add services.

Doesn't that really only work if all your location blocks are identical? For this little example sure, but in real production environments I've always ended up having microservice-specific tweaks to the location blocks.

from containerpilot.

donniev commented on May 20, 2024

My generator grabs domain information for each service from an Manta object and so far I have been able to add new services by just updating the Manta object when I add a new service. I would rather have that info in a tag in service but whenever I query consul for tags I get an empty array. It seems like cb doesn’t register those tags with consul or else I am doing something wrong.

On Feb 18, 2016, at 7:37 AM, Tim Gross [email protected] wrote:

My main goal is that this can be automated so you don't have to create a new template file manually as you add services.

Doesn't that really only work if all your location blocks are identical? For this little example sure, but in real production environments I've always ended up having microservice-specific tweaks to the location blocks.

—
Reply to this email directly or view it on GitHub #80 (comment).

from containerpilot.

tgross commented on May 20, 2024

It seems like cb doesn’t register those tags with consul or else I am doing something wrong.

@donniev sorry I let this issue sit for a couple days. If the tags aren't being registered (or queried properly) then that's definitely a bug. Is it possible for you to share the Containerbuddy configuration you're using to write the tags?

from containerpilot.

donniev commented on May 20, 2024

It was a while ago when I tried it. I will see if I can reproduce it.

On Feb 24, 2016, at 6:39 AM, Tim Gross [email protected] wrote:

It seems like cb doesn’t register those tags with consul or else I am doing something wrong.

@donniev https://github.com/donniev sorry I let this issue sit for a couple days. If the tags aren't being registered (or queried properly) then that's definitely a bug. Is it possible for you to share the Containerbuddy configuration you're using to write the tags?

—
Reply to this email directly or view it on GitHub #80 (comment).

from containerpilot.

donniev commented on May 20, 2024

Ok here is config

{
  "consul": "consul:8500",
  "onStart": "/opt/containerbuddy/reload-ninja.sh",
  "services": [
    {
      "name": "ninja",
      "port": 3500,
      "tags":["joyent.lendinclubninja.com"],
      "health": "/usr/bin/curl --fail -s  -o /dev/null http://localhost:3500/",
      "poll": 10,
      "ttl": 25
    }
  ],
  "backends": [
  ]
}

If I then open Consul the service is up and running and reports "No tags"
I also have this in nginx template

 ## Tags from Consul ## {{range .Tags }} ##{{.}} ##{{end}}

Which renders as

 ## Tags from Consul ##

from containerpilot.

tgross commented on May 20, 2024

@donniev I tested this locally and was able to see tags being added. The example application actually uses a tag ("application" for the app service). Can you verify what version of Containerbuddy you're using?

from containerpilot.

donniev commented on May 20, 2024

The cb executable I have been using I pulled from containerbuddy-0.0.1-alpha.tar.gz which is clearly out of date. What is process for building executeable for current version that I can include in my images.

from containerpilot.

tgross commented on May 20, 2024

Yeah, that version didn't support tags yet (they were added in 0.1.0). For getting a new release you've got two options:

You can pull the latest version (1.0 was released today!) directly from GitHub https://github.com/joyent/containerbuddy/releases/latest.
You can checkout this repo, make sure your Docker client is pointed to a local Docker engine and run make, which will create a build container and output the binary in the build/ directory.

from containerpilot.

donniev commented on May 20, 2024

getting the following error:

Status: Downloaded newer image for quay.io/coreos/etcd:v2.0.8
542e81f70b118465fc8b875963645948a9ef99b60af7477fd128cc3c6b310908
docker run --rm --link containerbuddy_consul:consul --link containerbuddy_etcd:etcd -v /Volumes/DataDrive/containerbuddy/vendor:/go/src -v /Volumes/DataDrive/containerbuddy:/go/src/github.com/joyent/containerbuddy -v /Volumes/DataDrive/containerbuddy/build:/build -v /Volumes/DataDrive/containerbuddy/cover:/cover -v /Volumes/DataDrive/containerbuddy/examples:/root/examples:ro -v /Volumes/DataDrive/containerbuddy/Makefile.docker:/go/makefile:ro -e LDFLAGS='-X containerbuddy.GitHash=afd088a -X containerbuddy.Version=dev-build-not-for-release' containerbuddy_build build
make: *** makefile: Is a directory.  Stop.
make: *** [build] Error 2

which makes no sense to me because it is a file not a dir

from containerpilot.

donniev commented on May 20, 2024

pulling latest directly from gihub solves the build issue but now I have a new problem. None of my services are registering with consul. (except nginx). I have made no changes to their config json files and the only change I have made to their docker images is to use the new cb executable. No clue at this point why they are no longer registering. Registered fine in previous version of cb.

from containerpilot.

tgross commented on May 20, 2024

getting the following error:
which makes no sense to me because it is a file not a dir

You may have build artifacts that are controlled by .gitignore in your local working directory after having built older versions. Might want to try make clean build.

No clue at this point why they are no longer registering. Registered fine in previous version of cb.

Hard to debug that from here... can you provide logs from those containers, their configuration files, and/or the docker-compose file you're using to run them?

from containerpilot.

donniev commented on May 20, 2024

Ok. I have tried to simplify by running the additional service by just adding to the examples docker-compose.yml

ninja:
    image: quay.io/donniev/projects:lendingclubninja.tests
    mem_limit: 128m
    ports:
    - 3500
    links:
    - consul:consul
    restart: always
    environment:
    - AWS_SECRET_ACCESS_KEY
    - AWS_REGION
    - AWS_ACCESS_KEY_ID
    - CONTAINERBUDDY=file:///opt/containerbuddy/ninja.json
    command: >
      /opt/containerbuddy/containerbuddy
      /usr/local/bin/node /application/bin/www.js

ninja.json

{
  "consul": "consul:8500",
  "onStart": "/opt/containerbuddy/reload-ninja.sh",
  "services": [
    {
      "name": "ninja",
      "port": 3500,
      "health": "/usr/bin/curl --fail -s  -o /dev/null http://localhost:3500/",
      "poll": 10,
      "ttl": 25
    }
  ],
  "backends": [

  ]
}

reload-ninja.sh

#!/bin/bash

# get all the health application servers and write the json to file

/usr/bin/curl -s consul:8500/v1/health/service/ninja?passing | /usr/local/bin/json > /tmp/lastQuery.json
/usr/local/bin/node /application/bin/www.js

TheBeast:consul donvawter$ docker logs examples_ninja_1
Thu, 25 Feb 2016 14:14:10 GMT express-session deprecated undefined resave option; provide resave option at app.js:50:9
Thu, 25 Feb 2016 14:14:10 GMT express-session deprecated undefined saveUninitialized option; provide saveUninitialized option at app.js:50:9
{"name":"LendingClub","hostname":"4c9eddd37438","pid":10193,"level":30,"msg":"Loggers initialized","time":"2016-02-25T14:14:22.190Z","v":0}
{ address: '::', family: 'IPv6', port: 3500 }

TheBeast:consul donvawter$ docker inspect examples_ninja_1
[
    {
        "Id": "4c9eddd374384cc98e0ec5fb8bc6ea9f3d82b9ae1941437b92a0f66d82ba5099",
        "Created": "2016-02-25T14:13:39.967Z",
        "Config": {
            "AttachStderr": false,
            "AttachStdin": false,
            "AttachStdout": false,
            "CpuShares": 8,
            "Cpuset": "",
            "Domainname": "local",
            "ExposedPorts": {
                "3500/tcp": {}
            },
            "Hostname": "4c9eddd37438",
            "MacAddress": "",
            "Memory": 134217728,
            "MemorySwap": 536870912,
            "NetworkDisabled": false,
            "OnBuild": null,
            "OpenStdin": false,
            "PortSpecs": null,
            "SecurityOpt": null,
            "StdinOnce": false,
            "Tty": false,
            "User": "",
            "Volumes": null,
            "WorkingDir": "/application",
            "Image": "quay.io/donniev/projects:lendingclubninja.tests",
            "Cmd": [
                "/opt/containerbuddy/containerbuddy",
                "/usr/local/bin/node",
                "/application/bin/www.js"
            ],
            "Entrypoint": null,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "NPM_CONFIG_LOGLEVEL=info",
                "NODE_VERSION=4.3.1",
                "BUCKET=dvawterconfig",
                "CREDSFILE=lcconfig/creds.json",
                "APPCONFIGFILE=lcconfig/app_config.json",
                "COMMONCONFIGFILE=lcconfig/common_config.json",
                "CONTAINERBUDDY=file:///opt/containerbuddy/ninja.json",
                "AWS_REGION=us-east-1",
                "AWS_ACCESS_KEY_ID=AKIAJCEIUUTOZQURVVPA",
                "AWS_SECRET_ACCESS_KEY=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"
            ],
            "Labels": {
                "com.docker.compose.service": "ninja",
                "com.docker.compose.project": "examples",
                "com.docker.compose.config-hash": "7913f4db1de3b1db2bf5e174fe68026cea9f47b91dac597e0df0b9f90628e26f",
                "com.docker.compose.version": "1.6.0",
                "com.docker.compose.oneoff": "False",
                "com.docker.compose.container-number": "1",
                "com.joyent.package": "t4-standard-128M"
            }
        },
        "Path": "/opt/containerbuddy/containerbuddy",
        "Args": [
            "/usr/local/bin/node",
            "/application/bin/www.js"
        ],
        "Driver": "sdc",
        "ExecDriver": "sdc-0.1",
        "HostConfig": {
            "Binds": null,
            "CapAdd": null,
            "CapDrop": null,
            "ContainerIDFile": "",
            "Devices": [],
            "Dns": [
                "8.8.8.8",
                "8.8.4.4"
            ],
            "DnsSearch": null,
            "ExtraHosts": null,
            "IpcMode": "",
            "Links": [
                "/examples_consul_1:/examples_ninja_1/consul",
                "/examples_consul_1:/examples_ninja_1/examples_consul_1",
                "/examples_consul_1:/examples_ninja_1/consul_1"
            ],
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "LxcConf": [],
            "NetworkMode": "bridge",
            "PortBindings": {
                "3500/tcp": [
                    {
                        "HostIp": "",
                        "HostPort": "3500"
                    }
                ]
            },
            "Privileged": false,
            "PublishAllPorts": false,
            "RestartPolicy": {
                "MaximumRetryCount": 0,
                "Name": "always"
            },
            "VolumesFrom": null,
            "CgroupParent": "",
            "CpuShares": 8,
            "CpusetCpus": "",
            "Memory": 134217728,
            "MemorySwap": 536870912,
            "PidMode": "",
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "Ulimits": null
        },
        "Volumes": null,
        "VolumesRW": null,
        "RestartCount": 0,
        "HostnamePath": "/etc/hostname",
        "HostsPath": "/etc/hosts",
        "Image": "023542d04daa044288f33d7c9719f91188d0a85568b26d4b783b7defcc39f82d",
        "MountLabel": "",
        "Name": "/examples_ninja_1",
        "NetworkSettings": {
            "Bridge": "eth1",
            "Gateway": "165.225.136.1",
            "IPAddress": "165.225.139.122",
            "IPPrefixLen": 22,
            "MacAddress": "90:b8:d0:32:01:2e",
            "PortMapping": null,
            "Ports": {
                "3500/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "3500"
                    }
                ]
            }
        },
        "ProcessLabel": "",
        "ResolvConfPath": "/etc/resolv.conf",
        "State": {
            "Error": "",
            "ExitCode": 0,
            "FinishedAt": "0001-01-01T00:00:00Z",
            "OOMKilled": false,
            "Paused": false,
            "Pid": 9498,
            "Restarting": false,
            "Running": true,
            "StartedAt": "2016-02-25T14:14:03.000Z"
        },
        "ExecIDs": null
    }
]

curl 165.225.139.122:3500
<html lang="en" >
  <head>
    <meta charset="utf-8">
    <meta http-equiv="X-UA-Compatible" 
......

The app is up and running but does not register with consul

from containerpilot.

donniev commented on May 20, 2024

make clean build did not help so am still using direct download of executable

Digest: sha256:4f2cc8a5d9641e7faf54fcaf7227aab382c9f181ac3304d9540f72469af3fa9a
Status: Downloaded newer image for quay.io/coreos/etcd:v2.0.8
d35d7b2dd78281479a5a15b8d7113676d6cd49e1689c96140506a36484dd297a
docker run --rm --link containerbuddy_consul:consul --link containerbuddy_etcd:etcd -v /Volumes/DataDrive/containerbuddy/vendor:/go/src -v /Volumes/DataDrive/containerbuddy:/go/src/github.com/joyent/containerbuddy -v /Volumes/DataDrive/containerbuddy/build:/build -v /Volumes/DataDrive/containerbuddy/cover:/cover -v /Volumes/DataDrive/containerbuddy/examples:/root/examples:ro -v /Volumes/DataDrive/containerbuddy/Makefile.docker:/go/makefile:ro -e LDFLAGS='-X containerbuddy.GitHash=afd088a -X containerbuddy.Version=dev-build-not-for-release' containerbuddy_build build
make: *** makefile: Is a directory.  Stop.
make: *** [build] Error 2

from containerpilot.

tgross commented on May 20, 2024

Ok, so you have reload-ninja.sh running as your onStart handler. This means that before the ninja application is launched you're:

curling Consul for Ninja nodes (of which there will be none at this point), and then passing that null value as JSON to /tmp/lastQuery.json (so that file will just say null I think?)
and then you're running node /application/bin/www.js.

I don't know what www.js is, but if www.js is a server and doesn't exit, then your onStart handler will never return its exit code to Containerbuddy, and so Containerbuddy will be hanging and never starting the ninja app nor registering it with Consul. You could confirm this by running ps inside the container. If my hunch is right, you'll see Containerbuddy running and www.js running but not Ninja.

make: *** makefile: Is a directory. Stop.

I dug into this and you typically see this error from a Makefile syntax error. Is your make file modified in any way? (The reason I ask is because it Works On My Machine™)

from containerpilot.

donniev commented on May 20, 2024

Doh. Obviously a complete misunderstanding on my part of purpose of onStart. Working now. Will try the make again on a clean pull. I did not intentionally change the makefile and git didn't say I had modified it but to make sure I will pull fresh but I need to get these other things working first. Thanks for your help. and yes tags do now show up in template. I am planning to use those to supply server names in the nginx config. When I didn't have tags I had a node plugin which returned them but I would like to get rid of that dependency

from containerpilot.

donniev commented on May 20, 2024

Thanks for your help on the new version.
Now back to the original topic.
With the addition of tags I have reworked my generation of nginx.conf
There were three aspects I didn't like

It relied on the "links" in nginx to determine whether to generate an upstream which is not the purpose of links
It relied on an external program yaml2json
To generate the server names it relied on a node plugin. Again an external dependency

In the new approach I determine which services need an upstream by adding a comment line in docker-compose

## 'forupstream+"consul" "crypto" "hexo" "nginx" "ninja" "socketserver" "notify"+'"

and in maketemplate.sh I parse that

declare -a rArray=$(echo '('; cat  docker-compose.yml |grep 'forupstream' |sed 's/\([^+]*\)\+\(.*\)\+\(.*\)$/\2/'|sed 's/\\//g'; echo ')')

To generate the server block I now expect a service to have an array of tags containing the server names

{
    "consul"  : "consul:8500",
    "services": [
        {
            "name"  : "crypto",
            "port"  : 8081,
            "tags"  : [
                "joyent.dailycryptogram.com",
                "www.icryptogram.com",
                "www.dailycryptogram.com",
                "dailycryptogram.com",
                "secure.dailycryptogram.com",
                "icryptogram.com",
                "dailycrypto.com",
                "www.dailycrypto.com",
                "www.onlinecryptograms.com",
                "onlinecryptograms.com"
            ],
            "health": "/usr/bin/curl --fail -s  -o /dev/null http://localhost:8081/testhealth.html",
            "poll"  : 60,
            "ttl"   : 120
        }
    ],
    "backends": [
    ]
}

Since the tags are available in the consul template I need no external calls. If .Tags is empty I don't create the server block

{{if .Tags}}
 server {
 server_name {{range .Tags }} {{.}} {{end}} ;
 listen 80 ;
 location / {
 proxy_redirect off;
 proxy_http_version 1.1;
 proxy_set_header Upgrade $http_upgrade;
 proxy_set_header Connection "upgrade";
 proxy_set_header Host $host;
 proxy_set_header X-Forwarded-Host $host;
 proxy_set_header X-Forwarded-Server $host;
 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 proxy_set_header X-Real-IP $remote_addr;
 proxy_pass http://{{.Name}}/;
 }
 location = "" {
 return 302 /;
 }
 }
 {{end}}

I think this is a lot cleaner. I am sure that is not what tags were designed for but it is an easy way to get the information into the consul template. I suppose you could also create key-value pairs.

from containerpilot.

donniev commented on May 20, 2024

gist for new template https://gist.github.com/donniev/60cd7c97522c538f6d11

from containerpilot.

tgross commented on May 20, 2024

@donniev I think I get where you're going with this, but the example app isn't necessarily intended to be a universal applies-to-all-situations sort of thing. I wonder if it might make more sense to move this idea over to https://github.com/tgross/triton-nginx where we've built an Nginx configuration that is intended to be more universal?

from containerpilot.

tgross commented on May 20, 2024

I've opened autopilotpattern/nginx#4 to continue work on this issue. Please join me over there!

from containerpilot.

Example Nginx config grabs all services about containerpilot HOT 22 CLOSED

Comments (22)

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent