blablacar / ggn Goto Github PK

Link container builds with envs to end up into low level orchestration

Go 89.49% Shell 10.51%

ggn's Introduction

GGN

GGN uses a tree structure to describe envs and services in envs. It will generate systemd units based on information taken in the directories and send them to the environment using fleet.

directory structure

env
|-- development
|   |-- attributes
|   |   `-- dns.yml                      # Attributes of this env (dns suffix, dns servers IPs, zookeeper IPs, ...)
|   |-- services                         # list of services in this env
|   |   |-- loadbalancer                 
|   |   |   |-- attributes               # loadbalancer attributes in this env
|   |   |   |   `-- nginx.yml            # any structure configuration
|   |   |   |-- unit.tmpl                # template uses to generate the systemd's units for loadbalancer
|   |   |   `-- service-manifest.yml     # manifest for this service
|   |   |-- cassandra
|   |   |   |-- attributes
|   |   |   |   |-- cassandra.yml        # cassandra configuration for this env (DC name, seeds nodes, cluster name)
|   |   |   |   `-- datastax-agent.yml   # another configuration file that will be merged with the other one
|   |   |   `-- service-manifest.yml    
|   |   ...
|   `-- config.yml                       # configuration of this env (fleet)
|-- prod-DC1
|   ...
|-- prod-DC2
|   ...
|-- preprod
|   ...

Attributes from envs will be merged with attributes from each service and be available in unit template as object tree and also as a json string to embbed it in the start command line.

configuration of where the unit will be started can be done in the attribute+template system to generate a [X-Fleet] unit part (example below).

commands

Some command example :

ggn dev redis srv1 start           start redis server1 unit
ggn preprod check                  check that all units of all services in prepod are running and are up to date
ggn prod cassandra cass1 journal   see journal of cass1 systemd unit 
ggn prod lb lb1 stop               stop lb1 unit in prod
ggn prod cassandra update          rolling update of cassandra servers in prod

Envs commands :

- check                            check that all units are up to date and running in this env
- fleetctl                         run custom fleetctl command
- list-units                       list all units in this env
- generate                         generate all units in this env

Services commands:

- generate                         generate units of this service
- check                            check that all units of this service are up to date and running
- diff                             display diff of units between when is generated and what is running in this env
- lock                             lock this service. nobody will be able to run mutable actions
- unlock                           unlock the service
- list-units                       list-units of this service in fleet
- update                           run a update of all units for this service

Units commands:

- start                            start the service
- stop                             stop the service
- update                           update the service
- restart                          restart the service
- destroy                          destroy the service
- status                           display fleet status of the unit
- journal                          display unit's journal
- diff                             display the diff between genetated and what is running
- check                            check that the service is running and is up to date
- unload                           unload the unit
- load                             load the unit
- ssh                              ssh on the server's running this unit

global configuration file

The configuration file is located at ~/.config/green-garden/config.yml

workPath: /home/myuser/build-tools           # root directory of environments. all envs have to be in an env/ directory in it

service manifest structure

concurrentUpdater: 2                            # concurrent run when updating the service
containers:
  - aci.example.com/pod-cassandra               # list of aci or pod

nodes:                                          # list of nodes for this service
  - hostname: cass1                             # hostname of the service
    ip: 10.2.135.136                            # any other property used in the template
    node-id: 1     # node special attribute
    fleet:
      - MachineMetadata="rack=113" "pos=4"

  - hostname: cass2
    ip: 10.2.143.136
    node-id: 2
    fleet:
      - MachineMetadata="rack=213" "pos=4"

example of unit template

[Unit]
Description=pod-cassandra {{.hostname}}
After=mnt-sda9-{{.hostname}}-mount-sdb1.mount \
      mnt-sda9-{{.hostname}}-mount-sdc1.mount \
      mnt-sda9-{{.hostname}}-mount-sdd1.mount \
      mnt-sda9-{{.hostname}}-mount-sde1.mount \

[Service]
{{.environmentAttributes}}
ExecStartPre=/opt/bin/rkt gc --grace-period=0s --expire-prepared=0s
ExecStartPre=-/opt/bin/rkt image gc
ExecStart=/opt/bin/rkt --insecure-skip-verify run \
    --private-net='bond0:IP={{.ip}}' \
    --volume=cassandra-mount-1,kind=host,source=/mnt/sda9/{{.hostname}}/mount/sdb1 \
    --volume=cassandra-mount-2,kind=host,source=/mnt/sda9/{{.hostname}}/mount/sdc1 \
    --volume=cassandra-mount-3,kind=host,source=/mnt/sda9/{{.hostname}}/mount/sdd1 \
    --volume=cassandra-mount-4,kind=host,source=/mnt/sda9/{{.hostname}}/mount/sde1 \
    --volume=cassandra-commitlog,kind=host,source=/mnt/sda9/{{.hostname}}/commitlog \
    --volume=cassandra-savedcaches,kind=host,source=/mnt/sda9/{{.hostname}}/saved_caches \
    --set-env=CONFD_OVERRIDE='{{.environmentAttributesVars}}' \
    --set-env=HOSTNAME={{.hostname}} \
    --set-env=DOMAINNAME="{{.domainname}}" \
    {{.acis}}

[X-Fleet]
{{range .fleet -}}
    {{- . }}
{{end -}}

ggn's People

Contributors

Stargazers

Watchers

Forkers

puckel ramitsurana zeysh nyodas prfalken julienbalestra obiesmans geckosplinter

ggn's Issues

Scheduled tasks deployment

ggn update should be able to deploy a .service unit without starting it

[Output] Update a whole service

When you update a whole service like this
ggn environment service update -y

The output is not does not have much information

09:26:14 INFO      ggn/work/service-update.go:38  Remote service is already up to date         env=prod-pa3 service=zabbix-core-agent
09:26:17 INFO      ggn/work/service-update.go:40  But service is not running                   env=prod-pa3 service=zabbix-core-agent
09:26:18 INFO      ggn/work/service-update.go:38  Remote service is already up to date         env=prod-pa3 service=zabbix-core-agent
09:26:18 INFO      ggn/work/service-update.go:77  Updating unit                                env=prod-pa3 service=zabbix-core-agent

Witch container was not running ?
Witch container is updated ?
In order to better investigate :)

Karles

Manage Arrays for attributes in service-manifest

The generate failed when we put an array in attributes, for instance:

nodes:
  - hostname: "test1"
    ip: "XXX.XXX.XXX.XXX"
    attributes:
      nerve:
        services:
          - svc_name: "name1"
            svc_port: 3333
            zk_path: "/services/test/name1"
            check_interval: 2
            weight: 200

The error is error=json: unsupported type: map[interface {}]interface {}

display full unit file in diff when remote unit not found

ggn diff should generate first

right now ggn diff crashes if you don't run generate first.

parallelize process of units on env check

ggn generate should accept a local pod manifest.

As for now I can't generate a local unit file if the pod has not been uploaded to the registry yet.
It would be good to be able to pass the locally generated manifest. That way we can test the pod before pushing.

Ensure service will globally work before stopping node ?

having multiple nodes on services is always a cluster. depending on the service, we may check that we will not stop the quorum of cluster nodes to crash the service.

We can add an info in the manifest to give the rules and check it's ok before stopping/destroying a node ??

add check state of units

check if attributes files end with .yml or .yaml ignore otherwise

ggn check should continue on discovery error

currently:
ggn $env check
stops when a pod is not available on remote repository. It should just report the error and continue.

service update, ask even if already up to date for timer

on timer service, when you do an update, ggn find that the service is not running and ask to update.
but it's normal that it's not running since it's trigger by timer.

add command to get unit logs

support external action when processing service / units

this will be usefull for slack notification, etc...

Exclude aci from service-manifest

Add ability to exclude ACIs in the service-manifest.yml.

For instance, pod-foobar contains aci-foo, aci-bar and aci-zabbix. I'd like to:

containers:
  - pod-foobar
exclude:
  - aci-zabbit # this will remove pod-foobar_aci-zabbit from the {{.acis}} variable

Unconfigured GGN should indicate how to setup configuration instead of cryptic error message

Running an unconfigured ggn gives:

10:42:50 ERROR      green-garden/work/work.go:29  env directory not found in

It should indicate instead how to configure it, ie:

> mkdir -p $HOME/.config/green-garden
> echo 'workPath: /path_to_repo/build-tools' > $HOME/.config/green-garden/config.yml

allow running update on unit

Does the lowest level of granularity of ggn fleet is the env? Can we manipule (update for instance) directly a single node?

Support checking service running after start

Store log copy

check command should probably verify repo being up to date

Problem with node and env attributes - leaf overrided

I created a attributes directory with some galera config on my environment:

vi ../env/prod/services/db-log/attributes/galera.cnf.yml


---
override:
  galera:
    mysqld:
      wsrep_provider_options:
        gcache.size: "1G"
        gmcast.segment: 1
        gcs.fc_limit: 500
        gcs.fc_master_slave: "YES"
        gcs.fc_factor: "1.0"

It is basically some "environment" attributes.

I added some "node attributes" in the manifest as:

vi ../env/prod/services/db-log/service-manifest.yml

...
nodes:
  - hostname: mydb5
    ip: XX.XX.XX.XX
    fleet:
      - MachineMetadata="rack=1" "pos=1"
    attributes:
      galera:
        mysqld:
          wsrep_cluster_address:"gcomm://mydb1.prod,mydb2.prod,mydb3.prod,mydb4prod"

My problem is that the mysqld section is overrided by the node attribute.

In the unit I have:

[...]"galera":{"mysqld":"wsrep_cluster_address:\\\"gcomm://mydb1.prod,mydb2.prod,mydb3.prod,mydb4prod\\\""}[...]

Instead of

[...]"galera":{"mysqld":"wsrep_cluster_address:"gcomm://mydb1.prod,mydb2.prod,mydb3.prod,mydb4prod","wsrep_provider_options":{"gcache.size":"1G","gcs.fc_factor":"1.0","gcs.fc_limit":500,"gcs.fc_master_slave":"YES","gmcast.segment":1}}[...]

Note that the attribute wsrep_cluster_address is part of the mysqld value,

Key => "mysqld": 
Value => "wsrep_cluster_address:\\\"gcomm://bbcdblog4,bbcdblog5,bbcdblog6,dblog8\\\""

Thanks

support auto complete of commands

list-machines should not fail argument for all envs

when you have an env that is not running in ggn, no command can be executed at all since list-machines is needed to generate commands.

we should extract the environment from the command line before calling cobra

[confd] Scientific Notation for big integer

Confd formats large integer with the scientific notation. Can be avoided by adding quotes on attributes (temporary fix).

Why doed ggn want to absolutly list-machines

$ ggn --help
16:43:53 FATAL                ggn/work/env.go:123 Cannot list-machines                         env=prod-pa3 error=exit status 1

$ ggn genautocomplete
16:44:13 FATAL                ggn/work/env.go:123 Cannot list-machines                         env=prod-pa3 error=exit status 1

missing generate before ggn check

running ggn check doesn't show any difference.
you have to run ggn generate before, or any other command including a generate.

ggn check should include a generate prior to actually running the command.

Implement fleetctl list-machines for ggn

please !

ggn journal -f crashes when Ctrl+C

ggn prod-pa3 service node journal -f

works perfectly, but crashes when stopped with Ctrl+C :

FATAL ggn/work/env/service/unit.go:98 Failed to run journal error=exit status 130

use User config attribute for lock

Better Process Panic and Fatal

Fleetctl endpoint

The ggn prod-pa3 check seems to use the local endpoint for fleetctl. I have an Unknow Unit for all checks

argument parsing is dirty

How do we support remove a server in node list

hook check can stop update on a service

Service Manifest Templating

ggn should be able to template it's own manifest.

We would have to read the attributes from the command line and the attributes folder.
If the attributes are provided on the command line could be stored in etcd to be able to resolve template for the people that do not have the proper configuration.

Deploy the service front with a particular version on the command line.
- The version is available to be handled for the other user of the cluster.

check on env is too slow

Concurrency update

Flag to skip already up to date units
Flag to force units update
Flag to set concurrency for parallel updates