Giter VIP home page Giter VIP logo

maestro's Introduction

maestro

Deploy only what's changed for your multiple services in mono-repos

Go Report Card Code Climate

How it Works

Maestro pulls a given repository then builds a dependency graph based on a given config file. Once the dependency graph is created, Maestro diffs against a given previous commit, with the pathspec being the root directory for each artifact. Maestro then flags only the changed artifacts for the pipeline, which are then ran concurrently per teir of dependencies, therefore siblings in the graph will build, test and deploy concurrently, but parents and children dependencies will always be built in the correct order.

Used with Maestrod you can have a build manager that integrates with Github webhooks and runs on Kubernetes or a single-host Docker setup.

For more details see this talk: https://www.youtube.com/watch?v=dGM8mYj8nz4&feature=youtu.be

Install

    git clone [email protected]:cpg1111/maestro
    cd maestro
    make docker

or

    docker pull cpg1111/maestro:<release>

Test

    go test ./...

Run

    Usage of maestro:
      --branch string
            Git branch to checkout for project (default "master")
      --clone-path string
            Local path to clone repo to defaults to PWD (default "./")
      --config string
            Path to the config for maestro to use (default "./conf.toml")
      --deploy
            Whether or not to deploy this build                             # defaults to false
      --prev-commit string
            Previous commit to compare to                                   # required
    maestro --branch <git branch to build> --conf <project config> --prev-commit <commit to compare to> --deploy <whether to deploy build or not> --clone-path <tmp path to clone repo into>

or

    docker run -v <path of conf>:<target> -v <path to ssh credentials if using ssh for git>:<target> maestro --branch <git branch to build> --conf <project config> --prev-commit <commit to compare to> --deploy <whether to deploy build or not> --clone-path <tmp path to clone repo into>

Example Config

    [Environment] # Environment will run before anything else, ExecSync will execute commands in the array synchronously, while exec will execute them concurrently
    Env=["node_env:test", "docker_tls_verify:1"] # set environment variables all lowercase keys and keys are separated from values with ':'
    ExecSync=["apt-get install -y docker node go"]
    Exec=["docker pull someOrg/logger", "docker pull someOrg/models", "docker pull someOrg/auth", "docker pull someOrg/client"]

    [Project]
    RepoURL="[email protected]:someOrg/someRepo.git"
    CloneCMD="git clone"
    AuthType="SSH"
    SSHPrivKeyPath="~/.ssh/id_rsa"
    SSHPubKeyPath="~/.ssh/id_rsa.pub"
    Username="git" # github's ssh user is git, but this can vary
    Password=""
    PromptForPWD=false # when requiring a password, you prompt for a password

    [[Services]] # Services are either actual services or libraries / packages / separately compiled objects
    Name="logger"
    Tag="0.1.0"
    TagType="git"
    Path="./src/logger"
    BuildCMD=["docker build -t logger ."] # '.' is relative to the given path field of the service
    TestCMD=["go test ./..."]
    CheckCMD=["bash -c 'docker images -a | grep logger'"]
    CreateCMD=[
        "docker tag logger <org>/logger:{{.Curr}}",
        "docker push <org>/logger:{{.Curr}}"
    ]
    UpdateCMD=[
        "docker tag logger <org>/logger:{{.Curr}}",  # {{.Curr}} will template the current commit hash into the command
        "docker push <org>/logger:{{.Curr}}"
    ]
    DependsOn=[]

    [[Services]]
    Name="models"
    Tag="0.1.0"
    TagType="git"
    Path="./src/models"
    BuildCMD=["docker build -t models ."]
    TestCMD=["go test ./..."]
    CheckCMD=["bash -c 'docker images -a | grep models'"]
    CreateCMD=[
        "docker tag models <org>/models:{{.Curr}}",
        "docker push <org>/models:{{.Curr}}"
    ]
    UpdateCMD=[
        "docker tag models <org>/models:{{.Curr}}",
        "docker push <org>/models:{{.Curr}}"
    ]
    DependsOn=["logger"] # Assume Dockerfile contains FROM logger

    [[Services]]
    Name="auth"
    Tag="0.1.0"
    TagType="git"
    Path="./src/auth"
    BuildCMD=["docker build -t auth ."]
    TestCMD=["go test ./..."]
    CheckCMD=["bash -c 'docker ps -a | grep auth'"]
    CreateCMD=[
        "docker tag auth <org>/auth:{{.Curr}}",
        "docker tag auth <org>/auth:{{.Curr}}",
        "docker run --rm -d <org>/auth:{{.Curr}}"
    ]
    UpdateCMD=[
        "docker tag auth <org>/auth:{{.Curr}}",
        "docker tag auth <org>/auth:{{.Curr}}",
        "docker run --rm -d <org>/auth:{{.Curr}}"
    ]
    DependsOn=["models"] # Assume Dockerfile contains FROM models

    [[Services]]
    Name="client"
    Tag="0.1.0"
    TagType="git"
    Path="./src/client"
    BuildCMD=["docker build -t client ."]
    TestCMD=["npm test"]
    CheckCMD=["bash -c 'docker ps -a | grep client'"]
    CreateCMD=[
        "docker tag client <org>/client:{{.Curr}}",
        "docker push <org>/client:{{.Curr}}",
        "docker run --rm -d <org>/client:{{.Curr}}"
    ]
    UpdateCMD=[
        "docker tag client <org>/client:{{.Curr}}",
        "docker push <org>/client:{{.Curr}}",
        "docker run --rm -d <org>/client:{{.Curr}}"
    ]
    DependsOn=[]

    [CleanUp]
    AdditionalCMDs=["docker inspect auth", "docker export -o ./dist/auth.tgz auth"] # Will execute synchronously
    InDaemon=false # COMING SOON for maestrod
        [[CleanUp.Artifacts]] # Artifacts are saved concurrently
        RuntimeFilePath="./dist/auth.tgz"
        SaveFilePath="/opt/data/auth.tgz"

Roadmap

  • Allow larger log buffers
  • More possible dependency structures
  • Encrypted Environment variable values
  • Log Versbosity control
  • Debug with bash session

Daemon

See this (https://github.com/cpg1111/maestrod) for a manager daemon for handling git push hooks and multiple concurrent builds and repos.

Warning

Some dependency structures are not supported yet. Best to refer to immediate dependencies only and avoid circlular references.

maestro's People

Contributors

cpg1111 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

Forkers

numonium

maestro's Issues

Statically Compile LibGit2

Currently, LibGit2 is dynamically linked due to dependency issues.

It should be statically linked for easier and cleaner releases.

Allow Sibling Services to finish when a service has failed.

Currently, when a service fails either to build, test, or deploy, all siblings in the dependency tree stop wherever they may be in the process.

It should be such that when a service fails, its siblings are able to complete and children that do not depend on the failed service but do depend on the sibling services may complete as well.

Clean up blocking builds for siblings to finish

Maestro builds siblings in its dependency tree concurrently, it then blocks further builds from building until all siblings in the tree are done. Currently, this is done through the use of for... select loops and channels.

This is a proposal to wrap this logic in Go's context package. Such that context.Context.Done() will handle blocking at a higher level until all siblings in the tree have finished their builds.

State should be more detailed

Currently, state is expressed as Started or Finished externally. Considering the internal state is much more detailed, Maestro should communicate this with Maestrod in order to better manage its lifecycle.

Proposal:
Create a new package that observes the internal lifecycle in its own goroutine and communicates the state to Maestrod.

Additional SCM support

Currently, maestro only supports git, this is a proposal to support at least SVN and Mercurial, and others may be tacked on later. This should be implemented in such a way that any SCM can be pluggable and removable.

Encrypt sensitive data in environment configs

Currently all data in the config is displayed in plain text or would need to execute a child process in order to decrypt any encrypted data.

Proposal:
In config specify what encryption followed by the encrypted value. Should look like:

Env=["SHA512:password:a7CXd14WsfeEewQsfr"]

Grant it, those values are made up, but assume the value is encrypted with sha512.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.