Giter VIP home page Giter VIP logo

beetle's Introduction

Beetle

Kubernetes multi-cluster deployment automation service



Application deployment and management should be automated, auditable, and easy to understand and that's what beetle tries to achieve in a simple manner. Beetle automates the deployment and rollback of your applications in a multi-cluster, multi-namespaces kubernetes environments. Easy to integrate with through API endpoints & webhooks to fit a variety of workflows.

Documentation

Deployment

On a Linux Server

Download the latest beetle binary.

$ curl -sL https://github.com/Clivern/Beetle/releases/download/vx.x.x/beetle_x.x.x_OS.tar.gz | tar xz

Create your config file as explained on development part and run beetle with systemd or anything else you prefer.

$ ./beetle serve -c /custom/path/config.prod.yml

Development

Beetle uses Go Modules to manage dependencies. First Create a prod config file.

$ git clone https://github.com/Clivern/Beetle.git
$ cp config.dist.yml config.prod.yml

Then add your default configs. You probably wondering how the following configs even work! let's pick one and explain.

The item mode: ${BEETLE_APP_MODE:-dev} means that the mode is dev unless environment variable BEETLE_APP_MODE is defined. so you can always override the value by defining the environment variable export BEETLE_APP_MODE=prod. and same for others

# App configs
app:
    # Env mode (dev or prod)
    mode: ${BEETLE_APP_MODE:-dev}
    # HTTP port
    port: ${BEETLE_API_PORT:-8080}
    # App URL
    domain: ${BEETLE_APP_DOMAIN:-http://127.0.0.1:8080}
    # TLS configs
    tls:
        status: ${BEETLE_API_TLS_STATUS:-off}
        pemPath: ${BEETLE_API_TLS_PEMPATH:-cert/server.pem}
        keyPath: ${BEETLE_API_TLS_KEYPATH:-cert/server.key}

    # Message Broker Configs
    broker:
        # Broker driver (native)
        driver: ${BEETLE_BROKER_DRIVER:-native}
        # Native driver configs
        native:
            # Queue max capacity
            capacity: ${BEETLE_BROKER_NATIVE_CAPACITY:-5000}
            # Number of concurrent workers
            workers: ${BEETLE_BROKER_NATIVE_WORKERS:-4}

    # API Configs
    api:
        key: ${BEETLE_API_KEY:- }

    # Runtime, Requests/Response and Beetle Metrics
    metrics:
        prometheus:
            # Route for the metrics endpoint
            endpoint: ${BEETLE_METRICS_PROM_ENDPOINT:-/metrics}

    # Application Database
    database:
        # Database driver (sqlite3, mysql)
        driver: ${BEETLE_DATABASE_DRIVER:-sqlite3}
        # Database Host
        host: ${BEETLE_DATABASE_MYSQL_HOST:-localhost}
        # Database Port
        port: ${BEETLE_DATABASE_MYSQL_PORT:-3306}
        # Database Name
        name: ${BEETLE_DATABASE_MYSQL_DATABASE:-beetle.db}
        # Database Username
        username: ${BEETLE_DATABASE_MYSQL_USERNAME:-root}
        # Database Password
        password: ${BEETLE_DATABASE_MYSQL_PASSWORD:-root}

    # Kubernetes Clusters
    clusters:
        -
            name: ${BEETLE_KUBE_CLUSTER_01_NAME:-production}
            inCluster: ${BEETLE_KUBE_CLUSTER_01_IN_CLUSTER:-false}
            kubeconfig: ${BEETLE_KUBE_CLUSTER_01_CONFIG_FILE:-/app/configs/production-cluster-kubeconfig.yaml}
        -
            name: ${BEETLE_KUBE_CLUSTER_02_NAME:-staging}
            inCluster: ${BEETLE_KUBE_CLUSTER_02_IN_CLUSTER:-false}
            kubeconfig: ${BEETLE_KUBE_CLUSTER_02_CONFIG_FILE:-/app/configs/staging-cluster-kubeconfig.yaml}

    # HTTP Webhook
    webhook:
        url: ${BEETLE_WEBHOOK_URL:- }
        retry: ${BEETLE_WEBHOOK_RETRY:-3}
        apiKey: ${BEETLE_WEBHOOK_API_KEY:- }

# Log configs
log:
    # Log level, it can be debug, info, warn, error, panic, fatal
    level: ${BEETLE_LOG_LEVEL:-info}
    # output can be stdout or abs path to log file /var/logs/beetle.log
    output: ${BEETLE_LOG_OUTPUT:-stdout}
    # Format can be json
    format: ${BEETLE_LOG_FORMAT:-json}

And then run the application.

$ go build beetle.go
$ ./beetle serve -c /custom/path/config.prod.yml

// OR

$ make run

// OR

$ go run beetle.go serve -c /custom/path/config.prod.yml

API Documentation

Go to https://editor.swagger.io/ and import this file https://raw.githubusercontent.com/Clivern/Beetle/main/swagger.yaml.

Versioning

For transparency into our release cycle and in striving to maintain backward compatibility, Beetle is maintained under the Semantic Versioning guidelines and release process is predictable and business-friendly.

See the Releases section of our GitHub project for changelogs for each release version of Beetle. It contains summaries of the most noteworthy changes made in each release.

Bug tracker

If you have any suggestions, bug reports, or annoyances please report them to our issue tracker at https://github.com/clivern/beetle/issues

Security Issues

If you discover a security vulnerability within Beetle, please send an email to [email protected]

Contributing

We are an open source, community-driven project so please feel free to join us. see the contributing guidelines for more details.

License

ยฉ 2020, clivern. Released under MIT License.

Beetle is authored and maintained by @clivern.

beetle's People

Contributors

butuzov avatar clivern avatar hyeio avatar mergify[bot] avatar renovate-bot avatar renovate[bot] avatar todd-the-bot avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

beetle's Issues

Create App Endpoint

/api/v1/cluster/<clusterName>/namespace/<namespaceName>/application/<AppID>

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

Ignored or Blocked

These are blocked by an existing closed PR and will not be recreated unless you click a checkbox below.

Detected dependencies

docker-compose
deployment/docker/docker-compose.yml
  • redis 7.2-alpine
  • prom/prometheus v2.53.0
  • grafana/grafana 9.5.20
dockerfile
Dockerfile
  • golang 1.20.2
github-actions
.github/workflows/build.yml
  • actions/checkout v4
  • actions/setup-go v4
.github/workflows/release.yml
  • actions/checkout v4
  • actions/setup-go v4
  • goreleaser/goreleaser-action v3
.github/workflows/release_pkg.yml
  • actions/checkout v4
  • actions/setup-go v4
gomod
go.mod
  • go 1.20
  • github.com/briandowns/spinner v1.23.0
  • github.com/drone/envsubst v1.0.3
  • github.com/gin-gonic/gin v1.10.0
  • github.com/jinzhu/gorm v1.9.16
  • github.com/logrusorgru/aurora/v3 v3.0.0
  • github.com/olekukonko/tablewriter v0.0.5
  • github.com/prometheus/client_golang v1.18.0
  • github.com/satori/go.uuid v1.2.0
  • github.com/sirupsen/logrus v1.9.3
  • github.com/spf13/cobra v1.8.1
  • github.com/spf13/viper v1.18.2
  • k8s.io/api v0.27.4
  • k8s.io/apimachinery v0.27.4
  • k8s.io/client-go v0.27.4

  • Check this box to trigger a request for Renovate to run again on this repository

What is this?

This project looks very cool, and is speaking my language (I think) when it comes to solving a very real need that I have experienced when running applications on k8s. But, back to the "I think" from above... what exactly is it? Is it meant to compete with something like Helm? Does it use/interoperate with Helm? Is it a wrapper around Helm? Is it a wrapper around the terraform k8s plugin? Is it a service harness/chassis/runtime and a central server and some other glue for introspection on running/apps deployments (the example config file makes me feel like it might be something like that... but I honestly can't tell from the README...)

Anyway, saw this on the Changelog newsletter, and I'm intrigued... but I still am not really sure what it is.

So I guess my feature request would be a section in the README that explains that.

Create Apps Endpoint

GET /api/v1/cluster/<clusterName>/namespace/<namespaceName>/application

Cluster Configs

If only version (eg. 1.0.0 or 1.1.0) will be provided to deployment & rollback endpoints

    applications:
      -
        name: Toad App
        id: toad
        image: clivern/toad:release-[.Release]

If only version or version with suffix (eg. 1.0.0 or release-1.1.0 or version-1.0.0) will be provided to deployment & rollback endpoints

    applications:
      -
        name: Toad App
        id: toad
        image: clivern/toad:[.Release]

If the whole image will be provided (eg. clivern/toad:1.0.0 or clivern/toad:release-1.1.0 or clivern/toad:version-1.0.0) will be provided to deployment & rollback endpoints

    applications:
      -
        name: Toad App
        id: toad
        image: [.Release]

Action Required: Fix Renovate Configuration

There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.

Error type: undefined. Note: this is a nested preset so please contact the preset author if you are unable to fix it yourself.

Graph showing beetle in action

showing beetle, beetle workers, database, kubernetes clusters, namespaces, beetle reading from configMaps ...etc

It could be a cmd video of how to run beetle

Automate releases with different deployment strategies & rollback plan

Figure out how to do that. Also a way to get all releases available for deployment or text input/api parameter

Deployment strategies that can be supported.

recreate: terminate the old version and release the new one.
ramped: release a new version on a rolling update fashion, one after the other.
blue/green: release a new version alongside the old version then switch traffic.
canary: release a new version to a subset of users, then proceed to a full rollout.

Client -->
https://github.com/kubernetes/client-go/blob/master/kubernetes/typed/core/v1/core_client.go

Build Deployment Module

ref.
https://semaphoreci.com/blog/kubernetes-deployment
https://www.weave.works/technologies/manually-updating-kubernetes-workloads/

$ kubectl patch deployment patch-demo --patch '{
	"spec": {
		"template": {
			"spec": {
				"containers": [
					{
						"name": "nginx",
						"image": "nginx:1.7.9"
					}
				]
			}
		}
	}
}'

https://github.com/kubernetes/client-go/blob/5194bac869679e2d68716e9eeddf23d3c13661e1/kubernetes/typed/apps/v1beta1/deployment.go#L183-L194

	payload := []model.PatchStringValue{
		model.PatchStringValue{
			Op:    "replace",
			Path:  "/spec/template/spec/containers/0/image",
			Value: "clivern/toad:release-0.2.3",
		},
	}

	data, _ := util.ConvertToJSON(payload)

	fmt.Println(cluster.PatchDeployment(
		context.Background(),
		ns,
		"toad-deployment",
		data,
	))

Tutorial how to run beetle

  • Setup cluster and deploy a test application.
  • Beetle In cluster deployment.
  • Beetle outside cluster deployment.
  • Beetle local environment deployment.
  • How to Call beetle api server to deploy our application.
  • Future features.

Clusters REST API Endpoints

GET          /api/v1/cluster
{
    "_metadata": {
        "totalCount": 2,
        "limit": 10,
        "offset": 0
    },
    "cluster": [
            {"id":"default"},
            {"id":"prod"},
            {"id":"staging"}
     ]
}

Change clientset to be a parameter/configurable for testing

Change clientset to be a parameter/configurable so i can use https://github.com/kubernetes/client-go/blob/master/kubernetes/fake/clientset_generated.go#L114 for unit/func testing.

prototype

clientset := fake.NewSimpleClientset(&v1.Pod{
    ObjectMeta: metav1.ObjectMeta{
        Name:        "toad",
        Namespace:   "default",
        Annotations: map[string]string{},
    },
}, &v1.Pod{
    ObjectMeta: metav1.ObjectMeta{
        Name:        "clivern",
        Namespace:   "default",
        Annotations: map[string]string{},
    },
})

Separate global configs from others that change per cluster & namespace

Global Configs -> beetle yaml config file.
Other Configs that change per cluster & namespace -> namespaced configMaps

Something like that to load cluster.namespace configs?

"github.com/clivern/beetle/internal/app/model"
    
var data = []byte(`
version: 0.1.0
`)

// ideally from here 
item, _ := cluster.GetConfigMap(context.Background(), "namespace-here", "k8s-cluster-configmap-name")
item.Data

nc := model.NamespaceConfigs{}
nc.LoadFromYAML([]byte(item.Data))

narrowed down to

cluster.GetConfig(context.Background(), "~namespace-name~")

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.