Kubernetes deployment strategies explained

Home Page: https://container-solutions.com/kubernetes-deployment-strategies/

Go 89.52% Makefile 5.07% Dockerfile 5.42%

kubernetes kubernetes-deployment-strategies canary grafana prometheus helm canary-deployment blue-green-deployment ab-testing deployment-automation

k8s-deployment-strategies's Introduction

Kubernetes deployment strategies

In Kubernetes there are a few different ways to release an application, you have to carefully choose the right strategy to make your infrastructure resilient.

recreate: terminate the old version and release the new one
ramped: release a new version on a rolling update fashion, one after the other
blue/green: release a new version alongside the old version then switch traffic
canary: release a new version to a subset of users, then proceed to a full rollout
a/b testing: release a new version to a subset of users in a precise way (HTTP headers, cookie, weight, etc.). This doesn’t come out of the box with Kubernetes, it imply extra work to setup a smarter loadbalancing system (Istio, Linkerd, Traeffik, custom nginx/haproxy, etc).
shadow: release a new version alongside the old version. Incoming traffic is mirrored to the new version and doesn't impact the response.

Before experimenting, checkout the following resources:

Getting started

These examples were created and tested on Minikube running with Kubernetes v1.25.2 and Rancher Desktop running with Kubernetes 1.23.6.

On MacOS the hypervisor VM does not have external connectivity so docker image pulls will fail. To resolve this, install another driver such as VirtualBox and add --vm-driver virtualbox to the command to be able to pull images.

$ minikube start --kubernetes-version v1.25.2 --memory 8192 --cpus 2

Visualizing using Prometheus and Grafana

The following steps describe how to setup Prometheus and Grafana to visualize the progress and performance of a deployment.

Install Helm3

To install Helm3, follow the instructions provided on their website.

Install Prometheus

$ helm install prometheus prometheus-community/prometheus \
    --create-namespace --namespace=monitoring \
    -f prometheus-override.yaml

Install Grafana

$ helm install grafana \
    --namespace=monitoring \
    --set=adminUser=admin \
    --set=adminPassword=admin \
    --set=service.type=NodePort \
    grafana/grafana

Setup Grafana

Now that Prometheus and Grafana are up and running, you can access Grafana:

$ minikube service grafana -n monitoring

To login, username: admin, password: admin.

Then you need to connect Grafana to Prometheus, to do so, add a DataSource:

Name: prometheus
Type: Prometheus
Url: http://prometheus-server
Access: Server

Create a dashboard with a Time series or import the JSON export - update the UID first of the Prometheus source. Use the following query:

sum(rate(http_requests_total{app="my-app"}[2m])) by (version)

Since we installed Prometheus with default settings, it is using the default scrape interval of 1m so the range cannot be lower than that.

To have a better overview of the version, add {{version}} in the legend field.

Example graph

Recreate:

Ramped:

Blue/Green:

Canary:

A/B testing:

Shadow:

k8s-deployment-strategies's People

Contributors

Stargazers

Watchers

Forkers

duke-lv azooinmyluggage dreadbird nkhare ycliuhw shoneslab thienngho franklinharry mirnujatom vikasgubbi gustavomr azavadskiy etsangsplk lorgine-li kagkarlsson yyzi lorenzo-biava ambertch nuaays abdelmuniem copley sdileep philchung tburdett internet-com xinzhang end-of-game alfredglover rpremgit enixdark anandmohnish vignesh97 brucewu16899 stafot morganyvm yamaszone foundjem benjaziasadok moccamc chuanma surechen123 samaddico amoahcp sudharsan-prodapt vikramhansawat arjundandagi lizhibin-ata pathcl pamir luebken resouer net-ai webofmars yuzisun hookttg justcoworker davansy rgodishela sxauyhz arosequist codeponpon leewalter trytynichenko raymondzhaoy lijianwh hustshawn birui sailinnthu veerendarreddy hellolijj jisd2089 coderwangke tomllt hpfish marinflorin ljdtc cloudowski az-art hjgraca ekeren ahmedbenabid asceticboy chbharathkumar bigsiri andaok iledarn currycan dpittner bhrikuty jackwen jangocheng 8tomat8 ssrinivasreddys tvollmer mtocchetto sharma-abhishek songhui1984 adesanluj satishrawat hulaoan

k8s-deployment-strategies's Issues

No target for "my-app" on prometheus

Hi,

When I follow the tutorial for the canary deployment, Grafana does not pull any data from Prometheus and thus cannot draw the graph. I've changed the prometheus service to be a NodePort, so that I can access it UI. The /targets path only show http://localhost:9090/metrics, despite there is much more targets definitions on configuration. Is this normal? How is Prometheus able to get data from my-app if there isn't any target to it?

Best Regards.

Canary by weight is not working properly while having perfect configuration using nginx-ingress

deployment_nginx_canary_conf.zip

we are using nginx version 0.27.0 and we tried it with version 0.30.0 also.

we tried implementing canary first by header value and then we tried to implement canary by header weight but the issue is canary itself is not working.

A little question about Grafana visualization of blue-green strategies.

I am learning Kubernetes deployment strategies with this repository. It's interesting to deploy test App and show the process in Grafana :D

When deploy blue-green (Single-service) the Graph is difference with README.md, maybe the problem is Grafana configure that has nothing to do with this repository, but I try to submit this issue to find out the reason.

Desired

Real

It look like Ramped but actual is Blue/Green deploying.

Look at the terminal:

# Switch version
kubectl patch service my-app -p '{"spec":{"selector":{"version":"v2.0.0"}}}'

# Output of `while sleep 0.1; do curl "$service"; done`
# We can see version was switched immediately
# but the Graph is not, it switch slowly just like ramped.
...
Host: my-app-v1-d9d8fb4c4-7nmhd, Version: v1.0.0
Host: my-app-v1-d9d8fb4c4-ppkz4, Version: v1.0.0
Host: my-app-v1-d9d8fb4c4-mzj6n, Version: v1.0.0
Host: my-app-v2-5856879d68-9kh2m, Version: v2.0.0
Host: my-app-v2-5856879d68-54xd7, Version: v2.0.0
Host: my-app-v2-5856879d68-9kh2m, Version: v2.0.0
...

Environmental

# base env version
macOS 11.2.3 + minikue  v1.18.1
prometheus 2.20.1 + grafana 7.1.1

# Grafana Panel
Metrics: sum(rate(http_requests_total{app="my-app"}[5m])) by (version)
Legend: {{version}} 

Visualization > Graph
Display-Bars > ON
Display-Lines > ON
Stacking and null value - Stack > ON
Stacking and null value - Percent > ON

Graphana visualization doesn't work

For some reason, the visualation isn't working, although the data source is properly configured

The repository is still very useful, without the grafana part, but I'd be happy to know how to get grapha working.

This is my configuration for the prometheus chart:

serverFiles:
  prometheus.yml:
    rule_files:
      - /etc/config/rules
      - /etc/config/alerts
    scrape_configs:
    - job_name: 'kube-kubelet'
      honor_labels: false
      scheme: https

      tls_config:
      # This is needed because the kubelets' certificates are not generated
      # for a specific pod IP
        insecure_skip_verify: true
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

      kubernetes_sd_configs:
      - role: node
      relabel_configs:
      - target_label: __metrics_path__
        replacement: /metrics
      - source_labels: [__meta_kubernetes_node_address_InternalIP]
        target_label: instance
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)

    - job_name: 'kube-kubelet-cadvisor'
      honor_labels: false
      scheme: https

      tls_config:
      # This is needed because the kubelets' certificates are not generated
      # for a specific pod IP
        insecure_skip_verify: true
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

      kubernetes_sd_configs:
      - role: node
      relabel_configs:
      - target_label: __metrics_path__
        replacement: /metrics/cadvisor
      - source_labels: [__meta_kubernetes_node_address_InternalIP]
        target_label: instance
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)

    # Example scrape config for probing services via the Blackbox Exporter.
    #
    # Relabelling allows to configure the actual service scrape endpoint using the following annotations:
    #
    # * `prometheus.io/probe`: Only probe services that have a value of `true`
    - job_name: 'kubernetes-services'
      metrics_path: /probe
      params:
        module: [http_2xx]
      kubernetes_sd_configs:
        - role: service
      relabel_configs:
        - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe]
          action: keep
          regex: true
        - source_labels: [__address__]
          target_label: __param_target
        - target_label: __address__
          replacement: blackbox
        - source_labels: [__param_target]
          target_label: instance
        - action: labelmap
          regex: __meta_kubernetes_service_label_(.+)
        - source_labels: [__meta_kubernetes_namespace]
          target_label: kubernetes_namespace
        - source_labels: [__meta_kubernetes_service_name]
          target_label: kubernetes_name
    # Example scrape config for pods
    #
    # Relabelling allows to configure the actual service scrape endpoint using the following annotations:
    #
    # * `prometheus.io/scrape`: Only scrape pods that have a value of `true`
    # * `prometheus.io/path`: If the metrics path is not `/metrics` override this.
    # * `prometheus.io/port`: Scrape the pod on the indicated port instead of the default of `9102`.
    - job_name: 'kubernetes-pods'
      kubernetes_sd_configs:
        - role: pod
      relabel_configs:
        - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
          action: keep
          regex: true
        - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
          action: replace
          target_label: __metrics_path__
          regex: (.+)
        - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
          action: replace
          regex: (.+):(?:\d+);(\d+)
          replacement: ${1}:${2}
          target_label: __address__
        - action: labelmap
          regex: __meta_kubernetes_pod_label_(.+)
        - source_labels: [__meta_kubernetes_namespace]
          action: replace
          target_label: kubernetes_namespace
        - source_labels: [__meta_kubernetes_pod_name]
          action: replace
          target_label: kubernetes_pod_name
    # Scrape config for service endpoints.
    #
    # The relabeling allows the actual service scrape endpoint to be configured
    # via the following annotations:
    #
    # * `prometheus.io/scrape`: Only scrape services that have a value of `true`
    # * `prometheus.io/scheme`: If the metrics endpoint is secured then you will need
    # to set this to `https` & most likely set the `tls_config` of the scrape config.
    # * `prometheus.io/path`: If the metrics path is not `/metrics` override this.
    # * `prometheus.io/port`: If the metrics are exposed on a different port to the
    # service then set this appropriately.
    - job_name: 'kubernetes-service-endpoints'
      kubernetes_sd_configs:
        - role: endpoints
      relabel_configs:
        - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
          action: keep
          regex: true
        - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
          action: replace
          target_label: __scheme__
          regex: (https?)
        - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
          action: replace
          target_label: __metrics_path__
          regex: (.+)
        - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
          action: replace
          target_label: __address__
          regex: (.+)(?::\d+);(\d+)
          replacement: $1:$2
        - action: labelmap
          regex: __meta_kubernetes_service_label_(.+)
        - source_labels: [__meta_kubernetes_namespace]
          action: replace
          target_label: kubernetes_namespace
        - source_labels: [__meta_kubernetes_service_name]
          action: replace
          target_label: kubernetes_name # Add your additional configuration here...

containersolutions / k8s-deployment-strategies Goto Github PK