Giter VIP home page Giter VIP logo

tekton-pipelines-elastic-tutorials's Introduction

Tekton Pipelines and the Elastic Stack

In a few minutes, you will have the following components up and running in the tekton-pipelines of your Kubernetes cluster:

  • Tekton v0.10.1

  • Kube Stats Metric v1.4.0

  • Elastic Stack 7.5.2, Elasticsearch / Kibana / Beats (filebeat and metricbeat) deployed using the official Elastic Helm Charts

By default, some TaskRuns and PipelineRuns from the official Tekton repositories are executed in the tutorials namespace to generate data.

TL;DR
# Insall all resources and run TaskRuns/PipelineRuns examples
./scripts/set-up-env.sh

# wait for Forwarding from [::1]:5601 -> 5601
open http://localhost:5601/

Prerequisites

  • Local Kubernetes environment (Docker for Desktop, Minikube…​)

  • Helm v2.16+ installed

NOTE: for now this tutorial is only tested on Docker Desktop for Mac

See it in action:

kaniko stream logs
Figure 1. tekton example with Kaniko (Dockerfile in git → Docker image pushed to Dockerhub)

Tekton Pipelines Tutorials

The samples folder contains several examples using Tekton Tasks and Pipelines.
The Elastic components (Elasticsearch, Beats and Kibana) are deployed to the Kubernetes cluster to collect the logs and metrics from the pods deployed by Tekton.
The Tekton components deployed are the official ones. There is no need to update those components to monitor them using the Elastic Stack.

Set up

# make sure you are using the right k8s context
$ kubectl config current-context
docker-for-desktop

# install all the components
$ ./scripts/set-up-env.sh init

Then wait until the pods are running before starting the tutorials

# Check All Elastic and Tekton pods
$ kubectl get pods -w -n tekton-pipelines
NAME                                           READY   STATUS    RESTARTS   AGE
tekton-pipelines-controller-77f74f5bcf-rbj8s   1/1     Running   0          1m
tekton-pipelines-webhook-f76c97965-5xkxq       1/1     Running   0          1m

Get Access to Kibana

When the set-up is done, you should have a Kibana pod up and running:

# Check Elastic pods
$ kubectl get pods -n tekton-pipelines
NAME                                 READY     STATUS    RESTARTS   AGE
elasticsearch-7bf6cd96cd-r6llt       1/1       Running   0          4m
filebeat-5lrwg                       1/1       Running   0          4m
kibana-694998774-dxwgm               1/1       Running   0          4m
kube-state-metrics-b8845b4d7-ch9v7   1/1       Running   0          4m
metricbeat-zctb6                     1/1       Running   0          4m

Then you can access to the Kibana Infrastructure app:

kibana infrastructure app

Check the Tekton Pipeline Controller logs to make sure everything is OK

kibana k8s logs

Ready to start!

CONGRATS you are now ready to start the tutorials!

Observability

To check the Tekton metrics

kubectl port-forward deployment/tekton-pipelines-controller 9090 --namespace tekton-pipelines
Forwarding from 127.0.0.1:9090 -> 9090
Forwarding from [::1]:9090 -> 9090
Handling connection for 9090
Handling connection for 9090

and then

curl http://localhost:9090/metrics
# HELP tekton_reconcile_count Number of reconcile operations
# TYPE tekton_reconcile_count counter
tekton_reconcile_count{key="tutorials/echo-hello-world-task-run",reconciler="TaskRun",success="false"} 2
tekton_reconcile_count{key="tutorials/echo-hello-world-task-run",reconciler="TaskRun",success="true"} 9
# HELP tekton_reconcile_latency Latency of reconcile operations
# TYPE tekton_reconcile_latency histogram
tekton_reconcile_latency_bucket{key="tutorials/echo-hello-world-task-run",reconciler="TaskRun",success="false",le="10"} 0
tekton_reconcile_latency_bucket{key="tutorials/echo-hello-world-task-run",reconciler="TaskRun",success="false",le="100"} 1
tekton_reconcile_latency_bucket{key="tutorials/echo-hello-world-task-run",reconciler="TaskRun",success="false",le="1000"} 2
tekton_reconcile_latency_bucket{key="tutorials/echo-hello-world-task-run",reconciler="TaskRun",success="false",le="10000"} 2
tekton_reconcile_latency_bucket{key="tutorials/echo-hello-world-task-run",reconciler="TaskRun",success="false",le="30000"} 2
tekton_reconcile_latency_bucket{key="tutorials/echo-hello-world-task-run",reconciler="TaskRun",success="false",le="60000"} 2
tekton_reconcile_latency_bucket{key="tutorials/echo-hello-world-task-run",reconciler="TaskRun",success="false",le="+Inf"} 2
tekton_reconcile_latency_sum{key="tutorials/echo-hello-world-task-run",reconciler="TaskRun",success="false"} 274
tekton_reconcile_latency_count{key="tutorials/echo-hello-world-task-run",reconciler="TaskRun",success="false"} 2
tekton_reconcile_latency_bucket{key="tutorials/echo-hello-world-task-run",reconciler="TaskRun",success="true",le="10"} 4
tekton_reconcile_latency_bucket{key="tutorials/echo-hello-world-task-run",reconciler="TaskRun",success="true",le="100"} 9
tekton_reconcile_latency_bucket{key="tutorials/echo-hello-world-task-run",reconciler="TaskRun",success="true",le="1000"} 9
tekton_reconcile_latency_bucket{key="tutorials/echo-hello-world-task-run",reconciler="TaskRun",success="true",le="10000"} 9
tekton_reconcile_latency_bucket{key="tutorials/echo-hello-world-task-run",reconciler="TaskRun",success="true",le="30000"} 9
tekton_reconcile_latency_bucket{key="tutorials/echo-hello-world-task-run",reconciler="TaskRun",success="true",le="60000"} 9
tekton_reconcile_latency_bucket{key="tutorials/echo-hello-world-task-run",reconciler="TaskRun",success="true",le="+Inf"} 9
tekton_reconcile_latency_sum{key="tutorials/echo-hello-world-task-run",reconciler="TaskRun",success="true"} 187
tekton_reconcile_latency_count{key="tutorials/echo-hello-world-task-run",reconciler="TaskRun",success="true"} 9
# HELP tekton_running_taskruns_count Number of taskruns executing currently
# TYPE tekton_running_taskruns_count gauge
tekton_running_taskruns_count 0
# HELP tekton_taskrun_count number of taskruns
# TYPE tekton_taskrun_count counter
tekton_taskrun_count{status="success"} 1
# HELP tekton_taskrun_duration_seconds The taskrun's execution time in seconds
# TYPE tekton_taskrun_duration_seconds histogram
tekton_taskrun_duration_seconds_bucket{namespace="tutorials",status="success",task="echo-hello-world",taskrun="echo-hello-world-task-run",le="10"} 0
tekton_taskrun_duration_seconds_bucket{namespace="tutorials",status="success",task="echo-hello-world",taskrun="echo-hello-world-task-run",le="30"} 1
tekton_taskrun_duration_seconds_bucket{namespace="tutorials",status="success",task="echo-hello-world",taskrun="echo-hello-world-task-run",le="60"} 1
tekton_taskrun_duration_seconds_bucket{namespace="tutorials",status="success",task="echo-hello-world",taskrun="echo-hello-world-task-run",le="300"} 1
tekton_taskrun_duration_seconds_bucket{namespace="tutorials",status="success",task="echo-hello-world",taskrun="echo-hello-world-task-run",le="900"} 1
tekton_taskrun_duration_seconds_bucket{namespace="tutorials",status="success",task="echo-hello-world",taskrun="echo-hello-world-task-run",le="1800"} 1
tekton_taskrun_duration_seconds_bucket{namespace="tutorials",status="success",task="echo-hello-world",taskrun="echo-hello-world-task-run",le="3600"} 1
tekton_taskrun_duration_seconds_bucket{namespace="tutorials",status="success",task="echo-hello-world",taskrun="echo-hello-world-task-run",le="5400"} 1
tekton_taskrun_duration_seconds_bucket{namespace="tutorials",status="success",task="echo-hello-world",taskrun="echo-hello-world-task-run",le="10800"} 1
tekton_taskrun_duration_seconds_bucket{namespace="tutorials",status="success",task="echo-hello-world",taskrun="echo-hello-world-task-run",le="21600"} 1
tekton_taskrun_duration_seconds_bucket{namespace="tutorials",status="success",task="echo-hello-world",taskrun="echo-hello-world-task-run",le="43200"} 1
tekton_taskrun_duration_seconds_bucket{namespace="tutorials",status="success",task="echo-hello-world",taskrun="echo-hello-world-task-run",le="86400"} 1
tekton_taskrun_duration_seconds_bucket{namespace="tutorials",status="success",task="echo-hello-world",taskrun="echo-hello-world-task-run",le="+Inf"} 1
tekton_taskrun_duration_seconds_sum{namespace="tutorials",status="success",task="echo-hello-world",taskrun="echo-hello-world-task-run"} 13
tekton_taskrun_duration_seconds_count{namespace="tutorials",status="success",task="echo-hello-world",taskrun="echo-hello-world-task-run"} 1
# HELP tekton_taskruns_pod_latency scheduling latency for the taskruns pods
# TYPE tekton_taskruns_pod_latency gauge
tekton_taskruns_pod_latency{namespace="tutorials",pod="echo-hello-world-task-run-pod-5xqd7",task="echo-hello-world",taskrun="echo-hello-world-task-run"} 0
# HELP tekton_work_queue_depth Depth of the work queue
# TYPE tekton_work_queue_depth gauge
tekton_work_queue_depth{reconciler="TaskRun"} 0

Utils

Check the API Resources

kubectl api-resources | grep tekton.dev
clustertasks                                   tekton.dev                     false        ClusterTask
conditions                                     tekton.dev                     true         Condition
pipelineresources                              tekton.dev                     true         PipelineResource
pipelineruns                      pr,prs       tekton.dev                     true         PipelineRun
pipelines                                      tekton.dev                     true         Pipeline
taskruns                          tr,trs       tekton.dev                     true         TaskRun
tasks                                          tekton.dev                     true         Task

tekton-pipelines-elastic-tutorials's People

Contributors

mgreau avatar conky5 avatar

Watchers

James Cloos avatar  avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.