Hi! Reviving #8 now that it's the new year and things are in full swing.

This section(s) might include some best practices for testing operators and mention kuttl (and/or other tools if applicable)

Starting to manage the group and get thinks all set up here. @caniszczyk Thank you

"Finish Related Work / What’s next? / Where to get help?" section on Operator white paper.

"Technical Implementation" section/label for the operator white paper.

This is a bigger task that will have subtasks. We need to create a label "technical implementation" and add to each subtask.
Please suggest subtasks in this issue.

Current suggestions are:

On a very high level:

• How to write an operator (What to consider? Where to start?)
• How to test an operator (e.g. KUTTL)
• Delivery Pipeline for Operators

Litmus is a toolset to do chaos engineering in a kubernetes native way. Litmus provides chaos CRDs for Cloud-Native developers and SREs to inject, orchestrate and monitor chaos to find weaknesses in Kubernetes deployment in production.

We would love to demonstrate Litmus at sig-app-delivery meeting.

Proposed Agenda:
Introduction, architecture - 15 minutes
Demo - 10 minutes
Road map - 5 minutes

Important links:
GitHub: https://github.com/litmuschaos
Slack: https://kubernetes.slack.com/messages/CNXNB0ZTN
Website: https://litmuschaos.io
Twitter: https://twitter.com/LitmusChaos

"Operator Frameworks for Kubernetes" section on Operator white paper

Description of a few most popular frameworks used to create operators. It would be best if we could get input from the framework contributors.

"Summary/Conclusion" section on Operator white paper.

Hi, everyone! We would really like the chance to demo Brigade at SIG App Delivery.

Brigade is an in-cluster runtime that interprets scripts and executes generic pipelines of work in Kubernetes. Using JavaScript, developers can chain containers together in a script to create an event-driven workflow in Kubernetes. Scripts can define a series of jobs that run in parallel or serially, and take advantage of having full programming language support for error handling and async functionality, as well as benefit from the rich JavaScript ecosystem of libraries.

Brigade is a CNCF Sandbox project, it integrates with other cloud-native projects (Helm, Prometheus, CloudEvents, Virtual Kubelet), and has varied use cases (from a foundation for opinionated CI/CD platforms, to building training pipelines for predictive modeling, or asynchronous processing for orders).

cc @resouer, @AloisReitbauer, @bryanl

References:

1. GitHub: @brigadecore
2. Documentation: https://docs.brigade.sh

Use case session - Prometheus Operator.

See the PR: #66

(Created for project management workflow as requested in #58.)

• Recovery path should be definable in the spec
• Data is restored from the recovery path
• Recover the same application-version as the backup

i have read the doc for App Delivery and agree with the point of separated layers.
but there are still some confusion.
by the design layey2 and layer3 are two different layers and layer3 is a controller in local k8s while layer2 is a micro-service in our manage cluster. how do they do interaction? and if there are multiple k8s clusters(one layer2 and multiple layer3). and if the deploy policy is complex like grouped release with multi-versions.

👋

We'd love the chance to demo Porter and CNAB at one of the upcoming sig-app-delivery meetings. Porter is a tool for installing applications using CNAB and we thought that sig-app-delivery would be a good place to demo and discuss. We’ve been really focused on trying to deliver the best experience around CNAB and we’d love to hear feedback from this group.

We were thinking something along the lines of:
5 mins - A whirlwind introduction to CNAB
10 mins - How to build and work with bundles with Porter

We think it aligns with sig-app-delivery's goals and would love to have people check it out and contribute.

At the SIG intro session at KubeCon NA 2019, there were a few questions about how air-gapped scenarios fit in or can be enabled by tooling in the landscape.

It would be good to have a discussion and maybe develop some best practices, an article or blog post that addresses these scenarios.

In the CNAB world, we’ve been considering these use cases and would be happy to share our experiments and learnings at a SIG meeting or on the mailing list!

Cc: @AloisReitbauer @radu-matei

For the community review, a single document version of the document will be created to make it reviewable.

Process:

• Create Review branch
• Merge all documents into one
• Create PR
• Reviews
• Make changes in this branch (PR against this branch)
• Remove "separated documents"
• Approve Merge PR (all WG Chairs)

• The Operator is responsible for managing the configuration of the deployed application
• The customer should be able to “tell” the operator which configurations could be auto-tuned
• Auto-Tuning features of the operator should be documented (the user should know which configuration will be auto-tuned)

Write Operator White Paper Introduction section.

kubebuilder subsection on "Operator Frameworks for Kubernetes" section on Operator white paper

Document the kubebuilder v2 framework and the underlying controller-runtime library

"Use-Cases for an Operator" section on Operator white paper.

KUDO presented to the TOC on Sep 17 and we were referred to SIG App Delivery for another review. Can someone clarify what this review entails? I don't see the process documented anywhere publicly.

TOC issue: cncf/toc#261

'Best Practices" section on Operator white paper.

This is a task that will have several subtasks. Suggestions are welcome. Currently suggested sections are:

Placement of Operators (Workloads vs. Control Processes)
Metrics: Prometheus/OpenTelemetry
Umbrella-Operators (Operator of Operators / Federated)
Best Practices for Operator Implementation

• How to write an operator (What to consider? Where to start?)

• How to test an operator (e.g. KUTTL)

• Delivery Pipeline for Operators

• See: https://cloud.google.com/blog/products/containers-kubernetes/best-practices-for-building-kubernetes-operators-and-stateful-apps

• Loose coupling

• As less features as possible (stability)

Best Practices for securing operators
Publishing an Operator / How to find an operator?

• [TODO, operatorhub, artifacthub, OLM]

TODO:

• When we have permissions on the Operator white paper Github project, create a label for "best practices" and add a task for each section with that label.

Security section on the Operator white paper.

Sections to be discussed with Security SIG and when those are defined, create tasks under the white paper Project on Github.

Currently the suggested sections are:

Build trust - documentations

• Threats
• which threats an operator brings?
• How do users that wants to use an operator can trust it? What API endpoints and permissions?
• CVEs disclosure process
• How transparent the process is?
• Audit log analysis, what should the user expect to see once this is deployed

Security constraints - implementation

• How can I as a customer or end user of an operator can make sure that it’s a secure one?
• API Permissions
• What are the security features of the operator?
• Is this operator considering security as a first class citizen?
• Operator Scope (Cluster vs. Namespaced vs. Outside Cluster)

Metrics - end user observation:

• Audit log

(Suggestion: Follow up: z Follow the recommendation on Sig Security white paper for securing workloads)

cncf/toc#303 - PR within the TOC repo

Please create a WG for GitOps. There are several community members that wish to maintain a document that describes GitOps.

A very rough starting point for such a doc is here
https://docs.google.com/document/d/1ZgdtI8ck3DWIig5gBnbS20A0cgzBRlOT3Yv-s1b2c54/edit?usp=drivesdk

cncf/toc#237

https://docs.google.com/presentation/d/1k1QcjeTmb1ZeYtwMntLw1IVTbJXGerGnGExow9oa7i4/

Hi, we would like to propose Chaos Mesh as a CNCF sandbox project. We have some minor pending details to fill in or update, but basically that's it.

I also created the PR for the proposal in TOC repo:

cncf/toc#367

Name of Project: Chaos Mesh

Description:

Chaos Mesh is a versatile Chaos Engineering platform that orchestrates chaos experiments on Kubernetes environments. It features all-around fault injection methods for complex systems on Kubernetes, covering faults in Pod, network, file system, and even the kernel.

Chaos Mesh was originated from the internal chaos engineering platform at PingCAP to ensure the resilience of TiDB, its distributed database system. As the system evolves and testing requirements multiply, the team realized that they need an easy to use, scalable and universal chaos testing platform. The combination of chaos and Kubernetes became the natural choice. Chaos Mesh uses CRD to define chaos objects, which makes it naturally integrate with the Kubernetes ecosystem.

At the current stage, it supports the following fault injection types:

• pod-kill: Simulates Kubernetes Pods being killed
• pod-failure: Simulates Kubernetes Pods being continuously unavailable
• container-kill: Simulates Kubernetes Pod’s container being killed
• network-delay: Simulates network delay
• network-loss: Simulates network packet loss
• network-duplication: Simulates network packet duplication
• network-corrupt: Simulates network packet corruption
• network-partition: Simulates network partition
• I/O delay: Simulates file system I/O delay
• I/O errno: Simulates file system I/O errors
• Time skew: Simulates time jumping forward or backward

Besides the versatile chaos types, it also offers Chaos Dashboard (under development), a visualized panel that shows the impacts of chaos experiments on the online services of the system, which makes chaos tests easily observable and manageable.

Sponsor from TOC: TBD

Unique Identifier: chaos-mesh

Preferred Maturity Level: Sandbox

License: Apache License, Version 2.0

Source control repositories: https://github.com/pingcap/chaos-mesh (to be moved to https://github.com/chaos-mesh)

Issue tracker: GitHub

Infrastructure Required:

Chaos Mesh uses in-house Jenkins CI cluster for integration tests. We plan to use CNCF test cluster to automatically run stability tests and performance tests in the future.

Website: https://github.com/pingcap/chaos-mesh

Documentation: https://github.com/pingcap/chaos-mesh/wiki

Release methodology and mechanics:

This is currently being defined. Releases every few months with RC process.

External dependencies (including licenses):

• Apache License 2.0：

  • github.com/containerd/containerd
  • github.com/docker/docker
  • github.com/ethercflow/hookfs
  • github.com/go-logr/logr
  • github.com/grpc-ecosystem/go-grpc-middleware
  • github.com/grpc-ecosystem/go-grpc-prometheus
  • github.com/oklog/run
  • github.com/prometheus/client_golang
  • github.com/vishvananda/netlink
  • github.com/vishvananda/netns
  • github.com/kubernetes-sigs/controller-runtime

• MIT：

  • github.com/ghodss/yaml
  • github.com/jmoiron/sqlx
  • github.com/onsi/ginkgo
  • github.com/onsi/gomega
  • github.com/robfig/cron
  • github.com/unrolled/render

• Mozilla Public License 2.0：

  • github.com/go-sql-driver/mysql

• BSD 3-Clause：

  • github.com/golang/protobuf
  • github.com/gorilla/mux

BSD：
* github.com/hanwen/go-fuse

Initial committers:

Community Stats:

Although it’s just been open-sourced since Dec 31, 2019, Chaos Mesh has been gaining recognition and popularity quickly in the community, with 1400+ stars received on GitHub in a month. It has also been listed under CNCF Cloud Native Interactive Landscape.

• Stars: 1500+
• Contributors: 20
• Commits: 175+
• Forks: 73+

Comparison

This comparison is intended simply to compare fault injection features supported by Chaos Mesh with other well-known chaos engineering platforms. It is not intended to favor or position one project over another. Any corrections are welcome.

Roadmap:

See https://github.com/pingcap/chaos-mesh/blob/master/ROADMAP.md

Social media accounts

Twitter: @chaos_mesh

Communication channels

GitHub

Slack channel: #sig-chaos-mesh

Community meetings (Planing)

Adopters or potential users?

TiKV/TiDB projects
Netease (testing)

Existing sponsorship

PingCAP

Statement on alignment with CNCF charter mission

Our team believes Chaos Mesh will be a great fit for CNCF._ As manifested in the CNCF mission:_

““These techniques enable loosely coupled systems that are resilient, manageable, and observable. Combined with robust automation, they allow engineers to make high-impact changes frequently and predictably with minimal toil. “

We believe Chaos Mesh is one of the essential enablements to this mission, and it’s also a great addition to the sandbox project scope. By covering comprehensive fault injection methods in Pod, network, file system, and even the kernel, Chaos Mesh aims at providing a neutral, universal Chaos Engineering platform that enables cloud-native applications to be as resilient as they should be. Chaos Mesh uses CRD to define chaos objects, making it naturally integrated with the Kubernetes ecosystem. In addition, it integrates several other projects in the cloud-native ecosystem, such as Helm, Prometheus, and Grafana.

What are we looking for from CNCF:

• Project governance - Chaos Mesh is still in the early phase of development, we would love to receive advice and guidance from CNCF on the community governance process and other community-related aspects.
• Neutral home for collaboration - As a universal Chaos Engineering platform, Chaos Mesh grows by incorporating feedback and collaborations from the community. We would love to see more collaborations between Chaos Mesh and other cloud-native projects.
• Community marketing support - Guidance or sponsorship in terms of events hosting
• Legal support - Any sort of legal assistance in aspects of potential IP, trademark issues

• Unlink objects from the operator
• After disconnecting all objects created by the operator are still present

"Operator Lifecycle Management" section on Operator white paper.

• Updating an Operator itself
• Updating managed CRDs

• All objects managed by the operator are removed

This section should include an overview what Operator Framework is and how it could help

Give a short overview what metacontroller is and how it could help ...

Operator white paper.

Acceptance criteria: Make "Operator Characteristics" section more k8s agnostic.

Hello!

We recently announced OAM & Rudr as open source projects and would love the opportunity to demo it at the upcoming sig-app-delivery meeting.

OAM is a spec for building cloud native applications and Rudr is an implementation on Kubernetes that accepts this specification. These projects fit in nicely with Topic 1 for this sig and feedback from this group especially would be amazing in these early stages of the release.

The agenda would be:

5 minutes: Introduction to OAM and the goals
10 minutes: Demo how to author the spec and deploy to Rudr

Look forward to this opportunity!

The @cncf/toc would like to request public feedback from CNCF SIG App Delivery on Argo: cncf/toc#299

This section should include an overview what KUDO is and how it could help

Should be located in a doc 052_kudo.md

Section for kubebuilder framework

Harbor is proposing Graduation: cncf/toc#311

There is a Helm specific component in Harbor that needs to be reviewed by SIG-App-Delivery.

Other relevant SIGs are reviewing (or have reviewed): SIG-Runtime, SIG-Storage, SIG-Security.

There is a consolidated document that needs to be updated once the review is complete: http://bit.ly/harbor-graduation-dd

"Operator Frameworks for Non-Kubernetes Platforms" section on Operator white paper.

Operator white paper "Operator Capabilities" section.

This is a bigger task, more like an epic.

TODO (blocked by when we have access to Operator White paper Github project):

• Create tasks for each section present in the current working document
• Create "operator capabilities" label and add to each task above

The TOC asked us to work on a definition for "What an operator is" Let us use this as a discussion thread.

• Operator is able to get the applications state
• E.g. Metrics indicating errors
• The operator knows how to deal with errors / system errors
• The operator takes actions

We are getting a logo!

Use this issue to share and discuss your ideas.

Operator White Paper:

Latest version of this section is available on https://docs.google.com/document/d/1daXHyR2yG5ArjO1PO83v4a0fmLCLo2kYNT3TwGZksng/edit#

Please review and add to PR: #30