https://github.com/kubernetes-sigs/cluster-addons/blob/master/bootstrap/app/addon-manager.go#L73 references v1beta1 cronjobs which will no longer be served in 1.25+

xref kubernetes/kubernetes#108797 (comment)

Current coredns operator doesn't have tests.
According to here, It is better to add "Golden File Tests".

FEATURE REQUEST

Allow users to easily provision a cluster with KubeVirt enabled. This would allow users provisioning there clusters with kubeadm who need to run VMs in containers and easy way to set up their cluster and be able to run their applications from day 1.

The addon would then deploy the manifest for the kubevirt operator, and then kubevirt CR to deploy the kubevirt components.

Create an end-to-end smoketest for coredns-operator. The test should run locally by default (via kind or kinder) and support running on a remote kubernetes cluster when provided.

The test should implement and validate the following scenarios:

• Creation 
• Deletion
• Configuration Change
• Upgrade the version of CoreDNS
• Downgrade the version of CoreDNS

This work will require adding multiple versions to the CoreDNS version directory.

Implementation Nodes:

• The test can be based off the existing dashboard-operator smoketest [src].
• If there is bandwidth, write a test framework in kubebuilder-declarative-pattern and consume it in the coredns-operator repo. 
  • Roughly: A test framework would perform the execution of the test (eg, install the operator, CRDs), provide a set of common validation functions (eg verifyNoWorkloadsWithLabel) and allow the operator author to pass in their own “AddonTest” object.
  • The kubebuilder/controller-runtime folks may be providing something like this (/cc @ecordell)

/cc @annp1987

In a conversation with @stealthybox we realised it might be good to add a project board to make it easier to spot what still needs to be done and who does what.

Referenced in #14, we would like installers to be able to load kustomize packages.
We also expect that addon-operators can benefit from this ComponentConfig work and packaging.

This implies improvements for packaging and distributing these kustomize bases and overlays.

git support with tags/refs and nested dirs already seems to be built into kustomize which is a great starting point.

Some good things to work on:

• OCI image package format
  • OCI registries are a pre-req to running a cluster
  • allows signing and verification
  • use https://github.com/deislabs/oras?
• Cluster Bundle based package
  • no deps other than CRD's in k8s API
  • can use typed refs to load related bundles inside the cluster from installer ComponentConfig (interesting MIME-type questions for the URI here)

(Tracking issue for GSoC proposal)

Now that we're making good progress on the various pieces of machinery and integrations for cluster-addons, it's feasible to start a real effort to create addon operators for the most popular and most interesting cluster addons. Good candidates would be the addons in the (deprecated) cluster directory of k/k, the packaged addons in kops and the other addons in kops. Or your favorite addons!

Working through packaging some of these has a lot of intrinsic value, but we also hope to discover areas where there are problems with the tooling and/or with the approach. Hopefully that will seed further interesting mini-projects which can ideally be solved in collaboration with the community, ranging from docs fixes to deeper technical fixes in the machinery.

Some likely areas of investigation:

• How should we package and distribute the base yaml files - e.g. the manifest bundle proposal
• Can we streamline the process further for simple / generic addons? Do they need their own controller?
• Can we address the lack of granularity of RBAC?
• What is the best UX for consuming addons in cluster management tooling?
• What information do we need on the cluster? (e.g. addon discovery )

The addons you build will likely end up being used on every kubernetes cluster! kubeadm and kops are actively working on supporting these operators, and Redhat is also contributing with their background from the Operator SDK / CoreOS / OpenShift.

Proposed mentors: @justinsb and @stealthybox

Users should be able to include arbitrary patches to operator resources.
One use-case would be to mutate the Service to annotate it for an ingress-controller.

@johnsonj is working on implementing Kustomize style patch support:

justinsb#1
kubernetes-sigs/kubebuilder-declarative-pattern#44

We should get the operator side of things merged into #7 either in justin's repo before merge.
We can also just re-open the PR on this repo which unblocks merging #7.

I noticed addon-operators/coredns/pkg/webhook/, but I can't see what it's there for, or why we would need a webhook to install DNS.

per @jwforres comment.

We should replace all references to k8s.io with x-k8s.io to encourage best practices and follow API review guidelines. As we iterate on the individual operators we should look into API review and promotion into k8s.io

/cc @dholbach

This should ideally cover

• quick intro: what's an addon operator
• explaining structure, bare minimum of work
• links to already existing bits
• caveats(?)

The csi-driver-image-populator is a CSI plugin that allows you to mount the contents of a container image as a volume in a container.

Example:

apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
  - name: nginx
    image: nginx:1.13-alpine
    ports:
    - containerPort: 80
    volumeMount:
    - name: data
      mountPath: /usr/share/nginx/html
  volumes:
  - name: data
    csi:
      driver: image.csi.k8s.io
      volumeAttributes:
          image: kfox1111/misc:test

The current driver is built around buildah, which uses the registries.conf configuration to set up a connection to registries.

This creates a dichotomy between connections from the cluster to external registries when pulling images for pods vs. pulling images for volumes. Pod image pulls are configured via the cluster CRI implementation, node, and Pod config, while volume image pulls are only configured via the configuration that buildah understands.

This means that proxy configuration, cert configuration, auth information, and mirror information is not shared unless the CRI implementation also understands registries.conf (currently this is only cri-o)

The goals of this project in decreasing order of importance:

• Use CRI to pull images that will be mounted as volumes
• Make updates to CRI/CSI to make their integration simpler
• Explore options to support OCI-artifacts directly
• Move csi-driver-image-populator in-tree in kube

CRI Endpoint

There is no standard location for the CRI Endpoint - making a CRI-agnostic CSI driver requires providing the endpoint up front:

- mountPath: /var/run/dockershim.sock
  name: cri-socket-dir

or a change to CRI to allow a well-known location (e.g. /var/run/cri.sock)

CRI Storage Location

There is no standard location or directory structure for an image on-disk across CRI implementations. This means that once an image is on a node, there's no standard way to get its contents.

A solution that requires no modifications to CRI (mocked up here) is to:

• Use CRI to start a pod with two containers and a shared volume
• The first container contains a statically-linked version of cp
• The second container is the target container to be used as a volume
• The first container copies the contents of the shared volume out to a standardized location on the node.
• The contents, now on the node, are mounted into the user-requested pod

It would be nice to find an alternative that:

• Doesn't require building or running a statically built cp
• Doesn't require running an image via CRI to get the final output filesystem
• Doesn't require the user to input the storage location / format for their CRI implementation

CRI

The CRI api is defined here

It is worth exploring what changes, if any, would make some of the above goals possible.

OCI Artifacts

Most CRI implementations do not support pulling non-runnable images.

Others

The metadata (image manifest, manifestlist, labels, etc) are all useful information for a consumer of an image, especially for OCI artifacts.

Some ideas:

• CRI updated to include explicit support for non-runnable images
• C*I defined for pulling / unpacking but not running
• Expose metadata (annotations/labels) as data that can be mounted as well
• Options for exposing the full oci manifest / manifestlist as a volume

We have an existing operator focused KEP here: https://github.com/kubernetes/enhancements/blob/master/keps/sig-cluster-lifecycle/addons/0035-20190128-addons-via-operators.md

We should suss out the technical approach for and propose solutions in these areas.
We may need to split out into multiple KEPS and cross-ref between them.

Packaging:

• kustomize packaging improvements /w git, OCI, and Bundles
  • allows packaging base manifests
  • allows packaging patch overlays (overlay use-case pre-packaged CNI configs)
  • ComponentConfig portion to load packages /w overlays from different package refs

Operators:

• kubebuilder declarative pattern for operator development
• kustomize integration using previously mentioned ComponentConfig for bases and overlays
• mention/document singleton pattern through the use of CRD related ResourceQuotas
• minimum docs + potential high-order tooling for creating aggregated Roles/ClusterRoles
• design early-exit /w convergence/timeout and Jobs for low-resource, low-privilege clusters

Management:

• libraries for using kustomize to install/patch things during cluster-lifecycle
• propose shelling out to kubectl apply -k as a dependency

Feature to consider:

The operator could hash the configMap bytes and then update the name by appending the hash to it. The Deployment would then be updated to reference this new name.
This would allow the Deployment with multiple replicas to undergo a proper (and safe) RollingUpdate. (Changing the value of a ConfigMap is a common source of error for config rollouts)

Currently, we are just changing the value of the configmap and relying on the CoreDNS pods to use the reload plugin, but this could be optional.

This mechanism is fully supported by kustomize's configMapGenerator should we integrate kustomize into the operator.

/kind feature

In the first Cluster Addons meeting it was decided to write a addon operator for core-dns. This would be a first concrete step to test the waters, inspire others and get some end-to-end testing mileage out of this.

@justinsb and others (who?) demonstrated interested in this.

Kubebuilder 2.0 is approaching and results in significant change in directory structure.

We should migrate the coredns operator to this (either re-create and retain the custom bits or manually diff?) once 2.0 is released.

Here's an example PR

Description
CoreDNS Operator aims to manage the lifecycle of the CoreDNS installation in the cluster. It is concerned with the installation, updating, and configuration of CoreDNS.

Project Goal
Define the set of milestones and criteria for productionizing the CoreDNS operator. The current operator is a proof of concept of the add-on library. The goal of the alpha operator is to provide a minimal feature set that is manageable, predictable, and reliable in a production environment.

• Issue: Implement CoreDNS operator smoke test.
  Implementing and validating various scenarios like creation, deletion, configuration change, etc.
  This work will require adding multiple versions to the CoreDNS version directory.
• Issue: Provide metrics in the kubebuilder-declarative-pattern library
  A user should be able to emit metrics on the Prometheus metrics endpoint. Metrics should be written to the controller-runtime Prometheus Registry.
• Issue: Integrate Kustomize with kubebuilder-declarative-pattern
  Finish and share design doc.
• Issue: Implement rolling updates for ConfigMap
  Rolling update for Configs should use hashed names (kustomize supports this) Issue

The KEP document and the first Cluster Addons meeting (video coming soon) will answer a lot of the questions new-comers will have. We should compile them and add them to the main README.md.

Each operator in this repo may have a set of unit tests and longer term (eg #36) we expect to have E2E tests.

We need to setup Prow integration for this repo.

This is an issue to request API review of our addon operator schema. We now have an example addon operator for CoreDNS that is our current thinking:
https://github.com/kubernetes-sigs/addon-operators/blob/master/coredns/config/crds/addons_v1alpha1_coredns.yaml

/label api-review

In order for the community to create and maintain addons, we need a shared interface for addon installation and overall lifecycle.

There are a number of considerations regarding packaging, runtime, security, dev tools, ops tools, use-case, and governance.

These concerns are separate from creating operators or other programs concerned with addon specifics such as #3. (ex: How do you install or maintain the coredns-operator?)

This google doc aims to list and weigh various approaches and trade-offs:
https://docs.google.com/document/d/1tpayzZ4teTKOXNUEsA2HK73eWzz9MhZOFTKYz_rRI_8

• Anyone can view and comment
• kubernetes-dev members can edit
• kubernetes-dev members can change permissions
  ( reach out if you are not a member but desire edit access )

This should enable fair, open, async discussion across time-zones.

From this doc, one or more parties may create proposals(KEP's) which can compose or compete.
These new proposals may build on the existing KEP: kubernetes/enhancements#746

Feel free to comment here and/or on the document.

/kind design
/priority important-soon
/sig cluster-lifecycle

Problem
We would like to provide the ability for multiple collaborators (controllers or humans) to add upstream DNS resolvers for DNS zones.

The Corefile is a global resource. This makes it difficult for a custom controller to add its own configuration to the Corefile. A custom controller might have to assume no one else would change the Corefile and overwrite any outside changes, which is not great because it reduces flexibility. Alternatively, a controller could have logic to determine what parts it added, and therefore can change, which isn’t impossible, but complex and potentially brittle e.g a human operator cannot tell if a part of the Corefile was added by a controller and they attempt to change it. Additionally, while the CoreDNS-operator can handle Corefile migrations, a custom controller would still have to be knowledgeable about compatibility of the Corefile if it is going to read and write to it adding additional complexity.

Strawdog Proposal
We propose a new CRD that allows someone/something to declare their intent to forward a DNS zone to a server and the CoreDNS-Operator can generate a server block and append it to the Corefile. This allows the users to not have to deal with managing the entire Corefile themselves (e.g migrating, merging).

This is what we were thinking the CRD would look like:

---
apiVersion: coredns.addons.x-k8s.io/v1alpha1
kind: DNSZone
metadata:
  name: my-company.internal
  namespace: kube-system
spec:
  zoneName: my-company.internal
  forwardTo: 10.100.1.1

The CoreDNS Operator would generate a CoreDNS server block and append it to the Corefile given in the CoreDNS resource.

This is more of a conversation starter and how we are thinking of it, but not tied to this solution.

Alternatives Considered

We considered a more generic “CorefileFragment” CRD that would take a CoreDNS server block to append to the Corefile. However, then the author of the “fragment” is responsible for migrations, which we’re trying to avoid.

We also considered adding a new field to the CoreDNS CRD that could take in the zones. Perhaps something similar to what RedHat’s DNS Operator does. This would then generate server blocks for each zone and append them to the contents of the corefile field in CoreDNS. I foresee a similar issue where modifying the Corefile field directly with custom controllers can result in update conflicts, but that isn’t impossible to overcome.

Tagging @rajansandeep, from what I understand you maintain CoreDNS-Operator and it would be good to get your feedback.

cc: @neolit123 @ncdc

Description:-
(GSOC proposal 2020)
We have some new features to be added this time to the CoreDNS operator.The current operator is a proof of concept of the add-on library.

Goal:-
Define the set of milestones and criteria for productionizing the CoreDNS operator. The current operator is a proof of concept of the add-on library. The goal of the alpha operator is to provide a minimal feature set that is manageable, predictable, and reliable in a production environment.

• Implement CoreDNS - operator smoke test.
  Implementing and validating various scenarios like creation, deletion, configuration change, etc.
  This work will require adding multiple versions to the CoreDNS version directory. Issue
• Provide metrics in the kubebuilder-declarative-pattern library
  A user should be able to emit metrics on the Prometheus metrics endpoint. Metrics should be written to the controller-runtime Prometheus Registry. Issue
• Integrate Kustomize with kubebuilder-declarative-pattern
  Finish and share design doc. Issue
• Implement rolling updates for ConfigMap
  Rolling update for Configs should use hashed names (kustomize supports this) Issue

Proposed mentors: Jeff Johnson [ @johnsonj ]

/assign @dholbach

• Review options again (don't clash with other SIG meetings)
• Set up doodle
• Ask for objections on mailing list: https://groups.google.com/forum/#!topic/kubernetes-sig-cluster-lifecycle/VXrokx1Kq5U
• Announce change on mailing list and Slack
• Document in README.md and community repo: #5 (comment)
• Get calendar invite updated

Discussed on the addon-operators call, the group would like to see a standalone binary from this repo capable of using some ComponentConfig to prepare several kustomize directories for application to the cluster.

This binary is a POC that can be used to implement installer/management functionality into tools like kops/kubeadm/eksctl as well as many other tools that went unmentioned/unrepresented.

The decision on kustomize is heavily inspired by the ability to overlay arbitrary patches without requiring deps external to the k8s org.
It is also already built into kubectl which helps simplify the POC.

We expect to shell out to kubectl for simplicity of maintenance as its sophisticated apply behavior cannot easily be replicated at this point in time. (see serviside-apply)

The config should allow users to select a kustomize base for installing their operator.
CoreDNS operator is a good starting point.
Users should be able to specify versioned/packaged patch sets.

This is accomplished with a package ref that can start by loading the kustomize base/overlay from a git ref or local file-system location.
Further work may allow loading from other sources.

/kind feature

A follow-up work item from #25 is adding config validation tests.

A follow-up work item from #25 is adding support for inline patches.

@justinsb is working on a self-contained using the previoulsy vetted CoreDNS DaemonSet.

The node intercept and eviction-fallback mechanisms are not completely general, but the immediate goal is to ship variants of the addon that are useful for cluster-owners on AWS where DNS latency can be very problematic. Similar issues are faced at scale in other environments. (TODO: link other issues)

Creating an operator for managing this and packaging it with kustomize for shipping the different variants should allow this to compose with the addon-installer library for clients like kops and kubeadm (kubernetes/kubeadm#1220).

/assign @justinsb
/kind feature
/sig cluster-lifecycle
/sig network

Add the ability for users using the CoreDNS operator to enable the DNS autoscaler (it would be an optional feature).

I have opened the issue to check whether this would be feasible.

/kind feature

A follow-up work item from #25 is adding prune logic.