kubernetes-sigs / cluster-addons Goto Github PK
View Code? Open in Web Editor NEWAddon operators for Kubernetes clusters.
License: Apache License 2.0
Addon operators for Kubernetes clusters.
License: Apache License 2.0
The KEP document and the first Cluster Addons meeting (video coming soon) will answer a lot of the questions new-comers will have. We should compile them and add them to the main README.md.
In the first Cluster Addons meeting it was decided to write a addon operator for core-dns. This would be a first concrete step to test the waters, inspire others and get some end-to-end testing mileage out of this.
@justinsb and others (who?) demonstrated interested in this.
Kubebuilder 2.0 is approaching and results in significant change in directory structure.
We should migrate the coredns operator to this (either re-create and retain the custom bits or manually diff?) once 2.0 is released.
Here's an example PR
(Tracking issue for GSoC proposal)
Now that we're making good progress on the various pieces of machinery and integrations for cluster-addons, it's feasible to start a real effort to create addon operators for the most popular and most interesting cluster addons. Good candidates would be the addons in the (deprecated) cluster directory of k/k, the packaged addons in kops and the other addons in kops. Or your favorite addons!
Working through packaging some of these has a lot of intrinsic value, but we also hope to discover areas where there are problems with the tooling and/or with the approach. Hopefully that will seed further interesting mini-projects which can ideally be solved in collaboration with the community, ranging from docs fixes to deeper technical fixes in the machinery.
Some likely areas of investigation:
The addons you build will likely end up being used on every kubernetes cluster! kubeadm and kops are actively working on supporting these operators, and Redhat is also contributing with their background from the Operator SDK / CoreOS / OpenShift.
Proposed mentors: @justinsb and @stealthybox
I noticed addon-operators/coredns/pkg/webhook/
, but I can't see what it's there for, or why we would need a webhook to install DNS.
Referenced in #14, we would like installers to be able to load kustomize packages.
We also expect that addon-operators can benefit from this ComponentConfig work and packaging.
This implies improvements for packaging and distributing these kustomize bases and overlays.
git support with tags/refs and nested dirs already seems to be built into kustomize which is a great starting point.
Some good things to work on:
Discussed on the addon-operators call, the group would like to see a standalone binary from this repo capable of using some ComponentConfig to prepare several kustomize directories for application to the cluster.
This binary is a POC that can be used to implement installer/management functionality into tools like kops/kubeadm/eksctl as well as many other tools that went unmentioned/unrepresented.
The decision on kustomize is heavily inspired by the ability to overlay arbitrary patches without requiring deps external to the k8s org.
It is also already built into kubectl which helps simplify the POC.
We expect to shell out to kubectl for simplicity of maintenance as its sophisticated apply behavior cannot easily be replicated at this point in time. (see serviside-apply)
The config should allow users to select a kustomize base for installing their operator.
CoreDNS operator is a good starting point.
Users should be able to specify versioned/packaged patch sets.
This is accomplished with a package ref that can start by loading the kustomize base/overlay from a git ref or local file-system location.
Further work may allow loading from other sources.
/kind feature
In a conversation with @stealthybox we realised it might be good to add a project board to make it easier to spot what still needs to be done and who does what.
Add the ability for users using the CoreDNS operator to enable the DNS autoscaler (it would be an optional feature).
I have opened the issue to check whether this would be feasible.
/kind feature
A follow-up work item from #25 is adding prune logic.
https://github.com/kubernetes-sigs/cluster-addons/blob/master/bootstrap/app/addon-manager.go#L73 references v1beta1 cronjobs which will no longer be served in 1.25+
A follow-up work item from #25 is adding support for inline patches.
This is an issue to request API review of our addon operator schema. We now have an example addon operator for CoreDNS that is our current thinking:
https://github.com/kubernetes-sigs/addon-operators/blob/master/coredns/config/crds/addons_v1alpha1_coredns.yaml
/label api-review
The csi-driver-image-populator is a CSI plugin that allows you to mount the contents of a container image as a volume in a container.
Example:
apiVersion: v1
kind: Pod
metadata:
name: nginx
spec:
containers:
- name: nginx
image: nginx:1.13-alpine
ports:
- containerPort: 80
volumeMount:
- name: data
mountPath: /usr/share/nginx/html
volumes:
- name: data
csi:
driver: image.csi.k8s.io
volumeAttributes:
image: kfox1111/misc:test
The current driver is built around buildah, which uses the registries.conf
configuration to set up a connection to registries.
This creates a dichotomy between connections from the cluster to external registries when pulling images for pods vs. pulling images for volumes. Pod image pulls are configured via the cluster CRI implementation, node, and Pod config, while volume image pulls are only configured via the configuration that buildah understands.
This means that proxy configuration, cert configuration, auth information, and mirror information is not shared unless the CRI implementation also understands registries.conf
(currently this is only cri-o
)
The goals of this project in decreasing order of importance:
There is no standard location for the CRI Endpoint - making a CRI-agnostic CSI driver requires providing the endpoint up front:
- mountPath: /var/run/dockershim.sock
name: cri-socket-dir
or a change to CRI to allow a well-known location (e.g. /var/run/cri.sock
)
There is no standard location or directory structure for an image on-disk across CRI implementations. This means that once an image is on a node, there's no standard way to get its contents.
A solution that requires no modifications to CRI (mocked up here) is to:
cp
It would be nice to find an alternative that:
cp
The CRI api is defined here
It is worth exploring what changes, if any, would make some of the above goals possible.
Most CRI implementations do not support pulling non-runnable images.
The metadata (image manifest, manifestlist, labels, etc) are all useful information for a consumer of an image, especially for OCI artifacts.
Some ideas:
Allow users to easily provision a cluster with KubeVirt enabled. This would allow users provisioning there clusters with kubeadm who need to run VMs in containers and easy way to set up their cluster and be able to run their applications from day 1.
The addon would then deploy the manifest for the kubevirt operator, and then kubevirt CR to deploy the kubevirt components.
@justinsb is working on a self-contained using the previoulsy vetted CoreDNS DaemonSet.
The node intercept and eviction-fallback mechanisms are not completely general, but the immediate goal is to ship variants of the addon that are useful for cluster-owners on AWS where DNS latency can be very problematic. Similar issues are faced at scale in other environments. (TODO: link other issues)
Creating an operator for managing this and packaging it with kustomize for shipping the different variants should allow this to compose with the addon-installer library for clients like kops
and kubeadm
(kubernetes/kubeadm#1220).
/assign @justinsb
/kind feature
/sig cluster-lifecycle
/sig network
Create an end-to-end smoketest for coredns-operator. The test should run locally by default (via kind or kinder) and support running on a remote kubernetes cluster when provided.
The test should implement and validate the following scenarios:
This work will require adding multiple versions to the CoreDNS version directory.
Implementation Nodes:
/cc @annp1987
Description:-
(GSOC proposal 2020)
We have some new features to be added this time to the CoreDNS operator.The current operator is a proof of concept of the add-on library.
Goal:-
Define the set of milestones and criteria for productionizing the CoreDNS operator. The current operator is a proof of concept of the add-on library. The goal of the alpha operator is to provide a minimal feature set that is manageable, predictable, and reliable in a production environment.
Proposed mentors: Jeff Johnson [ @johnsonj ]
Current coredns operator doesn't have tests.
According to here, It is better to add "Golden File Tests".
Problem
We would like to provide the ability for multiple collaborators (controllers or humans) to add upstream DNS resolvers for DNS zones.
The Corefile is a global resource. This makes it difficult for a custom controller to add its own configuration to the Corefile. A custom controller might have to assume no one else would change the Corefile and overwrite any outside changes, which is not great because it reduces flexibility. Alternatively, a controller could have logic to determine what parts it added, and therefore can change, which isn’t impossible, but complex and potentially brittle e.g a human operator cannot tell if a part of the Corefile was added by a controller and they attempt to change it. Additionally, while the CoreDNS-operator can handle Corefile migrations, a custom controller would still have to be knowledgeable about compatibility of the Corefile if it is going to read and write to it adding additional complexity.
Strawdog Proposal
We propose a new CRD that allows someone/something to declare their intent to forward a DNS zone to a server and the CoreDNS-Operator can generate a server block and append it to the Corefile. This allows the users to not have to deal with managing the entire Corefile themselves (e.g migrating, merging).
This is what we were thinking the CRD would look like:
---
apiVersion: coredns.addons.x-k8s.io/v1alpha1
kind: DNSZone
metadata:
name: my-company.internal
namespace: kube-system
spec:
zoneName: my-company.internal
forwardTo: 10.100.1.1
The CoreDNS Operator would generate a CoreDNS server block and append it to the Corefile given in the CoreDNS resource.
This is more of a conversation starter and how we are thinking of it, but not tied to this solution.
Alternatives Considered
We considered a more generic “CorefileFragment” CRD that would take a CoreDNS server block to append to the Corefile. However, then the author of the “fragment” is responsible for migrations, which we’re trying to avoid.
We also considered adding a new field to the CoreDNS CRD that could take in the zones. Perhaps something similar to what RedHat’s DNS Operator does. This would then generate server blocks for each zone and append them to the contents of the corefile field in CoreDNS. I foresee a similar issue where modifying the Corefile field directly with custom controllers can result in update conflicts, but that isn’t impossible to overcome.
Tagging @rajansandeep, from what I understand you maintain CoreDNS-Operator and it would be good to get your feedback.
cc: @neolit123 @ncdc
Description
CoreDNS Operator aims to manage the lifecycle of the CoreDNS installation in the cluster. It is concerned with the installation, updating, and configuration of CoreDNS.
Project Goal
Define the set of milestones and criteria for productionizing the CoreDNS operator. The current operator is a proof of concept of the add-on library. The goal of the alpha operator is to provide a minimal feature set that is manageable, predictable, and reliable in a production environment.
We have an existing operator focused KEP here: https://github.com/kubernetes/enhancements/blob/master/keps/sig-cluster-lifecycle/addons/0035-20190128-addons-via-operators.md
We should suss out the technical approach for and propose solutions in these areas.
We may need to split out into multiple KEPS and cross-ref between them.
Packaging:
Operators:
Management:
kubectl apply -k
as a dependency/assign @dholbach
This should ideally cover
In order for the community to create and maintain addons, we need a shared interface for addon installation and overall lifecycle.
There are a number of considerations regarding packaging, runtime, security, dev tools, ops tools, use-case, and governance.
These concerns are separate from creating operators or other programs concerned with addon specifics such as #3. (ex: How do you install or maintain the coredns-operator?)
This google doc aims to list and weigh various approaches and trade-offs:
https://docs.google.com/document/d/1tpayzZ4teTKOXNUEsA2HK73eWzz9MhZOFTKYz_rRI_8
This should enable fair, open, async discussion across time-zones.
From this doc, one or more parties may create proposals(KEP's) which can compose or compete.
These new proposals may build on the existing KEP: kubernetes/enhancements#746
Feel free to comment here and/or on the document.
/kind design
/priority important-soon
/sig cluster-lifecycle
Feature to consider:
The operator could hash the configMap bytes and then update the name by appending the hash to it. The Deployment would then be updated to reference this new name.
This would allow the Deployment with multiple replicas to undergo a proper (and safe) RollingUpdate. (Changing the value of a ConfigMap is a common source of error for config rollouts)
Currently, we are just changing the value of the configmap and relying on the CoreDNS pods to use the reload
plugin, but this could be optional.
This mechanism is fully supported by kustomize's configMapGenerator
should we integrate kustomize into the operator.
/kind feature
A follow-up work item from #25 is adding config validation tests.
Users should be able to include arbitrary patches to operator resources.
One use-case would be to mutate the Service to annotate it for an ingress-controller.
@johnsonj is working on implementing Kustomize style patch support:
justinsb#1
kubernetes-sigs/kubebuilder-declarative-pattern#44
We should get the operator side of things merged into #7 either in justin's repo before merge.
We can also just re-open the PR on this repo which unblocks merging #7.
Each operator in this repo may have a set of unit tests and longer term (eg #36) we expect to have E2E tests.
We need to setup Prow integration for this repo.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.