Application Status
Objective
The objective of this proposal is to provide a mechanism to aggregate the status of an Application. We propose a mechanism to compute the readiness, availability, errors, and disruptions associated with an Application. As black and white box health monitoring are a complicated topic that deserves its own treatment, we do not address it in this proposal.
Background
- We have decided that status should not computed from phases. Phases are deprecated since state-machine enumerations are antithetic to the level-triggered world of k8s. See the API conventions and this discussion .
- Based on the above conventions, and on previous community discussions in the main repository, we use conditions (see 51594) and fields in the
.status
of a resource to communicate information about its readiness. This allows existing CRDs to opt into the scheme without breaking compatibility with existing tooling and to evolve to use fields. Additionally, it provides a mechanism to provide additional, human readable, information with respect to the status of an Application's components.
Conditions
Conditions are used across the Kubernetes API surface in order to indicate the condition of a resource as its controllerseeks to realize the declared intent in its specification. They are described by the golang struct below. Throughout this proposal we use conditions in conjunction with fields.
type Condition struct {
// Type is the type of the condition.
Type ComponentConditionType `json:"type"`
// Status is one of True, False, Unknown.
Status v1.ConditionStatus `json:"status"`
// LastTransitionTime is the last time the condition's status changed.
LastTransitionTime metav1.Time `json:"lastTransitionTime,omitempty"`
// Reason is the reason for the condition's last transition.
Reason string `json:"reason,omitempty"`
// Message is a human readable message indicating details about the transition.
Message string `json:"message,omitempty"`
}
Readiness
From the perspective of Pods, readiness indicates the ability of the Pod to receive network traffic. It also indicates that the resource, from the perspective of the control loops that act on it, is ready for use. We use the same semantics for the components of an Application. Readiness for an Application implies that all of its components are ready. That is, an Application is ready if and only if all of its components are ready. Components that contain no user declared desired state (i.e. have no spec) (e.g. ConfigMaps and Secrets) are always ready post creation.
- A controller MAY communicate that a resource is ready by indicating
status.ready=true
or by including a condition like {"type":"Ready","status":"true"}
in the resource's status.conditions
.
- A controller MAY communicate that a resource is unready by indicating
status.ready=false
or by including a condition like {"type":"Ready","status":"false"}
in the resource's status.conditions
.
- If a controller communicates conflicting readiness (e.g setting
status.ready=true
and including a condition like {"type":"Ready","status":"false"}
) the value of the field takes precedence.
- If a controller uses both a status field and a condition to communicate readiness, and if the field and condition are consistent, the condition is treated as a decorator for the field (e.g setting
status.ready=true
and including a condition like {"type":"Ready","status":"false","message":"RDBMs is ready for use."}
provides a user readable message about the readiness of the resource).
Availability
From the perspective of Deployments, and many out of tree resources, availability, indicates that all Pods in the related to the resource remain ready after some configurable duration. This notion is useful for application components in general as indication that the resource is unlikely to fall victim to infant mortality after creation or mutation. We use availability for an Application's components in this context. As availability is not applicable to all components, it is not aggregated for an Application.
- A controller MAY communicate that a resource is available by indicating
status.available=true
or by including a condition like {"type":"Available","status":"true"}
in the resource's status.conditions
.
- A controller MAY communicate that a resource is unavailable by indicating
status.available=false
or by including a condition like {"type":"Available","status":"false"}
in the resource's status.conditions
.
- If a controller communicates conflicting availability (e.g setting
status.availabile=true
and including a condition like {"type":"Available","status":"false"}
) the value of the field takes precedence.
- If a controller uses both a status field and a condition to communicate availability, and if the field and condition are consistent, the condition is treated as a decorator for the field.
Observation
Kubernetes control loops communicate that they have observed modifications of the declared desired state contained in a resource by setting its status.observedGeneration
to its generation
. A resource for which this is true is said to be observed.
- Controllers SHOULD update
status.observedGeneration
to the value of meta.generantion
to communicate that they have observed the creation of, or a mutation to, a resource they control.
Progress
Kubernetes control loop use various methods to communicate that reconciliation between a resources specification and the observed state of the system is progressing. Deployments, and many non-core resources, communicate this using the Progressing
condition. For an Application, progressing components indicate that the application is updating.
- A controller MAY communicate that reconciliation is in progress by indicating setting
status.progress
to a true or to a non-negative 32-bit floating point number between in [0,100] (e.g status.progress=true
or status.progress=99.9
).
- A controller MAY communicate that reconciliation is in including a condition like
{"type":"Progressing","status":"true"}
in the resource's status.conditions
.
- If a controller communicates conflicting progress (e.g setting
status.progressing=true
and including a condition like {"type":"Progressing","status":"false"}
) the value of the field takes precedence.
- If a controller uses both a status field and a condition to communicate readiness, and if the field and condition are consistent, the condition is treated as a decorator for the field.
Disruptions
Application components may be affected by planned or unplanned disruptions. For instance the destruction of a Node may disrupt many replicated Pod sets. The application controller MAY use other resources, e.g PodDisruptionBudgets, to add this condition to a resource's status, and the controller for a resource MAY communicate this directly by adding such a condition.
- A controller MAY communicate disruptions using by adding a condition like
{"type":"Disruption","status":"true","reason":"Node unavailable","message":"Auto-scaling in progress"}
.
- Disruptions are considered to be orthogonal to readiness for the computation of Application readiness. If a disruption makes a resource unready the resources controller MUST communicate this via readiness.
Errors
At any point in their lifetime controller may encounter errors when realizing the declared intent of the user. The are communicating using the status.conditions
of the resource.
- A controller MAY communicate an error by including a condition like
{"type":"Error","status":"true","reason":"Controller Wedged","message":"SharedInformer sycn failing."}
.
- Errors are considered to be orthogonal to readiness for the computation of Application readiness. If an error makes a resource unready the resources controller MUST communicate this via readiness.
Compatibility Requirements
Resources and controllers that wish to be compatible with the Application Controller status computation need only implement the following.
- Any resource that does not implement a
.spec
contains no declarative intent. It is always ready.
- The representative resource MUST implement a
.status
field.
- The
.status
field MUST indicate readiness.
- The
.status
field MAY indicate availability.
- The
.status
field SHOULD indicate observation by the controller.
- The
.status
field MAY contain conditions.
- Errors MAY be reported using error conditions in the
status.conditions
field.
- Disruptions MAY be reported using disruption conditions in the
status.conditions
field.
Core Resource Adaptations
The core resources do not all conform to the schema above. In the future, we may modify them to do so. For the time being, the following describes how the Application controller will compute the status of these resources.
- Deployment
- Readiness -
.spec.replicas
is equal to .status.readyReplicas
is equal to .status.replicas
and all are greater than zero.
- Availability -
.status.conditions
contains an Available
condition.
- Progress - true if
spec.conditions
contains a Progressing
condition.
- Observed - when
.status.observedGeneration
is equal to spec.generation
.
- Errors -
Failure
conditions are converted to Error
conditions.
- ReplicaSet and ReplicationController
- You should not use these directly in an Application. Deployment should be used instead.
- Readiness -
.spec.replicas
is equal to .status.replicas
and .status.readyReplicas
and all are greater than zero.
- Availability -
.spec.replicas
is equal to .status.replicas
and .status.availableReplicas
and all are greater than zero.
- Progress - true if
.spec.replicas
is not equal to .status.replicas
.
- Observed - when
.status.obloadbalancerservedGeneration
is equal to spec.generation
.
- Errors -
ReplicaFailure
conditions will be converted to Error
conditions.
- StatefulSet
- Readiness -
.spec.replicas
is equal to .status.replicas
and .status.readyReplicas
and all are greater than zero.
- Availability - N/A, StatefulSet Pods are available when ready.
- Progress - true if
.status.currentReplicas
is not equal to status.updateReplicas
or if
.status.replicas
is not equal to .spec.replicas
.
- Observed - when
.status.observedGeneration
is equal to spec.generation
.
- Errors - N/A
- DaemonSet
- Readiness -
.status.currentNumberScheduled
is equal to .status.desiredNumberScheduled
and .status.numberReady
and all are greater than 0.
- Availability - if
.status.currentNumberScheduled
is equal to .status.desiredNumberScheduled
and .status.numberAvailable
and all are greater than 0.
- Progress - DaemonSet is sensitive to Node cardinality. It is making progress when
status.numberUpdated
is greater than 0.
- Observed - when
.status.observedGeneration
is equal to spec.generation
.
- Errors - N/A
- Pod
- Readiness - The Pods has a
Ready
condition.
- Availability - N/A
- Progress - N/A
- Observed - N/A
- Errors - N/A
- Ingress
- Readiness -
.status.loadbalancer.ingress
list is not empty.
- Availability - N/A
- Progress - N/A
- Observed - N/A
- Errors - N/A
- Service
- Readiness - All non-load balanced Services are ready when created. All load balanced Services are ready when
.status.loadbalancer.ingerss
is not empty.
- Availability - N/A
- Progress - N/A
- Observed - N/A
- Errors - N/A
- PersistentVolumeClaim
- A PersistentVolumeClaim claim is
Ready
when its .status.phase
is Bound
. This may seem strange as PVC implements a Ready
phase, but a PVC is not useful to the application until it is bound, and most errors post creation and during binding.
- Availability - N/A
- Progress - N/A
- Observed - N/A
- Errors - N/A
API
This section contains the proposed modifications to the API. Here, we modify the ApplicationStatus type to report the observed status of its components. Each ComponetStatus
contains a link, resource identifying information, and the ComponentConditions
of the components indicated by the Application's .Spec.ComponentKinds
and .Spec.Selector
. The status of the applications components is used to compute ApplicationConditions
that apply to the application as a whole.
// ComponentConditionType indicates the type of a ComponentCondition.
type ComponentConditionType string
const (
// ComponentAvailable indicates that the component is available. This is used by controller to indicate that the
// resource has been ready for a sufficient period of time after creation or mutation that it is unlikely to suffer
// from infant mortality.
ComponentAvailable ComponentConditionType = "Available"
// ComponentReady indicates that component is ready to use.
ComponentReady = "Ready"
// ComponentProgressing is used to communicate that a component is in updating (i.e. A Deployment with a Rollout in
// progress or a StatefulSet with a RollingUpdate in progress).
ComponentProgressing = "Progressing"
// ComponentDisrupted is used to indicate that the component is affected by a planned or unplanned disruption.
ComponentDisrupted = "Disrupted"
// Error is used to communicate and error condition for a component.
ComponentError = "Error"
)
// ComponentCondition represents the condition of a component of an application. It is modeled after the Conditions
// use for the Kubernetes workloads objects.
type ComponentCondition struct {
// Type is the type of the condition.
Type ComponentConditionType `json:"type"`
// Status is one of True, False, Unknown.
Status v1.ConditionStatus `json:"status"`
// LastTransitionTime is the last time the condition's status changed.
LastTransitionTime metav1.Time `json:"lastTransitionTime,omitempty"`
// Reason is the reason for the condition's last transition.
Reason string `json:"reason,omitempty"`
// Message is a human readable message indicating details about the transition.
Message string `json:"message,omitempty"`
}
// ComponentStatus contains the status of a generic component selected by the Application.
type ComponentStatus struct {
// Name is the name of the component to which the condition pertains.
Name string `json:"name"` ComponentConditionTyp
// Namespace is the name containing the component to which the condition pertains.
Namespace string `json:"namespace,omitempty"`
// GroupKind indicates the group and kind of the component.
GroupKind metav1.GroupKind `json:groupKind"`
// Link is a link to the resource that represents the component.
Link string `json:"link,omitempty"`
// Available indicates that the component is available. This is used by controller to indicate that the
// resource has been ready for a sufficient period of time after creation or mutation that it is unlikely to suffer
// from infant mortality.
Available *bool `json:"available,omitempty"`
// tReady indicates that component is ready to use.
Ready *bool `json:"ready,omitempty"`
// Progressing is used to communicate that a component is in updating (i.e. A Deployment with a Rollout in
// progress or a StatefulSet with a RollingUpdate in progress).
Progressing *bool `json:"progressing,bool"`
// Observed indicates that the controller for the component's resource has observed the current generation.
// (i.e .Meta.Generation == .Status.ObservedGeneration)
Observed bool = `json:"observed,omitempty"`
// Conditions contains the conditions that are applicable to the component.
Conditions [] ComponentCondition `json:"conditions,omitempty"`
}
// ApplicationConditionType indicates the type of an application condition.
type ApplicationConditionType string
const (
// ApplicationError is used to communicate that an Application has an Error condition.
ApplicationError ApplicationConditionType = "Error"
)
// ApplicationCondition represents the condition of an Application.
type ApplicationCondition struct {
// Type is the type of the condition.
Type ApplicationConditionType `json:"type"`
// Status is one of True, False, Unknown.
Status v1.ConditionStatus `json:"status"`
// LastTransitionTime is the last time the condition's status changed.
LastTransitionTime metav1.Time `json:"lastTransitionTime,omitempty"`
// Reason is the reason for the condition's last transition.
Reason string `json:"reason,omitempty"`
// Message is a human readable message indicating details about the transition.
Message string `json:"message,omitempty"`
}
type ApplicationStatus struct {
// Ready indicates that all of an Application's components are Available.
Ready *bool `json:"available,omitempty"`
// Updating is used to communicate that a component is in updating (i.e. A Deployment with a Rollout in
// progress or a StatefulSet with a RollingUpdate in progress.
Updating *bool `json:"progressing,bool"`
// Conditions is a list of ApplicationConditions for the application.
Conditions [] ApplicationCondition
//ComponentStatus is a list of the statues of the applications components.
Components [] ComponentStatus
}
Application Status Computation
The Application controller will periodically list the applications residing on the API Server. For each Application resource the controller will do the following.
- If the
spec.assemblyPhase
of the Application is pending the controller will not update the .status
of the Application. This allows application installers time apply all necessary components prior to application status computation.
- If the Application is assembled, the controller will do the following for each GroupKind indicated by the
Application's .spec.componentKinds
.
- Use the discovery API to get the default version of the resource.
- Retrieve the resource from the API Server.
- Determine the readiness, availability, progress and observed status of each component with respect to the core resource adaptations and use this create a corresponding ComponentStatus and to append it to the
.status.components
of the Application.
- If one or more components of the Application are
Progressing
the status.updating
field is set to true.
- If the Application is assembled and all of its components are ready the
status.ready
field is set to true.
- If the Application is assembled and any of its components are not ready the
status.ready
field is set to false.
- The Application controller will then update the Application to communicate the status to the end user.
Example
apiVersion: app.k8s.io/v1beta1
kind: Application
metadata:
name: "wordpress-01"
labels:
app.kubernetes.io/name: "wordpress-01"
spec:
type: "wordpress"
selector:
matchLabels:
app.kubernetes.io/name: "wordpress-01"
componentKinds:
- group: core
kind: Service
- group: apps
kind: StatefulSet
version: "4.9.4"
description: "WordPress is open source software you can use to create a beautiful website, blog, or app."
icons:
- src: "https://s.w.org/style/images/about/WordPress-logotype-wmark.png"
type: "image/png"
size: "1000x1000"
- src: "https://s.w.org/style/images/about/WordPress-logotype-standard.png"
type: "image/png"
size: "2000x680"
maintainers:
- name: Kenneth Owens
email: [email protected]
owners:
- name: Kenneth Owens
email: [email protected]
keywords:
- "cms"
- "blog"
- "wordpress"
links:
- description: About
url: "https://wordpress.org/"
- description: Web Server Dashboard
url: "https://metrics/internal/wordpress-01/web-app"
- description: Mysql Dashboard
url: "https://metrics/internal/wordpress-01/mysql"
status:
ready: true
updating: false
components:
- name: wordpress-mysql-hvc
namespace: default
groupKind: Service
link: /apis/v1/namespaces/default/services/wordpress-mysql-hvc
ready: true
- name: wordpress-mysql
namespace: default
groupKind: apps/StatefulSet
link: /apis/apps/v1/namespaces/default/statefulsets/wordpress-mysql
ready: true
avialable: true
- name: wordpress-webserver-svc
namespace: default
groupKind: Service
link: /apis/v1/namespaces/default/services/wordpress-webserver-svc
ready: true
- name: wordpress-webserver
namespace: default
groupKind: apps/StatefulSet
link: /apis/apps/v1/namespaces/default/statefulsets/wordpress-webserver
ready: true
avialable: true