projectsyn / component-backup-k8up Goto Github PK

In https://github.com/projectsyn/component-backup-k8up/blob/master/class/backup-k8up.yml#L9 the chart version is used to retrieve the versioned CRD of k8up. Since the chart version can be different from the app version this can result in version conflicts.

Steps to Reproduce the Problem

1. Configure backup_k8up:charts:k8up as 1.0.5
2. Compile catalog

Actual Behavior

Error:

404 Client Error: Not Found for url: https://github.com/vshn/k8up/releases/download/v1.0.5/k8up-crd.yaml

Expected Behavior

The CRD is being retrieved with the app version instead of the chart version

This issue provides visibility into Renovate updates and their statuses. Learn more

This repository currently has no open or pending branches.

• Check this box to trigger a request for Renovate to run again on this repository

Context

Currently the alert filtering rules are only applied to the alert for metric baas_backup_restic_last_errors. However, the component also configures a number of alerts for k8up_* metrics.

To improve consistency, the alert filtering rules should also be applied to the alerts based on k8up_* metrics.

Note that the namespace for the actual backup is exposed as label exported_namespace for the k8up_* metrics, while label namespace always refers to the namespace in which K8up runs.

Alternatives

The alternative is to leave the setup as is, which will lead to inconsistent alerting.

Steps to Reproduce the Problem

1. Cluster without a push gateway available

2. parameters:
     backup_k8up:
       prometheus_push_gateway: ''
       monitoring_enabled: false

Actual Behavior

Wrestic compains about push gateway not accessible

E1016 11:15:32.306402       1 backup.go:145] wrestic "msg"="prometheus send failed" "error"="Post \"http://127.0.0.1/metrics/job/restic_backup/instance/syn-cluster-backup\": dial tcp 127.0.0.1:80: connect: connection refused"  

Expected Behavior

Wrestic does not complain about missing push gateway.

When overwriting parameter images.k8up.registry with a reference to a hierarchy parameter which itself has been set more than once, resolving the reference ${backup_k8up:_backupImageRepository:${backup_k8up:majorVersion}} in the Helm values results in

k8up:
  backupImage:
    repository: <reclass.values.valuelist.ValueList object at 0x7f657281b9a0>/k8up-io/k8up

instead of the overriden parameter value.

Steps to Reproduce the Problem

Text form

1. Set global.registries.quay=quay.io in the hierarchy
2. Overwrite global.registries.quay=quay-mirror.example.com in a different class
3. Overwrite the k8up registry with a reference to the multi-valued global.registries.quay: registry: ${global:registries:quay}
4. Run kapitan inventory
5. Observe that you get a <reclass.values.valuelist.ValueList > reference in the YAML instead of the resolved reference

Code form

1. Create a minimal test case (this assumes that you're in the component root directory)

mkdir -p inventory/classes/defaults
mkdir -p inventory/classes/components
mkdir -p inventory/classes/global
mkdir -p inventory/classes/tenant
mkdir -p inventory/targets

cat > inventory/classes/global/common.yml <<EOF
parameters:
  global:
    registries:
      quay: quay.io
  cluster:
    name: c-cluster-id-1234
    tenant: t-tenant-id-1234
EOF

cat > inventory/classes/tenant/cluster.yml <<EOF
parameters:
  global:
    registries:
      quay: quay-mirror.example.com

  backup_k8up:
    images:
      k8up:
        registry: \${global:registries:quay}
EOF

cat > inventory/classes/global/commodore.yml <<EOF
classes:
- global.common
- tenant.cluster
EOF

cat > inventory/targets/backup-k8up.yml <<EOF
classes:
- defaults.backup-k8up
- global.commodore
- components.backup-k8up
parameters:
  _instance: backup-k8up
  _base_directory: /tmp/backup-k8up # won't actually compile, but required for kapitan inventory
EOF

ln -s ${PWD}/class/defaults.yml inventory/classes/defaults/backup-k8up.yml
ln -s ${PWD}/class/backup-k8up.yml inventory/classes/components/backup-k8up.yml

2. Render the inventory

kapitan inventory -t backup-k8up

Actual Behavior

Kapitan renders the following for backup_k8up.helmValues using the minimal test case above:

helmValues:
  k8up:
    backupImage:
      repository: <reclass.values.valuelist.ValueList object at 0x7f64eb3f25b0>/k8up-io/k8up
      tag: v2.5.1

Expected Behavior

Kapitan renders the following for backup_k8up.helmValues:

helmValues:
  k8up:
    backupImage:
      repository: quay-mirror.example.com/k8up-io/k8up
      tag: v2.5.1

Initial debugging notes

• The issue can be worked around by directly setting helmValues.k8up.backupImage.repository=${global:registries:quay}/k8up-io/k8up in cluster.yml of the minimal example

Context

The current format of the configuration is the following

parameters:
  backup_k8up:
    alert_rule_filters:
      namespace: namespace=~"syn.*"

However, while the key in alert_rule_filters is called namespace, nothing stops users from using different labels to filter alerts.

Proposal

This issue proposes an alternative structure for the field parameters.backup_k8up.alert_rule_filters with the following properties:

1. Keys in alert_rule_filters are used to indicate the metric label on which to apply the filter
2. Each key in alert_rule_filters holds a dict with one or more of the keys match, match_re, exclude and exclude_re.
3. The match/exclude keys are mapped to filtering operations =, =~, !=, and !~ respectively
4. The value of each of the match/exclude keys is used as the value for the corresponding expression and wrapped in double quotes.

An example might look as follows:

parameters:
  backup_k8up:
    alert_rule_filters:
      namespace: 
        match_re: "syn.*"
        exclude_re: ".*(demo|dev).*"

Alternatives

We could go with an array-based approach, but removing array elements in a reclass hierarchy is much trickier than overwriting dict keys.

In projectsyn/commodore#192, some parameter values were deprecated. This component is affected by this change and needs to adapt. See the commodore change log for the details.

Context

K8up 2.0 contains breaking changes. Adapt the component to support K8up 2.0 and provide an upgrade path.

• https://k8up.io/k8up/explanations/what-has-changed-in-v2.html

Alternatives

None

Context

Upon upgrading from v2.1.4 to v2.2.0 of the component, we realized that none of our failed job alerts seemed to be working as intended. After some investigation, we found that labels are not scraped from jobs by default when using the kube-state-metrics helmchart for exporting metrics.

In order to have these labels scraped so that the failed job alerts will work correctly, we had to add label scraping configuration here: https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-state-metrics/values.yaml#L141

Alternatives

As the default AlertManager rules rely on the scraping of the relevant job labels in prometheus, I feel there should be some documentation or warning that the default AlertManager rules in component versions >= v2.2.0 require the scraping of these job labels, as this scraping does not happen by default in the typical kube-state-metrics stack installation.

The Helm Chart for K8up 1.0+ is being updated in appuio/charts#224. This also needs adjustments in the Commodore Component to support this new version of the Helm Chart and K8up.