oskoperator / osko Goto Github PK

Right now, our development environment is relatively tightly coupled to having a running Mimir instance reachable over the network (we are using our company's mimir dev instance).

It would be great if that coupling wasn't there to make hacking on osko easier for people who don't have that luxury.

By default, it would be a good idea not to add them, however if someone, for any reason, doesn't have the prometheus-operator or the CRDs installed to their cluster, we should have a mechanism of adding them ourselves.

Let's get inspired here

Check out mimirrule_controller.go

OpenSLO expects this to be possible and is currently not supported by our types

Search for TODO with this

Inspiration

This issue is a record of a conversation that we had around multitenancy in Mimir and our company's specific use-case and how OSKO could support that use-case without it being intrusive for other people using the project

Brainstorm

• We sync PrometheusRule resources to Mimir based annotation at the SLO resource level - if no annotation, we don't do this and just create the PrometheusRule (osko.com/mimir-sync: true)
• Mimir sync that's either tenant based or not from PrometheusRule based on if osko.com/target-tenant is set or not, default no tenant
• TenantID based on annotation or default (namespace name) (osko.com/target-tenant: "infra")
• Source tenants (osko.com/source-tenants: ["infra", "logging"])

• source_tenants, if multi-tenant, have to include target
• osko.com/source-tenants should be Datasource-based (therefore, SLI-based)
• osko.com/target-tenant defaults to namespace name, unless it's set at the META Datasource or SLO level
• Datasource META resource - osko.com/meta-ds: true
  • includes connectionDetails
  • will include target-tenant for our use-case
  • may include source-tenants, appended to all SLIs

RESULT (GOLDEN PATH)

1 datasource with:

• connectionDetails
• targetTenant
• sourceTenants

Example:

apiVersion: openslo.com/v1
kind: Datasource
metadata:
  labels:
    app.kubernetes.io/name: datasource
    app.kubernetes.io/instance: datasource-sample
    app.kubernetes.io/part-of: slo-kubernetes-operator
    app.kubernetes.io/managed-by: kustomize
    app.kubernetes.io/created-by: slo-kubernetes-operator
  name: datasource-main # should it have some special name/annotation?
spec:
  description: Mimir Example Datasource
  type: mimir
  connectionDetails:
    address: https://mimir.monitoring.dev.heu.group/
    syncPrometheusRules: true
    targetTenant: billing
    sourceTenants:
      - billing
      - infra

If neither targetTenant or sourceTenants are present, treat the type: mimir datasource as not multitenant. If either is present, the other has to be as well.

Process

We will create PrometheusRule resources and sync them to the Mimir cluster using another controller as part of osko.

With that same controller, sync other PrometheusRule resources from this namespace to the cluster as well, as defined by the optional targetTenant (if syncPrometheusRules is true)

This can be useful if one is using an authentication proxy in front of Mimir/Cortex

ref.: https://grafana.com/docs/mimir/latest/operators-guide/secure/authentication-and-authorization/