From https://github.com/grafana-operator/grafana-operator/blob/master/documentation/roadmap/roadmap-v5.md
As part of 5.0 we would like to change how the grafana dashboard discovery is working.

Today we have a selector on the grafana instance and it finds the grafanadashboard it want to use.
Instead we would like to change that of having the dashboards finding the grafana instance that it should apply to.

Deploying a folder with the field:

spec:
   folder: my-folder

Creates the folder on the grafana API of the matching grafana instance, however, the dashboard is never assigned to the folder, resulting in an empty

Example of bug:

---
apiVersion: grafana.integreatly.org/v1beta1
kind: Grafana
metadata:
  name: grafana
  labels:
    dashboards: "grafana"
spec:
  client:
    preferIngress: true
  config:
    log:
      mode: "console"
    auth:
      disable_login_form: "false"
    security:
      admin_user: root
      admin_password: secret
---
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDashboard
metadata:
  name: grafanadashboard-from-url-with-folder
spec:
  folder: "my-custom-folder"
  instanceSelector:
    matchLabels:
      dashboards: "grafana"
  url: "https://raw.githubusercontent.com/integr8ly/grafana-operator/master/deploy/examples/remote/grafana-dashboard.json"

I'm currently evaluating the experimental Grafana operator for future use in our organization. I stumbled across the following problem:

This documentation describes the use of environment variables in datasources: using-environment-variables-in-data-sources

I tried to set an access token as described in the documentation:

apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDatasource
...
    secureJsonData:
      httpHeaderValue1: Bearer ${ACCESS_TOKEN}

Access did not work. Since the Grafana GUI hides the actual value it is using, I also set the name of a header based on the environment variable:

apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDatasource
...
    jsonData:
      httpHeaderName1: Authorization
      httpHeaderName2: Bearer ${ACCESS_TOKEN}

The Grafana GUI now showed me the literal value that was set instead of replacing with the environment variable. I also tried removing the curly braces, but no luck.

Does the new operator somehow escape the dollar sign?

From https://github.com/grafana-operator/grafana-operator/blob/master/documentation/roadmap/roadmap-v5.md

As part of 5.0 we would like to see grafanadatasources to be managed by the grafana API instead of using a configmap that is mounted in the the grafana instance.

Create metrics for the grafana-operator.
Things like:

• number of dashboards, datasource etc having issues being added to there grafana instance. Since we support many we need to think about labeling.
• How many api calls is done towards the k8s API, how long does it take
• how long does it take to talk to the different grafana API:s?
• How many hits does the dashboard cache get
• etc

This issue will never be perfect but lets start with something and we go from there.

Extra gold star if you also create a cool grafana dashboard that we can ship together with the operator :)

missing features for v5.0.0-rc

• datasource env vars
• website with documentation grafana/grafana-operator#912
• manifest overlays to produce both namespaced and cluster scoped resources
• e2e tests, at least some unit tests
• jsonnet and grafonnet support #67

We've noticed, that the field spec.grafanaContainer.baseImage goes unused during reconciliation of the Deployment.

Although, it is possible to override the image via

spec:
  deployment:
    spec:
      template:
        spec:
          containers:
            - name: grafana
              image: harbor.registry/grafana:1234

But that's kinda a lot of overhead, just to overwrite the image. So I hope we can keep the baseImage parameter (as it's already in the current v4 as well) :)

There's also the initImage parameter right next to it. But I guess that's obsolete now?

Currently we provide the community with a single manifest bundle that mostly supports OCP.
It might be a good idea to only use kustomiaztion overlays to be able to separate config like seucirty context.

For inspiration look at:

grafana/loki#7308

From https://github.com/grafana-operator/grafana-operator/blob/master/documentation/roadmap/roadmap-v5.md
As part of 5.0 we would like to see the same configuration options available in both route and ingress, e.g. no TLS options exposed for Routes.

At the same time we would like to use the power of openshift when it's phaseable.
Today we assume routes should be behind TLS by default. This is very simple to do in OCP.

Since we don't know enough about none OCP environments we can't do the same assumptions. Instead the default value for ingress should be http.
We have already implemented was of making differences between OCP and none OCP and we should use them to solve the same issue here.

Today we are using logrus which is something that flux also uses.

At the same time I'm wondering if it could be a good idea to go over to the official k8s log solution, klog:
https://kubernetes.io/docs/concepts/cluster-administration/system-logs/
https://pkg.go.dev/k8s.io/klog/v2

This isn't a issue to be implemented, more a issue to think of the best long-term solution for the operator.
What is other operators like

• loki-operator
• kubebuilder in general
• flux
• etc etc

In short good to think about this before releasing v5.

Running on kind 1.23.3

2022-03-05T17:34:35.649+0100    ERROR   controller.grafana      error determining platform      {"reconciler group": "grafana.integreatly.org", "reconciler kind": "Grafana", "name": "grafana", "namespace": "grafana", "error": "the server could not find the requested resource"}

The issue is: https://github.com/pb82/grafana-operator-experimental/blob/066726eed8b8ddce10bb133398436f1dca11a6da/controllers/reconcilers/grafana/ingress_reconciler.go#L40

My grafana looks like this.

apiVersion: grafana.integreatly.org/v1beta1
kind: Grafana
metadata:
  name: grafana
  labels:
    app: grafana
spec:
  ingress:
    enabled: false
  config:
    log:
      mode: "console"
      level: "error"
    log.frontend:
      enabled: true
    auth:
      disable_login_form: False
      disable_signout_menu: True
    auth.anonymous:
      enabled: True
  resources:
    # Optionally specify container resources
    limits:
      cpu: 200m
      memory: 200Mi
    requests:
      cpu: 100m
      memory: 100Mi

I will try to fix it and create a PR.

Issue scenario: A big company want to use the grafana-operator to host multiple different grafana instances with multiple different dashboards/datasources.

The company use different grafana instances to separate data sharing between the teams.
If a malicious user with namespace access want to get access to the grafanadatasource of another team all he needs to do is to create a grafana instances that matches the same labels that the other team uses.

Solution

You could argue that this is something that the cloud administrators should think about and hinder by setting ValidatingAdmissionWebhooks.
But a general best practice to follow is to make a project as easy as possible to use securely.

To make this I think that we should introduce a new config value to the grafanadatasource and grafanadashboard being something like namespaceLocalOnly which should be true by default.
If this value is true the operator won't sync to any grafana instance outside of that namespace.

I still think it's important to support being able to have a single grafana instances that multiple users/teams/namespace can contribute to. But I think the general use case will be one grafana instance in one namespace with the same datasource and dashboards in the same namespace.

There is a golang client to talk to the grafana API.
Evaluate it and see if it's a good fit for the operator.

https://github.com/grafana/grafana-api-golang-client

If it's a good fit, implement it.

Is there any reasons why not to make instanceSelector required?

This applies to all resources that we have created.

➜ k explain grafanadashboards.grafana.integreatly.org.spec
KIND:     GrafanaDashboard
VERSION:  grafana.integreatly.org/v1beta1

RESOURCE: spec <Object>

DESCRIPTION:
     <empty>

FIELDS:
   instanceSelector	<Object>
   json	<string>
   plugins	<[]Object>

Hi @pb82 tried using the operator and I get some issues when creating a service:

2022-03-05T16:59:23.371+0100    ERROR   controller.grafana      reconciler error in stage   
    {"reconciler group": "grafana.integreatly.org", "reconciler kind": "Grafana",
     "name": "grafana", "namespace": "grafana", "stage": "service", 
     "error": "Service \"grafana-service\" is invalid: [spec.clusterIPs[0]: Invalid value: []string(nil): 
     primary clusterIP can not be unset, spec.ipFamilies[0]: Invalid value: []core.IPFamily(nil): primary ipFamily can not be unset]"}

How to recreate:

cat <<EOF | kind create cluster --image=kindest/node:v1.21.2 --config=-
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  kubeadmConfigPatches:
  - |
    kind: InitConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        node-labels: "ingress-ready=true"
  extraPortMappings:
  - containerPort: 80
    hostPort: 80
    protocol: TCP
  - containerPort: 443
    hostPort: 443
    protocol: TCP
EOF

I took a quick look and it seems like this was fixed in 1.22
kubernetes/kubernetes#91459
https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.22.md#api-change-2

It's not impossible that we will release this version after the 1.24 is released thus making 1.21 not supported. But at the same time we know that there will always be stranglers and it would be nice to support 1.21.

My grafana.yaml looks like this:

apiVersion: grafana.integreatly.org/v1beta1
kind: Grafana
metadata:
  name: grafana
  labels:
    app: grafana
spec:
  ingress:
    enabled: false
  config:
    log:
      mode: "console"
      level: "error"
    log.frontend:
      enabled: true
    auth:
      disable_login_form: False
      disable_signout_menu: True
    auth.anonymous:
      enabled: True
  resources:
    # Optionally specify container resources
    limits:
      cpu: 200m
      memory: 200Mi
    requests:
      cpu: 100m
      memory: 100Mi

When looking at the functions inside dashboard_list.go i notice that they are general purpose for the whole v1beta1 package.
We should probably change it's name to make it a bit easier to understand that.

And we should write it using more generic variables to show that it's generic used by multiple controllers.

Using examples/basic/resources.yaml
I get

1.6713644067525926e+09  ERROR   reconciler error in stage       {"controller": "grafana", "controllerGroup": "grafana.integreatly.org", "controllerKind": "Grafana", "Grafana": {"name":"grafana","namespace":"grafana-operator-system"}, "namespace": "grafana-operator-system", "name": "grafana", "reconcileID": "5a10f411-13ad-4d57-9200-c0a39d3f5d24", "stage": "ingress", "error": "ingress is not ready yet"}

Probably something simple that I have missed in my kind/ingress and all I guess we need to do is to write some docs around it.

In line

It's not a good idea but it's not impossible that the user uses a http endpoint and not https.
I guess we could look if ingress.spec.tls != nil, and if so we could assume that the user uses https.

This is very low priority.

From https://github.com/grafana-operator/grafana-operator/blob/master/documentation/roadmap/roadmap-v5.md
As part of 5.0 we would like to see a automatic cleanup of empty folders that was created by the operator.

Work was started in grafana/grafana-operator#657, but never finished and some could argue that this would be a breaking change.

If I create a dashboard/datasource/folder that don't have any matching instance it's kind of hard to know.

For example with the bellow config i missed the grafana-a and I had to look in the logs to find it.
It's kind of painful unless you know how the logs looks like and it would have saved me allot of time to understand what was going on to have it in the status field.

➜ k get grafanadatasources.grafana.integreatly.org grafanadatasource-sample -o yaml
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDatasource
metadata:
  name: grafanadatasource-sample
  namespace: grafana-operator-system
spec:
  datasource:
    access: proxy
    editable: false
    isDefault: true
    jsonData:
      timeInterval: 5s
      tlsSkipVerify: true
    name: prometheus
    type: prometheus
    url: http://prometheus-service:9090
  instanceSelector:
    matchLabels:
      dashboards: grafana-a
  plugins:
  - name: grafana-clock-panel
    version: 1.3.0

In version 4 we got support for jsonnet and grafonnet.
Implement the same features in version 5.

We are unsure if this is a feature that we want to support for version 5.
If this something that you want please do 👍 on the issue.

I know this is still in the works, but before it would be forgotten: The option editable=false in the GrafanaDatasource doesn't seem to do anything. I can still change the data source in the GUI and save the changes.

From https://github.com/grafana-operator/grafana-operator/blob/master/documentation/roadmap/roadmap-v5.md
As part of 5.0 we would like to see the grafana-opreator to be secure by default + being more configurable.

We have done a design document that describes the new ideas in: https://github.com/grafana-operator/grafana-operator/blob/master/documentation/design_proposal/002_grafana_v1beta1.md

In #52 I added functionality to generate rbac settings out of the box using kubebuilder.
The rbac settings that get generated is cluster wide.

In #30 @pb82 added the functionality of running the operator in namespace mode.

The question is how do we generate rbac rules for namespace mode and how do we distribute them through OLM?

When deploying a Grafana CR without having any Ingress configuration specified, the Operator will create an Ingress object without any host attached to it.

Expected behaviour should be to not provision an Ingress CR, as long as there's no valid host defined. One use case (our's, actually), where an Ingress is unwanted, is when Grafana is behind a separate proxy application and authentication is done via JWT for example.

Example Grafana CR

apiVersion: grafana.integreatly.org/v1beta1
kind: Grafana
metadata:
  name: grafana-test
spec:
  config:
    log:
      mode: "console"
    auth:
      disable_login_form: "false"
    security:
      admin_user: root
      admin_password: secret

Resulting Ingress

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  creationTimestamp: "2023-03-09T14:49:13Z"
  generation: 2
  name: grafana-test-ingress
  namespace: grafana
  ownerReferences:
  - apiVersion: grafana.integreatly.org/v1beta1
    kind: Grafana
    name: grafana-test
    uid: 49f3e944-f1d6-48ba-bb12-7a9de8dcbce0
  resourceVersion: "206126146"
  uid: f0676b71-15fd-4415-aaf7-c3c6692cbb74
spec:
  rules:
  - http:
      paths:
      - backend:
          service:
            name: grafana-test-service
            port:
              number: 3000
        path: /
        pathType: Prefix
status:
  loadBalancer: {}

Can we extend the Grafana backend and provider a webhook server by using the Go plugin SDK (https://github.com/grafana/grafana-plugin-sdk-go)?

This would allow us to only trigger reconciliation of existing resources when an actual change has happend.

Previously, some users complained about grafana-operator making too many requests to kube-apiserver (e.g. grafana/grafana-operator#460).
It'd be useful to:

• have a way to track such interactions (e.g. through metrics);
• optimize interactions if needed (use WATCH + cache results).

I'm having issues trying to create grafana instance in kubernetes when using make run

I have created a simple grafana instance using the example files with some extra additions.

---
apiVersion: grafana.integreatly.org/v1beta1
kind: Grafana
metadata:
  name: grafana
  labels:
    dashboards: "grafana"
spec:
  client:
    preferIngress: true
  config:
    log:
      mode: "console"
    auth:
      disable_login_form: "false"
    security:
      admin_user: root
      admin_password: secret
  ingress:
    spec:
      ingressClassName: nginx
      rules:
        - host: test.io
          http:
            paths:
              - backend:
                  service:
                    name: grafana-service
                    port:
                      number: 3000
                path: /
                pathType: Prefix
---
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDashboard
metadata:
  name: grafana-dashboard
spec:
  instanceSelector:
    matchLabels:
      dashboards: "grafana"
  json: >
    {
      "id": null,
      "title": "Simple Dashboard",
      "tags": [],
      "style": "dark",
      "timezone": "browser",
      "editable": true,
      "hideControls": false,
      "graphTooltip": 1,
      "panels": [],
      "time": {
        "from": "now-6h",
        "to": "now"
      },
      "timepicker": {
        "time_options": [],
        "refresh_intervals": []
      },
      "templating": {
        "list": []
      },
      "annotations": {
        "list": []
      },
      "refresh": "5s",
      "schemaVersion": 17,
      "version": 0,
      "links": []
    }
---
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDatasource
metadata:
  name: example-grafanadatasource
spec:
  datasource:
    access: proxy
    database: prometheus
    jsonData:
      timeInterval: 5s
      tlsSkipVerify: true
    name: Prometheus
    url: http://prometheus-service:9090
  instanceSelector:
    matchLabels:
      dashboards: "grafana"

1.6686266712433786e+09  INFO    found matching Grafana instances for datasource {"controller": "grafanadatasource", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaDatasource", "GrafanaDatasource": {"name":"example-grafanadatasource","namespace":"grafana-operator-system"}, "namespace": "grafana-operator-system", "name": "example-grafanadatasource", "reconcileID": "44852e11-9b41-4be7-9f23-3594cc4a6544", "count": 1}
1.6686266712433953e+09  INFO    grafana instance not ready      {"controller": "grafanadatasource", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaDatasource", "GrafanaDatasource": {"name":"example-grafanadatasource","namespace":"grafana-operator-system"}, "namespace": "grafana-operator-system", "name": "example-grafanadatasource", "reconcileID": "44852e11-9b41-4be7-9f23-3594cc4a6544", "grafana": "grafana"}
1.6686267822545404e+09  INFO    found matching Grafana instances for dashboard  {"controller": "grafanadashboard", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaDashboard", "GrafanaDashboard": {"name":"grafana-dashboard","namespace":"grafana-operator-system"}, "namespace": "grafana-operator-system", "name": "grafana-dashboard", "reconcileID": "44d02bbc-3696-4d19-9839-2f2b67f0b609", "count": 1}
1.668626782254582e+09   INFO    grafana instance not ready      {"controller": "grafanadashboard", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaDashboard", "GrafanaDashboard": {"name":"grafana-dashboard","namespace":"grafana-operator-system"}, "namespace": "grafana-operator-system", "name": "grafana-dashboard", "reconcileID": "44d02bbc-3696-4d19-9839-2f2b67f0b609", "grafana": "grafana"}

I assume this is due to the missing of URL in the status:

status:
  stage: complete
  stageStatus: success

But if I instead use the example grafana straight up and don't define any custom ingress config:

---
apiVersion: grafana.integreatly.org/v1beta1
kind: Grafana
metadata:
  name: grafana
  labels:
    dashboards: "grafana"
spec:
  client:
    preferIngress: true
  config:
    log:
      mode: "console"
    auth:
      disable_login_form: "false"
    security:
      admin_user: root
      admin_password: secret

Then I get

status:
  adminUrl: https://
  stage: complete
  stageStatus: success

It might be something dumb how I have setup my local env.
Any pointers would be great.

So I realized that we don't have any way of re-triggering a new grafana instance that gets created with the same label as another instance.

Say that we have grafana below together with a random GrafanaDatasource

apiVersion: grafana.integreatly.org/v1beta1
kind: Grafana
metadata:
  name: grafana
  labels:
    dashboards: "grafana"
spec:
  client:
    preferIngress: false
  config:
    log:
      mode: "console"
    auth:
      disable_login_form: "false"
    security:
      admin_user: root
      admin_password: secret
----
apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaDatasource
metadata:
  name: grafanadatasource-sample
spec:
  instanceSelector:
    matchLabels:
      dashboards: "grafana"
  plugins:
    - name: grafana-clock-panel
      version: 1.3.0
  datasource:
    name: prometheus
    type: prometheus
    access: proxy
    url: http://prometheus-service:9090
    isDefault: true
    jsonData:
      'tlsSkipVerify': true
      'timeInterval': "5s"
    editable: false

When the operator have reconciled both of them and the dashboard is applied to the grafana instance.
What happens if you create a new grafana instance with the same label selector. It should apply that dashboard to the new grafana instance as well.

But we don't have any reconciler that tells the grafanadatasource that it should trigger a new reconcile.

apiVersion: grafana.integreatly.org/v1beta1
kind: Grafana
metadata:
  name: grafana-b
  labels:
    dashboards: "grafana"
spec:
  client:
    preferIngress: false
  config:
    log:
      mode: "console"
    auth:
      disable_login_form: "false"
    security:
      admin_user: root
      admin_password: secret
  ingress:
    spec:
      ingressClassName: nginx
      rules:
        - host: test.io
          http:
            paths:
              - backend:
                  service:
                    name: grafana-service
                    port:
                      number: 3000
                path: /
                pathType: Prefix

So other then it's a bug is this realistic that it would happen.
And I think the answer is maybe.
I can defiantly see some big org wanting the same datasource on multiple grafana instances and applying that with a simple label is the way to go and probably a new grafana instance would be created after some time if a new team is created or something like that.

Not the end of the world but it would be good to support this use case.