For example, we could use the discourse API to periodically download posts per topic, and run woke.

There's a bug in our CI workflow that opens a PR to OCI Factory.

As you can see in this file (produced by our CI), multiple images are assigned the major 10-22.04 tag, which is obviously wrong (only 10.3.3 should have it, not 10.3.1).

The way we decide what to tag with major.minor and major is based on checking against the list of already-existing tags. However, this means that if we now add 11.0, 11.1, and 11.2 in the same PR, they will all be tagged as 11, which is wrong.

The CI needs to be modified to not only check against already-existing tags, but on the union of that and the tags being added/modified by the PR itself.

Enhancement Proposal

pytest-operator has this:

jobs:
  call-inclusive-naming-check:
    name: Inclusive naming
    uses: canonical-web-and-design/Inclusive-naming/.github/workflows/woke.yaml@main
    with:
      fail-on-error: "true"

Need to verify and remove the following workaround from the itests CI:

• Juju agent pinning to 2.9.29
• Hostpath provisioner pinning to 1.3.0

grafana-agent-operator is a machine charm the utilizes our centralized CI (canonical/grafana-agent-operator/pull/2).
We never ran machine itests in CI when it was part of the hybrid k8s-machine charm repo.

Now, for itests to work, we need to be able to bootstrap a lxd controller.

Currently all repos have the same duplicated codeql-analysis workflow, for example:

• https://github.com/canonical/prometheus-k8s-operator/blob/main/.github/workflows/codeql-analysis.yml
• https://github.com/canonical/grafana-k8s-operator/blob/main/.github/workflows/codeql-analysis.yml
• etc.

On a rock repository, we've been duplicating the rock-release-dev.yaml file.

We quickly had the need to release branches on different tags (to test a k8s charm for example).

We modified our flow as followed: https://github.com/ubuntu-robotics/foxglove-studio-rock/pull/12/files

This way, when manually triggered on a branch (branch_A) the flow is publishing the rock with the associated tag (:branch_A).

Would you be interested in integrating a similar flow?

See latest "release to edge" for example:
https://github.com/canonical/catalogue-k8s-operator/actions/runs/5952084611

The charm is uploaded but not released.

Had to release manually today:

Revision 24 of charm 'catalogue-k8s' released to edge (attaching resources: 'catalogue-image' r25)

I discovered this by trying to relate catalogue to ca:

$ juju relate ca catalogue                     
ERROR no relations found

For some reason, the release-libraries CI started failing, like this traefik CI run.

The rock-pull-request workflow does nothing.

The reason is that it relies on the tj-actions/changed-files action to get a list of all modified rockcraft.yaml files but then uses the wrong output variable from this action. It uses changed_files -- which was renamed a while back to type_changed_files -- which gets the list of files which extension have been changed.
This results in an empty list, which is happily and successfully iterated over by the job resulting in a happy CI.

Seems like rock-update.yaml workflow is adding an extra section to rockcraft.yaml that breaks rockcraft pack command.

For instance this is is the original rockcraft.yaml, and this is the new one

The added section is:

  prometheus-pushgateway:
    source-tag: v1.6.1
    build

In canonical/prometheus-k8s-operator#442, there was a force-push, followed by a merge from main, which caused CI to fail.

After running Promote charm from edge to beta, the following happens:

simme@willow:~$ charmcraft status blackbox-exporter-k8s
Track    Base                  Channel    Version    Revision    Resources                                            
latest   ubuntu 20.04 (amd64)  stable     -          -           -                                                    
                               candidate  -          -           -                                                    
                               beta       1          1           blackbox-exporter-image (r1)                         
                               edge       1          1           blackbox-exporter-image (r1)                         
         ubuntu 22.04 (amd64)  stable     -          -           -                                                    
                               candidate  -          -           -                                                    
                               beta       -          -           -                                                    
                               edge       3          3           blackbox-exporter-image (r3)   

Expectation would be for this to happen for both bases. Similarly, the revs do not match between the bases from the get-go.

If for example rockcraft fails and then a new release of a workload comes out, current CI would result in having our rock tags skip a version.

Seems like rock-update.yaml workflow is not picking the right source-tag

For instance in this PR it is upgrading to 1.6.2 but picking source-tag: "v1.6.1"

@simskij

Deployment Graph

Workloads:

• coordinator: nginx + Grafana agent
• worker: Mimir

graph LR

subgraph mimir
worker1 ---|mimir-cluster| coordinator+gagent
worker2 ---|mimir-cluster| coordinator+gagent
worker3 ---|mimir-cluster| coordinator+gagent


end

subgraph cos-lite
coordinator+gagent ---|logging| loki
coordinator+gagent ---|remote-write| prometheus
coordinator+gagent ---|dashboards| grafana
coordinator+gagent ---|trace| tempo

end

Alert rules and dashboards belong to the coordinator.

Issues to complete

Mimir Worker

• canonical/mimir-worker-k8s-operator#11
• canonical/mimir-worker-k8s-operator#12

Mimir Coordinator

• canonical/mimir-coordinator-k8s-operator#13
• canonical/mimir-coordinator-k8s-operator#14
• canonical/mimir-coordinator-k8s-operator#15
• canonical/mimir-coordinator-k8s-operator#16

Others

• canonical/charm-relation-interfaces#120

Thanks, @rbarry82 for bringing this to light.

We are currently using a community action for getting the latest releases of the upstreams, which occasionally hits the rate limit for unauthenticated requests.

We should set up a generic ROCKs pipeline here and, while we're at it, implement something along the lines of https://github.com/gregziegan/fetch-latest-release/blob/main/action.js ourselves instead.

Example of the issue occurring

Currently, on merge, libs are released separately from release to edge.
This is a problem because if release to edge fails, then charmhub would have a lib version that is not yet part of the charm itself.

Maybe releasing the libs should be a step in the release-to-edge workflow?

The CI for catalogue-k8s fails because

ERROR: tox config file (either pyproject.toml, tox.ini, setup.cfg) not found

It would be handy if we could pass custom workdir location:

jobs:
  pull-request:
    name: PR
    uses: canonical/observability/.github/workflows/pull-request.yaml@main
+   with:
+     workdir: charm/
    secrets: inherit

When a PR is modifying multiple rocks, the rock-pull-request.yaml workflow will try to build all of them. However, without some cleanup we fill up the disk of the runner, which in turn makes CI fail.

We should run at least rockcraft clean (which might be enough if we run it as we build every rock), or something along the lines of:

for instance in $(lxc list --project= rockcraft --format json | jq -r '.[] | select(.name | startswith("rockcraft-")) | .name'); do
    lxc --project=rockcraft delete $instance
done

Sometimes just building the rock isn't enough. It would be handy if we had an optional arg to test the app is ok.
For example, for prometheus it could be /api/v1/status/buildinfo that should return a 200.

Inspired by kubeflow, shall we add a log dump to the itests workflow?

    - name: Dump deployments
      if: failure()
      run: kubectl describe deployments -A
    - name: Dump replicasets
      if: failure()
      run: kubectl describe replicasets -A
    - name: Dump pods and their logs
      if: failure()
      shell: bash
      run: |
        juju status --relations --storage
        kubectl get pods -A -o=jsonpath='{range.items[*]}{.metadata.namespace} {.metadata.name}{"\n"}' --sort-by=.metadata.namespace | while read namespace pod; do kubectl -n $namespace describe pod $pod; kubectl -n $namespace logs $pod --all-containers=true --tail=100; done

cc: @lucabello @ca-scribner

Related issues:

• canonical/charmcraft#642
• canonical/charmcraft#913

We should implement automation so that we can tag GitHub issues and run automatically post the message by linking the issue; this should happen through multiple mediums like Mattermost, RSS feed, a mailing list, etc.

We should figure out if we want to write a bot to run somewhere, use the GitHub bot, or different things.

tox will happily pass for non-existing testenvs:

$ tox -e volkswagen
  volkswagen: OK (0.84 seconds)
  congratulations :) (0.87 seconds)

This is especially misleading when instead of unit we have unit-k8s and unit-machine.

We should probably change the CI command to run tox to:

if tox -l | grep -q '^unit$'; then tox -e unit; else echo "Error: 'unit' testenv does not exist"; fi

Enhancement Proposal

Might be useful for detecting syntax errors: promtool check rules.