Currently, we've added runtime conditions to the stctl command for use during execution from Cloud Build. Instead, we want a runtime environment that interprets conditional directives like --execute_in_project=<foo> before calling exec on the remaining parameters passed to the docker image.

I imagine a new command cbenv or cbexec or cbopt or cbcond or similar, that accepts parameters from the environment and then execs the rest of argv if the conditions for execution are satisfied.

The first condition it should support is "EXECUTE_IN_PROJECTS=". With this directive it will be possible to have a single cloud build config with commands that should only be run in specific projects.

- name: gcp-config
  args: [ 'echo', 'hello world' ]
  env:
  - EXECUTE_IN_PROJECTS=mlab-sandbox,mlab-staging,mlab-oti

Additional ideas include conditional action on built-in environment variables -- https://cloud.google.com/cloud-build/docs/configuring-builds/substitute-variable-values For example, IF_TAG_SET, or IF_TAG_MATCHES=<pattern>.

The daily archive transfers have been taking hours to run, such that the second transfer to archive-measurement-lab only has a fraction of the day's data.

With the single time API, we can't tell whether we should disable existing transfers at different times.

By supporting multiple times in a single CLI call, we could then compare all existing jobs to see which ones should be updated, and which ones should be removed. This would work for changes in configs, times, or number of transfers per day.

Cloud builds regular fail for transient reasons that we do not detect until much later because there is no notification mechanism for failed builds. This guide describes how to setup Slack notifications for Cloud builds.

• https://cloud.google.com/build/docs/configuring-notifications/configure-slack

Their examples filter on SUCCESS. But we could setup notifications for any non-SUCCESSful build, and send to the appropriate slack alert channel, #alerts-staging or #alerts-oti.

Available Build.Status enum values: https://cloud.google.com/build/docs/api/reference/rest/v1/projects.builds#Build.Status

Derived from comments on m-lab/siteinfo#100

I suggest that we create a new command, perhaps cbctl, with a subcommand "trigger" that accepts a trigger ID. This command should live in the gcp-config repo. My vision is to create a standard set of containers that will exist in our projects by default that we can just reference in the cloudbuild.yamls of other repos (like this one).

The other operations that cbctl would support is registering the trigger definitions for the standard workflow that we support between sandbox, staging, and prod using a uniform name, for a new (or existing) repo. Basically to remove the manual steps for creating build configs through GCP web console.

The location for the archive-mlab-oti bucket in the mlab-oti project is "us-central1" (single region).

We want the bucket location to be "US" (multi-region) to allow for greater flexibility and for consistency with the corresponding buckets in mlab-sandbox and mlab-staging.

According to the documentation for GCS, it is not possible to change a bucket's location after it's created. We will have to create a new bucket with the location set to "US" and transfer the existing data to it.

The build error is:

Step #1: 2021/02/08 18:30:06 Failed to sync (error: googleapi: Error 400: Service account storage-transfer-13271960596232968425@partnercontent.gserviceaccount.com does not have required permissions {storage.objects.list, storage.objects.delete} for bucket pusher-mlab-oti., failedPrecondition)
Step #1: 2021/02/08 18:30:06 error: pid:9 code:1 err:exit status 1
Finished Step #1
ERROR
ERROR: build step 1 "gcp-config-cbif" failed: step exited with non-zero status: 1

This looks rather like the service account or GCS bucket being used for this build in production is lacking some permissions that may have been manually configured in sandbox and staging (where builds are working). @stephen-soltesz noted that @gfr10598 had been making some changes to this repo which could possibly have affected this.

@stephen-soltesz noticed that pushing to branch sandbox-soltesz-main-am was triggering Cloud Builds for prometheus-support in both mlab-sandbox and mlab-staging. Upon closer inspection, we noticed that the build trigger in mlab-staging had a value of "main" in the Build * field. The description for the Build * field states:

Trigger only for a branch that matches the given regular expression

The trigger was created using cbctl from this repository. The value of the Build * field should have been "^main$".

We need to update cbctl to create bounded regular expressions for matching on a branch name.

The allowed-projects flag should be eliminated once #7 is available. And, that support should not allow accidental substring matches.

Currently, these are excluded from transfers. It seems like we should be transferring everything.

If we want these to be backed up elsewhere, we should add more transfer rules.