taskcluster / community-tc-config Goto Github PK

View Code? Open in Web Editor NEW

11.0 11.0 37.0 2.3 MB

Configuration for Taskcluster at https://community-tc.services.mozilla.com/

License: Mozilla Public License 2.0

Python 29.71% Shell 43.64% PowerShell 25.85% Dockerfile 0.80%

community-tc-config's Introduction

Taskcluster

The task execution framework that supports Mozilla's continuous integration and release processes.

Usage

This repository is used to develop, build, and release the Taskcluster services.

Team Mentions

Do you need to reach a specific subset of the team? Use the team handles to mention us with GitHub's @mention feature.

Team Name	Use To...
`@taskcluster/Core`	ping members of the Taskcluster team at Mozilla
`@taskcluster/services-reviewers`	ping reviewers for changes to platform services and libraries
`@taskcluster/frontend-reviewers`	ping people who can review changes to frontend (and related) code in the services monorepo
`@taskcluster/security-folks`	ping people who do security things

Contributors

Thanks goes to these wonderful people (emoji key):

_{James Lal} 💻 👋	_{Selena Deckelmann} 💻 👋	_{Dustin J. Mitchell} 💻 👋	_{Wander Lairson Costa} 💻 👋	_{Greg Arndt} 💻 👋	_{Pete Moore} 💻 🔧	_{Hassan Ali} 💻 👋
_{John Whitlock} 💻 👋	_{Brian Stack} 💻 👋	_{John Ford} 💻 👋	_{Eli Perelman} 💻 👋	_{Jonas Finnemann Jensen} 💻 👋	_{owlishDeveloper} 💻 👋	_{Miles Crabill} 💻 👋
_{Chris Cooper} 💻 👋	_{Mathieu Leplatre} 💻 👋	_{Rob Thijssen} 💻	_Anup 💻	_{Hammad Akhtar} 💻	_{Chinmay Kousik} 💻	_{Anthony Miyaguchi} 💻
_{Ana Rute Mendes} 💻	_{Andrea Del Rio} 💻	_kristelteng 💻	_{Elena Solomon} 💻	_{Xavier L.} 💻	_{Yann Landry} 💻	_Ayub 💻
_lteigrob 💻	_{Bastien Abadie} 💻	_{Amjad Mashaal} 💻	_{Tom Prince} 💻	_{Samantha Yu} 💻	_{Auni Ahsan} 💻	_alex 💻
_{Alisha Aneja} 💻	_{Prachi Manchanda} 💻	_{Simon Fraser} 💻	_{Yashvardhan Didwania} 💻	_{Cynthia Pereira} 💻	_{Hashini Galappaththi} 💻	_{Fienny Angelina} 💻
_{Kanika Saini} 💻	_{Biboswan Roy} 💻	_{sudipt dabral} 💻	_Ojaswin 💻	_{Матрешка} 💻	_{Alok Kumar} 💻	_{Arshad Kazmi} 💻
_{Jason Yang} 💻	_{Shubham Gupta} 💻	_{Arun Kumar Mohan} 💻	_{Brian Pitts} 💻	_{E. Dunham} 💻	_{Shubham Chinda} 💻	_{Patrick Kang} 💻
_{Rishabh Budhiraja} 💻	_ededals 💻	_{Ajin Kabeer} 💻	_{Catherine Chepkurui} 💻	_Jo 💻	_vishakha 💻 📖	_{Noor Fatima} 💻
_Michael 💻	_{Mariana Zangrossi} 💻	_{ANURADHAJHA99} 💻	_Edil 💻	_Olympia 💻 📖	_{Michael Ozoemena} 💻	_lailahgrant 💻
_km-js 💻	_{Carolina Machado} 💻	_reenesa 💻	_{Kelli Blalock} 💻	_{naima shaikh} 💻	_{Jiwoon Kim} 💻	_{Michael Umanah} 💻
_{Fahd Jamal A.} 📖	_{shilpi verma} 💻	_somchi 💻	_Anastasia 💻	_Lubna 💻	_{Soundharya AM} 💻	_{Mustafa Jebara} 💻
_{Aryaman Puri} 💻	_{Simon Sapin} 💻	_thoran 💻	_{Manish Giri} 💻	_{Tiger Oakes} 💻	_{Ricky Taylor} 💻	_{Alex Lopez} 💻
_Michelle 🐛 🚇	_{Mrs. Velena} 💻	_{Ahmed A.} 💻	_{Matt Boris} 💻 🔧	_{Yaraslau Kurmyza} 💻 🔧	_{Bastien Orivel} 💻	_HamdiAmine 💻

This project follows the all-contributors specification. Contributions of any kind are welcome!

community-tc-config's People

Contributors

Stargazers

Watchers

community-tc-config's Issues

Update README

Community-TC pulse access for fuzzmanager staging instance

I'm working with @jschwartzentruber on a new staging environment for FuzzManager and we would need access to the communitytc pulse virtual host.

I already created a pulse account using pulse guardian: fuzzmanager-staging.

Could you do the same steps as in #377 ? Thanks !

Migrate `fuzzing` project to Generic Worker

Migrate `l10n` project to Generic Worker

Add a check that all repos for a project have scopes limited to those of the project

So e.g., repos for project taskcluster should have scopes that are satisfied by ["assume:project-admin:taskcluster"].

Interactive tasks use internal hostname under docker-worker

I'm trying to launch an interactive task on a worker using docker-worker.
An example is here: https://community-tc.services.mozilla.com/tasks/D_htooOhQKCGvHIS7VkjhQ

Once the task is running, I click the shell link, but I never see any activity there. Checking the console, I see websocket connection errors:

Content Security Policy: Ignoring ‘x-frame-options’ because of ‘frame-ancestors’ directive.
Firefox can’t establish a connection to the server at wss://proj-fuzzing-linux-pool1-aq1nr8jitc6i2wpppoey4w.c.community-tc-workers.internal:50314/Vk-fWoNIQuG82GjYOPcKVg/shell.sock?tty=true&command=sh&command=-c&command=if%20%5B%20-f%20%22%2Fetc%2Ftaskcluster-motd%22%20%5D%3B%20then%20cat%20%2Fetc%2Ftaskcluster-motd%3B%20fi%3Bif%20%5B%20-z%20%22%24TERM%22%20%5D%3B%20then%20export%20TERM%3Dxterm%3B%20fi%3Bif%20%5B%20-z%20%22%24HOME%22%20%5D%3B%20then%20export%20HOME%3D%2Froot%3B%20fi%3Bif%20%5B%20-z%20%22%24USER%22%20%5D%3B%20then%20export%20USER%3Droot%3B%20fi%3Bif%20%5B%20-z%20%22%24LOGNAME%22%20%5D%3B%20then%20export%20LOGNAME%3Droot%3B%20fi%3Bif%20%5B%20-z%20%60which%20%22%24SHELL%22%60%20%5D%3B%20then%20export%20SHELL%3Dbash%3B%20fi%3Bif%20%5B%20-z%20%60which%20%22%24SHELL%22%60%20%5D%3B%20then%20export%20SHELL%3Dsh%3B%20fi%3Bif%20%5B%20-z%20%60which%20%22%24SHELL%22%60%20%5D%3B%20then%20export%20SHELL%3D%22%2F.taskclusterutils%2Fbusybox%20sh%22%3B%20fi%3BSPAWN%3D%22%24SHELL%22%3Bif%20%5B%20%22%24SHELL%22%20%3D%20%22bash%22%20%5D%3B%20then%20SPAWN%3D%22bash%20-li%22%3B%20fi%3Bif%20%5B%20-f%20%22%2Fbin%2Ftaskcluster-interactive-shell%22%20%5D%3B%20then%20SPAWN%3D%22%2Fbin%2Ftaskcluster-interactive-shell%22%3B%20fi%3Bexec%20%24SPAWN%3B. Shell.4d2d9c5f.js:34:359597
uncaught exception: Object
Firefox can’t establish a connection to the server at wss://proj-fuzzing-linux-pool1-aq1nr8jitc6i2wpppoey4w.c.community-tc-workers.internal:50314/Vk-fWoNIQuG82GjYOPcKVg/shell.sock?tty=true&command=sh&command=-c&command=if%20%5B%20-f%20%22%2Fetc%2Ftaskcluster-motd%22%20%5D%3B%20then%20cat%20%2Fetc%2Ftaskcluster-motd%3B%20fi%3Bif%20%5B%20-z%20%22%24TERM%22%20%5D%3B%20then%20export%20TERM%3Dxterm%3B%20fi%3Bif%20%5B%20-z%20%22%24HOME%22%20%5D%3B%20then%20export%20HOME%3D%2Froot%3B%20fi%3Bif%20%5B%20-z%20%22%24USER%22%20%5D%3B%20then%20export%20USER%3Droot%3B%20fi%3Bif%20%5B%20-z%20%22%24LOGNAME%22%20%5D%3B%20then%20export%20LOGNAME%3Droot%3B%20fi%3Bif%20%5B%20-z%20%60which%20%22%24SHELL%22%60%20%5D%3B%20then%20export%20SHELL%3Dbash%3B%20fi%3Bif%20%5B%20-z%20%60which%20%22%24SHELL%22%60%20%5D%3B%20then%20export%20SHELL%3Dsh%3B%20fi%3Bif%20%5B%20-z%20%60which%20%22%24SHELL%22%60%20%5D%3B%20then%20export%20SHELL%3D%22%2F.taskclusterutils%2Fbusybox%20sh%22%3B%20fi%3BSPAWN%3D%22%24SHELL%22%3Bif%20%5B%20%22%24SHELL%22%20%3D%20%22bash%22%20%5D%3B%20then%20SPAWN%3D%22bash%20-li%22%3B%20fi%3Bif%20%5B%20-f%20%22%2Fbin%2Ftaskcluster-interactive-shell%22%20%5D%3B%20then%20SPAWN%3D%22%2Fbin%2Ftaskcluster-interactive-shell%22%3B%20fi%3Bexec%20%24SPAWN%3B. Shell.4d2d9c5f.js:34:359597
uncaught exception: Object

The hostname it tries connecting to is a subdomain of community-tc-workers.internal, which looks wrong to me.

@walac is this a configuration problem, or am I missing something in my task definition to make this work?

Migrate `bugbug` project to Generic Worker

Build new Docker Worker images for AWS

Our current worker pools running on docker worker in AWS are outdated. These machine images need to be rebuilt and updated.

Build new Docker Worker images for GCP

Our current worker pools running on docker worker in GCP are outdated. These machine images need to be rebuilt and updated.

`imageset.sh` script should not delete old AMIs until the script finishes successfully

There are some times where the image building process hangs up (I've only ever seen this happen on AWS) and it errors out before fully creating the new AMIs, while also deleting the old ones. This causes a backlog of worker pool errors because of non-existent AMIs.

Unapplied fuzzing changes

The diff below is from reality -> repo. @jschwartzentruber should we get some of this in the repo?

--- current
+++ generated
@@ -473,17 +473,17 @@ Hook=project-fuzzing/bugmon:
               },
               "env": {
                   "BUG_ACTION": "monitor"
               },
               "features": {
                   "taskclusterProxy": true
               },
               "image": "mozillasecurity/bugmon:latest",
-              "maxRunTime": 10800
+              "maxRunTime": 3600
           },
           "provisionerId": "proj-fuzzing",
           "scopes": [
               "docker-worker:capability:device:hostSharedMemory",
               "docker-worker:capability:device:loopbackAudio",
               "docker-worker:capability:privileged",
               "queue:create-task:highest:proj-fuzzing/bugmon-monitor",
               "queue:create-task:highest:proj-fuzzing/bugmon-processor",
@@ -561,57 +561,47 @@ Hook=project-fuzzing/grizzly-reduce-monitor:
     hookId: grizzly-reduce-monitor
     name: grizzly-reduce-monitor
     description:
       *DO NOT EDIT* - This resource is configured automatically.

       Hook for triggering Grizzly reduce monitor tasks
     owner: [email protected]
     emailOnError: True
-    schedule: - 0 0 */2 * * *
+    schedule:
     bindings:
     task:
       {
           "deadline": {
               "$fromNow": "3 hours"
           },
           "expires": {
               "$fromNow": "2 weeks"
           },
           "metadata": {
               "description": "Hook for triggering grizzly reduce monitor tasks",
               "name": "grizzly-reduce-monitor",
               "owner": "[email protected]",
               "source": "https://github.com/MozillaSecurity/grizzly"
           },
           "payload": {
-              "artifacts": {
-                  "project/fuzzing/private/logs": {
-                      "expires": {
-                          "$fromNow": "2 weeks"
-                      },
-                      "path": "/logs/",
-                      "type": "directory"
-                  }
-              },
               "command": [
                   "/usr/bin/grizzly-reduce-tc-monitor"
               ],
               "features": {
                   "taskclusterProxy": true
               },
               "image": {
                   "namespace": "project.fuzzing.reduce-monitor.main",
                   "path": "public/grizzly-reduce-tc-monitor.tar",
                   "type": "indexed-image"
               },
               "maxRunTime": 3600
           },
           "provisionerId": "proj-fuzzing",
-          "schedulerId": "fuzzing",
           "scopes": [
               "docker-worker:capability:device:hostSharedMemory",
               "docker-worker:capability:device:loopbackAudio",
               "queue:create-task:highest:proj-fuzzing/grizzly-reduce-worker",
               "queue:route:[email protected]",
               "queue:scheduler-id:fuzzing",
               "secrets:get:project/fuzzing/credstash-aws-auth",
               "secrets:get:project/fuzzing/fuzzmanagerconf",
@@ -661,17 +651,16 @@ Hook=project-fuzzing/grizzly-reduce-reset-error:
               "image": {
                   "namespace": "project.fuzzing.reduce-monitor.main",
                   "path": "public/grizzly-reduce-tc-monitor.tar",
                   "type": "indexed-image"
               },
               "maxRunTime": 600
           },
           "provisionerId": "proj-fuzzing",
-          "schedulerId": "fuzzing",
           "scopes": [
               "queue:route:[email protected]",
               "queue:scheduler-id:fuzzing",
               "secrets:get:project/fuzzing/fuzzmanagerconf"
           ],
           "workerType": "grizzly-reduce-monitor"
       }
     triggerSchema: {}
@@ -1032,17 +1021,16 @@ Role=hook-id:project-fuzzing/grizzly-reduce-monitor:
     roleId: hook-id:project-fuzzing/grizzly-reduce-monitor
     description:
       *DO NOT EDIT* - This resource is configured automatically.


     scopes:
       - docker-worker:capability:device:hostSharedMemory
       - docker-worker:capability:device:loopbackAudio
-      - queue:create-task:highest:proj-fuzzing/grizzly-reduce-monitor
       - queue:create-task:highest:proj-fuzzing/grizzly-reduce-worker
       - queue:route:[email protected]
       - queue:scheduler-id:fuzzing
       - secrets:get:project/fuzzing/credstash-aws-auth
       - secrets:get:project/fuzzing/fuzzmanagerconf
       - secrets:get:project/fuzzing/grizzly-reduce-tool-list

   Role=hook-id:project-fuzzing/grizzly-reduce-reset-error:
@@ -1757,17 +1745,16 @@ Role=repo:github.com/MozillaSecurity/*:
     roleId: repo:github.com/MozillaSecurity/*
     description:
       *DO NOT EDIT* - This resource is configured automatically.


     scopes:
       - queue:create-task:highest:proj-fuzzing/ci
       - queue:create-task:highest:proj-fuzzing/ci-*
-      - queue:scheduler-id:fuzzing
       - secrets:get:project/fuzzing/ci-*
       - secrets:get:project/fuzzing/codecov-*
       - secrets:get:project/fuzzing/deploy-*
       - secrets:get:project/fuzzing/pypi-*

   Role=repo:github.com/MozillaSecurity/fuzzing-tc-config:*:
     roleId: repo:github.com/MozillaSecurity/fuzzing-tc-config:*
     description:
@@ -1823,17 +1810,16 @@ Role=repo:github.com/MozillaSecurity/orion:*:
     roleId: repo:github.com/MozillaSecurity/orion:*
     description:
       *DO NOT EDIT* - This resource is configured automatically.


     scopes:
       - docker-worker:capability:privileged
       - queue:route:index.project.fuzzing.orion.*
-      - secrets:get:project/fuzzing/docker-hub

   Role=repo:github.com/Pike/user-journey-localized:*:
     roleId: repo:github.com/Pike/user-journey-localized:*
     description:
       *DO NOT EDIT* - This resource is configured automatically.


     scopes: - queue:create-task:highest:proj-l10n/*
@@ -10241,44 +10227,16 @@ WorkerPool=proj-fuzzing/ci:
                           ]
                       }
                   ],
                   "region": "us-east1",
                   "scheduling": {
                       "onHostMaintenance": "terminate"
                   },
                   "workerConfig": {
-                      "artifacts": {
-                          "skipCompressionExtensions": [
-                              ".7z",
-                              ".bz2",
-                              ".dmg",
-                              ".flv",
-                              ".gif",
-                              ".gz",
-                              ".jpeg",
-                              ".jpg",
-                              ".png",
-                              ".swf",
-                              ".tbz",
-                              ".tgz",
-                              ".webp",
-                              ".whl",
-                              ".woff",
-                              ".woff2",
-                              ".xz",
-                              ".zip",
-                              ".zst",
-                              ".lz4",
-                              ".tar"
-                          ]
-                      },
-                      "dockerConfig": {
-                          "allowPrivileged": true
-                      },
                       "shutdown": {
                           "afterIdleSeconds": 15,
                           "enabled": true
                       }
                   },
                   "zone": "us-east1-b"
               },
               {
@@ -10304,44 +10262,16 @@ WorkerPool=proj-fuzzing/ci:
                           ]
                       }
                   ],
                   "region": "us-east1",
                   "scheduling": {
                       "onHostMaintenance": "terminate"
                   },
                   "workerConfig": {
-                      "artifacts": {
-                          "skipCompressionExtensions": [
-                              ".7z",
-                              ".bz2",
-                              ".dmg",
-                              ".flv",
-                              ".gif",
-                              ".gz",
-                              ".jpeg",
-                              ".jpg",
-                              ".png",
-                              ".swf",
-                              ".tbz",
-                              ".tgz",
-                              ".webp",
-                              ".whl",
-                              ".woff",
-                              ".woff2",
-                              ".xz",
-                              ".zip",
-                              ".zst",
-                              ".lz4",
-                              ".tar"
-                          ]
-                      },
-                      "dockerConfig": {
-                          "allowPrivileged": true
-                      },
                       "shutdown": {
                           "afterIdleSeconds": 15,
                           "enabled": true
                       }
                   },
                   "zone": "us-east1-c"
               },
               {
@@ -10367,44 +10297,16 @@ WorkerPool=proj-fuzzing/ci:
                           ]
                       }
                   ],
                   "region": "us-east1",
                   "scheduling": {
                       "onHostMaintenance": "terminate"
                   },
                   "workerConfig": {
-                      "artifacts": {
-                          "skipCompressionExtensions": [
-                              ".7z",
-                              ".bz2",
-                              ".dmg",
-                              ".flv",
-                              ".gif",
-                              ".gz",
-                              ".jpeg",
-                              ".jpg",
-                              ".png",
-                              ".swf",
-                              ".tbz",
-                              ".tgz",
-                              ".webp",
-                              ".whl",
-                              ".woff",
-                              ".woff2",
-                              ".xz",
-                              ".zip",
-                              ".zst",
-                              ".lz4",
-                              ".tar"
-                          ]
-                      },
-                      "dockerConfig": {
-                          "allowPrivileged": true
-                      },
                       "shutdown": {
                           "afterIdleSeconds": 15,
                           "enabled": true
                       }
                   },
                   "zone": "us-east1-d"
               },
               {
@@ -10430,44 +10332,16 @@ WorkerPool=proj-fuzzing/ci:
                           ]
                       }
                   ],
                   "region": "us-east4",
                   "scheduling": {
                       "onHostMaintenance": "terminate"
                   },
                   "workerConfig": {
-                      "artifacts": {
-                          "skipCompressionExtensions": [
-                              ".7z",
-                              ".bz2",
-                              ".dmg",
-                              ".flv",
-                              ".gif",
-                              ".gz",
-                              ".jpeg",
-                              ".jpg",
-                              ".png",
-                              ".swf",
-                              ".tbz",
-                              ".tgz",
-                              ".webp",
-                              ".whl",
-                              ".woff",
-                              ".woff2",
-                              ".xz",
-                              ".zip",
-                              ".zst",
-                              ".lz4",
-                              ".tar"
-                          ]
-                      },
-                      "dockerConfig": {
-                          "allowPrivileged": true
-                      },
                       "shutdown": {
                           "afterIdleSeconds": 15,
                           "enabled": true
                       }
                   },
                   "zone": "us-east4-a"
               },
               {
@@ -10493,44 +10367,16 @@ WorkerPool=proj-fuzzing/ci:
                           ]
                       }
                   ],
                   "region": "us-east4",
                   "scheduling": {
                       "onHostMaintenance": "terminate"
                   },
                   "workerConfig": {
-                      "artifacts": {
-                          "skipCompressionExtensions": [
-                              ".7z",
-                              ".bz2",
-                              ".dmg",
-                              ".flv",
-                              ".gif",
-                              ".gz",
-                              ".jpeg",
-                              ".jpg",
-                              ".png",
-                              ".swf",
-                              ".tbz",
-                              ".tgz",
-                              ".webp",
-                              ".whl",
-                              ".woff",
-                              ".woff2",
-                              ".xz",
-                              ".zip",
-                              ".zst",
-                              ".lz4",
-                              ".tar"
-                          ]
-                      },
-                      "dockerConfig": {
-                          "allowPrivileged": true
-                      },
                       "shutdown": {
                           "afterIdleSeconds": 15,
                           "enabled": true
                       }
                   },
                   "zone": "us-east4-b"
               },
               {
@@ -10556,44 +10402,16 @@ WorkerPool=proj-fuzzing/ci:
                           ]
                       }
                   ],
                   "region": "us-east4",
                   "scheduling": {
                       "onHostMaintenance": "terminate"
                   },
                   "workerConfig": {
-                      "artifacts": {
-                          "skipCompressionExtensions": [
-                              ".7z",
-                              ".bz2",
-                              ".dmg",
-                              ".flv",
-                              ".gif",
-                              ".gz",
-                              ".jpeg",
-                              ".jpg",
-                              ".png",
-                              ".swf",
-                              ".tbz",
-                              ".tgz",
-                              ".webp",
-                              ".whl",
-                              ".woff",
-                              ".woff2",
-                              ".xz",
-                              ".zip",
-                              ".zst",
-                              ".lz4",
-                              ".tar"
-                          ]
-                      },
-                      "dockerConfig": {
-                          "allowPrivileged": true
-                      },
                       "shutdown": {
                           "afterIdleSeconds": 15,
                           "enabled": true
                       }
                   },
                   "zone": "us-east4-c"
               }
           ],

Migrate `mozci` project to Generic Worker

Migrate `wpt` project to Generic Worker

Add CI to check PRs

Perhaps even using Taskcluster itself? Just an idea :)

Run Black on the Python source
Run flake8 on the Python source
Run yamllint on the YAML
Run tc-admin diff --without-secrets

Add a new project for RTEMs

I've been chatting with @verm about Taskcluster, and it might be a good match for RTEMS.

The particular use-case involves bringing workers, so I think a good way to get started and see if there's a match is to set up a project in the community TC deployment, but not add any cloud-based workers. The incremental cost to Mozilla is minimal, and if the experiment is successful then RTEMS can look into hosting their own Taskcluster deployment.

Change master to main

Move development from the master branch to main.

Refactor provisioner references for docker-worker tests

Ref: #23 (comment)

Credentials needed for community-tc pulse server

For fuzzing tasks in community-tc, we maintain a fuzzing dashboard by scraping task status of all fuzzing hooks. Doing a full scrape takes ~15 minutes now. It would be better if we could listen to pulse messages instead.

role hook-id:project-fuzzing/fuzzing-tc-config-community-update contains cycle

tc-admin update failure:

taskcluster.exceptions.TaskclusterRestFailure: Roles may not contain dependency cycles, found: 'assume:hook-id:project-fuzzing/fuzzing-tc-config-community-update' -> 'assume:hook-id:project-fuzzing/fuzzing-tc-config-community-update'

Migrate `bors-ng` project to Generic Worker

Migrate `taskcluster` project to Generic Worker

Migrate `relman` project to Generic Worker

Add a check that no roles include redundant scopes

Sometimes we don't notice that we have directly assigned a scope to a role that it already possesses by virtue of an assume scope that it contains, or perhaps the list of scopes is long and we don't notice it is already included, or perhaps a * scope is already included which is more general, or perhaps it inherits the scope due to a parameterised role.

I would propose that if we can determine that the removal of a given scope from a role results in no change to the expanded scopes of the role, then we should fail with an error that the given scope assignment is redundant and can safely be removed.

Perhaps this is overkill, but on the other hand it might be a housekeeping aid to keep roles in check.

"tc-admin diff" failing since ea76ecf1d3f976b1c49f5df9655905c062bb8242

Since ea76ecf, the task is failing with:

RuntimeError: unmanaged resource: Role=repo:github.com/mozilla/bugbug:*

Generate AWS region us-east-2 launch configurations

The AWS EC2 worker pools are currently limited to regions us-east-1 / us-west-1 / us-west-2, which often results in Worker Manager failing to launch instances due to limited supply.

By adding the us-east-2 availability zones to the worker pool definitions, supply should be improved, and we should have better coverage when supply is low in the current regions. In particular, m5.2xlarge, which is our default instance type, is only supported in availability zones us-west-1a and us-west-1b with the currently configured regions, but is supported in many us-east-2 availability zones, so would provide a lot more coverage if us-east-2 was supported.

Note, before we add us-east-2 availability zones in the worker pool definitions, we will first need to add support for us-east-2 to the community cluster Worker Manager, which may involve ec2 configuration (e.g. creation of security groups, vpn configuration, IAM configuration, ...) and possibly additional Worker Manager configuration(?)

Scope auth:sentry:generic-worker missing from Client ID worker/static/proj-taskcluster/gw-ci-freebsd/proj-taskcluster/freebsd-1

Just dumping this here so I don't forget about it.

Presumably some role somewhere needs to be updated...

{
  "code": "InsufficientScopes",
  "message": "Client ID worker/static/proj-taskcluster/gw-ci-freebsd/proj-taskcluster/freebsd-1 does not have sufficient scopes and is missing the following scopes:\n\n```\nauth:sentry:generic-worker\n```\n\nThis request requires the client to satisfy the following scope expression:\n\n```\nauth:sentry:generic-worker\n```\n\n---\n\n* method:     sentryDSN\n* errorCode:  InsufficientScopes\n* statusCode: 403\n* time:       2023-07-13T14:53:45.665Z",
  "requestInfo": {
    "method": "sentryDSN",
    "params": {
      "project": "generic-worker"
    },
    "payload": {},
    "time": "2023-07-13T14:53:45.665Z"
  }
}

Migrate `taskcluster` to GCP

Update generic-worker Ubuntu worker pools to Ubuntu 22.04 LTS

Migrate `firefoxreality` project to Generic Worker

Move microannotate to misc from relman

microannotate shouldn't need anything special, so it could be in misc.

Ubuntu generic-worker Windows worker pools to Windows Server 2022

tc-admin apply throws an error

tc-admin apply --with-secrets --ids-only --grep proj-taskcluster throws:

Traceback (most recent call last):
  File "/Users/hassanali/Documents/Mozilla/projects/community-tc-config/sandbox/bin/tc-admin", line 11, in <module>
    load_entry_point('tc-admin==2.2.0', 'console_scripts', 'tc-admin')()
  File "/Users/hassanali/Documents/Mozilla/projects/community-tc-config/sandbox/lib/python3.7/site-packages/pkg_resources/__init__.py", line 490, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/Users/hassanali/Documents/Mozilla/projects/community-tc-config/sandbox/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2853, in load_entry_point
    return ep.load()
  File "/Users/hassanali/Documents/Mozilla/projects/community-tc-config/sandbox/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2453, in load
    return self.resolve()
  File "/Users/hassanali/Documents/Mozilla/projects/community-tc-config/sandbox/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2459, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/Users/hassanali/Documents/Mozilla/projects/community-tc-config/sandbox/lib/python3.7/site-packages/tc_admin-2.2.0-py3.7.egg/tcadmin/boot.py", line 10, in <module>
    from .main import main
  File "/Users/hassanali/Documents/Mozilla/projects/community-tc-config/sandbox/lib/python3.7/site-packages/tc_admin-2.2.0-py3.7.egg/tcadmin/main.py", line 13, in <module>
    from .util.sessions import with_aiohttp_session
  File "/Users/hassanali/Documents/Mozilla/projects/community-tc-config/sandbox/lib/python3.7/site-packages/tc_admin-2.2.0-py3.7.egg/tcadmin/util/sessions.py", line 7, in <module>
    import aiohttp
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
  File "/Users/hassanali/Documents/Mozilla/projects/community-tc-config/sandbox/lib/python3.7/site-packages/aiohttp-2.3.10-py3.7.egg/aiohttp/__init__.py", line 6, in <module>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
  File "/Users/hassanali/Documents/Mozilla/projects/community-tc-config/sandbox/lib/python3.7/site-packages/aiohttp-2.3.10-py3.7.egg/aiohttp/client.py", line 15, in <module>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
  File "/Users/hassanali/Documents/Mozilla/projects/community-tc-config/sandbox/lib/python3.7/site-packages/aiohttp-2.3.10-py3.7.egg/aiohttp/connector.py", line 12, in <module>
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
  File "/Users/hassanali/Documents/Mozilla/projects/community-tc-config/sandbox/lib/python3.7/site-packages/aiohttp-2.3.10-py3.7.egg/aiohttp/helpers.py", line 742, in <module>
TypeError: function() argument 1 must be code, not str

Migrate `releng` project to Generic Worker

Default idleTimeoutSecs should be 1

In order to reduce overhead, community workers should terminate themselves ASAP when there is no pending work. This can, of course, be overridden in the project configs, but workers should default to shutting down immediately.

Image building instructions out of date

https://github.com/mozilla/community-tc-config/edit/main/imagesets/README.md talks about the password store being on the mozilla internal git server. Now it lives elsewhere. Also the instructions should ensure that all keys are trusted that need to be, in order that pass commands do not fail. In fact, the imageset.sh script should probably take care of both gcloud auth login and checking gpg keys are all trusted before running, since this recently broke my run of imageset.sh.

Note instructions should include project requirements for gcloud configuration, such as .ssh config settings etc, so that pass git pull just works.

Another thing we could do, if the script needs to exit early, is to dump the remaining commands to standard err, so that an interrupted run can easily be completed post-failure. But that is just a possible optimisation, since when the script does exit early, it is a pain to work out what it hasn't done yet!

Migrate `bugbug` to GCP

Don't create worker-pool secrets when --without-secrets

$ tc-admin diff --without-secrets --ids-only
+ Secret=worker-pool:proj-deepspeech/win
+ Secret=worker-pool:proj-deepspeech/win-b
+ Secret=worker-pool:proj-git-cinnabar/win2012r2
+ Secret=worker-pool:proj-taskcluster/gw-ci-ubuntu-18-04
+ Secret=worker-pool:proj-taskcluster/gw-ci-ubuntu-18-04-staging
+ Secret=worker-pool:proj-taskcluster/gw-ci-windows2012r2-amd64
+ Secret=worker-pool:proj-taskcluster/gw-ci-windows2012r2-amd64-staging

when, in fact, those secrets do exist.

Add projectId scopes to projects, admins

Migrate `bors-ng` to GCP

Migrate `relman` to GCP

Run tc-admin apply at end of imageset.sh

This works reliably enough now, that it should run automatically, rather than instructing the user to run it.

Remove unused secret from webrender config

https://github.com/mozilla/community-tc-config/blob/ad04d3b692b4b46033711c0bc61790c60d0fe3e3/config/projects/webrender.yml#L31-L32

As far as I am aware, the secret referenced in this config file is not used by the WebRender setup. I think it was erroneously ported to the community TC instance and was never needed here. It was needed on the firefox TC instance, because it was needed by the WR sync task which runs in Firefox CI. So this secret can be deleted and the reference removed from the config file.

Note that since the TC split that created the community TC instance, https://hg.mozilla.org/mozilla-central/rev/03b4d21736d44c06e2aed6f52a229ba98fcebe04 also landed which renamed the secret as used on the Firefox instance. The current secret token is in gecko/gfx-github-sync/token, and that's the only one that needs to be kept. There is an obsolete one at project/webrender-ci/wrupdater-github-token that can also be deleted. I believe I can just delete that myself using the web interface but I wanted to confirm that it's not referenced in any TC config files or such before I delete it.

/cc @kvark @petemoore

Remove hardcoded blacklists of instance/machine type to (availability) zones

Originally from #428 (comment).

There are hardcoded blacklists of EC2 availability zones/GCP zones for a given instance type / machine type, that can be replaced by calls to aws ec2 describe-instance-type-offerings and gcloud compute machine-types list to determine if an instance type / machine type is available in a given EC2 availability zone / GCP zone. This is desirable, since this hardcoded list is constantly subject to change, but the API calls that would replace them should not need to change over time.

GCP image projects/community-tc-workers/global/images/generic-worker-ubuntu-22-04-o6zboandezv8dqoy8b1v broken

Commit f8eee98 was needed to revert some worker pools from GCP back to AWS since the gcp image seems not to work. Workers are spawned, but they don't claim tasks.

It looks like projects/community-tc-workers/global/images/generic-worker-ubuntu-22-04-o6zboandezv8dqoy8b1v was created shortly before its equivalent aws images.

Set up Azure worker pools

Currently we have no Azure worker pools, so we do not test Azure provider in Worker Manager or discover issues, until taskcluster is deployed to fxci. We should have at least one or two Azure worker pools in the community deployment so we discover issues early related to this provider.

Migrate `mwz` project to Generic Worker

Manage the `login-identity:*` role as suggested in the docs

The suggestion gives auth:create-client:<..>/* etc.

Migrate `webrender` project to Generic Worker

Requesting access to Community instance for testing

Hi,

Forwarded here from the Taskcluster channel on Matrix.

I'm really interested in Taskcluster and would like to explore it and learn it, testing things like workers and jobs. Since there are no docs yet on how to set up a Taskcluster instance yourself, I was told I could possible get access to the community instance for it, which is why I'm making this issue.

If you need any more information, please let me know.

taskcluster / community-tc-config Goto Github PK

community-tc-config's Introduction

Taskcluster

The task execution framework that supports Mozilla's continuous integration and release processes.

Usage

Table of Contents

Team Mentions

Contributors

community-tc-config's People

Contributors

Stargazers

Watchers

Forkers

community-tc-config's Issues

Recommend Projects

Recommend Topics

Recommend Org