akanieski / ado-agent-orchestrator Goto Github PK

When not specifying a K8s job definition, it creates a Job with environment variables that contain a sensitive PAT. That should be moved to a generic secret.

Currently, the Kube Config is expected to be provided explicitly via an environment variable. Update the KubernetesAgentHostService to read the well-known kube config locations including the in-cluster location.

Agent container start times can be long anywhere from 10-60s. After analysis it seems that a fair bit of time is spent downloading a fresh copy of the agent binaries at runtime. This shouldn't have to happen every time.

Proposal
Update the sample start.ps1 and start.sh to first check if there is a /mnt/agent-cache and if there is check to see if the specific agent package has been downloaded already. If so just use that package instead of having to download the package at runtime.

@akanieski this is an absolutely awesome implementation you wrote, and I'm excited to try it out!

When creating the Job object, I noticed some things that might be an issue depending on the Kubernetes distribution that's being targeted:

  HostPath = new V1HostPathVolumeSource() {
      Path = "/var/run/docker.sock"
  }

I've seen many cases where using a HostPath source is blocked by an Admission Controller based on the security controls of the cluster. Additionally, there is an assumption that the docker.sock is accessible. Again, this might not be the case for many clusters including those that use alternate container runtimes (cri-o, containerd, etc.)

I saw the start script references ./run-docker.sh. Does that require the socket to exist? I'm assuming that script is included when the agent client gets downloaded so I can't see the source at the moment to see what it requires.

It would be very useful to be able to cache build-time assets like docker layers, nuget packages, npm packages etc.

With Kubernetes hosts it would be straightforward. You can use whatever persistent storage solution available to you k8s cluster and then main commonly cached paths to that persistent storage.

• Cache Azure Pipelines Tasks by mounting persistent storage to /azp/_work/_tasks
• Cache 'Installer Tasks' by mounting persistent storage to /azp/_work/_tool
• Cache NPM by mounting persistent storage to /root/.npm
• Cache Nuget by mounting persistent storage to /root/.nuget/packages
• Cache Docker layers by mounting persistent storage to /var/lib/docker/overlay exposing docker.sock
• Update docs to explain the above strategies for caching

The above needs testing in case anyone in the community would like to pitch in!

It would be great to be able to adjust the MINIMUM_AGENT_COUNT at runtime without have to restart the agent orchestrator service. It may also be useful to change other configurables at runtime.

Proposal
Replace get properties in KuberenetesAgentHostService and ACIAgentHostService for configurables like MINIMUM_AGENT_COUNT so that they first attempt to read from a path on the filesystem. This would give people the opportunity to use Kubernetes ConfigMap objects to adjust values at runtime. You could for example, scale your minimum agent count from a high number during busy periods, and scale down to less during slow periods.