gocd / azure-elastic-agent-plugin Goto Github PK

GoCD Elastic Agent plugin for Microsoft Azure, which allows running GoCD elastic (dynamic) agents on Windows Azure virtual machines.

Home Page: https://www.gocd.org

License: Apache License 2.0

Java 95.33% HTML 3.80% FreeMarker 0.83% Fluent 0.04%

azure-elastic-agent-plugin's Introduction

GoCD Microsoft Azure Elastic Agent Plugin

This Elastic Agent plugin for Microsoft Azure allows you to run elastic agents on Windows Azure virtual machines. The plugin takes care of spinning up, shutting down virtual machines and installing go-agent based on the need of your deployment pipeline, thus removing bottlenecks and reducing the cost of your agent infrastructure. The plugin, based on the user-defined configuration, decides whether to spin up a new virtual machine or re-use an existing one.

The plugin can be configured to use a Azure virtual network, resource group and region where the virtual machines will be created. It supports both Windows and Linux virtual machines. The elastic profiles can be configured to use an image from Azure marketplace or a custom image from your subscription along with the size of the VM, disk space and more.

Once the builds finish, the virtual machine will be terminated or kept idle for a configured amount of time. This allows for more efficient usage of resources and reduces the overhead of creating virtual machines for each job. The VM will be terminated after the idle time and a fresh one will be created once required by a new job.

Building the code base

To build the jar, run ./gradlew clean check assemble

License

Copyright 2020 Thoughtworks, Inc.

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

azure-elastic-agent-plugin's People

Contributors

Stargazers

Watchers

Forkers

mgloder roamin9 maheshp arvindsv marques-work isabella232 joestr

azure-elastic-agent-plugin's Issues

Error installing elastic agent on new Azure VM

Issue Type

Bug Report

Summary

I'm getting this error with the Go Azure Elastic Agent not installing the agent on the VM that it spins up? I'm not even sure where to look to find this information for it to be edited.

Failed to install go agent on gocd-azure-c413d70b-f776-4d01-8435-5b77799a3ddf for job settingsrouter_experiment/2/build/1/windows due to error: freemarker.core.InvalidReferenceException: The following has evaluated to null or missing:
==> plugin_id [in template "windows_install_go_agent.template.flth" at line 20, column 17]

Tip: If the failing expression is known to legally refer to something that's sometimes null or missing, either specify a default value like myOptionalVar!myDefault, or use <#if myOptionalVar??>when-present<#else>when-missing</#if>. (These only cover the last step of the expression; to cover the whole expression, use parenthesis: (myOptionalVar.foo)!myDefault, (myOptionalVar.foo)??

FTL stack trace ("~" means nesting-related):
- Failed at: ${plugin_id} [in template "windows_install_go_agent.template.flth" at line 20, column 15]

Environment

Basic environment details

Go Version: 20.1
JAVA Version: 13
OS: Windows server 2012R2

Additional Environment Details

Steps to Reproduce

Set up elastic agent profile correctly
Run a build to have it spin up a VM in Azure based on a correct image
View the VM in Azure stuck in Creating state
Error in GO stating that it could not install the elastic agent on the newly created VM

Expected Results

It should install the agent based on the elastic agent plugin

Actual Results

I get the error posted above where it failed due to not being able to install the agent on the new vm.

Possible Fix

Log snippets

Needs verification: Leaving behind disks?

Need to check later: Does this plugin leave behind disks, even after the VM is terminated? It seemed like it, but needs confirmation.

Plugin settings and elastic profiles do not render correctly on GoCD 23.2.0+

plugin settings are cropped off after the "SSH public key" section on GoCD 23.2.0 and it's not possible to view or enter Windows etc configuration.
profile settings are cropped off after the "Custom Script" section on GoCD 23.2.0 and it's not possible to view or enter Agent idle timeout configuration etc.

Error Creating a New Elastic Profile

Hello,

I've setup a test GoCD server instance (version 20.3.0) and I'm experiencing a validation error when trying to create a new elastic profile. It looks like this may be part of a validation routine trying to check the the image URN entered in the UI? The specific fault is that the URN "OpenLogic:CentOS-CI:7-CI:latest" is being flagged as invalid even though I use this in other ARM templates that deploy fine. Watching the GoCD log files (specifically the go-server.log) when clicking the "Save" button I see the exception below get dumped out:

2020-06-30 20:40:45,443 ERROR [179@MessageListener for ServerPingListener] JMSMessageListenerAdapter:87 - Exception thrown in message handling by listener com.thoughtworks.go.server.messag ing.elasticagents.ServerPingListener@76b4eb12 java.lang.RuntimeException: java.lang.IllegalArgumentException: Parameter this.client.subscriptionId() is required and cannot be null. at com.thoughtworks.go.plugin.infra.FelixGoPluginOSGiFramework.executeActionOnTheService(FelixGoPluginOSGiFramework.java:214) at com.thoughtworks.go.plugin.infra.FelixGoPluginOSGiFramework.doOn(FelixGoPluginOSGiFramework.java:164) at com.thoughtworks.go.plugin.infra.DefaultPluginManager.submitTo(DefaultPluginManager.java:131) at com.thoughtworks.go.plugin.access.PluginRequestHelper.submitRequest(PluginRequestHelper.java:49) at com.thoughtworks.go.plugin.access.elastic.v4.ElasticAgentExtensionV4.serverPing(ElasticAgentExtensionV4.java:131) at com.thoughtworks.go.plugin.access.elastic.ElasticAgentExtension.serverPing(ElasticAgentExtension.java:63) at com.thoughtworks.go.plugin.access.elastic.ElasticAgentPluginRegistry.serverPing(ElasticAgentPluginRegistry.java:51) at com.thoughtworks.go.server.messaging.elasticagents.ServerPingListener.onMessage(ServerPingListener.java:31) at com.thoughtworks.go.server.messaging.elasticagents.ServerPingListener.onMessage(ServerPingListener.java:21) at com.thoughtworks.go.server.messaging.activemq.JMSMessageListenerAdapter.runImpl(JMSMessageListenerAdapter.java:83) at com.thoughtworks.go.server.messaging.activemq.JMSMessageListenerAdapter.run(JMSMessageListenerAdapter.java:63) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: java.lang.RuntimeException: java.lang.IllegalArgumentException: Parameter this.client.subscriptionId() is required and cannot be null. at com.thoughtworks.gocd.elasticagent.azure.AzurePlugin.handle(AzurePlugin.java:105) at com.thoughtworks.go.plugin.infra.DefaultPluginManager.lambda$submitTo$0(DefaultPluginManager.java:134) at com.thoughtworks.go.plugin.infra.FelixGoPluginOSGiFramework.executeActionOnTheService(FelixGoPluginOSGiFramework.java:208) ... 11 common frames omitted Caused by: java.lang.IllegalArgumentException: Parameter this.client.subscriptionId() is required and cannot be null. at com.microsoft.azure.management.compute.implementation.VirtualMachinesInner.listSinglePageAsync(VirtualMachinesInner.java:1689) at com.microsoft.azure.management.compute.implementation.VirtualMachinesInner.list(VirtualMachinesInner.java:1617) at com.microsoft.azure.management.resources.fluentcore.arm.collection.implementation.TopLevelModifiableResourcesImpl.list(TopLevelModifiableResourcesImpl.java:118) at com.thoughtworks.gocd.elasticagent.azure.client.GoCDAzureClient.virtualMachinesWithTag(GoCDAzureClient.java:205) at com.thoughtworks.gocd.elasticagent.azure.client.GoCDAzureClient.runningVirtualMachinesWithTag(GoCDAzureClient.java:177) at com.thoughtworks.gocd.elasticagent.azure.AzureInstanceManager.listInstances(AzureInstanceManager.java:65) at com.thoughtworks.gocd.elasticagent.azure.AzureAgentInstances.refreshAll(AzureAgentInstances.java:145) at com.thoughtworks.gocd.elasticagent.azure.AzurePlugin.refreshInstances(AzurePlugin.java:110) at com.thoughtworks.gocd.elasticagent.azure.AzurePlugin.handle(AzurePlugin.java:67) ... 13 common frames omitted

I don't see anywhere that a subscription ID is requested through either the plugin setup or the elastic profile creation so this is a bit confusing. In order to create the cluster profile I followed the steps documented on the page below; again I didn't see any mention of setting a subscription ID in these steps:

https://github.com/gocd/azure-elastic-agent-plugin/blob/master/docs/ELASTIC_PROFILE_CONFIGURATION.md#create-an-elastic-agent-profile-for-azure

Screenshot of error in the UI:

Thank you for any help you can provide in explaining or helping to resolve this issue.

Issues after plugin installs elastic agent into new Azure VM

When the elastic agent plugin creates a virtual machine and installs the agent into the VM, it appears under the static agent list and not the elastic agent list. This requires me to add resources for the agent for it to build, and doesn't get removed when the build is done. The VM is correctly removed, but since it's a static agent it would just show as lost contact in the GO portal.

To add to the complication, is there any possible way to have it create the agent so that it runs under a specific user account and not Local System? Would there be some way to implement a way for me to edit a file to provide the user account and password that the agent service should run under?

If there is any specific information you require about our setup just let me know.

Cannot create elastic profiles as plugin is deprecated

Using the plugin version 1.0.0-5 on the latest GoCD version.

The plugin will not allow to create elastic profiles because there is no cluster profile associated. And there is no option of create the said cluster profile.

Reason: GoCD 19.3.0 introduced the concept of cluster profile and deprecated the elastic agent extension v4 currently used by this plugin.

The said extension endpoint has been deprecated and a version v5 was introduced. This plugin needs to be updated to utilize the v5 version to support cluster profile and hence elastic profiles as the v4 version can be removed in the future releases and the plugin would no longer be loaded into the system.

Agent creation fails for Linux machines

The preparation for Linux VM is flawed.

While in the prepare script JDK 13 gets downloaded and installed in the start script references a JDK 12 installation.

Prepare script

azure-elastic-agent-plugin/src/main/resources/post_provision_script.template.ftlh

Line 5 in 7337443

 java_download_url="https://download.java.net/java/GA/jdk13.0.2/d4173c853231432d94f001e99d882ca7/8/GPL/openjdk-13.0.2_linux-x64_bin.tar.gz" 

azure-elastic-agent-plugin/src/main/resources/post_provision_script.template.ftlh

Line 15 in 7337443

tar -xvf openjdk-13.0.2_linux-x64_bin.tar.gz -C /var/lib/jdk/

Start script

azure-elastic-agent-plugin/src/main/resources/linux_start_go_agent.template.flth

Line 4 in 7337443

default_jre_dir="/var/lib/jdk/jdk-12"

JDK installation fails: Target directory not created

Version: 1.2.0-53 (PRE RELEASE)

Fails due to target extraction directory not created yet.

Log

2023-01-13 15:44:08,375 ERROR [162@MessageListener for CreateAgentListener] AzurePlugin:142 - Failed to create vm with name gocd-azure-ed17e034-c4a7-46f7-b858-de696d7187c6 for job test-maven/17/Build/1/default due to error: Async operation failed with provisioning state: Failed
com.microsoft.azure.CloudException: Async operation failed with provisioning state: Failed: VM has reported a failure when processing extension 'post-provision-script'. Error message: "Enable failed: failed to execute command: command terminated with exit status=2
[stdout]
Setting up unzip utility
unzip
Setting up JAVA

[stderr]
...... .......... 94%  225M 0s
 42500K .......... .......... .......... .......... .......... 94%  200M 0s
 42550K .......... .......... .......... .......... .......... 94%  228M 0s
<reduced>
 44950K .......... .......... .......... .......... .......... 99%  248M 0s
 45000K                                                       100% 27.6M=0.5s

2023-01-13 15:43:53 (81.9 MB/s) - ‘jre.tar.gz’ saved [46080826/46080826]

tar: /var/lib/jdk/17-jre: Cannot open: No such file or directory
tar: Error is not recoverable: exiting now
"

More information on troubleshooting is available at https://aka.ms/VMExtensionCSELinuxTroubleshoot  {}
2023-01-13 15:44:08,380 INFO  [162@MessageListener for CreateAgentListener] AzurePlugin:82 - Tearing down vm gocd-azure-ed17e034-c4a7-46f7-b858-de696d7187c6
2023-01-13 15:44:08,590 INFO  [162@MessageListener for CreateAgentListener] AzurePlugin:82 - Terminating vm gocd-azure-ed17e034-c4a7-46f7-b858-de696d7187c6
2023-01-13 15:44:49,183 INFO  [162@MessageListener for CreateAgentListener] AzurePlugin:82 - Terminating vm gocd-azure-ed17e034-c4a7-46f7-b858-de696d7187c6 disks
2023-01-13 15:45:19,879 INFO  [162@MessageListener for CreateAgentListener] AzurePlugin:82 - Terminating vm gocd-azure-ed17e034-c4a7-46f7-b858-de696d7187c6 nics
2023-01-13 15:45:24,502 INFO  [162@MessageListener for CreateAgentListener] AzurePlugin:82 - Clearing request fingerprint 1105851431 from the cache

Forcing pipeline to use only one agent ?