base2services / ciinabox-pipelines Goto Github PK

View Code? Open in Web Editor NEW

10.0 5.0 34.0 782 KB

Standard pipelines for ciinabox

Groovy 96.86% PowerShell 3.14%

jenkinsfile shared-library cloudformation aws amazon aws-cloudformation groovy declarative-pipeline

ciinabox-pipelines's Introduction

Ciinabox Shared Library Pipelines

A (Pipeline Shared Libraries)[https://github.com/jenkinsci/workflow-cps-global-lib-plugin] repo for Ciinabox

ciinabox-pipelines's People

Contributors

Stargazers

Watchers

ciinabox-pipelines's Issues

Add function to clear old snapshots after washery execution

Currently old washery snapshots aren't cleared up after washery runs are executed, this causes a build-up of old snapshots which ecentually exceeds the manual snapshot limit causing subsequent runs to fail.

Add function to clear up old washery snapshots.

Update cloudformation unable to delete parameter

getting error when trying to update a stack and removing a parameter.

com.amazonaws.services.cloudformation.model.AmazonCloudFormationException: Parameters: [parameterTag] do not exist in the template (Service: AmazonCloudFormation; Status Code: 400; Error Code: ValidationError; Request ID: b9ae3420-7e5d-11e8-a353-850e66563e2d)

Add "Detect Root Cause" to failed cloudformation outputs

AWS have added the ability to detect the root cause of a cloudformation failure. Add this to the output in jenkins whenever its made available in the SDK.

Add pagination to washeryCleanupSnapshots

Currently the washeryCleanupSnapshots function will not work as expected if there are more than 100 snapshots in the account. This is because the returned list of snapshots doesn't use pagination and thus doesn't return all snapshots.

cloudformation change set execition times out after an hour

we could use the sts GetSessionToken api to query how long is left in the sessions and if less than 10 minutes generate a new client and waiter.

https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/index.html?com/amazonaws/services/s3/AmazonS3ClientBuilder.html

CloudFormation stack update should fail if status is UPDATE_ROLLBACK_*

When updating a stack using the cloudformation.update() function, there is no error handling for when the stack fails to update (with status UPDATE_ROLLBACK_IN_PROGRESS). Instead, it returns as successful.

This causes tools like Jenkins to incorrectly report a successful job run.

Add a step for looking up the desired count of an ASG

Similar to the ECS one.

Packer step hasn't implemented root volume size

https://github.com/base2Services/ciinabox-pipelines/blob/master/src/com/base2/ciinabox/PackerTemplateBuilder.groovy#L60

tags aren't applied to baked packer AMI

Issue

option amiTags: ['key': 'value'] on the packer and packerAnsible steps isn't applying the tags to the baked AMI

Expectation

the tags should be applied to the baked AMI

17:34:16  java.lang.InterruptedException: sleep interrupted
17:34:16  	at java.lang.Thread.sleep(Native Method)
17:34:16  	at sun.reflect.GeneratedMethodAccessor11952.invoke(Unknown Source)
17:34:16  	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
17:34:16  	at java.lang.reflect.Method.invoke(Method.java:498)
17:34:16  	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
17:34:16  	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
17:34:16  	at org.codehaus.groovy.runtime.callsite.StaticMetaMethodSite$StaticMetaMethodSiteNoUnwrap.invoke(StaticMetaMethodSite.java:133)
17:34:16  	at org.codehaus.groovy.runtime.callsite.StaticMetaMethodSite.call(StaticMetaMethodSite.java:91)
17:34:16  	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:125)
17:34:16  	at ecrScan.waitForEcrScanResults(ecrScan.groovy:96)
17:34:16  	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
17:34:16  	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
17:34:16  	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
17:34:16  	at java.lang.reflect.Method.invoke(Method.java:498)
17:34:16  	at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
17:34:16  	at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
17:34:16  	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
17:34:16  	at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
17:34:16  	at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42)
17:34:16  	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
17:34:16  	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
17:34:16  	at com.cloudbees.groovy.cps.sandbox.DefaultInvoker.methodCall(DefaultInvoker.java:20)
17:34:16  	at ecrScan.call(ecrScan.groovy:27)
17:34:16  	at WorkflowScript.run(WorkflowScript:115)
17:34:16  	at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.delegateAndExecute(ModelInterpreter.groovy:137)
17:34:16  	at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.executeSingleStage(ModelInterpreter.groovy:661)
17:34:16  	at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.catchRequiredContextForNode(ModelInterpreter.groovy:395)
17:34:16  	at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.catchRequiredContextForNode(ModelInterpreter.groovy:393)
17:34:16  	at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.executeSingleStage(ModelInterpreter.groovy:660)
17:34:16  	at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.evaluateStage(ModelInterpreter.groovy:288)
17:34:16  	at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.toolsBlock(ModelInterpreter.groovy:544)
17:34:16  	at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.toolsBlock(ModelInterpreter.groovy:543)
17:34:16  	at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.evaluateStage(ModelInterpreter.groovy:276)
17:34:16  	at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.withEnvBlock(ModelInterpreter.groovy:443)
17:34:16  	at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.withEnvBlock(ModelInterpreter.groovy:442)
17:34:16  	at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.evaluateStage(ModelInterpreter.groovy:275)
17:34:16  	at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.withCredentialsBlock(ModelInterpreter.groovy:481)
17:34:16  	at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.withCredentialsBlock(ModelInterpreter.groovy:480)
17:34:16  	at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.evaluateStage(ModelInterpreter.groovy:274)
17:34:16  	at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.inDeclarativeAgent(ModelInterpreter.groovy:586)
17:34:16  	at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.inDeclarativeAgent(ModelInterpreter.groovy:585)
17:34:16  	at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.evaluateStage(ModelInterpreter.groovy:272)
17:34:16  	at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.stageInput(ModelInterpreter.groovy:356)
17:34:16  	at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.stageInput(ModelInterpreter.groovy:355)
17:34:16  	at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.evaluateStage(ModelInterpreter.groovy:261)
17:34:16  	at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.inWrappers(ModelInterpreter.groovy:613)
17:34:16  	at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.inWrappers(ModelInterpreter.groovy:612)
17:34:16  	at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.evaluateStage(ModelInterpreter.groovy:259)
17:34:16  	at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.withEnvBlock(ModelInterpreter.groovy:443)
17:34:16  	at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.withEnvBlock(ModelInterpreter.groovy:442)
17:34:16  	at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.evaluateStage(ModelInterpreter.groovy:254)
17:34:16  	at ___cps.transform___(Native Method)
17:34:16  	at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:86)
17:34:16  	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:113)
17:34:16  	at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:83)
17:34:16  	at sun.reflect.GeneratedMethodAccessor351.invoke(Unknown Source)
17:34:16  	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
17:34:16  	at java.lang.reflect.Method.invoke(Method.java:498)
17:34:16  	at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
17:34:16  	at com.cloudbees.groovy.cps.impl.LocalVariableBlock$LocalVariable.get(LocalVariableBlock.java:39)
17:34:16  	at com.cloudbees.groovy.cps.LValueBlock$GetAdapter.receive(LValueBlock.java:30)
17:34:16  	at com.cloudbees.groovy.cps.impl.LocalVariableBlock.evalLValue(LocalVariableBlock.java:28)
17:34:16  	at com.cloudbees.groovy.cps.LValueBlock$BlockImpl.eval(LValueBlock.java:55)
17:34:16  	at com.cloudbees.groovy.cps.LValueBlock.eval(LValueBlock.java:16)
17:34:16  	at com.cloudbees.groovy.cps.Next.step(Next.java:83)
17:34:16  	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
17:34:16  	at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
17:34:16  	at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129)
17:34:16  	at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268)
17:34:16  	at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
17:34:16  	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:18)
17:34:16  	at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:51)
17:34:16  	at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:185)
17:34:16  	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:400)
17:34:16  	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$400(CpsThreadGroup.java:96)
17:34:16  	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:312)
17:34:16  	at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:276)
17:34:16  	at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:67)
17:34:16  	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
17:34:16  	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:136)
17:34:16  	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
17:34:16  	at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
17:34:16  	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
17:34:16  	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
17:34:16  	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
17:34:16  	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
17:34:16  	at java.lang.Thread.run(Thread.java:748)
17:34:16  Finished: FAILURE

Execute changeset failed when state is 'UPDATE_COMPLETE_CLEANUP_IN_PROGRESS'

01:18:36 [Pipeline] echo
01:18:36 execute changeset update failed with status UPDATE_COMPLETE_CLEANUP_IN_PROGRESS

Changesets longer than 1 hour still fail due to expired security token

13:20:10  com.amazonaws.services.rds.model.AmazonRDSException: The security token included in the request is expired (Service: AmazonRDS; Status Code: 403; Error Code: ExpiredToken; Request ID: 72ad534e-23a7-4453-bdca-47b35098817e; Proxy: null)
13:20:10  	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1

Fix from commit 7ce555587794dd4a2ea88524f148900da5ec7907 not worked

This one worked:

def wait(clientBuilder, stackName, changeSetType) {
  def cfclient = clientBuilder.cloudformation()
  def waiter = null
  def count = 0
  switch(changeSetType) {
    case 'CREATE':
      waiter = cfclient.waiters().stackCreateComplete()
      break
    default:
      waiter = cfclient.waiters().stackUpdateComplete()
      break
  }

  try {
    def future = waiter.runAsync(
      new WaiterParameters<>(new DescribeStacksRequest().withStackName(stackName)),
      new NoOpWaiterHandler()
    )
    while(!future.isDone()) {
      try {
        echo "waiting for execute changeset to ${changeSetType.toLowerCase()} ..."
        Thread.sleep(10000)
        count++
        echo "Current Client - ${cfclient} & Current Waiter - ${waiter}"
        // Initialise new client and waiter if count exceeds set timeout value
        if (count > 300) { //3000 seconds = 50 minutes, thread sleep is 10 secs so 300 iterations
          future = waiter.runAsync(
             new WaiterParameters<>(new DescribeStacksRequest().withStackName(stackName)),
             new NoOpWaiterHandler()
          )
          cfclient = updateClient(clientBuilder) 
          waiter = updateWaiter(cfclient,changeSetType)
          count = 0
        }

      } catch(InterruptedException ex) {
          // suppress and continue
      }
    }
  } catch(Exception ex) {
    cfclient = null
    echo "execute changeset ${changeSetType.toLowerCase()} failed with error ${ex.getMessage()}"
    return false
  }
  
  // check the final state of the stack to check for any false positive stack status such as ROLLBACK_COMPLETE
  def request = new DescribeStacksRequest().withStackName(stackName)
  def stacks = cfclient.describeStacks(request).getStacks()
  def finalStatus = stacks[0].getStackStatus()
  cfclient = null
  if (!finalStatus.matches("UPDATE_COMPLETE|CREATE_COMPLETE")) {
    echo "execute changeset ${changeSetType.toLowerCase()} failed with status ${finalStatus}"
    return false
  }
  
  return true
}

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.