Giter VIP home page Giter VIP logo

aws-codeseeder's Introduction

aws-codeseeder

The aws-codeseeder project enables builders to easily "seed" python code to AWS CodeBuild for execution in their cloud environments.

The library and its CLI utility are typically used to simplify the development and deployment of complex Infrastructure as Code projects. These projects may have many dependencies and require multiple CLI utilities to orchestrate their deployments. For example, a project that deploys networking resources with the AWS CDK, an Amazon EKS Cluster with the eskctl CLI utility, and Kubernetes applications with kubectl and helm CLI utilities.

The aws-codeseeder eliminates the need to install and configure libraries and utilities locally or on a build system (i.e. Jenkins). Instead, the library enables builders to easily execute an AWS CodeBuild instance with the utilities they require, then seed local python code to, and execute it within, the CodeBuild instance. By bundling and executing local python code in AWS CodeBuild, aws-codeseeder can enable GitOps type deployments of complex, mixed technology projects.

Usage

See the Example project for basic usage info and the documentation for more advanced usage.

aws-codeseeder's People

Contributors

chamcca avatar dependabot[bot] avatar dgraeber avatar igorborgest avatar malachi-constant avatar rb201 avatar srinivasreddych avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

aws-codeseeder's Issues

[FEATURE] Support for China Regions

Is your feature request related to a problem? Please describe.
I would like to have CodeSeeder support for partitions other than aws...in particular for aws-cn and aws-us-gov

[FEATURE] Expose CFN methods to deploy seedkit

Describe the solution you'd like
I want call a codeseeder API to see if seedkit is deployed for my project (not via decorator patterns).
I want to be able to check:

  • if seedkit deployed, return the cfn output in a dict
  • if not, call an exposed method to deploy the seedkit and return the CFN output

** USE CASE:

  • my code needs artifacts created by seedkit, but the methods decorated for codeseeder are not called until after the needed artifacts are required to be set

Does not support deployments triggered from windows environment

Due the the direct string replacements used to transform the absolute local path of the source files to a bundle relative path we bundle the entire local path when running in a windows environment. Suggest fixing as follows within _bundle.py:

zip_relative_path = os.path.relpath(pathlib.Path(path), pathlib.Path(root_dir)) zf.write(path, zip_relative_path)

[FEATURE] Expose CFN stack commands for ProjectManagedPolicy

Lets discuss feasibility of below...?

Is your feature request related to a problem? Please describe.
Similar to the creation of the seedkit toolkit, our code calls the creation of a project managed policy that is digging deep into the codeseeder codebase.

Describe the solution you'd like
Create a command level function that will check if the managed policy stack is deployed, return a boolean, the stackname and output.
Create a command level function that will create the stack
Create a command level function to destroy the stack

A similar solution to this issue, but for the managed policy:
#20
#21

[FEATURE] asynchronous `remote_function` support

Is your feature request related to a problem? Please describe.
At times it may be necessary to asynchronously execute the remote_function code. Currently the processes blocks until the CodeBuild execution completes or errors.

Describe the solution you'd like
A mechanism for flagging a remote_function for asynchronous execution. When the function executes, rather than blocking until CodeBuild completes or errors, the library could block until CodeBuild starts, then return the ExecutionId. A secondary function to rejoin and/or check execution status and retrieve any return values will also be required.

Additional context
The use case being targeted is to begin execution of a remote_function within AWS Lambda function and then check execution status with another Lambda function. This would enable CodeSeeder workflows to be used in AWS StepFunctions StateMachines. For this to work, the asynchronous remote_function will need to return the CodeBuild ExecutionID so that it can be returned from the Lambda. subsequent Lambda functions would take the ExecutionID as input and use it to check Execution status.

[FEATURE] - Add data to error messaging from CodeSeeder

Is your feature request related to a problem? Please describe.
When a failure occurs, a Runtime error is thrown by CodeSeeder, and a customized string is added to the error message including the build id, but no other information.

Describe the solution you'd like
Create a new Error class that will contain a customized error string and a dict of any/all pertinent data (account id, region, build_id, etc)

Describe alternatives you've considered
Crafting a custom dict that serialized to a string as the error message

[BUG] Cannot add files to bundle that are nested / not at root

Describe the bug
When trying to add a file to the bundle via the decorators (extra_files) codeseeder throws an error if the file is in a directory.

For example, when adding seedfarmer.yaml to the bundle, that file is located at the root of the bundle, so it is added easily. But when trying to add an extra file located at data/somewhere/extradata.json, codeseeder issues an error:

FileNotFoundError: [Errno 2] No such file or directory: <userPath>/idf-modules/codeseeder.out/examples-extra-dummy-dummy/bundle/data/somewhere/extradata.json because the parent directory data/somewhere does not exists in the local bundle path under codeseeder.out.

Please see line 132 in aws_codeseeder/_bundle.py where the shutil.copy takes place

Make Codeartifact optional

At this point, when we deploy a seedkit using codeseeder, it deploys AWS Codeartifact domain and AWS Codeartifact Repository by default. This default behavior of creating AWS Codeartifact will cause an issue in the regions where it is not supported. AWS Codeartifact should be created optionally

[FEATURE] Windows support for CodeArtifact Management

We have added support for deploying on Windows OS (ref PR #81 ), but when trying to execute codeseeder deploy modules --module aws-codeseeder:. to upload and artifact, the upload fails as it is based on a bash script.

REF: aws-codeseeder/aws_codeseeder/commands/_module_commands.py
REF: aws-codeseeder/aws_codeseeder/resources/update_repo.sh

Add support for windows to upload a library?

NOTE: this does not affect deployments....only the management of CodeArtifact-enabled projects who want to upload libraries.

CodeSeeder not creating seedkit by default

First invocation of codeseeder from a client fails if the seedkit has not been deployed. It would be great if the seedkit would be created automatically if codeseeder detects that one does not exist on first invocation.

[BUG] Stack trace with failed SecretsManager pull

Describe the bug
When deploying without a SecretsManager configured, there is a stack trace stating that an error occurred. This is true, an error occurred, but the stack trace is confusing as the build process continues.

To Reproduce
Deploy a module via seedfarmer with a module that does not use SecretsManager (is not logging into DockerHub). See the stack trace

Expected behavior
When not using DockerHub or SecretsMananger, this stack trace should not indicate an error?

** Sample Log**

An error occurred (ResourceNotFoundException) when calling the GetSecretValue operation: Secrets Manager can't find the specified secret.
Traceback (most recent call last):
  File "/var/scripts/retrieve_docker_creds.py", line 24, in get_secret
    get_secret_value_response = client.get_secret_value(SecretId=secret_name)
  File "/root/.pyenv/versions/3.7.10/lib/python3.7/site-packages/botocore/client.py", line 415, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/root/.pyenv/versions/3.7.10/lib/python3.7/site-packages/botocore/client.py", line 745, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.ResourceNotFoundException: An error occurred (ResourceNotFoundException) when calling the GetSecretValue operation: Secrets Manager can't find the specified secret.

[FEATURE] Enable setting and updating of boto3 Session

Is your feature request related to a problem? Please describe.
In order to support cross-account or cross-region execution of the AWS CodeBuild it may be necessary to create a boto3 Session with credentials acquired from an sts:AssumeRole request or with a region other than the default. Currently the boto3 Session for all boto3 operations is created by CodeSeeder with default credentials and region.

Describe the solution you'd like
A method (function call? config?) for setting a boto3 Session that is used for boto3 calls. The method should enable updating of the boto3 Session so that short lived credentials can be refreshed during long running CodeBuild executions.

Describe alternatives you've considered
Overriding internal and/or private functions to manually set the boto3 Session

[BUG] Override of config parameters not working

Describe the bug
When passing in a decorator to override the configuration of aws-codeseeder, the changes are not being evaluated.

For example, the codebuild_image is configured as a config parameter (default) and the image itself is changed with a remote decorator. That image override is not referenced.

REF: codeseeder.py line 217

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.