CICD Management for the Organization

This document will summarize the design and concepts around a group of GitHub Actions workflows that will fulfill the following goals:

Create a centralized framework of CICD workflows based on GitHub Actions for use across the organization
Automate the onboarding of GitHub repositories into the CICD framework
Secure the framework from modification from anyone outside the infrastructure/CICD team
Limit the amount of work developers need to do integrate and use the CICD system
- Ideally only a configuration file to be processed by the CICD system
The framework should require minimal maintenance by the CICD team
Extending the framework should be simple and by convention

Assumptions

Promotion will be automated (i.e. promoting from one SDLC environment to the next will not be manual)
Every environment will run tests after deployment
The CICD system will assume that GitHub Action runners are self-hosted
Secrets are accessed from within the container orchestration platform, and not within GitHub
Each Git repository will consist of exactly one component or microservice.
Readers are familiar with Git, GitHub Actions, and Kubernetes.

Design

The design assumes three separate functional goals for the system:

Onboarding: Bringing a Git repository under control of the CICD system
Build from Source: Build a Git repository into an OCI compliant image and push to an image registry
Promotion and Deployment: Promote the image to the next environment if already deployed, and deploy the image into the container orchestration platform

Directory Structure

.github
- system-defs: Contains all system definitions
- workflows: Contains all reusable (centralized) GitHub Action workflows
.github-app-template
- workflows: Contains a YAML template of a workflow to be embedded in component repositories, and uses the reusable workflows in this repository
- CODEOWNERS (file): GitHub specific file for setting permissions withing a directory. Ensures only the infra and/or CICD will be able to push changes or approve pull requests to the workflows realized from this directory in a component Git repository.
run-step
- steps: Contains all directories for each codebase
  - <codebase id>: Directories named for each codebase ID, and the naming convention helps with leveraging the run step framework.

System Definitions

Collections of related software components, each in an individual Git repository, will each be defined in a YAML based configuration file owned and managed by the CICD team. Each configuration file will the define the codebase the component was developed with and the Git repo where the component resides. It will also contain a list of test environments the CICD system should provide to support each components SDLC.

sample-system-def.yml

organization: corp-org-name
components:
  - repo: microservice-a
    # java11 is an example of a codebase ID
    codebase: java11
    # status is an example of other possible fields that could add
    # further information for an organization or team
    status: active
  - repo: cicd-poc-app-2
    codebase: node16
    status: active
    overrides:
      # example of a component specific override the CICD system
      test: jest
# list of test environments used by the CICD system to support the SDLC
# development, pre-production, and production environments are assumed
test-environments:
  - qa
  - uat
  - stg

Codebase

An arbitrary ID created by the organization and assigned to each repository to designate the platform and language of a set of repositories. This ID is used by the run-step framework to determine which scripts to run to build, test, and scan a repository’s source code and/or build artifacts.

Reusable Workflows

GitHub Actions enables reusable workflows. This CICD system utilizes that feature to centralize the workflows for the organization. Any workflows that exist in an individual component repository should be skeletal and call the reusable workflows.

Run Steps

A homegrown convention-over-configuration framework for injecting scripts into the CICD system’s workflows. Depending on the ID used, scripts will be called from the run-step action; e.g.:

Example of a GitHub Action Step using the run-step action

    - id: test
      name: test
      uses: corp-org-name/cicd-manager/run-step@main
      with:
        label: ${{ env.CODEBASE }}
        step-name: test

In the above example, the label will determine which directory in the run-step directory in which to look for the script, and the step-name will determine the name of the script to run. In cases where the step-name needs to be overridden from it’s default value, it can be done with an override key/value pair in the System Definition File. In the system definition example above, the test step is overridden to run the jest.sh script in the node16 directory, rather than the default test.sh expected in that directory.

System Onboarding

The workflow (onboard-system.yml) for realizing and copying the .github-app-template into a system component repository. The template for a caller workflow that calls the centralized, reusable workflows is injected with the system name, copied into the target system component repository and renamed .github, and committed and pushed back to the repository.

In this way, the organization can control the CICD workflows across all component repositories from this central, CICD repository. Changes to the workflows in this repository can be instantly realized organization-wide simply by committing changes back into this repository.

Build

The build is represented by a single workflow (build-from-source.yml). The steps are summarized as follows:

Check out the source
Build the source (if necessary)
Run unit tests
Scan the source and build artifacts
Build the image
Scan the image
Push the image to a registry

Since building, testing, and scanning will be different given the platform, language, testing frameworks, etc., the codebase ID in the system definition file is used by the run step framework to determine how a repository is built, tested, and scanned.

Promote and Deploy

Image promotion and deployment is governed by a single workflow (promote.yml). The workflow uses a GithHub Actions matrix to manage deployment the development environment directly after the build takes place. The matrix is able to run serially instead of in parallel thanks to the max-parallel value being set to one.

Environment matrix from promote.yml

strategy:
  max-parallel: 1
  matrix:
    environment: ${{ fromJSON(inputs.environments) }}

Promotion will involve three basic steps:

Copying (or re-tagging) the image built in the build step
Deploying the image to the new environment
Running any integration tests in the new environment.

If the image deploys properly and the tests pass, then promotion to the next environment is automatically run; otherwise, the workflow fails.

The initial deployment to a "dev" environment will take place immediately after the image is built and pushed to the image registry. After that, the image will be promoted, deployed, and tested for each of the listed test environments in the System Definition File; e.g.

Sample list of environments in a system definition file

test-environments:
  - qa
  - uat
  - stg

Integration Test Step [TBD: requirements not fully defined]

The integration test step will launch a Job in the component environment to run the tests from there. The logs generated from the Job will be tailed for viewing in the GitHub Actions logs.

TDB: Integration Test Artifacts and Resources

Is the image for integration tests to be built with the component’s test artifacts along with the component image beforehand, or will there be a single image holding the test tools, and the test artifacts and resources pulled and/or copied from a separate artifact repository?

Production Promotion and Deployment

It is expected that promotion to a production environment for final release will happen in a separate workflow call than that of the initial build and promote workflow, but will still use this particular reusable workflow.

Component Repository Workflow Templates

Build and Promote

There is currently only one workflow templated for insertion into each component repository, and it first calls the centralized, reusable build workflow, parses the system definition file for itself to get the test environments for its SDLC, and then calls the promotion centralized, reusable workflow. When a component is onboarded the template will be realized and copied, committed, and pushed into the component repository for use by the developer.

Promote to Production

It is expected that the promotion to production will use a separate workflow. Some considerations that will differentiate it from the generic promotion workflow:

Canary or blue/green deployments
Approval process

training3 / cicd-manager Goto Github PK

cicd-manager's Introduction