Briefly describe what GitOps is and the main principles we want to adhere by in bedrock regardless of the CI/CD technology used this README.

We follow a release flow

1. Branch
2. Push
3. PR
4. Merge

We believe Git to be a source of truth.
We believe Git is enables collaboration at scale

We leverage operational features of the CI/CD platform to

• Configure
• Build
• Test
• Deploy
• Monitor

We align to the patterns of

• CI build artifacts
• Updates Config (in a git repo)
• Trigger Rollout(PR merge/commit)
• Observe
• Control

etc

Investigate and prototype.

Flux uses SSH based "deploy keys" to communicate with GitHub repos. It's unclear how we would authenticate with Azure DevOps git repos. Flux provides a guide on how to set up SSH keys on non GitHub repos here.

This will help unit tests run on any machine

Kube-Monkey may be the main options

Figure out what is currently available (to work with Azure)
What tests are possible
How would it integrate

As a developer, I want to focus on my application code and have all of the supporting infrastructure scaffolded out for me:

1. Application container builds (with buildpacks.io?)
2. Helm deployment charts

is there a minimum terraform version required?

and have you considered providing a docker container with binaries and scripts included?
If there is a need for this, i can send a PR.

All the best,

Benjamin

The helm chart in https://github.com/helm/charts/tree/master/incubator/jaeger is several versions behind now and has been superseded by the Jaeger project maintained Jaeger Operator.

We should be able to validate that the project works end to end before and after each commit with a CI system.

Some ideas on how to test a Terraform deployment here: https://devops.stackexchange.com/questions/863/how-to-test-a-terraform-configuration

in the setup section of cluster deployment, you need to install the Helm client... I had an error until I did so (I know its probably obvious - but I forgot to do it first, and got an error that helm was not found).

We're planning to use deploy keys for flux and ssh keys for pushing to the second repo from a script, but we would like to refresh these keys periodically and when we do that, it would need to be automatically updated on the github repo via a script. Explore the options for automating it (see github API)

Commit #21 fixed the Jaeger UI not showing up but no services show up in the UI. The correct way to pull the Istio based Jaeger UI is

kubectl port-forward -n istio-system $(kubectl get pod -n istio-system -l app=jaeger -o jsonpath='{.items[0].metadata.name}') 16686:16686 &

As referenced here.

After making this change I see a trace from the simple-service application with custom user-agent referenced. I only see one reference. This could be due to how tracing is implemented in simple-service. I'm going to investigate further to try other jaeger-client operations.

Running ./init asks for the var.container_repo variable, which looks to be docker.io/timfpark. What's that repo for?

Currently Flux requires a key that has read/write access to a GitHub repository. Need to investigate if this is actually needed, how to mitigate the security risk.

Add scripts to enable deploying multiple clusters and have them managed by a common ingress (traffic manager).

As a person in an operations role, I'd like to be able to spin up the Kubernetes cluster itself in an automated manner.

The deployment of the simple service is IMHO not working correctly.

the repo name refers to @timfpark and i cannot push to this docker repo.
https://github.com/Microsoft/bedrock/blob/master/services/common/deploy-service#L26

possbile way to go:

we change the values in the file:https://github.com/Microsoft/bedrock/blob/master/services/modules/simple-service/deploy-simple-service

or update the README with an overwrite of the default values.

or we could stop building it and relying on a prebuilt docker image.

The Dockerfile is currently missing the docker package and we would have to mount the docker socket into the container while starting it.

Steps done:
tried to test it via minikube:

docker build -t bedrock:latest .

docker run --rm -v /var/run/docker.sock:/var/run/docker.sock --net=host -it -v ~/.minikube:/.minikube -v ~/.kube/config:/.kube/config -e TF_VAR_grafana_admin_password="SECRETpass" bedrock:latest /bin/bash

cd services/environments/dev
./init
./apply

@timfpark what do you think?

All the best,

Benjamin

We need a repo similar to Getting Started directory that defines the high level definition deployments. This repo will be what our CI/CD will trigger off of.

Pin to a particular git commit in the Flux repo.

Provide configuration as an input to the cluster creation script for the git repo that you want Flux to pull from.

Per the research in #40, implement chaos for node/infra level.

Document available tests and how to enable/disable those tests.

Adjust Traefik configuration to provide TLS via Let's Encrypt

This should run fab generation as a verification to make sure components are valid and build passes before it's merged into master

Dropping terraform's state in the current directory is not a scalable solution, even if checked into git. Instead, persist this .tfstate in blob storage.

Expand on the README with detail instruction for steps that we currently can't automate.

The expectation here is that from scratch one can:

• Create an AKS cluster
  • Reference bedrock cluster instructions
• See an example of a HLD repo and a manifest repo
  • Suggest forking the repos?
  • Suggest repo policies to prevent commits to master
• Set up a build pipeline in Azure Pipelines associated with the HLD repo
  • Set appropriate environment variables
  • Setup a GitHub Personal Access Token
  • Reference provided shell scripts in build pipeline
• Experience a bedrock Gitops flow™
  • Describe how to make a PR against their HLD repo

Related to #27 we need to take the generated yam artifacts and git commit/push this files to a source of truth repo. For the moment we may be able to utilize the Azure Pipelines GitHub App to authenticate the git push command.

1. Refactor build.sh so that it is broken up into functions. That way, the unit test can source build.sh and reference reusable blocks of code from the script for testing.

2. Develop unit test shell script that will specify default environment variables and call functions from build.sh to run for testing. This should be able to run locally and report results.

Allow for the deployment of a Kubernetes / pod level chaos solution.

Implementation should include documenting available tests as well as implementing knobs to enable/disable tests.

• Implement Pod level chaos testing
• Implement latency injection istio fabrikate HDL
• Implement Fabrikate component for chaos stack
• Add Documentation for fabrikate component

Ala https://docs.microsoft.com/en-us/azure/aks/aad-integration for AD for AKS

The null resource provisioning requires bash shell. This should be at least called out in the readme or refactored to support both Windows and Linux.

As a person with operations responsibilities, I'd like to be able to:

1. Observe the success rate of requests to services in the system.
2. Secure east-west traffic between services with TLS.
3. Be able to flight new versions of services with traffic splitting.

Ideas for implementation include Istio, Linkerd2, or using a higher level management platform around these like supergloo

Look at kubediff (and similar solutions) and come up with a recommendation on how to:

• Deploy
• Execute
• Be notified of cluster and manifest repo parity

Also so: microsoft/fabrikate#29

Create a GitHub repo that will be the location that #28 publishes yam artifacts to

Suggest source repo to setup github repo rules that require at least 1 approver and checks to pass before pull request is allowed to merge.

Investigate the github API which might potentially allow setting these rules with the help of a personal access token. Currently it seems that a password is required to do this via the GUI. Reference

Ala Azure Key Vault or Hashicorp Vault

For moment create an Azure Pipelines instance that is triggered on commits to the HLDD repo. The CI/CD should run Fabrikate on the the components on the HLDD repo

Read here, and utilize azure pipelines to push to destination repo to avoid using ssh keys and personal access tokens.

Looked at a few options for developing unit tests, and it appears the using a shell script may be the simplest approach for now. Stumbled upon shUnit2, which is a unit test framework, and could be using it to aid in developing the unit test.

After discussing with the team, it appears that the following steps will take place:

1. Refactor build.sh so that it is broken up into functions. That way, the unit test can source build.sh and reference reusable blocks of code from the script for testing.

2. Develop unit test shell script that will specify default environment variables and call functions from build.sh to run for testing. This should be able to run locally and report results.

Currently you can not enable RBAC without a backing AAD service principal. In the meantime, use the az command line.
TODO: This support is expected within a couple of releases -- switch back when its available.

Lines 27 - 74 of providers/azure-aks/aks.tf