Giter VIP home page Giter VIP logo

dm885_project_devsecops's Introduction

DM885 MiniZinc solver service DevOps

General project structure

  • Use Google's Skaffold for development of local pods
  • Use Terraform to start all 3rd party services (Message queues, Databases, etc...)
  • Skaffold has a pipeline to build and deploy to Helm charts, and has guides on how to integrate with CI/CD. It also has profiles could maybe be used for production and development environments

Local development

Testing different tools for managing Kubernetes development.

How to run it

  • Make sure Minikube is running minikube start. Then make sure the tunnel is running by running minikube tunnel in a separate terminal.

  • In the ./terraform/development directory:

    • run terraform init
    • run terraform apply to start 3rd party dependencies
    • wait for it to completeley finish without errors before proceeding.
  • Before starting the local services set the PROJECT_DOMAIN environment variable appropriateley. For local development, this should be something like project.127.0.0.1.sslip.io, but with the external IP of the Istio Loadbalancer.

    This can be configured with this one-liner:

    export PROJECT_DOMAIN=$(kubectl get svc istio-ingressgateway -n istio-system -o jsonpath="project.{.status.loadBalancer.ingress[0].ip}.sslip.io")
    
  • Run skaffold dev in the ./services/ directory, to build and start all micro services at once. When changes are made to any service, Skaffold will automatically rebuild them.

The project can now be accessed at http://project.127.0.0.1.sslip.io/

Interacting with the services

The kubectl CLI can be used to interact with the local Kubernetes cluster.

  • kubectl get all -A get an overview of most of the running resources.
  • kubectl get kservice -n project get all running Knative services and their exposed URLs.
  • kubectl get virtualservice -n project get all Istio VirtualServices (API routes).
  • kubectl get vs -n project api-service-routes -o yaml get info about a specific VirtualService.

Project setup in cloud

Configuration

For every service in the services folder change the skaffold.yaml files to the github page where the packages should be hosted E.g. - image: ghcr.io/yousofmersal/dm885_auth-service to - image: ghcr.io/{YOU_REPO}/dm885_auth-service

Change cluster configuration in Terraform and in the deploy pipeline.

The configuration variables are in ´terraform.tfvars´.

By default it uses ´e2-standard-4´ instances'. Beaware small instances may not be able to run Istio.

Static IP-address

When the cluster is deployed then change the IP-address to static.

Run

Get the credentials and configure kubectl:

gcloud container clusters get-credentials steam-airfoil-367315-gke --region europe-west8-b

Provision the cluster by running the following commands in the terraform/google_cloud/ directory:

terraform init
terraform apply

When the cluster is up and running, run the same terraform commands as above but from the terraform/production/ directory, this will install and start all third party services such as databases and Knative on the newly created cluster.

List and set project

gcloud projects list

gcloud projects create dm885-dev

gcloud config set project dm885-dev

Enable GKE

gcloud --project=dm885-dev services enable container.googleapis.com

gcloud --project=dm885-dev services enable compute.googleapis.com

Get credentials for cluster

gcloud container clusters list

gcloud container clusters get-credentials dm885-dev-gke --region europe-north1

Get service-account key

  • Add secret in github action to enable pipeline

gcloud iam service-accounts list

gcloud iam service-accounts keys create key.json [email protected]

After retrieving your key in the repository settings under Secrets, Actions: Add the content of the key-file as a repository secret named GKE_SA_KEY.

dm885_project_devsecops's People

Contributors

yousofmersal avatar dannikiskov avatar jonasgroendahl avatar viktorstrate avatar henrikrossen avatar setin2 avatar

Stargazers

Morten Lund Jakobsen avatar

Watchers

James Cloos avatar  avatar

dm885_project_devsecops's Issues

Bug in skaffold when setting up

Steps to reproduce:

  • On a clean system follow the instructions in README.MD in root folder.
  • after terraform apply, in the services folder when running skaffold dev you get the error message panic: interface conversion: util.VersionedConfig is *v2.SkaffoldConfig, not *latest.SkaffoldConfig
with the trace
panic: interface conversion: util.VersionedConfig is *v2.SkaffoldConfig, not *latest.SkaffoldConfig

goroutine 1 [running]:
github.com/GoogleContainerTools/skaffold/pkg/skaffold/parser.getConfigs({_, _}, {{0xc000bd7dc0, 0x3f}, {0x0, 0x0, 0x0}, {0x436c300, 0x0, 0x0}, ...}, ...)
	/tmpfs/src/github/skaffold/pkg/skaffold/parser/config.go:137 +0xa54
github.com/GoogleContainerTools/skaffold/pkg/skaffold/parser.GetConfigSet({_, _}, {{0x0, 0x0}, {0xc000659d90, 0xd}, {0x27f53bd, 0x3}, {0x0, 0x0}, ...})
	/tmpfs/src/github/skaffold/pkg/skaffold/parser/config.go:85 +0x1a8
github.com/GoogleContainerTools/skaffold/cmd/skaffold/app/cmd.withFallbackConfig({_, _}, {_, _}, {{0x0, 0x0}, {0xc000659d90, 0xd}, {0x27f53bd, 0x3}, ...}, ...)
	/tmpfs/src/github/skaffold/cmd/skaffold/app/cmd/runner.go:110 +0x79
github.com/GoogleContainerTools/skaffold/cmd/skaffold/app/cmd.runContext({_, _}, {_, _}, {{0x0, 0x0}, {0xc000659d90, 0xd}, {0x27f53bd, 0x3}, ...})
	/tmpfs/src/github/skaffold/cmd/skaffold/app/cmd/runner.go:82 +0x85
github.com/GoogleContainerTools/skaffold/cmd/skaffold/app/cmd.createNewRunner({_, _}, {_, _}, {{0x0, 0x0}, {0xc000659d90, 0xd}, {0x27f53bd, 0x3}, ...})
	/tmpfs/src/github/skaffold/cmd/skaffold/app/cmd/runner.go:62 +0x7b
github.com/GoogleContainerTools/skaffold/cmd/skaffold/app/cmd.withRunner({0x2d63cd8, 0xc0001460c0}, {0x2d0c520, 0xc00058c370}, 0xc000b3fad0)
	/tmpfs/src/github/skaffold/cmd/skaffold/app/cmd/runner.go:50 +0x83
github.com/GoogleContainerTools/skaffold/cmd/skaffold/app/cmd.runDev({0x2d63cd8, 0xc0001460c0}, {0x2d0c520, 0xc00058c370})
	/tmpfs/src/github/skaffold/cmd/skaffold/app/cmd/dev.go:66 +0x1f3
github.com/GoogleContainerTools/skaffold/cmd/skaffold/app/cmd.(*builder).NoArgs.func1(0x9b4420, {0xc000b3fb80, 0xc000b3fb58, 0x0})
	/tmpfs/src/github/skaffold/cmd/skaffold/app/cmd/commands.go:138 +0x66
github.com/GoogleContainerTools/skaffold/cmd/skaffold/app/cmd.applyPostRunHooks.func1(0xc000211080, {0x436cad8, 0x0, 0x0})
	/tmpfs/src/github/skaffold/cmd/skaffold/app/cmd/commands.go:185 +0x2f
github.com/spf13/cobra.(*Command).execute(0xc000211080, {0x436cad8, 0x0, 0x0})
	/Users/kbuilder/go/pkg/mod/github.com/spf13/[email protected]/command.go:856 +0x60e
github.com/spf13/cobra.(*Command).ExecuteC(0xc000998000)
	/Users/kbuilder/go/pkg/mod/github.com/spf13/[email protected]/command.go:974 +0x3bc
github.com/spf13/cobra.(*Command).ExecuteContextC(...)
	/Users/kbuilder/go/pkg/mod/github.com/spf13/[email protected]/command.go:911
github.com/GoogleContainerTools/skaffold/cmd/skaffold/app.Run({0x2d0b6e0, 0xc000196008}, {0x2d0b6e0, 0xc000196010})
	/tmpfs/src/github/skaffold/cmd/skaffold/app/skaffold.go:50 +0x2ad
main.main()
	/tmpfs/src/github/skaffold/cmd/skaffold/skaffold.go:49 +0x1e5

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.