The GCP PubSec Declarative Toolkit is a collection of declarative solutions to help you on your Journey to Google Cloud. Solutions are designed using Config Connector and deployed using Config Controller.

License: Apache License 2.0

Shell 100.00%

pubsec-declarative-toolkit's Introduction

GCP PubSec Declarative Toolkit

Current Solutions

Name	Description	Documentation
Guardrails	Base Infrastructure for 30 Day Guardrail Deployment	link
Organization Policy Bundle	Package of Baseline Organization Policies	link
Guardrails Policy Bundle	Policy Bundle to help analyze compliance for Guardrails	link
KCC Namespaces	This solution is a simple fork of the KCC Project Namespaces blueprint found here	link
Landing Zone v2 (LZv2)	(In development) PBMM Landing Zone built in collaboration with Shared Services Canada	link
Gatekeeper Policy (LZv2)	Policy Bundle	link
Core Landing Zone (LZv2)	Foundational resources building the landing zone	link
Client Setup (LZv2)	Package to create the initial client folder and namespaces	link
Client Landing Zone (LZv2)	Package to create the client folder sub-structure and a standard Shared VPC	link
Client Project Setup (LZv2)	Package to create a service project and host workloads	link
GKE Setup (LZv2)	Package to prepare a service project for GKE clusters	link
GKE Defaults (LZv2)	A package to deploy common GKE resources	link
GKE Cluster Autopilot (LZv2)	A GKE Autopilot Cluster running in a service project	link
Cluster Defaults (LZv2)	This package deploys default resources that have to exist on all GKE clusters	link
Namespace Defaults (LZv2)	This package deploys a workload namespace and it's associated configuration	link

When getting a package you can use the @ to indicate what tag or branch you will be getting with the kpt pkg get command for example kpt pkg get https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit.git/solutions/core-landing-zone@main.

You can find the latest release versions in the releases page.

Quickstart

Deploying an example landing zone requires two steps:

A Config Connector enabled Kubernetes cluster
One or more solutions packages like the core-landing-zone and experimentation core-landing-zone documented in section 2 of landing-zone-v2

In order to deploy the solutions you will need a Kubernetes cluster with Config Connector installed.

We recommend using the Managed Config Controller service which comes bundled with Config Connector and Anthos Config Management, alternatively you can install Config Connector on any CNCF compliant Kubernetes cluster.

See the Google Cloud quickstart guide for getting up and running with Config Controller

A setup script is provided in the repository gcp-tools that will configure the Config Controller cluster. The instructions in the Advanced Install are automated as part of the setup-kcc.sh script.

We have also put together the following guide to deploy a standalone Config Controller instance or see the examples directory for example installation methods.

After the Kubernetes cluster is fully provisioned - proceed to Deploy a landing zone v2 package.

Additional Documentation

You may want to look at the documentation published by Shared Services Canada, providing a good level of details on how they have implemented the Landing Zone v2 solution to host workloads from any of the 43 departments of the Government of Canada.

For further documentation on the project, including the setup pre-requirements and supporting service such as Config Connector and Config Management.

Additional Resources

Disclaimer

This is not an officially supported Google product.

pubsec-declarative-toolkit's People

Contributors

Stargazers

Watchers

pubsec-declarative-toolkit's Issues

As a developer I need a single node GKE cluster down from the default 3 as part of overall FinOps cost reduction for the anthos config controller

Use case: we should be able to run the master and worker nodes on one single VM (8g I hope) - I know this defeats replicaset scaling and HA but this will reduce the cost from by close to 1/3 to keep the K8S cluster up.

The anthos config controller has a 2 part fee schedule - anthos/hr and the gke /hr. Both defaulting to .1
https://cloud.google.com/anthos-config-management/docs/pricing

Something like using Chris's private GKE cluster solution of 1 - as the KCC cluster itself
https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/main/services/private-gke/setters.yaml#L49

Finding the values/setters yaml override for use by arete

checking
https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/main/solutions/sandbox-gke/private-gke/gke/containernodepool.yaml#L27

Arete values.yaml generated or use the non arete advanced manual install

Cost/day is 13.35 CAD or $460/month with tax

Create a solution config yaml specification

Create a yaml specification for solution configuration items.

this should include items like:

pre-flight checks
iam
etc...

GCP User Analysis Dashboard Solution

Understanding the daily usage of the GCP platform will provide Google Cloud customer leadership with a clear line of sight as to the uptake and use. Building out a console to enable for this visibility would be across all of the projects leveraging the GCP platform.

[Solution Document] (https://docs.google.com/document/d/1RrbseQkkZd3rrz_9AiEbNA9Xk_xyfkv7EthA_USAf40/preview)

Adjust landing-zone readme instructions for --region flag

adjust readme instructions to pass in the region when standing up the landing zone solution

following
https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/tree/main/solutions/landing-zone#usage

admin_root@cloudshell:~/wse_github/20220831/pubsec-declarative-toolkit (pubsec-declarative-toolkit-cno)$ arete create landing-zone-controller
Error: required flag(s) "region" not set
Usage:
  arete create <instance-name> [flags]

Examples:
 arete create my-awesome-kcc --region=us-central1 # This will create a new project and prompt for billing ID
 arete create my-awesome-kcc --region=us-central1 --project=my-awesome-project # This will utilize an existing project

Flags:

change...

admin_root@cloudshell:~/wse_github/20220831/pubsec-declarative-toolkit (pubsec-declarative-toolkit-cno)$ arete create landing-zone-controller --region=northamerica-northeast1
5:46PM INF Project name will be set to: landing-zone-controller-e8dj3
Use the arrow keys to navigate: ↓ ↑ → ←
? Choose a billing account:
  ▸ billing-account-2 - 015DB..02
    My Billing Account - 019..E3D

Verified

admin_root@cloudshell:~/wse_github/20220831/pubsec-declarative-toolkit (pubsec-declarative-toolkit-cno)$ arete create landing-zone-controller --region=$REGION
5:50PM INF Project name will be set to: landing-zone-controller-9bomd
✔ My Billing Account - 0199≥...D
✔ nuage-cloud.org - 47...947
✔ Folder Level
✔ pdt - 346...8
5:51PM INF Create in progress for [https://cloudresourcemanager.googleapis.com/v1/projects/landing-zone-controller-9bomd].Waiting for [operations/cp.8322183786461892312] to finish.....done.Enabling service [cloudapis.googleapis.com] on project [landing-zone-controller-9bomd]...Operation "operations/acat.p2-890165283493-618b6b75-ab44-49c6-be35-11fbdef2a7a0" finished successfully.Updated property [core/project] to [landing-zone-controller-9bomd].
5:51PM INF Config Controller setup complete

Add Billing Export to GCS bucket Solution

A common problem for GCP customers is pulling billing data from Google Cloud Platform into their own reporting systems.

A couple different patterns are starting to emerge to enable billing data to be exported / imported into third-party systems. The most common pattern and the focus of the first release of this solution is:

Export the data from BigQuery using BigQuery Data Transfer API into a Cloud Storage Bucket

[ENHANCEMENT] add an additional template for enhancements to the single [BUG}

Describe the bug
Raising an issue is hardcoded to the bug template - we either need permission to use the empty template for enhancement or add a 2nd one
I would also add sections on bug peer/related links and a section for shell content and workarounds

To Reproduce
Steps to reproduce the behavior:

Go to '...'
Click on '....'
Scroll down to '....'
See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

Command Line tool for generating and Deploying Configs

In order to ease the consumption of services we should have a cli based installer to help adoption.

Cobra is a gobased CLI framework which we could leverage to help https://github.com/spf13/cobra/blob/master/projects_using_cobra.md

Kpt Go Pkg -> https://pkg.go.dev/github.com/GoogleContainerTools/kpt/commands

example

sandbox create package guadrails

Ideally this would generate a list of services available for the user to select from to provision the configs.

As a org admin i need to onboard a new L1 client via either profile: shared VPC or peered VPC under the 3 workload folders

TBD - procedure to add a new L1 client (dev/prod/uat (ideally automated, for now yaml adjustment procedure

Keep in mind the project architecture for various types of teams
Requirements

1: Shared VPC (with service projects) - PaaS + shared FaaS
all tenants share a shared VPC host subnet
some tenants share a particular shared VPC host subnet
single tenants get their own 1:1 shared VPC host subnet
2: Mix Shared VPC + Peered VPCs
some tenants have a mix of shared
3: Peered VPCs (single tenant PaaS, IaaS, FaaS)
some tenants want their own distinct VPC with their own servless VPC endpoints

Discussion
Define L1 as a completely separate folder - a business unit or even at the level of regional offices, an L2 would be a business/project client under that (they would shared the VPC of the L1 or as a dev client have their own peered VPC (if they need something different than their inherited K8S PaaS shared VPC for example - or a serverless endpoint not in the parent VPC).

We have 2 dimensions:
1 - different divisions + sub teams + projects per team
2 - serverless FaaS/SaaS/PaaS/IaaS workloads per team ( where teams need out of the box PaaS all the way to custom serverless and IaaS VPC's of their own)

See the org/folder/project diagram at https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/blob/main/docs/architecture.md#high-level-organizational-structure
The grey workload folder root should be a forest instead of a single tree (between the shared perimiter/audit/management/security folders)

GKE cluster will not come up with options set due to missing: Secure Boot is not enabled in the 'shielded_instance_config' field - for select environment with a policy override

GKE cluster will not come up in northamerica-northeast1 with options set due to missing: Secure Boot is not enabled in the 'shielded_instance_config' field - due to a policy override

Reproduction

follow the readme - post arete install - when creating the config controller environment
arete create.... using northamerica-northeast1
I will triage the issue - it looks to be either a missing parameter set for secure boot - and not likely the region itself

admin_@cloudshell:~/wse_github/20220829/pubsec-declarative-toolkit (pubsec-declarative-toolkit-arg)$ arete create pdt-arg-kcc --region=northamerica-northeast1 --project=pubsec-declarative-toolkit-arg
1:39PM INF Enabling required services...
1:40PM INF Operation "operations/acf.p2-716449377354-1b0788dc-9131-4ba1-8fa0-23c766f3ac6b" finished successfully.
1:40PM INF Creating Network...
1:40PM INF Creating subnet....
1:40PM INF Creating Config Controller Cluster....
1:50PM FTL  error="Create request issued for: [pdt-arg-kcc]Waiting for operation [projects/pubsec-declarative-toolkit-arg/locations/northamerica-northeast1/operations/operation-1661953241365-5e789a495bca1-4baa162a-23745e56] to complete........

Current errors: [CONDITION_NOT_MET]: Instance 'gke-krmapihost-pdt-arg-k-default-pool-36014ba0-fj66' creation failed: Constraint constraints/compute.requireShieldedVm violated for project projects/pubsec-declarative-toolkit-arg. Secure Boot is not enabled in the 'shielded_instance_config' field. See https://cloud.google.com/resource-manager/docs/organization-policy/org-policy-constraints for more information\t(2) Not all instances running in IGM after 23.407955739s. Expected 1, running 0, transitioning 1. Current errors: [CONDITION_NOT_MET]: Instance 'gke-krmapihost-pdt-arg-k-default-pool-f91b2aab-5zmw' creation failed: Constraint constraints/compute.requireShieldedVm violated for project projects/pubsec-declarative-toolkit-arg. Secure Boot is not enabled in the 'shielded_instance_config' field. See https://cloud.google.com/resource-manager/docs/organization-policy/org-policy-constraints for more information\t(3) Not all instances running in IGM after 24.065795965s. Expected 1, running 0, transitioning 1

looking to put in a local override - retesting on a clean org....

[ENHANCEMENT] avoid billing quota failure when deploying new (CC or Solution) Projects - for default/new organizations that have the quota of 5 per billing id - optionally add 1:1 billing/project KPT config

Describe the bug
Either the CC bootstrap or more likely the solution deployment fails on project creation because of the default 5 billing/project association quota on new organizations.
This occurs for all new accounts that have not specifically asked for a quota increase - usually 2 min with billing history and up to 2 days with a $50 deposit for new organizations.
Procedure workaround is detailed at https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/blob/main/docs/google-cloud-onboarding.md#quota-increase

Fill out the billing quota increase from the default 5 directly via https://support.google.com/code/contact/billing_quota_increase
Fill out the project quota increase from the default 20 directly via https://support.google.com/code/contact/project_quota_increase

To Reproduce
Steps to reproduce the behavior:
Deploy the CC or the landing-zone solution if you have not asked for increased billling/project quota

root_@cloudshell:~ (pubsec-declarative-tk-agz)$ arete create landing-zone-controller --region=northamerica-northeast1
5:55PM INF Project name will be set to: landing-zone-controller-3dy12
✔ My Billing Account - 011BCB-037F97-C9169E
✔ alternate.gcp.zone - 6839210352
✔ Folder Level
✔ pdt - 85066258830
5:56PM INF Create in progress for [https://cloudresourcemanager.googleapis.com/v1/projects/landing-zone-controller-3dy12].Waiting for [operations/cp.6843614771710427017] to finish.....done.Enabling service [[cloudapis.googleapis.com](http://cloudapis.googleapis.com/)] on project [landing-zone-controller-3dy12]...Operation "operations/acat.p2-755016227548-cf4437fa-1a07-415f-87e0-683eb742617c" finished successfully.Updated property [core/project] to [landing-zone-controller-3dy12].
5:56PM FTL Unable to assign billing account to project:  error="ERROR: (gcloud.beta.billing.projects.link) FAILED_PRECONDITION: Precondition check failed.\n- '@type': [type.googleapis.com/google.rpc.QuotaFailure\n](http://type.googleapis.com/google.rpc.QuotaFailure%5Cn)  violations:\n  - description: 'Cloud billing quota exceeded: https://support.google.com/code/contact/billing_quota_increase'\n    subject: billingAccounts/011BCB-037F97-C9169E\nexit status 1"

not a problem specific to the kcc lz - but this will occur for all new accounts (in particular dev accounts)

Expected behavior

Screenshots
see projects beyond the 5

Additional context
Mitigation is detailed in https://github.com/GoogleCloudPlatform/pbmm-on-gcp-onboarding/blob/main/docs/google-cloud-onboarding.md#quota-increase

Possible fix would be to allow for a billing ID per project (where we can add the super admin for the org to N other billing accounts as a billing account administrator - we then get 5 billing quotas per billing id

Actions

add readme prereq documentation and link to the workaround wiki
optionally adjust the landing-zone to allow for 1:1 billing/project KPT config

Add Documentation for Deploying Config Controller W/ Private Endpoint

This documentation should demonstrate how to create a Config Controller instance and only have it accessible from a set CIDR range using a Bastion Host and using a private endpoint.

The goal is to increase security by reducing the potential access points by following GKE best practices for minimizing the exposure of the control plane.

As a developer I wish to have a local Golang environment up to build arete (Google Cloud Shell is GO ready)

In the arete documentation as part of the overall pubsec-declarative-toolkit install we require Golang setup and configured.
Currently the Google Cloud Shell is good to go at https://shell.cloud.google.com/
This adjustment to the readme below should add a go install link or procedure for local gcloud users (local laptops)
https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/main/cli/README.md

todo: document several approaches to getting go locally (osx/darwin, jetbrains, docker) - see http://wiki.obrienlabs.cloud/display/DEV/Go+Lang+Developer+Guide

see install references in the onboarding task #33 (comment)

As Chris mentioned - a download option should be added as well

Scrum/Sprint: prioritized work items

This issue details a continuously modifiable list of issues being worked on, queued and deferred

Working on

Queued prioritized

periodic timeout on CC creation - #93

Queued non-prioritized

Deferred

closed

[BUG]: kpt fn render - periodically fails to connect to the cluster 1-2 times - may be vCore/ram 2/8 perf issue

Ref: https://cloud.google.com/architecture/managing-cloud-infrastructure-using-kpt
Describe the bug
Periodic failure connecting to the cluster during kpt fn render
org 1 = 2 runs to OK
org 2 = 3 runs to OK

To Reproduce
Steps to reproduce the behavior:

cc cluster was up and functional previously
modify your settings.yaml (For example switch billing accounts)
run "kpt fn render"
failure
wait
run "kpt fn render" a 2nd or 3rd time
ok

Expected behavior
kpt fn render - worked without error consistently
Fix is to warn on the readme to rerun the render
secondary fix is to determine if GKE cluster sizing is too small

Screenshots

Additional context

michael@cloudshell:~/wse_github/GoogleCloudPlatform/landing-zone (landing-zone-controller-e4g7d)$ kpt fn render
Package "landing-zone/environments/common/guardrails-policies":
Package "landing-zone/environments/common":
[RUNNING] "gcr.io/kpt-fn/set-namespace:v0.4.1"
[PASS] "gcr.io/kpt-fn/set-namespace:v0.4.1" in 2.8s
  Results:
    [info]: all namespaces are already "config-control". no value changed

Package "landing-zone/environments/nonprod":
[RUNNING] "gcr.io/kpt-fn/set-namespace:v0.4.1"
[PASS] "gcr.io/kpt-fn/set-namespace:v0.4.1" in 500ms
  Results:
    [info]: all namespaces are already "config-control". no value changed

Package "landing-zone/environments/prod":
[RUNNING] "gcr.io/kpt-fn/enable-gcp-services:v0.1.0"
[PASS] "gcr.io/kpt-fn/enable-gcp-services:v0.1.0" in 3.4s
  Results:
    [info] serviceusage.cnrm.cloud.google.com/v1beta1/Service/config-control/prod-nethost-service-compute: recreated service
    [info] serviceusage.cnrm.cloud.google.com/v1beta1/Service/config-control/prod-nethost-service-logging: recreated service
[RUNNING] "gcr.io/kpt-fn/set-namespace:v0.4.1"
[PASS] "gcr.io/kpt-fn/set-namespace:v0.4.1" in 400ms
  Results:
    [info]: all namespaces are already "config-control". no value changed

Package "landing-zone":
[RUNNING] "gcr.io/kpt-fn/apply-setters:v0.2"
[PASS] "gcr.io/kpt-fn/apply-setters:v0.2" in 2.5s
  Results:
    [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "925207728429"
    [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "925207728429"
    [info] spec.projectID: set field value to "net-perimeter-prj-common-gz1"
    [info] spec.parentRef.external: set field value to "925207728429"
    ...(87 line(s) truncated, use '--truncate-output=false' to disable)
[RUNNING] "gcr.io/kpt-fn/generate-folders:v0.1.1"
[PASS] "gcr.io/kpt-fn/generate-folders:v0.1.1" in 6s
[RUNNING] "gcr.io/kpt-fn/enable-gcp-services:v0.1.0"
[PASS] "gcr.io/kpt-fn/enable-gcp-services:v0.1.0" in 1.9s
  Results:
    [info] serviceusage.cnrm.cloud.google.com/v1beta1/Service/config-control/nonprod-nethost-service-compute: recreated service
    [info] serviceusage.cnrm.cloud.google.com/v1beta1/Service/config-control/nonprod-nethost-service-dns: recreated service
    [info] serviceusage.cnrm.cloud.google.com/v1beta1/Service/config-control/nonprod-nethost-service-logging: recreated service
    [info] serviceusage.cnrm.cloud.google.com/v1beta1/Service/config-control/prod-nethost-service-compute: recreated service
    ...(3 line(s) truncated, use '--truncate-output=false' to disable)
[RUNNING] "gcr.io/kpt-fn/gatekeeper:v0.2.1"
[PASS] "gcr.io/kpt-fn/gatekeeper:v0.2.1" in 4s
[RUNNING] "gcr.io/kpt-fn/kubeval:v0.3.0"
[FAIL] "gcr.io/kpt-fn/kubeval:v0.3.0" in 900ms
  Stderr:
    "docker: Error response from daemon: Get \"https://gcr.io/v2/\": dial tcp [2607:f8b0:400c:c03::52]:443: connect: cannot assign requested address."
    "See 'docker run --help'."
  Exit code: 125


michael@cloudshell:~/wse_github/GoogleCloudPlatform/landing-zone (landing-zone-controller-e4g7d)$ kubectl get pods --all-namespaces
NAMESPACE                         NAME                                                             READY   STATUS    RESTARTS        AGE
cnrm-system                       cnrm-controller-manager-ccdnqj4gkgtcjgejpi10-0                   2/2     Running   0               7h5m
cnrm-system                       cnrm-deletiondefender-0                                          1/1     Running   0               7h4m
cnrm-system                       cnrm-resource-stats-recorder-7d49746fc6-w42z7                    2/2     Running   0               7h4m
cnrm-system                       cnrm-webhook-manager-85f4848bc4-pm8bs                            1/1     Running   0               7h4m
cnrm-system                       cnrm-webhook-manager-85f4848bc4-q9km2                            1/1     Running   0               7h4m
config-management-monitoring      otel-collector-57586545db-jt2k6                                  1/1     Running   0               7h4m
config-management-system          config-management-operator-64ff79d555-fz9bt                      1/1     Running   0               7h5m
config-management-system          reconciler-manager-6c5967d99b-n2dh5                              2/2     Running   0               7h4m
configconnector-operator-system   configconnector-operator-0                                       1/1     Running   0               7h5m
gatekeeper-system                 gatekeeper-audit-68fb44f5bc-gqfds                                1/1     Running   0               7h4m
gatekeeper-system                 gatekeeper-controller-manager-5d768f8f49-xjr2m                   1/1     Running   0               7h4m
krmapihosting-monitoring          krmapihosting-metrics-agent-28lxd                                1/1     Running   0               7h5m
krmapihosting-monitoring          krmapihosting-metrics-agent-dm47c                                1/1     Running   0               7h5m
krmapihosting-monitoring          krmapihosting-metrics-agent-zxcsd                                1/1     Running   0               7h5m
krmapihosting-system              bootstrap-5ffd94d5cd-lgs5r                                       1/1     Running   3 (6h59m ago)   7h5m
kube-system                       event-exporter-gke-5479fd58c8-d92tl                              2/2     Running   0               7h11m
kube-system                       fluentbit-gke-6bvnb                                              2/2     Running   0               7h6m
kube-system                       fluentbit-gke-bblcs                                              2/2     Running   0               7h6m
kube-system                       fluentbit-gke-clxtb                                              2/2     Running   0               7h6m
kube-system                       gke-metadata-server-b6k26                                        1/1     Running   0               7h6m
kube-system                       gke-metadata-server-glj2w                                        1/1     Running   0               7h6m
kube-system                       gke-metadata-server-q2q8x                                        1/1     Running   0               7h6m
kube-system                       gke-metrics-agent-476g8                                          1/1     Running   0               7h6m
kube-system                       gke-metrics-agent-tctfm                                          1/1     Running   0               7h6m
kube-system                       gke-metrics-agent-xw6xv                                          1/1     Running   0               7h6m
kube-system                       kube-dns-85df8994db-6zfvk                                        4/4     Running   0               7h11m
kube-system                       kube-dns-85df8994db-rfdd6                                        4/4     Running   0               7h11m
kube-system                       kube-dns-autoscaler-f4d55555-rwcqm                               1/1     Running   0               7h11m
kube-system                       kube-proxy-gke-krmapihost-landi-krmapihost-landi-c558d5b5-k78c   1/1     Running   0               7h6m
kube-system                       kube-proxy-gke-krmapihost-landi-krmapihost-landi-cd502563-2fbx   1/1     Running   0               7h6m
kube-system                       kube-proxy-gke-krmapihost-landi-krmapihost-landi-ea7e5dd4-7npn   1/1     Running   0               7h6m
kube-system                       l7-default-backend-69fb9fd9f9-pk7hj                              1/1     Running   0               7h11m
kube-system                       metrics-server-v0.4.5-fb4c49dd6-4ksdj                            2/2     Running   0               7h11m
kube-system                       netd-2p4nj                                                       1/1     Running   0               7h6m
kube-system                       netd-bwggp                                                       1/1     Running   0               7h6m
kube-system                       netd-fkg5f                                                       1/1     Running   0               7h6m
kube-system                       pdcsi-node-pvdwn                                                 2/2     Running   0               7h6m
kube-system                       pdcsi-node-x7cx9                                                 2/2     Running   0               7h6m
kube-system                       pdcsi-node-zr7zq                                                 2/2     Running   0               7h6m
resource-group-system             resource-group-controller-manager-6c6774ff66-svplf               3/3     Running   0               7h4m
michael@cloudshell:~/wse_github/GoogleCloudPlatform/landing-zone (landing-zone-controller-e4g7d)$ kpt fn render
Package "landing-zone/environments/common/guardrails-policies":
Package "landing-zone/environments/common":
[RUNNING] "gcr.io/kpt-fn/set-namespace:v0.4.1"
[PASS] "gcr.io/kpt-fn/set-namespace:v0.4.1" in 600ms
  Results:
    [info]: all namespaces are already "config-control". no value changed

Package "landing-zone/environments/nonprod":
[RUNNING] "gcr.io/kpt-fn/set-namespace:v0.4.1"
[PASS] "gcr.io/kpt-fn/set-namespace:v0.4.1" in 400ms
  Results:
    [info]: all namespaces are already "config-control". no value changed

Package "landing-zone/environments/prod":
[RUNNING] "gcr.io/kpt-fn/enable-gcp-services:v0.1.0"
[PASS] "gcr.io/kpt-fn/enable-gcp-services:v0.1.0" in 1.7s
  Results:
    [info] serviceusage.cnrm.cloud.google.com/v1beta1/Service/config-control/prod-nethost-service-compute: recreated service
    [info] serviceusage.cnrm.cloud.google.com/v1beta1/Service/config-control/prod-nethost-service-logging: recreated service
[RUNNING] "gcr.io/kpt-fn/set-namespace:v0.4.1"
[PASS] "gcr.io/kpt-fn/set-namespace:v0.4.1" in 500ms
  Results:
    [info]: all namespaces are already "config-control". no value changed

Package "landing-zone":
[RUNNING] "gcr.io/kpt-fn/apply-setters:v0.2"
[PASS] "gcr.io/kpt-fn/apply-setters:v0.2" in 800ms
  Results:
    [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "925207728429"
    [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "925207728429"
    [info] spec.projectID: set field value to "net-perimeter-prj-common-gz1"
    [info] spec.parentRef.external: set field value to "925207728429"
    ...(87 line(s) truncated, use '--truncate-output=false' to disable)
[RUNNING] "gcr.io/kpt-fn/generate-folders:v0.1.1"
[PASS] "gcr.io/kpt-fn/generate-folders:v0.1.1" in 1s
[RUNNING] "gcr.io/kpt-fn/enable-gcp-services:v0.1.0"
[PASS] "gcr.io/kpt-fn/enable-gcp-services:v0.1.0" in 2s
  Results:
    [info] serviceusage.cnrm.cloud.google.com/v1beta1/Service/config-control/nonprod-nethost-service-compute: recreated service
    [info] serviceusage.cnrm.cloud.google.com/v1beta1/Service/config-control/nonprod-nethost-service-dns: recreated service
    [info] serviceusage.cnrm.cloud.google.com/v1beta1/Service/config-control/nonprod-nethost-service-logging: recreated service
    [info] serviceusage.cnrm.cloud.google.com/v1beta1/Service/config-control/prod-nethost-service-compute: recreated service
    ...(3 line(s) truncated, use '--truncate-output=false' to disable)
[RUNNING] "gcr.io/kpt-fn/gatekeeper:v0.2.1"
[PASS] "gcr.io/kpt-fn/gatekeeper:v0.2.1" in 1.5s
[RUNNING] "gcr.io/kpt-fn/kubeval:v0.3.0"
[PASS] "gcr.io/kpt-fn/kubeval:v0.3.0" in 26.9s

Successfully executed 9 function(s) in 5 package(s).

dev-exp: LZ section 4: project id/number is ambiguous - do we mean project name/id - should be the same for unique cc project previously generated

section 4 of https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/tree/main/solutions/landing-zone#usage
dev-experience: LZ section 4: project id/number is ambiguous - do we mean project name/id - should be the same for unique cc project previously generated

we need an example run in the docs
what is project number - I would expect it to be project id but we already have one of these - are they switched as in project name and project id (not always the same unless name is unique - or is it the random postfix
original

  management-project-id: management-project-12345
  management-project-number: "0000000000"

Why is one in quotes (i need to check their usage)
I would expect the same as the project id should be unique
modified

  management-project-id: landing-zone-controller-1z583
  management-project-number: "landing-zone-controller-1z583"

resulting run has issues connecting to the gke cluster - different issue

5.deploy

michael@cloudshell:~/github/GoogleCloudPlatform/pubsec-declarative-toolkit/landing-zone (landing-zone-controller-1z583)$ kpt fn render
Package "landing-zone/environments/common/guardrails-policies":
Package "landing-zone/environments/common":
[RUNNING] "gcr.io/kpt-fn/set-namespace:v0.4.1"
[PASS] "gcr.io/kpt-fn/set-namespace:v0.4.1" in 2.1s
  Results:
    [info]: namespace "common" updated to "config-control", 23 value(s) changed

Package "landing-zone/environments/nonprod":
[RUNNING] "gcr.io/kpt-fn/set-namespace:v0.4.1"
[PASS] "gcr.io/kpt-fn/set-namespace:v0.4.1" in 300ms
  Results:
    [info]: namespace "nonprod" updated to "config-control", 7 value(s) changed

Package "landing-zone/environments/prod":
[RUNNING] "gcr.io/kpt-fn/enable-gcp-services:v0.1.0"
[FAIL] "gcr.io/kpt-fn/enable-gcp-services:v0.1.0" in 700ms
  Stderr:
    "docker: Error response from daemon: Get \"https://gcr.io/v2/\": dial tcp [2607:f8b0:400c:c13::52]:443: connect: cannot assign requested address."
    "See 'docker run --help'."
  Exit code: 125

checking auth

As an arete create user I am running into a missing wait state between project creation and api enablement - running on an existing target project or running create twice is a workaround

running arete create after manually deleting the CC cluster - cluster creation does not kick in without deleting the .arete cache
running into existing .arete config - deleting - separate issue in #94
error - see end of details (timing/wait step between project creation and services enablement)

2:48PM INF Creating Config Controller Cluster....
2:48PM FTL  error="API [krmapihosting.googleapis.com] not enabled on project [153970848512]. Would you like to enable and retry (this will take a few minutes)? (y/N)?  ERROR: (gcloud.anthos.config.controller.create) PERMISSION_DENIED: KRM API Hosting API has not been used in project 153970848512 before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/krmapihosting.googleapis.com/overview?project=153970848512 then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.- '@type': type.googleapis.com/google.rpc.Help  links:  - description: Google developers console API activation    url: https://console.developers.google.com/apis/api/krmapihosting.googleapis.com/overview?project=153970848512- '@type': type.googleapis.com/google.rpc.ErrorInfo  domain: googleapis.com  metadata:    consumer: projects/153970848512    service: krmapihosting.googleapis.com  reason: SERVICE_DISABLED"

details

admin_root@cloudshell:~ (landing-zone-controller-w8hwa)$ arete create pdt-cno-kcc --region=northamerica-northeast1 --project=pubsec-declarative-toolkit-cno
2:37PM INF Config Controller setup complete
admin_root@cloudshell:~ (landing-zone-controller-w8hwa)$ rm -rf
.arete/                .bashrc                .docker/               .npm/                  .redhat/
.bash_history          .cache/                gopath/                .profile               .theia/
.bash_logout           .config/               .kube/                 README-cloudshell.txt  wse_github/
admin_root@cloudshell:~ (landing-zone-controller-w8hwa)$ ls -la .arete/
total 20
drwxr--r--  2 admin_root admin_root 4096 Sep  2 18:15 .
drwxr-xr-x 12 admin_root admin_root 4096 Sep  6 14:37 ..
-rw-r--r--  1 admin_root admin_root   46 Aug 31 15:06 config.yaml
-rw-------  1 admin_root admin_root  100 Aug 31 15:32 .create
-rw-r--r--  1 admin_root admin_root 1318 Sep  2 18:15 solutions.yaml
admin_root@cloudshell:~ (landing-zone-controller-w8hwa)$ rm -rf .arete/

admin_root@cloudshell:~ (landing-zone-controller-w8hwa)$ arete create pdt-cno-kcc --region=northamerica-northeast1 --project=pubsec-declarative-toolkit-cno
2:40PM INF Enabling required services...
2:40PM INF Operation "operations/acat.p2-491974186555-2e6beaa9-f3df-4413-9a28-419db485c8e0" finished successfully.
2:41PM INF Creating Config Controller Cluster....
2:41PM FTL  error="ERROR: (gcloud.anthos.config.controller.create) ALREADY_EXISTS: Resource 'projects/pubsec-declarative-toolkit-cno/locations/northamerica-northeast1/krmApiHosts/pdt-cno-kcc' already exists- '@type': type.googleapis.com/google.rpc.ResourceInfo  resourceName: projects/pubsec-declarative-toolkit-cno/locations/northamerica-northeast1/krmApiHosts/pdt-cno-kcc"

deleting project - attempt to reuse may fail on 30 day deleted cache - will try
admin_root@cloudshell:~ (landing-zone-controller-w8hwa)$ gcloud projects delete pubsec-declarative-toolkit-cno
Your project will be deleted.

Do you want to continue (Y/n)?  y

Deleted [https://cloudresourcemanager.googleapis.com/v1/projects/pubsec-declarative-toolkit-cno].

You can undo this operation for a limited period by running the command below.
    $ gcloud projects undelete pubsec-declarative-toolkit-cno

See https://cloud.google.com/resource-manager/docs/creating-managing-projects for information on shutting down projects.
admin_root@cloudshell:~ (landing-zone-controller-w8hwa)$ arete create pdt-cno-kcc --region=northamerica-northeast1 --project=pubsec-declarative-toolkit-cno
✔ My Billing Account - 019..3D
✔ nuage-cloud.org - 471..7
✔ Folder Level
✔ pdt - 346..8
2:44PM FTL  error="ERROR: (gcloud.projects.create) Project creation failed. The project ID you specified is already in use by another project. Please try an alternative ID."

admin_root@cloudshell:~ (landing-zone-controller-w8hwa)$ arete create pdt-cno-kcc --region=northamerica-northeast1 --project=pubsec-declarative-toolkit-cno2
✔ My Billing Account - 01..3D
✔ nuage-cloud.org - 471924274947
✔ Folder Level
✔ pdt - 346242644868
2:45PM FTL  error="ERROR: (gcloud.projects.create) argument PROJECT_ID: Bad value [pubsec-declarative-toolkit-cno2]: Project IDs are immutable and can be set only during project creation. They must start with a lowercase letter and can have lowercase ASCII letters, digits or hyphens. Project IDs must be between 6 and 30 characters.Usage: gcloud projects create [PROJECT_ID] [optional flags]  optional flags may be  --enable-cloud-apis | --folder | --help | --labels |                         --name | --organization | --set-as-defaultFor detailed information on this command and its flags, run:  gcloud projects create --help"

30 char limit

admin_root@cloudshell:~ (landing-zone-controller-w8hwa)$ arete create pdt-cno-kcc --region=northamerica-northeast1 --project=pubsec-declarative-tk-cno2
✔ My Billing Account - 019952-0D0AAC-777E3D
✔ nuage-cloud.org - 471924274947
✔ Folder Level
✔ pdt - 346242644868
2:48PM INF Create in progress for [https://cloudresourcemanager.googleapis.com/v1/projects/pubsec-declarative-tk-cno2].Waiting for [operations/cp.7885851846085518239] to finish.....done.Enabling service [cloudapis.googleapis.com] on project [pubsec-declarative-tk-cno2]...Operation "operations/acat.p2-153970848512-8ffc1200-8c5a-42fd-b142-e11cdaf69191" finished successfully.Updated property [core/project] to [pubsec-declarative-tk-cno2].
2:48PM INF Creating Config Controller Cluster....
2:48PM FTL  error="API [krmapihosting.googleapis.com] not enabled on project [153970848512]. Would you like to enable and retry (this will take a few minutes)? (y/N)?  ERROR: (gcloud.anthos.config.controller.create) PERMISSION_DENIED: KRM API Hosting API has not been used in project 153970848512 before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/krmapihosting.googleapis.com/overview?project=153970848512 then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.- '@type': type.googleapis.com/google.rpc.Help  links:  - description: Google developers console API activation    url: https://console.developers.google.com/apis/api/krmapihosting.googleapis.com/overview?project=153970848512- '@type': type.googleapis.com/google.rpc.ErrorInfo  domain: googleapis.com  metadata:    consumer: projects/153970848512    service: krmapihosting.googleapis.com  reason: SERVICE_DISABLED"

rerun on recently created project - or run on an existing project to avoid the service enablement missing wait timer

                                                      arete create pdt-cno-kcc --region=northamerica-northeast1 --project=pubsec-declarative-tk-cno2
4:39PM INF Enabling required services...
4:40PM INF Operation "operations/acf.p2-153970848512-b3d4a2a6-fe02-4a5b-8f5d-d27d917f6527" finished successfully.
4:40PM INF Creating Network...


........................................................................................done.Created instance [pdt-cno-kcc].Fetching cluster endpoint and auth data.kubeconfig entrgenerated for krmapihost-pdt-cno-kcc.
5:09PM INF Add SA to roles/owner role...
5:09PM INF Config Controller setup complete

reviewing
https://cloud.google.com/anthos-config-management/docs/tutorials/landing-zone#removing_resources

Private GitHub repo access

Right now the CLI uses a git token that is ephemeral and can be very frustrating to constantly update the git_token key in the config.yaml file for private repos.

Switching to a personal access token will make the git token access pattern more long lived. The usage of the git token is for pulling the solutions.yaml file from private repos and it utilizes https so the url is encrypted in transit. The config file is local to the callers machine so it's not much different then having a private cert on your machine.

Add More Enforcement Policies

Client asked for policy around enforcing naming conventions on resources in addition we have noticed errors when deploying resources where names entered exceed the limits and/or conventions of the resources.

We should do two things. Add policies to both allow for users to enforce naming conventions (most likely by regex) and add some GCP specific policies to enforce compliance on system limits (ie project name char limits).

Add HA Fortigate as a Solution or Service

Some implementations of the Landing Zone will require a fortigate appliance and in order to provide that functionality we should have either a service or solution we can distribute as a standalone or package with the LZ as needed.

The reference terraform module is located here.

This would need to be converted into the Config Connector format in order to keep in line with the goal of keeping the LZs at parity.

As a CD automation service account or developer I need to completely delete/recreate the Anthos cluster and associated solution projects from the target organization - in step

use case: full 2 part CD automation of the KCC cluster and the lz solution - with full tear down of everything or just the lz solution

action: add a delete action to the arete cli
this case is for a complete reuse of the org - back to a clean state (minus required roles)

Indirect collaboration on: GoogleCloudPlatform/pbmm-on-gcp-onboarding#166

Notes:

removing the solution via arete

pending (discussions with Shaun and Chris)
your prompts do work well - I like them in the create
just thinking automated CD - will need a cli double minus flag for scripts
I would target an addition (where we run 2+ solutions) - thinking your option 2 (from the cluster)
https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/tree/main/solutions/landing-zone

create
arete create landing-zone-controller --region=$REGION

delete (option 2)
arete delete landing-zone-controller --region=$REGION

removing the solution via kubernetes cli (thanks Chris for reminding me to think granular k8s again)

stick to this for now

admin_root@cloudshell:~/wse_github/20220831/pubsec-declarative-toolkit (pubsec-declarative-toolkit-cno)$ export REGION=northamerica-northeast1
admin_root@cloudshell:~/wse_github/20220831/pubsec-declarative-toolkit (pubsec-declarative-toolkit-cno)$ arete create landing-zone-controller --region=$REGION
8:41PM INF Project name will be set to: landing-zone-controller-1pw8k

removing the solution via kpt cli

removing the solution via gcloud cli

removing the solution manually

all projects under pdt

check arete cache - pending on delete command

admin_root@cloudshell:~$ ls -la .arete/
total 20
drwxr--r--  2 admin_root admin_root 4096 Sep  2 18:15 .
drwxr-xr-x 11 admin_root admin_root 4096 Sep  1 19:27 ..
-rw-r--r--  1 admin_root admin_root   46 Aug 31 15:06 config.yaml
-rw-------  1 admin_root admin_root  100 Aug 31 15:32 .create
-rw-r--r--  1 admin_root admin_root 1318 Sep  2 18:15 solutions.yaml
admin_root@cloudshell:~$ cat .arete/solutions.yaml
solutions:
    - solution: guardrails
      description: |
        Implementation of the GC Cloud Guardrails Checks. More Info: https://github.com/canada-ca/cloud-guardrails-gcp
      url: https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit.git/solutions/guardrails@main
    - solution: guardrails-policy-bundle
      description: |
        Policy Bundle to help analyze compliance for Guardrails
      url: https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit.git/solutions/guardrails-policies@main
    - solution: kcc-namespaces
      description: |
        Simplified declarative multi-tenancy with project namespaces taken from: https://cloud.google.com/anthos-config-management/docs/tutorials/project-namespace-blueprint
      url: https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit.git/solutions/kcc-namespaces@main
    - solution: sandbox-gke
      description: |
        A private GKE cluster with so many bells and whistles!
      url: https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit.git/solutions/sandbox-gke@main
    - solution: landing-zone
      description: |
        This is a reimplementation of pbmm-on-gcp-onboarding Landing Zone using KRM.
      url: https://github.com/GoogleCloudPlatform/gcp-pbmm-sandbox.git/solutions/landing-zone@main
admin_root@cloudshell:~$ cat .arete/config.yaml
cache: /home/admin_root/.arete
verbose: false
admin_root@cloudshell:~$ cat .arete/.create
steps:
- step: services
- step: network
- step: subnet
- step: config-controller
- step: add-policy

delete the solution project off the target folder
delete the anthos GKE cluster off the host projectd
remove the kcc-controller VPC

Rerun the CC cluster creation and LZ solution

running into existing .arete config - deleting

admin_root@cloudshell:~ (landing-zone-controller-w8hwa)$ arete create pdt-cno-kcc --region=northamerica-northeast1 --project=pubsec-declarative-toolkit-cno
2:37PM INF Config Controller setup complete
admin_root@cloudshell:~ (landing-zone-controller-w8hwa)$ rm -rf
.arete/                .bashrc                .docker/               .npm/                  .redhat/
.bash_history          .cache/                gopath/                .profile               .theia/
.bash_logout           .config/               .kube/                 README-cloudshell.txt  wse_github/
admin_root@cloudshell:~ (landing-zone-controller-w8hwa)$ ls -la .arete/
total 20
drwxr--r--  2 admin_root admin_root 4096 Sep  2 18:15 .
drwxr-xr-x 12 admin_root admin_root 4096 Sep  6 14:37 ..
-rw-r--r--  1 admin_root admin_root   46 Aug 31 15:06 config.yaml
-rw-------  1 admin_root admin_root  100 Aug 31 15:32 .create
-rw-r--r--  1 admin_root admin_root 1318 Sep  2 18:15 solutions.yaml
admin_root@cloudshell:~ (landing-zone-controller-w8hwa)$ rm -rf .arete/

admin_root@cloudshell:~ (landing-zone-controller-w8hwa)$ arete create pdt-cno-kcc --region=northamerica-northeast1 --project=pubsec-declarative-toolkit-cno
2:40PM INF Enabling required services...
2:40PM INF Operation "operations/acat.p2-491974186555-2e6beaa9-f3df-4413-9a28-419db485c8e0" finished successfully.
2:41PM INF Creating Config Controller Cluster....
2:41PM FTL  error="ERROR: (gcloud.anthos.config.controller.create) ALREADY_EXISTS: Resource 'projects/pubsec-declarative-toolkit-cno/locations/northamerica-northeast1/krmApiHosts/pdt-cno-kcc' already exists- '@type': type.googleapis.com/google.rpc.ResourceInfo  resourceName: projects/pubsec-declarative-toolkit-cno/locations/northamerica-northeast1/krmApiHosts/pdt-cno-kcc"

deleting project - attempt to reuse may fail on 30 day deleted cache - will try
admin_root@cloudshell:~ (landing-zone-controller-w8hwa)$ gcloud projects delete pubsec-declarative-toolkit-cno
Your project will be deleted.

Do you want to continue (Y/n)?  y

Deleted [https://cloudresourcemanager.googleapis.com/v1/projects/pubsec-declarative-toolkit-cno].

You can undo this operation for a limited period by running the command below.
    $ gcloud projects undelete pubsec-declarative-toolkit-cno

See https://cloud.google.com/resource-manager/docs/creating-managing-projects for information on shutting down projects.
admin_root@cloudshell:~ (landing-zone-controller-w8hwa)$ arete create pdt-cno-kcc --region=northamerica-northeast1 --project=pubsec-declarative-toolkit-cno
✔ My Billing Account - 019..3D
✔ nuage-cloud.org - 471..7
✔ Folder Level
✔ pdt - 346..8
2:44PM FTL  error="ERROR: (gcloud.projects.create) Project creation failed. The project ID you specified is already in use by another project. Please try an alternative ID."

admin_root@cloudshell:~ (landing-zone-controller-w8hwa)$ arete create pdt-cno-kcc --region=northamerica-northeast1 --project=pubsec-declarative-toolkit-cno2
✔ My Billing Account - 01..3D
✔ nuage-cloud.org - 471924274947
✔ Folder Level
✔ pdt - 346242644868
2:45PM FTL  error="ERROR: (gcloud.projects.create) argument PROJECT_ID: Bad value [pubsec-declarative-toolkit-cno2]: Project IDs are immutable and can be set only during project creation. They must start with a lowercase letter and can have lowercase ASCII letters, digits or hyphens. Project IDs must be between 6 and 30 characters.Usage: gcloud projects create [PROJECT_ID] [optional flags]  optional flags may be  --enable-cloud-apis | --folder | --help | --labels |                         --name | --organization | --set-as-defaultFor detailed information on this command and its flags, run:  gcloud projects create --help"

30 char limit

admin_root@cloudshell:~ (landing-zone-controller-w8hwa)$ arete create pdt-cno-kcc --region=northamerica-northeast1 --project=pubsec-declarative-tk-cno2
✔ My Billing Account - 019952-0D0AAC-777E3D
✔ nuage-cloud.org - 471924274947
✔ Folder Level
✔ pdt - 346242644868
2:48PM INF Create in progress for [https://cloudresourcemanager.googleapis.com/v1/projects/pubsec-declarative-tk-cno2].Waiting for [operations/cp.7885851846085518239] to finish.....done.Enabling service [cloudapis.googleapis.com] on project [pubsec-declarative-tk-cno2]...Operation "operations/acat.p2-153970848512-8ffc1200-8c5a-42fd-b142-e11cdaf69191" finished successfully.Updated property [core/project] to [pubsec-declarative-tk-cno2].
2:48PM INF Creating Config Controller Cluster....
2:48PM FTL  error="API [krmapihosting.googleapis.com] not enabled on project [153970848512]. Would you like to enable and retry (this will take a few minutes)? (y/N)?  ERROR: (gcloud.anthos.config.controller.create) PERMISSION_DENIED: KRM API Hosting API has not been used in project 153970848512 before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/krmapihosting.googleapis.com/overview?project=153970848512 then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.- '@type': type.googleapis.com/google.rpc.Help  links:  - description: Google developers console API activation    url: https://console.developers.google.com/apis/api/krmapihosting.googleapis.com/overview?project=153970848512- '@type': type.googleapis.com/google.rpc.ErrorInfo  domain: googleapis.com  metadata:    consumer: projects/153970848512    service: krmapihosting.googleapis.com  reason: SERVICE_DISABLED"

rerun on recently created project - or run on an existing project to avoid the service enablement missing wait timer

                                                      arete create pdt-cno-kcc --region=northamerica-northeast1 --project=pubsec-declarative-tk-cno2
4:39PM INF Enabling required services...
4:40PM INF Operation "operations/acf.p2-153970848512-b3d4a2a6-fe02-4a5b-8f5d-d27d917f6527" finished successfully.
4:40PM INF Creating Network...

........................................................................................done.Created instance [pdt-cno-kcc].Fetching cluster endpoint and auth data.kubeconfig entrgenerated for krmapihost-pdt-cno-kcc.
5:09PM INF Add SA to roles/owner role...
5:09PM INF Config Controller setup complete

Reference

deletion using anthos delete

Trying Chris's suggested

gcloud anthos config controller delete

Or the previous anthos cluster deletion doc
http://wiki.obrienlabs.cloud/display/DEV/Deploying+a+Landing+Zone+on+Google+Cloud#DeployingaLandingZoneonGoogleCloud-Pausingtheanthoscluster
via
http://wiki.obrienlabs.cloud/display/DEV/Deploying+a+Landing+Zone+on+Google+Cloud#DeployingaLandingZoneonGoogleCloud-Pausingtheanthoscluster

reviewing
https://cloud.google.com/anthos-config-management/docs/tutorials/landing-zone#removing_resources

Temporary CC cluster shutdown and restart

Bootstrap not failing after error

I ran the bootstrap and despite having a permission issue with billing detected at the start it tried to complete the script to the end. The standard out said that the Standalone GKE cluster was completed but it was never created.

�[32mCreating Project (<PROJECT-NAME>)....
Create in progress for [https://cloudresourcemanager.googleapis.com/v1/projects/<PROJECT-NAME>].
Waiting for [operations/cp.7124038129352777749] to finish...
..done.
Enabling service [cloudapis.googleapis.com] on project [<PROJECT-NAME>]...
Operation "operations/acf.p2-<PROJECT-ID>-6ae11fd4-f751-4068-a416-57e4acfb57d0" finished successfully.
�[32mSetting gcloud context to active project...
�[32mEnabling Services....
ERROR: (gcloud.services.enable) FAILED_PRECONDITION: Billing account for project '<PROJECT-ID>' is not found. Billing must be enabled for activation of service(s) 'compute.googleapis.com,compute.googleapis.com,compute.googleapis.com' to proceed.
Help Token: Ae-hA1MI2f6yfF6FCKxtgKeybzKvN9GsvNj-W7L8UZ2UmHL-c27E3Qwp-XM3uyln2MnKMWV2JejfEtvNaaTHa6Bbtfe5N4gUM7-3p-wf9fRBl32X
- '@type': type.googleapis.com/google.rpc.PreconditionFailure
  violations:
  - subject: ?error_code=390001&project=<PROJECT-ID>&services=compute.googleapis.com&services=compute.googleapis.com&services=compute.googleapis.com
    type: googleapis.com/billing-enabled
- '@type': type.googleapis.com/google.rpc.ErrorInfo
  domain: serviceusage.googleapis.com/billing-enabled
  metadata:
    project: '<PROJECT-ID>'
    services: compute.googleapis.com,compute.googleapis.com,compute.googleapis.com
  reason: UREQ_PROJECT_BILLING_NOT_FOUND
ERROR: (gcloud.services.enable) FAILED_PRECONDITION: Billing account for project '<PROJECT-ID>' is not found. Billing must be enabled for activation of service(s) 'container.googleapis.com,container.googleapis.com,compute.googleapis.com,compute.googleapis.com,compute.googleapis.com,containerregistry.googleapis.com' to proceed.
Help Token: Ae-hA1PQBC0uHSGowaddWMshRQ9wX272fbjjXHXIPHRz3yBOianBYeeAqDfAuirPnhjQyWGFjgnLHLbt6sGz0M9bFAYbxlqd9zAwoVt-2CZ-ORgI
- '@type': type.googleapis.com/google.rpc.PreconditionFailure
  violations:
  - subject: ?error_code=390001&project=<PROJECT-ID>&services=container.googleapis.com&services=container.googleapis.com&services=compute.googleapis.com&services=compute.googleapis.com&services=compute.googleapis.com&services=containerregistry.googleapis.com
    type: googleapis.com/billing-enabled
- '@type': type.googleapis.com/google.rpc.ErrorInfo
  domain: serviceusage.googleapis.com/billing-enabled
  metadata:
    project: '<PROJECT-ID>'
    services: container.googleapis.com,container.googleapis.com,compute.googleapis.com,compute.googleapis.com,compute.googleapis.com,containerregistry.googleapis.com
  reason: UREQ_PROJECT_BILLING_NOT_FOUND
Operation "operations/acf.p2-<PROJECT-ID>-677c2e1f-71b3-4d2c-852d-c67c3519c5f5" finished successfully.
�[32mCreating VPC (config-network)....
API [compute.googleapis.com] not enabled on project [<PROJECT-ID>]. Would you like to enable and retry (this will take a few minutes)? (y/N)?  
ERROR: (gcloud.compute.networks.create) PERMISSION_DENIED: Compute Engine API has not been used in project <PROJECT-ID> before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/compute.googleapis.com/overview?project=<PROJECT-ID> then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.
- '@type': type.googleapis.com/google.rpc.Help
  links:
  - description: Google developers console API activation
    url: https://console.developers.google.com/apis/api/compute.googleapis.com/overview?project=<PROJECT-ID>
- '@type': type.googleapis.com/google.rpc.ErrorInfo
  domain: googleapis.com
  metadata:
    consumer: projects/<PROJECT-ID>
    service: compute.googleapis.com
  reason: SERVICE_DISABLED
�[32mConfiguring Org Policies for GKE on folder (<FOLDER-ID>)....
ERROR: (gcloud.resource-manager.org-policies.disable-enforce) User [<USERNAME>] does not have permission to access folders instance [<FOLDER-ID>:setOrgPolicy] (or it may not exist): The caller does not have permission
ERROR: (gcloud.resource-manager.org-policies.disable-enforce) User [<USERNAME>] does not have permission to access folders instance [<FOLDER-ID>:setOrgPolicy] (or it may not exist): The caller does not have permission
ERROR: (gcloud.resource-manager.org-policies.set-policy) User [<USERNAME>] does not have permission to access folders instance [<FOLDER-ID>:setOrgPolicy] (or it may not exist): The caller does not have permission
�[32mCreating GKE Cluster (config-controller) (may take several minutes)....
WARNING: The `--enable-stackdriver-kubernetes` flag is deprecated and will be removed in an upcoming release. Please use `--logging` and `--monitoring` instead. For more information, please read: https://cloud.google.com/stackdriver/docs/solutions/gke/installing.
WARNING: The Pod address range limits the maximum size of the cluster. Please refer to https://cloud.google.com/kubernetes-engine/docs/how-to/flexible-pod-cidr to learn how to optimize IP address allocation.
ERROR: (gcloud.container.clusters.create) ResponseError: code=400, message=Failed precondition when calling the ServiceConsumerManager: tenantmanager::185014: Consumer <PROJECT-ID> should enable service:container.googleapis.com before generating a service account.
com.google.api.tenant.error.TenantManagerException: Consumer <PROJECT-ID> should enable service:container.googleapis.com before generating a service account.
�[32mCreating Service Account and adding to owner role (config-control)....
Created service account [config-control].
Updated IAM policy for project [<PROJECT-NAME>].
bindings:
- members:
  - serviceAccount:config-control@<PROJECT-NAME>.iam.gserviceaccount.com
  - user:<USERNAME>
  role: roles/owner
etag: BwXPbR2mRIo=
version: 1
�[32mAdding SA GKE Workload Identity (config-control)....
ERROR: Policy modification failed. For a binding with condition, run "gcloud alpha iam policies lint-condition" to identify issues in condition.
ERROR: (gcloud.iam.service-accounts.add-iam-policy-binding) INVALID_ARGUMENT: Identity Pool does not exist (<PROJECT-NAME>.svc.id.goog). Please check that you specified a valid resource name as returned in the `name` attribute in the configuration API.
�[32mAdding SA to requires roles (config-control@<PROJECT-NAME>.iam.gserviceaccount.com)....
ERROR: (gcloud.organizations.add-iam-policy-binding) User [<USERNAME>] does not have permission to access organizations instance [<ORG-ID>:getIamPolicy] (or it may not exist): The caller does not have permission
ERROR: (gcloud.organizations.add-iam-policy-binding) User [<USERNAME>] does not have permission to access organizations instance [<ORG-ID>:getIamPolicy] (or it may not exist): The caller does not have permission
ERROR: (gcloud.organizations.add-iam-policy-binding) User [<USERNAME>] does not have permission to access organizations instance [<ORG-ID>:getIamPolicy] (or it may not exist): The caller does not have permission
ERROR: (gcloud.organizations.add-iam-policy-binding) User [<USERNAME>] does not have permission to access organizations instance [<ORG-ID>:getIamPolicy] (or it may not exist): The caller does not have permission
ERROR: (gcloud.organizations.add-iam-policy-binding) User [<USERNAME>] does not have permission to access organizations instance [<ORG-ID>:getIamPolicy] (or it may not exist): The caller does not have permission
ERROR: (gcloud.organizations.add-iam-policy-binding) User [<USERNAME>] does not have permission to access organizations instance [<ORG-ID>:getIamPolicy] (or it may not exist): The caller does not have permission
ERROR: (gcloud.organizations.add-iam-policy-binding) User [<USERNAME>] does not have permission to access organizations instance [<ORG-ID>:getIamPolicy] (or it may not exist): The caller does not have permission
ERROR: (gcloud.organizations.add-iam-policy-binding) User [<USERNAME>] does not have permission to access organizations instance [<ORG-ID>:getIamPolicy] (or it may not exist): The caller does not have permission
ERROR: (gcloud.organizations.add-iam-policy-binding) User [<USERNAME>] does not have permission to access organizations instance [<ORG-ID>:getIamPolicy] (or it may not exist): The caller does not have permission
ERROR: (gcloud.organizations.add-iam-policy-binding) User [<USERNAME>] does not have permission to access organizations instance [<ORG-ID>:getIamPolicy] (or it may not exist): The caller does not have permission
ERROR: (gcloud.organizations.add-iam-policy-binding) User [<USERNAME>] does not have permission to access organizations instance [<ORG-ID>:getIamPolicy] (or it may not exist): The caller does not have permission
ERROR: (gcloud.organizations.add-iam-policy-binding) User [<USERNAME>] does not have permission to access organizations instance [<ORG-ID>:getIamPolicy] (or it may not exist): The caller does not have permission
ERROR: (gcloud.organizations.add-iam-policy-binding) User [<USERNAME>] does not have permission to access organizations instance [<ORG-ID>:getIamPolicy] (or it may not exist): The caller does not have permission
ERROR: (gcloud.organizations.add-iam-policy-binding) User [<USERNAME>] does not have permission to access organizations instance [<ORG-ID>:getIamPolicy] (or it may not exist): The caller does not have permission
ERROR: (gcloud.organizations.add-iam-policy-binding) User [<USERNAME>] does not have permission to access organizations instance [<ORG-ID>:getIamPolicy] (or it may not exist): The caller does not have permission
ERROR: (gcloud.organizations.add-iam-policy-binding) User [<USERNAME>] does not have permission to access organizations instance [<ORG-ID>:getIamPolicy] (or it may not exist): The caller does not have permission
Fetching cluster endpoint and auth data.
ERROR: (gcloud.container.clusters.get-credentials) ResponseError: code=404, message=Not found: projects/<PROJECT-NAME>/locations/northamerica-northeast1/clusters/config-controller.
No cluster named 'config-controller' in <PROJECT-NAME>.
�[32mSetting up config connector on standalone cluster....
The connection to the server localhost:8080 was refused - did you specify the right host or port?
The connection to the server localhost:8080 was refused - did you specify the right host or port?
The connection to the server localhost:8080 was refused - did you specify the right host or port?
�[32mStandalone GKE cluster completed�[0m

getting-started.html 404 reference to external cloudshell docs/cloudshell-tutorial.md needs either direct ext github link or site reference

https://reimagined-meme-7df92d3b.pages.github.io/getting-started.html
to
docs/cloudshell-tutorial.md

Add a simplified KCC multi-tenancy namespace solution

Adapt the blueprint here: https://cloud.google.com/anthos-config-management/docs/tutorials/project-namespace-blueprint to create a single namespace per project for multi-tenancy solution.

Leave the role assignment up-to the user and simplify the blueprint so it's not reliant on the landing zone blueprint found here: https://cloud.google.com/anthos-config-management/docs/tutorials/landing-zone

Add logging role binding and region/org_id instructions during logging storage bucket update - pre CC creation

See the addition at step 0
https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/tree/main/solutions/landing-zone#usage

cloud alpha logging settings update --organization=$ORG_ID --storage-location=$REGION

on a clean org we need to set the logging admin role as well as derive the org id

michael@cloudshell:~/wse_github/GoogleCloudPlatform (pubsec-declarative-tk-gz)$ export REGION=northamerica-northeast1
michael@cloudshell:~/wse_github/GoogleCloudPlatform (pubsec-declarative-tk-gz)$ gcloud alpha logging settings update --organization=$ORG_ID --storage-location=$REGION
ERROR: (gcloud.alpha.logging.settings.update) User [[email protected]] does not have permission to access organizations instance [925207728429] (or it may not exist): Permission 'logging.cmekSettings.update' denied on resource (or it may not exist).

fix...
michael@cloudshell:~/wse_github/GoogleCloudPlatform (pubsec-declarative-tk-gz)$ export PROJECT_ID=$(gcloud config list --format 'value(core.project)')
michael@cloudshell:~/wse_github/GoogleCloudPlatform (pubsec-declarative-tk-gz)$ echo $PROJECT
pubsec-declarative-tk-gz
michael@cloudshell:~/wse_github/GoogleCloudPlatform (pubsec-declarative-tk-gz)$ export ORG_ID=$(gcloud projects get-ancestors $PROJECT_ID --format='get(id)' | tail -1)
michael@cloudshell:~/wse_github/GoogleCloudPlatform (pubsec-declarative-tk-gz)$ echo $ORG_ID
925207728429
michael@cloudshell:~/wse_github/GoogleCloudPlatform (pubsec-declarative-tk-gz)$ export [email protected]
michael@cloudshell:~/wse_github/GoogleCloudPlatform (pubsec-declarative-tk-gz)$ gcloud organizations add-iam-policy-binding "${ORG_ID}" --member "user:${EMAIL}" --role roles/logging.admin
Updated IAM policy for organization [925207728429].

michael@cloudshell:~/wse_github/GoogleCloudPlatform (pubsec-declarative-tk-gz)$ gcloud alpha logging settings update --organization=$ORG_ID --storage-location=$REGION
name: organizations/925207728429
storageLocation: northamerica-northeast1



continue KCC CC creation
michael@cloudshell:~/wse_github/GoogleCloudPlatform (pubsec-declarative-tk-gz)$ arete create landing-zone-controller --region=northamerica-northeast1
5:36PM INF Project name will be set to: landing-zone-controller-e4g7d
✔ My Billing Account - 015F84-8FD578-D96F04
✔ gcp.zone - 925207728429
✔ Folder Level
✔ pdt - 16115302749
5:36PM INF Create in progress for [https://cloudresourcemanager.googleapis.com/v1/projects/landing-zone-controller-e4g7d].Waiting for [operations/cp.8884632004916483157] to finish.....done.Enabling service [cloudapis.googleapis.com] on project [landing-zone-controller-e4g7d]...Operation "operations/acat.p2-791826419490-8b10ad41-7c22-4a31-88bb-30abad9f31d6" finished successfully.Updated property [core/project] to [landing-zone-controller-e4g7d].
5:36PM INF Enabling required services...

5:37PM INF Operation "operations/acf.p2-791826419490-2789779d-61a9-46a0-b3db-8c6217fadace" finished successfully.
5:37PM INF Creating Network...
5:37PM INF Creating subnet....
5:38PM INF Creating Config Controller Cluster...

ller].Fetching cluster endpoint and auth data.kubeconfig entry generated for krmapihost-landing-zone-controller.
6:02PM INF Add SA to roles/owner role...
6:02PM INF Config Controller setup complete

add roles (step 1)

michael@cloudshell:~/wse_github/GoogleCloudPlatform (landing-zone-controller-e4g7d)$ pwd
/home/michael/wse_github/GoogleCloudPlatform
michael@cloudshell:~/wse_github/GoogleCloudPlatform (landing-zone-controller-e4g7d)$ kpt pkg get https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit.git/solutions/landing-zone landing-zone
Package "landing-zone":
Fetching https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit@main
From https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit
 * branch            main       -> FETCH_HEAD
 * [new branch]      main       -> origin/main
Adding package "solutions/landing-zone".

Fetched 1 package(s).


edit settings.yaml

 billing-id: "015F84-8FD578-D96F04"
  org-id: "925207728429"
  #############
  # Management Project
  # This is the project where the config controller instance is running
  # Values can be viewed in the Project Dashboard
  management-project-id: landing-zone-controller-e4g7d
  management-project-number: "791826419490"
  #############
  # Project  IDs
  # These are the IDs for the projects that will be created by the LZ script
  # All IDs should be universally unique
  # Must be 6 to 30 characters in length.
  # Can only contain lowercase letters, numbers, and hyphens.
  # Must start with a letter.
  # Cannot end with a hyphen.
  # Cannot be in use or previously used; this includes deleted projects.
  # Cannot contain restricted strings, such as google and ssl.
  net-host-prj-nonprod-id: net-host-prj-nonprod-gz1
  net-host-prj-prod-id: net-host-prj-prod-gz1
  net-perimeter-prj-common-id: net-perimeter-prj-common-gz1
  audit-prj-id: audit-prj-id-gz1
  guardrails-project-id: guardrails-project-gz1
  #############
  # Groups
  # Permissions will be assigned to the specified group email
  audit-viewer: [email protected]
  log-writer: [email protected]
  log-reader: [email protected]
  organization-viewer: [email protected]

add policies exclusion

#103

michael@cloudshell:~/wse_github/GoogleCloudPlatform (landing-zone-controller-e4g7d)$ cat landing-zone/.krmignore
cicd-examples/
environments/common/policies


michael@cloudshell:~/wse_github/GoogleCloudPlatform (landing-zone-controller-e4g7d)$ cd landing-zone/
michael@cloudshell:~/wse_github/GoogleCloudPlatform/landing-zone (landing-zone-controller-e4g7d)$ kpt fn render
Package "landing-zone/environments/common/guardrails-policies":
Package "landing-zone/environments/common":
[RUNNING] "gcr.io/kpt-fn/set-namespace:v0.4.1"
[PASS] "gcr.io/kpt-fn/set-namespace:v0.4.1" in 2.8s
  Results:
    [info]: namespace "common" updated to "config-control", 23 value(s) changed

Package "landing-zone/environments/nonprod":
[RUNNING] "gcr.io/kpt-fn/set-namespace:v0.4.1"
[PASS] "gcr.io/kpt-fn/set-namespace:v0.4.1" in 400ms
  Results:
    [info]: namespace "nonprod" updated to "config-control", 7 value(s) changed

Package "landing-zone/environments/prod":
[RUNNING] "gcr.io/kpt-fn/enable-gcp-services:v0.1.0"
[PASS] "gcr.io/kpt-fn/enable-gcp-services:v0.1.0" in 3.7s
  Results:
    [info] serviceusage.cnrm.cloud.google.com/v1beta1/Service/config-control/prod-nethost-service-compute: generated service
    [info] serviceusage.cnrm.cloud.google.com/v1beta1/Service/config-control/prod-nethost-service-logging: generated service
[RUNNING] "gcr.io/kpt-fn/set-namespace:v0.4.1"
[PASS] "gcr.io/kpt-fn/set-namespace:v0.4.1" in 400ms
  Results:
    [info]: namespace "prod" updated to "config-control", 4 value(s) changed

Package "landing-zone":
[RUNNING] "gcr.io/kpt-fn/apply-setters:v0.2"
[PASS] "gcr.io/kpt-fn/apply-setters:v0.2" in 2.6s
  Results:
    [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "925207728429"
    [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "925207728429"
    [info] spec.projectID: set field value to "net-perimeter-prj-common-gz1"
    [info] spec.parentRef.external: set field value to "925207728429"
    ...(87 line(s) truncated, use '--truncate-output=false' to disable)
[RUNNING] "gcr.io/kpt-fn/generate-folders:v0.1.1"
[PASS] "gcr.io/kpt-fn/generate-folders:v0.1.1" in 5.9s
[RUNNING] "gcr.io/kpt-fn/enable-gcp-services:v0.1.0"
[PASS] "gcr.io/kpt-fn/enable-gcp-services:v0.1.0" in 2.5s
  Results:
    [info] serviceusage.cnrm.cloud.google.com/v1beta1/Service/config-control/nonprod-nethost-service-compute: generated service
    [info] serviceusage.cnrm.cloud.google.com/v1beta1/Service/config-control/nonprod-nethost-service-dns: generated service
    [info] serviceusage.cnrm.cloud.google.com/v1beta1/Service/config-control/nonprod-nethost-service-logging: generated service
    [info] serviceusage.cnrm.cloud.google.com/v1beta1/Service/config-control/prod-nethost-service-compute: recreated service
    ...(3 line(s) truncated, use '--truncate-output=false' to disable)
[RUNNING] "gcr.io/kpt-fn/gatekeeper:v0.2.1"
[PASS] "gcr.io/kpt-fn/gatekeeper:v0.2.1" in 4.4s
[RUNNING] "gcr.io/kpt-fn/kubeval:v0.3.0"

[PASS] "gcr.io/kpt-fn/kubeval:v0.3.0" in 27.5s

Successfully executed 9 function(s) in 5 package(s).


michael@cloudshell:~/wse_github/GoogleCloudPlatform/landing-zone (landing-zone-controller-e4g7d)$ kpt live init landing-zone --namespace config-control
Error: invalid directory argument: landing-zone
michael@cloudshell:~/wse_github/GoogleCloudPlatform/landing-zone (landing-zone-controller-e4g7d)$ cd ..
michael@cloudshell:~/wse_github/GoogleCloudPlatform (landing-zone-controller-e4g7d)$ kpt live init landing-zone --namespace config-control
initializing Kptfile inventory info (namespace: config-control)...success

michael@cloudshell:~/wse_github/GoogleCloudPlatform (landing-zone-controller-e4g7d)$ kpt live apply landing-zone --reconcile-timeout=2m --output=table
Error: 4 resource types could not be found in the cluster or as CRDs among the applied resources.

Resource types:
constraints.gatekeeper.sh/v1beta1, Kind=NamingPolicy
constraints.gatekeeper.sh/v1beta1, Kind=DataLocation
constraints.gatekeeper.sh/v1beta1, Kind=LimitEgressTraffic
constraints.gatekeeper.sh/v1beta1, Kind=CloudMarketPlaceConfig
michael@cloudshell:~/wse_github/GoogleCloudPlatform (landing-zone-controller-e4g7d)$ cat landing-zone/.krmignore
cicd-examples/
environments/common/policies


wrong exclusion
environments/common/guardrails-policies

one of them is not in the gr policies
michael@cloudshell:~/wse_github/GoogleCloudPlatform (landing-zone-controller-e4g7d)$ kpt live apply landing-zone --reconcile-timeout=2m --output=table
Error: 1 resource types could not be found in the cluster or as CRDs among the applied resources.

Resource types:
constraints.gatekeeper.sh/v1beta1, Kind=NamingPolicy


add
environments/common/general-policies/naming-rules

working

Custom Roles per Solution

As a solution developer I would like to be able to add custom permissions to a solution on top of what is given to config-controller as a default.

For Example I have to run the following commands to assign permissions to the primary SA.

gcloud organizations add-iam-policy-binding "${ORG_ID}" --member 
"serviceAccount:${SA_EMAIL}" --role "roles/resourcemanager.projectCreator"
gcloud organizations add-iam-policy-binding "${ORG_ID}" --member "serviceAccount:${SA_EMAIL}" --role "roles/resourcemanager.projectDeleter"
gcloud organizations add-iam-policy-binding "${ORG_ID}" --member "serviceAccount:${SA_EMAIL}" --role "roles/resourcemanager.folderAdmin"
gcloud organizations add-iam-policy-binding "${ORG_ID}" --member "serviceAccount:${SA_EMAIL}" --role "roles/billing.user"
gcloud organizations add-iam-policy-binding "${ORG_ID}" --member "serviceAccount:${SA_EMAIL}" --role "roles/iam.securityAdmin"
gcloud organizations add-iam-policy-binding "${ORG_ID}" --member "serviceAccount:${SA_EMAIL}" --role roles/iam.serviceAccountAdmin
gcloud organizations add-iam-policy-binding "${ORG_ID}" --member "serviceAccount:${SA_EMAIL}" --role "roles/orgpolicy.policyAdmin"
gcloud organizations add-iam-policy-binding "${ORG_ID}" --member "serviceAccount:${SA_EMAIL}" --role "roles/serviceusage.serviceUsageConsumer"
gcloud organizations add-iam-policy-binding "${ORG_ID}" --member "serviceAccount:${SA_EMAIL}" --role roles/logging.admin

It would be great to potentially pass the required roles in via arete on a per solution basis.

For example we could use a config yaml to hold a list of permissions
roles.yaml

roles/billing.user
roles/resourcemanager.projectDeleter
roles/iam.serviceAccountAdmin
roles/orgpolicy.policyAdmin

Pass them in via a flag

arete solution deploy guardrails --roles roles.yaml

developer quickstart session documentation

Inline comments

Markup Service YAML with comments explaining what the component does/intends to do. This will especially be helpful with the org policies.

dev-exp: LZ section 5: kpt fn render has issues connecting to the cc gke cluster - setters.yaml needs an example

section 5 of https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/tree/main/solutions/landing-zone#usage
5.deploy part 1

michael@cloudshell:~/github/GoogleCloudPlatform/pubsec-declarative-toolkit/landing-zone (landing-zone-controller-1z583)$ kpt fn render
Package "landing-zone/environments/common/guardrails-policies":
Package "landing-zone/environments/common":
[RUNNING] "gcr.io/kpt-fn/set-namespace:v0.4.1"
[PASS] "gcr.io/kpt-fn/set-namespace:v0.4.1" in 2.1s
  Results:
    [info]: namespace "common" updated to "config-control", 23 value(s) changed

Package "landing-zone/environments/nonprod":
[RUNNING] "gcr.io/kpt-fn/set-namespace:v0.4.1"
[PASS] "gcr.io/kpt-fn/set-namespace:v0.4.1" in 300ms
  Results:
    [info]: namespace "nonprod" updated to "config-control", 7 value(s) changed

Package "landing-zone/environments/prod":
[RUNNING] "gcr.io/kpt-fn/enable-gcp-services:v0.1.0"
[FAIL] "gcr.io/kpt-fn/enable-gcp-services:v0.1.0" in 700ms
  Stderr:
    "docker: Error response from daemon: Get \"https://gcr.io/v2/\": dial tcp [2607:f8b0:400c:c13::52]:443: connect: cannot assign requested address."
    "See 'docker run --help'."
  Exit code: 125

checking auth

using setters.yaml

Note: there is an ambiguity issue on the management id/number vs name/id in #98

apiVersion: v1
kind: ConfigMap
metadata: # kpt-merge: /setters
  name: setters
data:
  #############
  # General Settings Values
  #
  # The following are Settings for the environment to bootstrap with
  #
  #############
  billing-id: "019283-6F1AB5-7AD576"
  org-id: "583675367868"
  #############
  # Management Project
  # This is the project  landing-zone-controller-1z583er instance is running
  # Values can be viewed in the Project Dashboard
  management-project-id: landing-zone-controller-1z583
  management-project-number: "landing-zone-controller-1z583"
  #############
  # Project  IDs
  # These are the IDs for the projects that will be created by the LZ script
  # All IDs should be universally unique
  # Must be 6 to 30 characters in length.
  # Can only contain lowercase letters, numbers, and hyphens.
  # Must start with a letter.
  # Cannot end with a hyphen.
  # Cannot be in use or previously used; this includes deleted projects.
  # Cannot contain restricted strings, such as google and ssl.
  net-host-prj-nonprod-id: net-host-prj-nonprod-old1
  net-host-prj-prod-id: net-host-prj-prod-old1
  net-perimeter-prj-common-id: net-perimeter-prj-common-old1
  audit-prj-id: audit-prj-id-old1
  guardrails-project-id: guardrails-project-old1
  #############
  # Groups (allow for user:)
  # Permissions will be assigned to the specified group email
  audit-viewer: [email protected]
  log-writer: [email protected]
  log-reader: [email protected]
  organization-viewer: [email protected]

Add Guardrails Gatekeeper Policies

Convert Rego rules from https://github.com/canada-ca/cloud-guardrails-gcp to Gatekeeper variants. This will allow us to validate Guardrails checks on deployment of the sandboxes and also deploy them into the target GKE or Config Controller clusters for constant validation and prevention.

kpt supports Gatekeeper via function which can be added to thekptfile of the sandboxes and would trigger on kpt fn render

Enhance error handling

Currently the CLI's error handling is pretty simple and utilizes the zerlog error or fatal methods to print messages to the console. The Cobra package offers error handling (runE) which needs to be investigated including proper stack tracing.

An overhaul of the error handling needs to be investigated and implemented.

As a landingzone solution deployer i need to automate iam role creation during the bootstrap phase of create

Part of maintaining write/undo/rewrite functionality

Section 1b of https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/tree/main/solutions/landing-zone#usage requires the use of multiple identical runs of 13 iam role additions. These require read and write automation including optional key/value return state tracking (we don't undelete the usual billing user or folder admin roles on a solution delete

function apply_roles () {
if [[ -z "$USER" ]]; then
  auth
fi
# Set Vars for Permissions application
project_id=pubsec-declarative-agz
#USER="$(kubectl get ConfigConnectorContext -n config-control -o jsonpath='{.items[0].spec.googleServiceAccount}' 2> /dev/null)"
#ORGID=$(gcloud organizations list --format="get(name)" --filter=displayName=$DOMAIN)
ORGID=$(gcloud projects get-ancestors $project_id --format='get(id)' | tail -1)
ROLES=("roles/resourcemanager.folderAdmin" "roles/resourcemanager.projectCreator" "roles/resourcemanager.projectDeleter" "roles/iam.securityAdmin" "roles/orgpolicy.policyAdmin" "roles/serviceusage.serviceUsageConsumer" "roles/billing.user" "roles/accesscontextmanager.policyAdmin" "roles/compute.xpnAdmin" "roles/iam.serviceAccountAdmin" "roles/serviceusage.serviceUsageConsumer" "roles/logging.admin") 
for i in "${ROLES[@]}" ; do
  # requires iam.securityAdmin
  #ROLE=`gcloud organizations get-iam-policy $ORGID --filter="bindings.members:$USER" --flatten="bindings[].members" --format="table(bindings.role)" | grep $i`
  #if [ -z "$ROLE" ]
    #then
      echo "Applying role $i to $USER"
      gcloud organizations add-iam-policy-binding $ORGID  --member=user:$USER --role=$i --quiet > /dev/null 1>&1
    #else
      #echo "Role $i already set on $USER"
    #fi
done
}

from

export ORG_ID=your-org-id
export SA_EMAIL="$(kubectl get ConfigConnectorContext -n config-control -o jsonpath='{.items[0].spec.googleServiceAccount}' 2> /dev/null)"
gcloud organizations add-iam-policy-binding "${ORG_ID}" --member "serviceAccount:${SA_EMAIL}" --role "roles/resourcemanager.folderAdmin"
gcloud organizations add-iam-policy-binding "${ORG_ID}" --member "serviceAccount:${SA_EMAIL}" --role "roles/resourcemanager.projectCreator"
gcloud organizations add-iam-policy-binding "${ORG_ID}" --member "serviceAccount:${SA_EMAIL}" --role "roles/resourcemanager.projectDeleter"
gcloud organizations add-iam-policy-binding "${ORG_ID}" --member "serviceAccount:${SA_EMAIL}" --role "roles/iam.securityAdmin"
gcloud organizations add-iam-policy-binding "${ORG_ID}" --member "serviceAccount:${SA_EMAIL}" --role "roles/orgpolicy.policyAdmin"
gcloud organizations add-iam-policy-binding "${ORG_ID}" --member "serviceAccount:${SA_EMAIL}" --role "roles/serviceusage.serviceUsageConsumer"
gcloud organizations add-iam-policy-binding "${ORG_ID}" --member "serviceAccount:${SA_EMAIL}" --role "roles/billing.user" 
gcloud organizations add-iam-policy-binding "${ORG_ID}" --member "serviceAccount:${SA_EMAIL}" --role roles/accesscontextmanager.policyAdmin
gcloud organizations add-iam-policy-binding "${ORG_ID}" --member "serviceAccount:${SA_EMAIL}" --role roles/compute.xpnAdmin
gcloud organizations add-iam-policy-binding "${ORG_ID}" --member "serviceAccount:${SA_EMAIL}" --role roles/iam.serviceAccountAdmin
gcloud organizations add-iam-policy-binding "${ORG_ID}" --member "serviceAccount:${SA_EMAIL}" --role roles/serviceusage.serviceUsageConsumer
gcloud organizations add-iam-policy-binding "${ORG_ID}" --member "serviceAccount:${SA_EMAIL}" --role roles/logging.admin

preliminary unit testing results

Applying role roles/resourcemanager.folderAdmin to [email protected]
Updated IAM policy for organization [6839210352].
Applying role roles/resourcemanager.projectCreator to [email protected]
Updated IAM policy for organization [6839210352].
Applying role roles/resourcemanager.projectDeleter to [email protected]
Updated IAM policy for organization [6839210352].
Applying role roles/iam.securityAdmin to [email protected]
Updated IAM policy for organization [6839210352].
Applying role roles/orgpolicy.policyAdmin to [email protected]
Updated IAM policy for organization [6839210352].
Applying role roles/serviceusage.serviceUsageConsumer to [email protected]
Updated IAM policy for organization [6839210352].
Applying role roles/billing.user to [email protected]
Updated IAM policy for organization [6839210352].
Applying role roles/accesscontextmanager.policyAdmin to [email protected]
Updated IAM policy for organization [6839210352].
Applying role roles/compute.xpnAdmin to [email protected]
Updated IAM policy for organization [6839210352].
Applying role roles/iam.serviceAccountAdmin to [email protected]
Updated IAM policy for organization [6839210352].
Applying role roles/serviceusage.serviceUsageConsumer to [email protected]
Updated IAM policy for organization [6839210352].
Applying role roles/logging.admin to [email protected]
Updated IAM policy for organization [6839210352].

regression testing results pending
...

As an arete user I need to delete the .arete cache when recreating the CC cluster after a previous delete - to kick in CC creation and avoid a KCC name collision

running arete create after manually deleting the CC cluster - cluster creation does not kick in without deleting the .arete cache
running into existing KCC in .arete config - deleting
timing between project creation and service enablement is a separate issue: #93
details

admin_root@cloudshell:~ (landing-zone-controller-w8hwa)$ arete create pdt-cno-kcc --region=northamerica-northeast1 --project=pubsec-declarative-toolkit-cno
2:37PM INF Config Controller setup complete
admin_root@cloudshell:~ (landing-zone-controller-w8hwa)$ rm -rf
.arete/                .bashrc                .docker/               .npm/                  .redhat/
.bash_history          .cache/                gopath/                .profile               .theia/
.bash_logout           .config/               .kube/                 README-cloudshell.txt  wse_github/
admin_root@cloudshell:~ (landing-zone-controller-w8hwa)$ ls -la .arete/
total 20
drwxr--r--  2 admin_root admin_root 4096 Sep  2 18:15 .
drwxr-xr-x 12 admin_root admin_root 4096 Sep  6 14:37 ..
-rw-r--r--  1 admin_root admin_root   46 Aug 31 15:06 config.yaml
-rw-------  1 admin_root admin_root  100 Aug 31 15:32 .create
-rw-r--r--  1 admin_root admin_root 1318 Sep  2 18:15 solutions.yaml
admin_root@cloudshell:~ (landing-zone-controller-w8hwa)$ rm -rf .arete/

admin_root@cloudshell:~ (landing-zone-controller-w8hwa)$ arete create pdt-cno-kcc --region=northamerica-northeast1 --project=pubsec-declarative-toolkit-cno
2:40PM INF Enabling required services...
2:40PM INF Operation "operations/acat.p2-491974186555-2e6beaa9-f3df-4413-9a28-419db485c8e0" finished successfully.
2:41PM INF Creating Config Controller Cluster....
2:41PM FTL  error="ERROR: (gcloud.anthos.config.controller.create) ALREADY_EXISTS: Resource 'projects/pubsec-declarative-toolkit-cno/locations/northamerica-northeast1/krmApiHosts/pdt-cno-kcc' already exists- '@type': type.googleapis.com/google.rpc.ResourceInfo  resourceName: projects/pubsec-declarative-toolkit-cno/locations/northamerica-northeast1/krmApiHosts/pdt-cno-kcc"

deleting project - attempt to reuse may fail on 30 day deleted cache - will try
admin_root@cloudshell:~ (landing-zone-controller-w8hwa)$ gcloud projects delete pubsec-declarative-toolkit-cno
Your project will be deleted.

Do you want to continue (Y/n)?  y

Deleted [https://cloudresourcemanager.googleapis.com/v1/projects/pubsec-declarative-toolkit-cno].

You can undo this operation for a limited period by running the command below.
    $ gcloud projects undelete pubsec-declarative-toolkit-cno

See https://cloud.google.com/resource-manager/docs/creating-managing-projects for information on shutting down projects.
admin_root@cloudshell:~ (landing-zone-controller-w8hwa)$ arete create pdt-cno-kcc --region=northamerica-northeast1 --project=pubsec-declarative-toolkit-cno
✔ My Billing Account - 019..3D
✔ nuage-cloud.org - 471..7
✔ Folder Level
✔ pdt - 346..8
2:44PM FTL  error="ERROR: (gcloud.projects.create) Project creation failed. The project ID you specified is already in use by another project. Please try an alternative ID."

admin_root@cloudshell:~ (landing-zone-controller-w8hwa)$ arete create pdt-cno-kcc --region=northamerica-northeast1 --project=pubsec-declarative-toolkit-cno2
✔ My Billing Account - 01..3D
✔ nuage-cloud.org - 471924274947
✔ Folder Level
✔ pdt - 346242644868
2:45PM FTL  error="ERROR: (gcloud.projects.create) argument PROJECT_ID: Bad value [pubsec-declarative-toolkit-cno2]: Project IDs are immutable and can be set only during project creation. They must start with a lowercase letter and can have lowercase ASCII letters, digits or hyphens. Project IDs must be between 6 and 30 characters.Usage: gcloud projects create [PROJECT_ID] [optional flags]  optional flags may be  --enable-cloud-apis | --folder | --help | --labels |                         --name | --organization | --set-as-defaultFor detailed information on this command and its flags, run:  gcloud projects create --help"

30 char limit

admin_root@cloudshell:~ (landing-zone-controller-w8hwa)$ arete create pdt-cno-kcc --region=northamerica-northeast1 --project=pubsec-declarative-tk-cno2
✔ My Billing Account - 019952-0D0AAC-777E3D
✔ nuage-cloud.org - 471924274947
✔ Folder Level
✔ pdt - 346242644868
2:48PM INF Create in progress for [https://cloudresourcemanager.googleapis.com/v1/projects/pubsec-declarative-tk-cno2].Waiting for [operations/cp.7885851846085518239] to finish.....done.Enabling service [cloudapis.googleapis.com] on project [pubsec-declarative-tk-cno2]...Operation "operations/acat.p2-153970848512-8ffc1200-8c5a-42fd-b142-e11cdaf69191" finished successfully.Updated property [core/project] to [pubsec-declarative-tk-cno2].
2:48PM INF Creating Config Controller Cluster....
2:48PM FTL  error="API [krmapihosting.googleapis.com] not enabled on project [153970848512]. Would you like to enable and retry (this will take a few minutes)? (y/N)?  ERROR: (gcloud.anthos.config.controller.create) PERMISSION_DENIED: KRM API Hosting API has not been used in project 153970848512 before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/krmapihosting.googleapis.com/overview?project=153970848512 then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.- '@type': type.googleapis.com/google.rpc.Help  links:  - description: Google developers console API activation    url: https://console.developers.google.com/apis/api/krmapihosting.googleapis.com/overview?project=153970848512- '@type': type.googleapis.com/google.rpc.ErrorInfo  domain: googleapis.com  metadata:    consumer: projects/153970848512    service: krmapihosting.googleapis.com  reason: SERVICE_DISABLED"

rerun on recently created project - or run on an existing project to avoid the service enablement missing wait timer

                                                      arete create pdt-cno-kcc --region=northamerica-northeast1 --project=pubsec-declarative-tk-cno2
4:39PM INF Enabling required services...
4:40PM INF Operation "operations/acf.p2-153970848512-b3d4a2a6-fe02-4a5b-8f5d-d27d917f6527" finished successfully.
4:40PM INF Creating Network...

........................................................................................done.Created instance [pdt-cno-kcc].Fetching cluster endpoint and auth data.kubeconfig entrgenerated for krmapihost-pdt-cno-kcc.
5:09PM INF Add SA to roles/owner role...
5:09PM INF Config Controller setup complete

reviewing
https://cloud.google.com/anthos-config-management/docs/tutorials/landing-zone#removing_resources

admin_root@cloudshell:~$ gcloud config set project pubsec-declarative-tk-cno2
Updated property [core/project].
admin_root@cloudshell:~ (pubsec-declarative-tk-cno2)$ kubectl get nodes
NAME                                                  STATUS   ROLES    AGE    VERSION
gke-krmapihost-pdt-c-krmapihost-pdt-c-23345ad9-6094   Ready    <none>   7h9m   v1.22.11-gke.400
gke-krmapihost-pdt-c-krmapihost-pdt-c-6515adfd-2kgt   Ready    <none>   7h9m   v1.22.11-gke.400
gke-krmapihost-pdt-c-krmapihost-pdt-c-ce2512f1-46tj   Ready    <none>   7h9m   v1.22.11-gke.400
admin_root@cloudshell:~ (pubsec-declarative-tk-cno2)$ gcloud anthos config controller list
NAME: pdt-cno-kcc
LOCATION: northamerica-northeast1
STATE: RUNNING
admin_root@cloudshell:~ (pubsec-declarative-tk-cno2)$ gcloud anthos config controller delete pdt-cno-kcc --location=northamerica-northeast1
You are about to delete instance [pdt-cno-kcc]

Do you want to continue (Y/n)?  y

Delete request issued for: [pdt-cno-kcc]
Waiting for operation [projects/pubsec-declarative-tk-cno2/locations/northamerica-northeast1/operations/operation-1662509152408-5e80b137713f2-3980bef0-55096225] to complete...working.

[BUG]: selective failure applying to the cc via kpt live apply - for those that fail add --inventory-policy adopt

Describe the bug
kpt live apply - having periodic issues applying to the cc (after a successful replacement render (see #103 and #111)
same issue as
kptdev/kpt#1724

To Reproduce

michael@cloudshell:~/github/GoogleCloudPlatform/20220909-103 (lz-20220910-oldev)$ kubens
cnrm-system
config-control
config-management-monitoring
config-management-system
configconnector-operator-system
default
gatekeeper-system
krmapihosting-monitoring
krmapihosting-system
kube-node-lease
kube-public
kube-system
resource-group-system
michael@cloudshell:~/github/GoogleCloudPlatform/20220909-103 (lz-20220910-oldev)$ cd landing-zone/
michael@cloudshell:~/github/GoogleCloudPlatform/20220909-103/landing-zone (lz-20220910-oldev)$ kpt fn render
Package "landing-zone/environments/common/guardrails-policies":
Package "landing-zone/environments/common":
[RUNNING] "gcr.io/kpt-fn/set-namespace:v0.4.1"
[PASS] "gcr.io/kpt-fn/set-namespace:v0.4.1" in 500ms
  Results:
    [info]: all namespaces are already "config-control". no value changed

Package "landing-zone/environments/nonprod":
[RUNNING] "gcr.io/kpt-fn/set-namespace:v0.4.1"
[PASS] "gcr.io/kpt-fn/set-namespace:v0.4.1" in 400ms
  Results:
    [info]: all namespaces are already "config-control". no value changed

Package "landing-zone/environments/prod":
[RUNNING] "gcr.io/kpt-fn/enable-gcp-services:v0.1.0"
[PASS] "gcr.io/kpt-fn/enable-gcp-services:v0.1.0" in 1.6s
  Results:
    [info] serviceusage.cnrm.cloud.google.com/v1beta1/Service/config-control/prod-nethost-service-compute: recreated service
    [info] serviceusage.cnrm.cloud.google.com/v1beta1/Service/config-control/prod-nethost-service-logging: recreated service
[RUNNING] "gcr.io/kpt-fn/set-namespace:v0.4.1"
[PASS] "gcr.io/kpt-fn/set-namespace:v0.4.1" in 400ms
  Results:
    [info]: all namespaces are already "config-control". no value changed

Package "landing-zone":
[RUNNING] "gcr.io/kpt-fn/apply-setters:v0.2"
[PASS] "gcr.io/kpt-fn/apply-setters:v0.2" in 600ms
  Results:
    [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "583675367868"
    [info] metadata.annotations.cnrm.cloud.google.com/organization-id: set field value to "583675367868"
    [info] spec.projectID: set field value to "net-perimeter-prj-common-old1"
    [info] spec.parentRef.external: set field value to "583675367868"
    ...(87 line(s) truncated, use '--truncate-output=false' to disable)
[RUNNING] "gcr.io/kpt-fn/generate-folders:v0.1.1"
[PASS] "gcr.io/kpt-fn/generate-folders:v0.1.1" in 900ms
[RUNNING] "gcr.io/kpt-fn/enable-gcp-services:v0.1.0"
[PASS] "gcr.io/kpt-fn/enable-gcp-services:v0.1.0" in 1.8s
  Results:
    [info] serviceusage.cnrm.cloud.google.com/v1beta1/Service/config-control/nonprod-nethost-service-compute: recreated service
    [info] serviceusage.cnrm.cloud.google.com/v1beta1/Service/config-control/nonprod-nethost-service-dns: recreated service
    [info] serviceusage.cnrm.cloud.google.com/v1beta1/Service/config-control/nonprod-nethost-service-logging: recreated service
    [info] serviceusage.cnrm.cloud.google.com/v1beta1/Service/config-control/prod-nethost-service-compute: recreated service
    ...(3 line(s) truncated, use '--truncate-output=false' to disable)
[RUNNING] "gcr.io/kpt-fn/gatekeeper:v0.2.1"
[PASS] "gcr.io/kpt-fn/gatekeeper:v0.2.1" in 1.3s
[RUNNING] "gcr.io/kpt-fn/kubeval:v0.3.0"
[PASS] "gcr.io/kpt-fn/kubeval:v0.3.0" in 14s

Successfully executed 9 function(s) in 5 package(s).
michael@cloudshell:~/github/GoogleCloudPlatform/20220909-103/landing-zone (lz-20220910-oldev)$ cd ..
michael@cloudshell:~/github/GoogleCloudPlatform/20220909-103 (lz-20220910-oldev)$ kpt live init landing-zone --namespace config-control
initializing Kptfile inventory info (namespace: config-control)...failed
Error: Inventory information has already been added to the package Kptfile. Changing it after a package has been applied to the cluster can lead to undesired results. Use the --force flag to suppress this error. 
michael@cloudshell:~/github/GoogleCloudPlatform/20220909-103 (lz-20220910-oldev)$ kpt live apply landing-zone
installing inventory ResourceGroup CRD.
namespace/config-control apply failed: can't adopt an object without the annotation config.k8s.io/owning-inventory
namespace/config-control reconcile skipped
configmap/setters unchanged

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

Solution

running with
--inventory-policy adopt

looks to work
michael@cloudshell:~/github/GoogleCloudPlatform/20220909-103 (lz-20220910-oldev)$ kpt live apply landing-zone --inventory-policy adopt
installing inventory ResourceGroup CRD.
namespace/config-control configured
namespace/config-control reconcile pending
namespace/config-control reconciled
configmap/setters created

folders coming up




using --inventory-policy adopt via https://github.com/GoogleContainerTools/kpt/issues/1724
works well

before

michael@cloudshell:~/github/GoogleCloudPlatform/20220909-103 (lz-20220910-oldev)$ kpt live apply landing-zone
installing inventory ResourceGroup CRD.
namespace/config-control apply failed: can't adopt an object without the annotation config.k8s.io/owning-inventory
namespace/config-control reconcile skipped
configmap/setters unchanged
..
iampolicymember.iam.cnrm.cloud.google.com/log-sink-writer reconcile skipped
0 resource(s) reconciled, 90 skipped, 0 failed to reconcile, 0 timed out
1 resources failed


after
michael@cloudshell:~/github/GoogleCloudPlatform/20220909-103 (lz-20220910-oldev)$ kpt live apply landing-zone --inventory-policy adopt
installing inventory ResourceGroup CRD.
namespace/config-control configured
namespace/config-control reconcile pending
namespace/config-control reconciled
configmap/setters created
accesscontextmanageraccesslevel.accesscontextmanager.cnrm.cloud.google.com/commonaccesslevels created
accesscontextmanageraccesslevel.accesscontextmanager.cnrm.cloud.google.com/nonprodperimaccesslevel created
accesscontextmanageraccesslevel.accesscontextmanager.cnrm.cloud.google.com/prodaccesslevels created
accesscontextmanageraccesspolicy.accesscontextmanager.cnrm.cloud.google.com/orgaccesspolicy created
computefirewall.compute.cnrm.cloud.google.com/allow-egress-internet created
computefirewall.compute.cnrm.cloud.google.com/allow-egress-internet-pr created
computefirewall.compute.cnrm.cloud.google.com/allow-egress-internet-pu created
computefirewall.compute.cnrm.cloud.google.com/allow-ssh-ingress-pr created
computefirewall.compute.cnrm.cloud.google.com/allow-ssh-ingressp created
computefirewall.compute.cnrm.cloud.google.com/computefirewall-sample-deny created
computefirewall.compute.cnrm.cloud.google.com/deny-ssh-ingress created
computefirewall.compute.cnrm.cloud.google.com/prod-firewall-default-deny created
computenetwork.compute.cnrm.cloud.google.com/common-ha-perimeter created
computenetwork.compute.cnrm.cloud.google.com/common-mgmt-perimeter created
computenetwork.compute.cnrm.cloud.google.com/nonprod-sharedvpc created
computenetwork.compute.cnrm.cloud.google.com/priv-perimeter created
computenetwork.compute.cnrm.cloud.google.com/prod-sharedvpc created
computenetwork.compute.cnrm.cloud.google.com/public-perimeter created
computeprojectmetadata.compute.cnrm.cloud.google.com/nonprod-oslogin-meta created
computeroute.compute.cnrm.cloud.google.com/egress-internet-nonprod created
computeroute.compute.cnrm.cloud.google.com/egress-internet-prod created
computesharedvpchostproject.compute.cnrm.cloud.google.com/computesharedvpchostproject-sample created
computesharedvpchostproject.compute.cnrm.cloud.google.com/nonprod-shared-vpc-host created
computesubnetwork.compute.cnrm.cloud.google.com/common-ha-perimeter-subnet created
computesubnetwork.compute.cnrm.cloud.google.com/management created
computesubnetwork.compute.cnrm.cloud.google.com/nonprod-sharedvpc-subnet created
computesubnetwork.compute.cnrm.cloud.google.com/priv-perimeter-subnet created
computesubnetwork.compute.cnrm.cloud.google.com/prod-sharedvpc-subnet created
computesubnetwork.compute.cnrm.cloud.google.com/public-perimeter-subnet created
iampolicymember.iam.cnrm.cloud.google.com/audit-viewer created
iampolicymember.iam.cnrm.cloud.google.com/billing-iam-member created
iampolicymember.iam.cnrm.cloud.google.com/log-reader created
iampolicymember.iam.cnrm.cloud.google.com/log-writer created
iampolicymember.iam.cnrm.cloud.google.com/organization-viewer created
iamserviceaccount.iam.cnrm.cloud.google.com/billing-service-account created
logginglogsink.logging.cnrm.cloud.google.com/audit-bucket-sink created
logginglogsink.logging.cnrm.cloud.google.com/logs-bucket-sink created
folder.resourcemanager.cnrm.cloud.google.com/audit-and-security created
folder.resourcemanager.cnrm.cloud.google.com/audit-and-security.audit created
folder.resourcemanager.cnrm.cloud.google.com/audit-and-security.security created
folder.resourcemanager.cnrm.cloud.google.com/automation created
folder.resourcemanager.cnrm.cloud.google.com/infrastructure created
folder.resourcemanager.cnrm.cloud.google.com/infrastructure.networking created
folder.resourcemanager.cnrm.cloud.google.com/infrastructure.networking.nonprodnetworking created
folder.resourcemanager.cnrm.cloud.google.com/infrastructure.networking.prodnetworking created
folder.resourcemanager.cnrm.cloud.google.com/infrastructure.sharedinfrastructure created
folder.resourcemanager.cnrm.cloud.google.com/sandbox created
folder.resourcemanager.cnrm.cloud.google.com/shared-services created
folder.resourcemanager.cnrm.cloud.google.com/workloads created
folder.resourcemanager.cnrm.cloud.google.com/workloads.dev created
folder.resourcemanager.cnrm.cloud.google.com/workloads.prod created
folder.resourcemanager.cnrm.cloud.google.com/workloads.uat created
project.resourcemanager.cnrm.cloud.google.com/audit-prj-id-old1 created
project.resourcemanager.cnrm.cloud.google.com/guardrails-project-old1 created
project.resourcemanager.cnrm.cloud.google.com/net-host-prj-nonprod-old1 created
project.resourcemanager.cnrm.cloud.google.com/net-host-prj-prod-old1 created
project.resourcemanager.cnrm.cloud.google.com/net-perimeter-prj-common-old1 created
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/allowed-contact-domains created
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/allowed-policy-member-domain created
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/disable-guest-attribute-access created
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/disable-nested-virtualization created
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/disable-serial-port-access created
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/disable-serviceaccount-key-creation created
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/disable-vpc-external-ipv6 created
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/require-shielded-vm created
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/require-trusted-images created
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/restrict-loadbalancer-creation-types created
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/restrict-os-login created
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/restrict-resource-locations created
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/restrict-sql-public-ip created
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/restrict-vm-external-access created
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/restrict-vpc-lien-removal created
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/restrict-vpc-peering created
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/skip-default-network-creation created
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/storage-public-access-prevention created
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/uniform-bucket-level-access created
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/vm-can-ip-forward created
service.serviceusage.cnrm.cloud.google.com/common-nethost-service-compute created
service.serviceusage.cnrm.cloud.google.com/common-nethost-service-logging created
service.serviceusage.cnrm.cloud.google.com/nonprod-nethost-service-compute created
service.serviceusage.cnrm.cloud.google.com/nonprod-nethost-service-dns created
service.serviceusage.cnrm.cloud.google.com/nonprod-nethost-service-logging created
service.serviceusage.cnrm.cloud.google.com/prod-nethost-service-compute created
service.serviceusage.cnrm.cloud.google.com/prod-nethost-service-logging created
storagebucket.storage.cnrm.cloud.google.com/audit-audit-prj-id-old1 created
storagebucket.storage.cnrm.cloud.google.com/log-bucket-audit-prj-id-old1 created
configmap/setters reconciled
accesscontextmanageraccesslevel.accesscontextmanager.cnrm.cloud.google.com/commonaccesslevels reconcile pending
accesscontextmanageraccesslevel.accesscontextmanager.cnrm.cloud.google.com/nonprodperimaccesslevel reconcile pending
accesscontextmanageraccesslevel.accesscontextmanager.cnrm.cloud.google.com/prodaccesslevels reconcile pending
accesscontextmanageraccesspolicy.accesscontextmanager.cnrm.cloud.google.com/orgaccesspolicy reconcile pending
computefirewall.compute.cnrm.cloud.google.com/allow-egress-internet reconcile pending
computefirewall.compute.cnrm.cloud.google.com/allow-egress-internet-pr reconcile pending
computefirewall.compute.cnrm.cloud.google.com/allow-egress-internet-pu reconcile pending
computefirewall.compute.cnrm.cloud.google.com/allow-ssh-ingress-pr reconcile pending
computefirewall.compute.cnrm.cloud.google.com/allow-ssh-ingressp reconcile pending
computefirewall.compute.cnrm.cloud.google.com/computefirewall-sample-deny reconcile pending
computefirewall.compute.cnrm.cloud.google.com/deny-ssh-ingress reconcile pending
computefirewall.compute.cnrm.cloud.google.com/prod-firewall-default-deny reconcile pending
computenetwork.compute.cnrm.cloud.google.com/common-ha-perimeter reconcile pending
computenetwork.compute.cnrm.cloud.google.com/common-mgmt-perimeter reconcile pending
computenetwork.compute.cnrm.cloud.google.com/nonprod-sharedvpc reconcile pending
computenetwork.compute.cnrm.cloud.google.com/priv-perimeter reconcile pending
computenetwork.compute.cnrm.cloud.google.com/prod-sharedvpc reconcile pending
computenetwork.compute.cnrm.cloud.google.com/public-perimeter reconcile pending
computeprojectmetadata.compute.cnrm.cloud.google.com/nonprod-oslogin-meta reconcile pending
computeroute.compute.cnrm.cloud.google.com/egress-internet-nonprod reconcile pending
computeroute.compute.cnrm.cloud.google.com/egress-internet-prod reconcile pending
computesharedvpchostproject.compute.cnrm.cloud.google.com/computesharedvpchostproject-sample reconcile pending
computesharedvpchostproject.compute.cnrm.cloud.google.com/nonprod-shared-vpc-host reconcile pending
computesubnetwork.compute.cnrm.cloud.google.com/common-ha-perimeter-subnet reconcile pending
computesubnetwork.compute.cnrm.cloud.google.com/management reconcile pending
computesubnetwork.compute.cnrm.cloud.google.com/nonprod-sharedvpc-subnet reconcile pending
computesubnetwork.compute.cnrm.cloud.google.com/priv-perimeter-subnet reconcile pending
computesubnetwork.compute.cnrm.cloud.google.com/prod-sharedvpc-subnet reconcile pending
computesubnetwork.compute.cnrm.cloud.google.com/public-perimeter-subnet reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/audit-viewer reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/billing-iam-member reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/log-reader reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/log-writer reconcile pending
iampolicymember.iam.cnrm.cloud.google.com/organization-viewer reconciled
iamserviceaccount.iam.cnrm.cloud.google.com/billing-service-account reconcile pending
logginglogsink.logging.cnrm.cloud.google.com/audit-bucket-sink reconcile pending
logginglogsink.logging.cnrm.cloud.google.com/logs-bucket-sink reconcile pending
folder.resourcemanager.cnrm.cloud.google.com/audit-and-security reconcile pending
folder.resourcemanager.cnrm.cloud.google.com/audit-and-security.audit reconcile pending
folder.resourcemanager.cnrm.cloud.google.com/audit-and-security.security reconcile pending
folder.resourcemanager.cnrm.cloud.google.com/automation reconcile pending
folder.resourcemanager.cnrm.cloud.google.com/infrastructure reconcile pending
folder.resourcemanager.cnrm.cloud.google.com/infrastructure.networking reconcile pending
folder.resourcemanager.cnrm.cloud.google.com/infrastructure.networking.nonprodnetworking reconcile pending
folder.resourcemanager.cnrm.cloud.google.com/infrastructure.networking.prodnetworking reconcile pending
folder.resourcemanager.cnrm.cloud.google.com/infrastructure.sharedinfrastructure reconcile pending
folder.resourcemanager.cnrm.cloud.google.com/sandbox reconcile pending
folder.resourcemanager.cnrm.cloud.google.com/shared-services reconcile pending
folder.resourcemanager.cnrm.cloud.google.com/workloads reconcile pending
folder.resourcemanager.cnrm.cloud.google.com/workloads.dev reconcile pending
folder.resourcemanager.cnrm.cloud.google.com/workloads.prod reconcile pending
folder.resourcemanager.cnrm.cloud.google.com/workloads.uat reconcile pending
project.resourcemanager.cnrm.cloud.google.com/audit-prj-id-old1 reconcile pending
project.resourcemanager.cnrm.cloud.google.com/guardrails-project-old1 reconcile pending
project.resourcemanager.cnrm.cloud.google.com/net-host-prj-nonprod-old1 reconcile pending
project.resourcemanager.cnrm.cloud.google.com/net-host-prj-prod-old1 reconcile pending
project.resourcemanager.cnrm.cloud.google.com/net-perimeter-prj-common-old1 reconcile pending
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/allowed-contact-domains reconciled
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/allowed-policy-member-domain reconciled
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/disable-guest-attribute-access reconciled
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/disable-nested-virtualization reconciled
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/disable-serial-port-access reconciled
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/disable-serviceaccount-key-creation reconciled
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/disable-vpc-external-ipv6 reconciled
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/require-shielded-vm reconciled
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/require-trusted-images reconciled
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/restrict-loadbalancer-creation-types reconcile pending
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/restrict-os-login reconciled
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/restrict-resource-locations reconcile pending
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/restrict-sql-public-ip reconciled
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/restrict-vm-external-access reconcile pending
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/restrict-vpc-lien-removal reconcile pending
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/restrict-vpc-peering reconcile pending
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/skip-default-network-creation reconcile pending
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/storage-public-access-prevention reconcile pending
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/uniform-bucket-level-access reconcile pending
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/vm-can-ip-forward reconcile pending
service.serviceusage.cnrm.cloud.google.com/common-nethost-service-compute reconcile pending
E0910 15:28:03.551790    9336 task.go:270] Empty object UID from ResourceCache (status: NotFound): config-control_common-nethost-service-logging_serviceusage.cnrm.cloud.google.com_Service
E0910 15:28:03.553439    9336 task.go:270] Empty object UID from ResourceCache (status: NotFound): config-control_nonprod-nethost-service-compute_serviceusage.cnrm.cloud.google.com_Service
service.serviceusage.cnrm.cloud.google.com/common-nethost-service-logging reconcile pending
service.serviceusage.cnrm.cloud.google.com/nonprod-nethost-service-compute reconcile pending
E0910 15:28:03.553505    9336 task.go:270] Empty object UID from ResourceCache (status: NotFound): config-control_nonprod-nethost-service-dns_serviceusage.cnrm.cloud.google.com_Service
E0910 15:28:03.553533    9336 task.go:270] Empty object UID from ResourceCache (status: NotFound): config-control_nonprod-nethost-service-logging_serviceusage.cnrm.cloud.google.com_Service
service.serviceusage.cnrm.cloud.google.com/nonprod-nethost-service-dns reconcile pending
service.serviceusage.cnrm.cloud.google.com/nonprod-nethost-service-logging reconcile pending
E0910 15:28:03.553578    9336 task.go:270] Empty object UID from ResourceCache (status: NotFound): config-control_prod-nethost-service-compute_serviceusage.cnrm.cloud.google.com_Service
E0910 15:28:03.553607    9336 task.go:270] Empty object UID from ResourceCache (status: NotFound): config-control_prod-nethost-service-logging_serviceusage.cnrm.cloud.google.com_Service
service.serviceusage.cnrm.cloud.google.com/prod-nethost-service-compute reconcile pending
service.serviceusage.cnrm.cloud.google.com/prod-nethost-service-logging reconcile pending
E0910 15:28:03.553645    9336 task.go:270] Empty object UID from ResourceCache (status: NotFound): config-control_audit-audit-prj-id-old1_storage.cnrm.cloud.google.com_StorageBucket
E0910 15:28:03.553666    9336 task.go:270] Empty object UID from ResourceCache (status: NotFound): config-control_log-bucket-audit-prj-id-old1_storage.cnrm.cloud.google.com_StorageBucket
storagebucket.storage.cnrm.cloud.google.com/audit-audit-prj-id-old1 reconcile pending
storagebucket.storage.cnrm.cloud.google.com/log-bucket-audit-prj-id-old1 reconcile pending
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/restrict-vpc-peering reconciled
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/uniform-bucket-level-access reconciled
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/restrict-loadbalancer-creation-types reconcile failed
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/vm-can-ip-forward reconciled
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/restrict-vpc-lien-removal reconciled
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/storage-public-access-prevention reconcile failed
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/skip-default-network-creation reconciled
storagebucket.storage.cnrm.cloud.google.com/audit-audit-prj-id-old1 reconcile failed
storagebucket.storage.cnrm.cloud.google.com/log-bucket-audit-prj-id-old1 reconcile failed
folder.resourcemanager.cnrm.cloud.google.com/audit-and-security reconciled
computenetwork.compute.cnrm.cloud.google.com/priv-perimeter reconcile failed
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/storage-public-access-prevention reconciled
service.serviceusage.cnrm.cloud.google.com/common-nethost-service-compute reconcile failed
service.serviceusage.cnrm.cloud.google.com/common-nethost-service-logging reconcile failed
service.serviceusage.cnrm.cloud.google.com/nonprod-nethost-service-compute reconcile failed
folder.resourcemanager.cnrm.cloud.google.com/automation reconciled
folder.resourcemanager.cnrm.cloud.google.com/infrastructure reconciled
folder.resourcemanager.cnrm.cloud.google.com/sandbox reconciled
folder.resourcemanager.cnrm.cloud.google.com/shared-services reconciled
service.serviceusage.cnrm.cloud.google.com/nonprod-nethost-service-dns reconcile failed
service.serviceusage.cnrm.cloud.google.com/nonprod-nethost-service-logging reconcile failed
service.serviceusage.cnrm.cloud.google.com/prod-nethost-service-compute reconcile failed
service.serviceusage.cnrm.cloud.google.com/prod-nethost-service-logging reconcile failed
iamserviceaccount.iam.cnrm.cloud.google.com/billing-service-account reconcile failed
folder.resourcemanager.cnrm.cloud.google.com/workloads reconciled
folder.resourcemanager.cnrm.cloud.google.com/audit-and-security.audit reconciled
folder.resourcemanager.cnrm.cloud.google.com/audit-and-security.security reconciled
computenetwork.compute.cnrm.cloud.google.com/common-mgmt-perimeter reconcile failed
computesharedvpchostproject.compute.cnrm.cloud.google.com/nonprod-shared-vpc-host reconcile failed
project.resourcemanager.cnrm.cloud.google.com/audit-prj-id-old1 reconcile failed
project.resourcemanager.cnrm.cloud.google.com/guardrails-project-old1 reconcile failed
project.resourcemanager.cnrm.cloud.google.com/net-perimeter-prj-common-old1 reconcile failed
iampolicymember.iam.cnrm.cloud.google.com/audit-viewer reconciled
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/restrict-loadbalancer-creation-types reconcile pending
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/restrict-loadbalancer-creation-types reconcile failed
iampolicymember.iam.cnrm.cloud.google.com/log-reader reconciled
iampolicymember.iam.cnrm.cloud.google.com/log-writer reconciled
folder.resourcemanager.cnrm.cloud.google.com/workloads.prod reconciled
folder.resourcemanager.cnrm.cloud.google.com/infrastructure.networking reconciled
folder.resourcemanager.cnrm.cloud.google.com/workloads.uat reconciled
computesharedvpchostproject.compute.cnrm.cloud.google.com/computesharedvpchostproject-sample reconcile failed
project.resourcemanager.cnrm.cloud.google.com/guardrails-project-old1 reconcile pending
project.resourcemanager.cnrm.cloud.google.com/guardrails-project-old1 reconcile failed
folder.resourcemanager.cnrm.cloud.google.com/infrastructure.networking.nonprodnetworking reconciled
folder.resourcemanager.cnrm.cloud.google.com/infrastructure.networking.prodnetworking reconciled
folder.resourcemanager.cnrm.cloud.google.com/infrastructure.sharedinfrastructure reconciled
folder.resourcemanager.cnrm.cloud.google.com/workloads.dev reconciled

will take over an hour to bring up the system
state so far is

michael@cloudshell:~ (magellan-01)$ kubectl get gcp
NAME                                                                                          AGE   READY   STATUS         STATUS AGE
accesscontextmanageraccesspolicy.accesscontextmanager.cnrm.cloud.google.com/orgaccesspolicy   21m   False   UpdateFailed   21m

NAME                                                                                                 AGE   READY   STATUS               STATUS AGE
accesscontextmanageraccesslevel.accesscontextmanager.cnrm.cloud.google.com/commonaccesslevels        21m   False   DependencyNotReady   21m
accesscontextmanageraccesslevel.accesscontextmanager.cnrm.cloud.google.com/nonprodperimaccesslevel   21m   False   DependencyNotReady   21m
accesscontextmanageraccesslevel.accesscontextmanager.cnrm.cloud.google.com/prodaccesslevels          21m   False   DependencyNotReady   21m

NAME                                                                        AGE   READY   STATUS         STATUS AGE
computeprojectmetadata.compute.cnrm.cloud.google.com/nonprod-oslogin-meta   21m   False   UpdateFailed   21m

NAME                                                                         AGE   READY   STATUS               STATUS AGE
computesubnetwork.compute.cnrm.cloud.google.com/common-ha-perimeter-subnet   21m   False   DependencyNotReady   21m
computesubnetwork.compute.cnrm.cloud.google.com/management                   21m   False   DependencyNotReady   21m
computesubnetwork.compute.cnrm.cloud.google.com/nonprod-sharedvpc-subnet     21m   False   DependencyNotReady   21m
computesubnetwork.compute.cnrm.cloud.google.com/priv-perimeter-subnet        21m   False   DependencyNotReady   21m
computesubnetwork.compute.cnrm.cloud.google.com/prod-sharedvpc-subnet        21m   False   DependencyNotReady   21m
computesubnetwork.compute.cnrm.cloud.google.com/public-perimeter-subnet      21m   False   DependencyNotReady   21m

NAME                                                                                           AGE   READY   STATUS         STATUS AGE
computesharedvpchostproject.compute.cnrm.cloud.google.com/computesharedvpchostproject-sample   21m   False   UpdateFailed   21m
computesharedvpchostproject.compute.cnrm.cloud.google.com/nonprod-shared-vpc-host              21m   False   UpdateFailed   21m

NAME                                                                 AGE   READY   STATUS         STATUS AGE
computenetwork.compute.cnrm.cloud.google.com/common-ha-perimeter     21m   False   UpdateFailed   21m
computenetwork.compute.cnrm.cloud.google.com/common-mgmt-perimeter   21m   False   UpdateFailed   21m
computenetwork.compute.cnrm.cloud.google.com/nonprod-sharedvpc       21m   False   UpdateFailed   21m
computenetwork.compute.cnrm.cloud.google.com/priv-perimeter          21m   False   UpdateFailed   21m
computenetwork.compute.cnrm.cloud.google.com/prod-sharedvpc          21m   False   UpdateFailed   21m
computenetwork.compute.cnrm.cloud.google.com/public-perimeter        21m   False   UpdateFailed   21m

NAME                                                                 AGE   READY   STATUS               STATUS AGE
computeroute.compute.cnrm.cloud.google.com/egress-internet-nonprod   21m   False   DependencyNotReady   21m
computeroute.compute.cnrm.cloud.google.com/egress-internet-prod      21m   False   DependencyNotReady   21m

NAME                                                                        AGE   READY   STATUS               STATUS AGE
computefirewall.compute.cnrm.cloud.google.com/allow-egress-internet         21m   False   DependencyNotReady   21m
computefirewall.compute.cnrm.cloud.google.com/allow-egress-internet-pr      21m   False   DependencyNotReady   21m
computefirewall.compute.cnrm.cloud.google.com/allow-egress-internet-pu      21m   False   DependencyNotReady   21m
computefirewall.compute.cnrm.cloud.google.com/allow-ssh-ingress-pr          21m   False   DependencyNotReady   21m
computefirewall.compute.cnrm.cloud.google.com/allow-ssh-ingressp            21m   False   DependencyNotReady   21m
computefirewall.compute.cnrm.cloud.google.com/computefirewall-sample-deny   21m   False   DependencyNotReady   21m
computefirewall.compute.cnrm.cloud.google.com/deny-ssh-ingress              21m   False   DependencyNotReady   21m
computefirewall.compute.cnrm.cloud.google.com/prod-firewall-default-deny    21m   False   DependencyNotReady   21m

NAME                                                            AGE   READY   STATUS               STATUS AGE
iampolicymember.iam.cnrm.cloud.google.com/audit-viewer          21m   True    UpToDate             21m
iampolicymember.iam.cnrm.cloud.google.com/billing-iam-member    21m   False   DependencyNotReady   21m
iampolicymember.iam.cnrm.cloud.google.com/log-reader            21m   True    UpToDate             20m
iampolicymember.iam.cnrm.cloud.google.com/log-writer            21m   True    UpToDate             20m
iampolicymember.iam.cnrm.cloud.google.com/organization-viewer   21m   True    UpToDate             21m

NAME                                                                  AGE   READY   STATUS         STATUS AGE
iamserviceaccount.iam.cnrm.cloud.google.com/billing-service-account   21m   False   UpdateFailed   21m

NAME                                                             AGE   READY   STATUS               STATUS AGE
logginglogsink.logging.cnrm.cloud.google.com/audit-bucket-sink   21m   False   DependencyNotReady   21m
logginglogsink.logging.cnrm.cloud.google.com/logs-bucket-sink    21m   False   DependencyNotReady   21m

NAME                                                                                               AGE   READY   STATUS         STATUS AGE
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/allowed-contact-domains                21m   True    UpToDate       21m
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/allowed-policy-member-domain           21m   True    UpToDate       21m
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/disable-guest-attribute-access         21m   True    UpToDate       21m
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/disable-nested-virtualization          21m   True    UpToDate       21m
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/disable-serial-port-access             21m   True    UpToDate       21m
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/disable-serviceaccount-key-creation    21m   True    UpToDate       21m
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/disable-vpc-external-ipv6              21m   True    UpToDate       21m
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/require-shielded-vm                    21m   True    UpToDate       21m
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/require-trusted-images                 21m   True    UpToDate       21m
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/restrict-loadbalancer-creation-types   21m   False   UpdateFailed   21m
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/restrict-os-login                      21m   True    UpToDate       21m
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/restrict-resource-locations            21m   False   UpdateFailed   21m
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/restrict-sql-public-ip                 21m   True    UpToDate       21m
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/restrict-vm-external-access            21m   False   UpdateFailed   21m
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/restrict-vpc-lien-removal              21m   True    UpToDate       21m
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/restrict-vpc-peering                   21m   True    UpToDate       21m
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/skip-default-network-creation          21m   True    UpToDate       21m
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/storage-public-access-prevention       21m   True    UpToDate       21m
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/uniform-bucket-level-access            21m   True    UpToDate       21m
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/vm-can-ip-forward                      21m   True    UpToDate       21m

NAME                                                                                       AGE   READY   STATUS     STATUS AGE
folder.resourcemanager.cnrm.cloud.google.com/audit-and-security                            21m   True    UpToDate   21m
folder.resourcemanager.cnrm.cloud.google.com/audit-and-security.audit                      21m   True    UpToDate   21m
folder.resourcemanager.cnrm.cloud.google.com/audit-and-security.security                   21m   True    UpToDate   21m
folder.resourcemanager.cnrm.cloud.google.com/automation                                    21m   True    UpToDate   21m
folder.resourcemanager.cnrm.cloud.google.com/infrastructure                                21m   True    UpToDate   21m
folder.resourcemanager.cnrm.cloud.google.com/infrastructure.networking                     21m   True    UpToDate   20m
folder.resourcemanager.cnrm.cloud.google.com/infrastructure.networking.nonprodnetworking   21m   True    UpToDate   19m
folder.resourcemanager.cnrm.cloud.google.com/infrastructure.networking.prodnetworking      21m   True    UpToDate   19m
folder.resourcemanager.cnrm.cloud.google.com/infrastructure.sharedinfrastructure           21m   True    UpToDate   19m
folder.resourcemanager.cnrm.cloud.google.com/sandbox                                       21m   True    UpToDate   21m
folder.resourcemanager.cnrm.cloud.google.com/shared-services                               21m   True    UpToDate   21m
folder.resourcemanager.cnrm.cloud.google.com/workloads                                     21m   True    UpToDate   21m
folder.resourcemanager.cnrm.cloud.google.com/workloads.dev                                 21m   True    UpToDate   19m
folder.resourcemanager.cnrm.cloud.google.com/workloads.prod                                21m   True    UpToDate   20m
folder.resourcemanager.cnrm.cloud.google.com/workloads.uat                                 21m   True    UpToDate   20m

NAME                                                                          AGE   READY   STATUS         STATUS AGE
project.resourcemanager.cnrm.cloud.google.com/audit-prj-id-old1               21m   False   UpdateFailed   21m
project.resourcemanager.cnrm.cloud.google.com/guardrails-project-old1         21m   False   UpdateFailed   21m
project.resourcemanager.cnrm.cloud.google.com/net-host-prj-nonprod-old1       21m   False   UpdateFailed   21m
project.resourcemanager.cnrm.cloud.google.com/net-host-prj-prod-old1          21m   False   UpdateFailed   21m
project.resourcemanager.cnrm.cloud.google.com/net-perimeter-prj-common-old1   21m   False   UpdateFailed   21m

NAME                                                                         AGE   READY   STATUS         STATUS AGE
service.serviceusage.cnrm.cloud.google.com/common-nethost-service-compute    21m   False   UpdateFailed   21m
service.serviceusage.cnrm.cloud.google.com/common-nethost-service-logging    21m   False   UpdateFailed   21m
service.serviceusage.cnrm.cloud.google.com/nonprod-nethost-service-compute   21m   False   UpdateFailed   21m
service.serviceusage.cnrm.cloud.google.com/nonprod-nethost-service-dns       21m   False   UpdateFailed   21m
service.serviceusage.cnrm.cloud.google.com/nonprod-nethost-service-logging   21m   False   UpdateFailed   21m
service.serviceusage.cnrm.cloud.google.com/prod-nethost-service-compute      21m   False   UpdateFailed   21m
service.serviceusage.cnrm.cloud.google.com/prod-nethost-service-logging      21m   False   UpdateFailed   21m

NAME                                                                       AGE   READY   STATUS         STATUS AGE
storagebucket.storage.cnrm.cloud.google.com/audit-audit-prj-id-old1        21m   False   UpdateFailed   21m
storagebucket.storage.cnrm.cloud.google.com/log-bucket-audit-prj-id-old1   21m   False   UpdateFailed   21m

[BUG]: sm: During kpt live apply - 4 CRDs are missing from the CC GKE cluster - NamingPolicy, DataLocation, LimitEgressTraffic, CloudMarketPlaceConfig

Effort: sm
Priority: high
type BUG

Use case: LZ deploy section 5 - kpt
https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/tree/main/solutions/landing-zone#usage

part of #33

see setters.yaml in #99

We need to run from the parent dir

michael@cloudshell:~/github/GoogleCloudPlatform/pubsec-declarative-toolkit/landing-zone (landing-zone-controller-1z583)$ cd ..
michael@cloudshell:~/github/GoogleCloudPlatform/pubsec-declarative-toolkit (landing-zone-controller-1z583)$ kpt live init landing-zone --namespace config-control
initializing Kptfile inventory info (namespace: config-control)...success

michael@cloudshell:~/github/GoogleCloudPlatform/pubsec-declarative-toolkit (landing-zone-controller-1z583)$ kpt live apply landing-zone --reconcile-timeout=2m --output=table
Error: 4 resource types could not be found in the cluster or as CRDs among the applied resources.

Resource types:
constraints.gatekeeper.sh/v1beta1, Kind=NamingPolicy
constraints.gatekeeper.sh/v1beta1, Kind=DataLocation
constraints.gatekeeper.sh/v1beta1, Kind=LimitEgressTraffic
constraints.gatekeeper.sh/v1beta1, Kind=CloudMarketPlaceConfig

Unit tests for CLI

Right now there are no unit test in the GO cli code. This is bad, very bad and needs to be addressed.

LZ infra HA active/standby: As an Org Admin I need to modify the Landing Zone shared infrastructure via promotion of the staging side of a dual prod/stg - in eviction scenarios

use case: landing zone shared infrastructure needs an upgrade that meets the threashold of workload eviction - we need an HA configuration that uses a staging or standby perimeter/management/

Note: most changes may be non-global atomic like changing firewall rules
Note: migrating workload projects under subfolders is also a separate work item depending on whether the test environment can be used for staging

Collaboration: GoogleCloudPlatform/pbmm-on-gcp-onboarding#165

New service for a private Cloud SQL instance.

Create a new service for a private cloud SQL instance. Not sure if one service can cover both postgresql and mysql or 2 services need to created. Follow security best practices.

[ENHANCEMENT] add an update section for kpt processing of solution or settings.yaml - ie: billing id change fails to take affect

The solutions (lz) readme has initial setup but it requires sections for the following

solution update (either solution changes or changes to settings.yaml - like a billing id switch)
optional (delete solution)

Example:
It is not clear for a user new to kpt whether they should run all 3 commands when deploying modified content
In this case a billing change to an account that has more project/billing quota up from the default 5 - which will cause the solution deployment to eventually fail

michael@cloudshell:~/wse_github/GoogleCloudPlatform/landing-zone (landing-zone-controller-e4g7d)$ kpt live init landing-zone --namespace config-control
Error: invalid directory argument: landing-zone
michael@cloudshell:~/wse_github/GoogleCloudPlatform/landing-zone (landing-zone-controller-e4g7d)$ cd ..
michael@cloudshell:~/wse_github/GoogleCloudPlatform (landing-zone-controller-e4g7d)$ kpt live init landing-zone --namespace config-control
initializing Kptfile inventory info (namespace: config-control)...failed
Error: Inventory information has already been added to the package Kptfile. Changing it after a package has been applied to the cluster can lead to undesired results. Use the --force flag to suppress this error.
michael@cloudshell:~/wse_github/GoogleCloudPlatform (landing-zone-controller-e4g7d)$

michael@cloudshell:~/wse_github/GoogleCloudPlatform (landing-zone-controller-e4g7d)$ kpt live apply landing-zone --reconcile-timeout=2m
installing inventory ResourceGroup CRD.
namespace/config-control apply failed: can't adopt an object without the annotation config.k8s.io/owning-inventory
namespace/config-control reconcile skipped
configmap/setters unchanged
accesscontextmanageraccesslevel.accesscontextmanager.cnrm.cloud.google.com/commonaccesslevels unchanged
accesscontextmanageraccesslevel.accesscontextmanager.cnrm.cloud.google.com/nonprodperimaccesslevel unchanged
accesscontextmanageraccesslevel.accesscontextmanager.cnrm.cloud.google.com/prodaccesslevels unchanged
accesscontextmanageraccesspolicy.accesscontextmanager.cnrm.cloud.google.com/orgaccesspolicy unchanged
computefirewall.compute.cnrm.cloud.google.com/allow-egress-internet unchanged
computefirewall.compute.cnrm.cloud.google.com/allow-egress-internet-pr unchanged
computefirewall.compute.cnrm.cloud.google.com/allow-egress-internet-pu unchanged
computefirewall.compute.cnrm.cloud.google.com/allow-ssh-ingress-pr unchanged
computefirewall.compute.cnrm.cloud.google.com/allow-ssh-ingressp unchanged
computefirewall.compute.cnrm.cloud.google.com/computefirewall-sample-deny unchanged
computefirewall.compute.cnrm.cloud.google.com/deny-ssh-ingress unchanged
computefirewall.compute.cnrm.cloud.google.com/prod-firewall-default-deny unchanged
computenetwork.compute.cnrm.cloud.google.com/common-ha-perimeter unchanged
computenetwork.compute.cnrm.cloud.google.com/common-mgmt-perimeter unchanged
computenetwork.compute.cnrm.cloud.google.com/nonprod-sharedvpc unchanged
computenetwork.compute.cnrm.cloud.google.com/priv-perimeter unchanged
computenetwork.compute.cnrm.cloud.google.com/prod-sharedvpc unchanged
computenetwork.compute.cnrm.cloud.google.com/public-perimeter unchanged
computeprojectmetadata.compute.cnrm.cloud.google.com/nonprod-oslogin-meta unchanged
computeroute.compute.cnrm.cloud.google.com/egress-internet-nonprod unchanged
computeroute.compute.cnrm.cloud.google.com/egress-internet-prod unchanged
computesharedvpchostproject.compute.cnrm.cloud.google.com/computesharedvpchostproject-sample unchanged
computesharedvpchostproject.compute.cnrm.cloud.google.com/nonprod-shared-vpc-host unchanged
computesubnetwork.compute.cnrm.cloud.google.com/common-ha-perimeter-subnet unchanged
computesubnetwork.compute.cnrm.cloud.google.com/management unchanged
computesubnetwork.compute.cnrm.cloud.google.com/nonprod-sharedvpc-subnet unchanged
computesubnetwork.compute.cnrm.cloud.google.com/priv-perimeter-subnet unchanged
computesubnetwork.compute.cnrm.cloud.google.com/prod-sharedvpc-subnet unchanged
computesubnetwork.compute.cnrm.cloud.google.com/public-perimeter-subnet unchanged
iampolicymember.iam.cnrm.cloud.google.com/audit-viewer unchanged
iampolicymember.iam.cnrm.cloud.google.com/billing-iam-member unchanged
iampolicymember.iam.cnrm.cloud.google.com/log-reader unchanged
iampolicymember.iam.cnrm.cloud.google.com/log-writer unchanged
iampolicymember.iam.cnrm.cloud.google.com/organization-viewer unchanged
iamserviceaccount.iam.cnrm.cloud.google.com/billing-service-account unchanged
logginglogsink.logging.cnrm.cloud.google.com/audit-bucket-sink unchanged
logginglogsink.logging.cnrm.cloud.google.com/logs-bucket-sink unchanged
folder.resourcemanager.cnrm.cloud.google.com/audit-and-security unchanged
folder.resourcemanager.cnrm.cloud.google.com/audit-and-security.audit unchanged
folder.resourcemanager.cnrm.cloud.google.com/audit-and-security.security unchanged
folder.resourcemanager.cnrm.cloud.google.com/automation unchanged
folder.resourcemanager.cnrm.cloud.google.com/infrastructure unchanged
folder.resourcemanager.cnrm.cloud.google.com/infrastructure.networking unchanged
folder.resourcemanager.cnrm.cloud.google.com/infrastructure.networking.nonprodnetworking unchanged
folder.resourcemanager.cnrm.cloud.google.com/infrastructure.networking.prodnetworking unchanged
folder.resourcemanager.cnrm.cloud.google.com/infrastructure.sharedinfrastructure unchanged
folder.resourcemanager.cnrm.cloud.google.com/sandbox unchanged
folder.resourcemanager.cnrm.cloud.google.com/shared-services unchanged
folder.resourcemanager.cnrm.cloud.google.com/workloads unchanged
folder.resourcemanager.cnrm.cloud.google.com/workloads.dev unchanged
folder.resourcemanager.cnrm.cloud.google.com/workloads.prod unchanged
folder.resourcemanager.cnrm.cloud.google.com/workloads.uat unchanged
project.resourcemanager.cnrm.cloud.google.com/audit-prj-id-gz1 unchanged
project.resourcemanager.cnrm.cloud.google.com/guardrails-project-gz1 unchanged
project.resourcemanager.cnrm.cloud.google.com/net-host-prj-nonprod-gz1 unchanged
project.resourcemanager.cnrm.cloud.google.com/net-host-prj-prod-gz1 unchanged
project.resourcemanager.cnrm.cloud.google.com/net-perimeter-prj-common-gz1 unchanged
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/allowed-contact-domains unchanged
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/allowed-policy-member-domain unchanged
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/disable-guest-attribute-access unchanged
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/disable-nested-virtualization unchanged
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/disable-serial-port-access unchanged
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/disable-serviceaccount-key-creation unchanged
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/disable-vpc-external-ipv6 unchanged
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/require-shielded-vm unchanged
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/require-trusted-images unchanged
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/restrict-loadbalancer-creation-types unchanged
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/restrict-os-login unchanged
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/restrict-resource-locations unchanged
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/restrict-sql-public-ip unchanged
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/restrict-vm-external-access unchanged
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/restrict-vpc-lien-removal unchanged
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/restrict-vpc-peering unchanged
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/skip-default-network-creation unchanged
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/storage-public-access-prevention unchanged
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/uniform-bucket-level-access unchanged
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/vm-can-ip-forward unchanged
service.serviceusage.cnrm.cloud.google.com/common-nethost-service-compute unchanged
service.serviceusage.cnrm.cloud.google.com/common-nethost-service-logging unchanged
service.serviceusage.cnrm.cloud.google.com/nonprod-nethost-service-compute unchanged
service.serviceusage.cnrm.cloud.google.com/nonprod-nethost-service-dns unchanged
service.serviceusage.cnrm.cloud.google.com/nonprod-nethost-service-logging unchanged
service.serviceusage.cnrm.cloud.google.com/prod-nethost-service-compute unchanged
service.serviceusage.cnrm.cloud.google.com/prod-nethost-service-logging unchanged
storagebucket.storage.cnrm.cloud.google.com/audit-audit-prj-id-gz1 unchanged
storagebucket.storage.cnrm.cloud.google.com/log-bucket-audit-prj-id-gz1 unchanged
configmap/setters reconcile skipped
accesscontextmanageraccesslevel.accesscontextmanager.cnrm.cloud.google.com/commonaccesslevels reconcile skipped
accesscontextmanageraccesslevel.accesscontextmanager.cnrm.cloud.google.com/nonprodperimaccesslevel reconcile skipped
accesscontextmanageraccesslevel.accesscontextmanager.cnrm.cloud.google.com/prodaccesslevels reconcile skipped
accesscontextmanageraccesspolicy.accesscontextmanager.cnrm.cloud.google.com/orgaccesspolicy reconcile skipped
computefirewall.compute.cnrm.cloud.google.com/allow-egress-internet reconcile skipped
computefirewall.compute.cnrm.cloud.google.com/allow-egress-internet-pr reconcile skipped
computefirewall.compute.cnrm.cloud.google.com/allow-egress-internet-pu reconcile skipped
computefirewall.compute.cnrm.cloud.google.com/allow-ssh-ingress-pr reconcile skipped
computefirewall.compute.cnrm.cloud.google.com/allow-ssh-ingressp reconcile skipped
computefirewall.compute.cnrm.cloud.google.com/computefirewall-sample-deny reconcile skipped
computefirewall.compute.cnrm.cloud.google.com/deny-ssh-ingress reconcile skipped
computefirewall.compute.cnrm.cloud.google.com/prod-firewall-default-deny reconcile skipped
computenetwork.compute.cnrm.cloud.google.com/common-ha-perimeter reconcile skipped
computenetwork.compute.cnrm.cloud.google.com/common-mgmt-perimeter reconcile skipped
computenetwork.compute.cnrm.cloud.google.com/nonprod-sharedvpc reconcile skipped
computenetwork.compute.cnrm.cloud.google.com/priv-perimeter reconcile skipped
computenetwork.compute.cnrm.cloud.google.com/prod-sharedvpc reconcile skipped
computenetwork.compute.cnrm.cloud.google.com/public-perimeter reconcile skipped
computeprojectmetadata.compute.cnrm.cloud.google.com/nonprod-oslogin-meta reconcile skipped
computeroute.compute.cnrm.cloud.google.com/egress-internet-nonprod reconcile skipped
computeroute.compute.cnrm.cloud.google.com/egress-internet-prod reconcile skipped
computesharedvpchostproject.compute.cnrm.cloud.google.com/computesharedvpchostproject-sample reconcile skipped
computesharedvpchostproject.compute.cnrm.cloud.google.com/nonprod-shared-vpc-host reconcile skipped
computesubnetwork.compute.cnrm.cloud.google.com/common-ha-perimeter-subnet reconcile skipped
computesubnetwork.compute.cnrm.cloud.google.com/management reconcile skipped
computesubnetwork.compute.cnrm.cloud.google.com/nonprod-sharedvpc-subnet reconcile skipped
computesubnetwork.compute.cnrm.cloud.google.com/priv-perimeter-subnet reconcile skipped
computesubnetwork.compute.cnrm.cloud.google.com/prod-sharedvpc-subnet reconcile skipped
computesubnetwork.compute.cnrm.cloud.google.com/public-perimeter-subnet reconcile skipped
iampolicymember.iam.cnrm.cloud.google.com/audit-viewer reconcile skipped
iampolicymember.iam.cnrm.cloud.google.com/billing-iam-member reconcile skipped
iampolicymember.iam.cnrm.cloud.google.com/log-reader reconcile skipped
iampolicymember.iam.cnrm.cloud.google.com/log-writer reconcile skipped
iampolicymember.iam.cnrm.cloud.google.com/organization-viewer reconcile skipped
iamserviceaccount.iam.cnrm.cloud.google.com/billing-service-account reconcile skipped
logginglogsink.logging.cnrm.cloud.google.com/audit-bucket-sink reconcile skipped
logginglogsink.logging.cnrm.cloud.google.com/logs-bucket-sink reconcile skipped
folder.resourcemanager.cnrm.cloud.google.com/audit-and-security reconcile skipped
folder.resourcemanager.cnrm.cloud.google.com/audit-and-security.audit reconcile skipped
folder.resourcemanager.cnrm.cloud.google.com/audit-and-security.security reconcile skipped
folder.resourcemanager.cnrm.cloud.google.com/automation reconcile skipped
folder.resourcemanager.cnrm.cloud.google.com/infrastructure reconcile skipped
folder.resourcemanager.cnrm.cloud.google.com/infrastructure.networking reconcile skipped
folder.resourcemanager.cnrm.cloud.google.com/infrastructure.networking.nonprodnetworking reconcile skipped
folder.resourcemanager.cnrm.cloud.google.com/infrastructure.networking.prodnetworking reconcile skipped
folder.resourcemanager.cnrm.cloud.google.com/infrastructure.sharedinfrastructure reconcile skipped
folder.resourcemanager.cnrm.cloud.google.com/sandbox reconcile skipped
folder.resourcemanager.cnrm.cloud.google.com/shared-services reconcile skipped
folder.resourcemanager.cnrm.cloud.google.com/workloads reconcile skipped
folder.resourcemanager.cnrm.cloud.google.com/workloads.dev reconcile skipped
folder.resourcemanager.cnrm.cloud.google.com/workloads.prod reconcile skipped
folder.resourcemanager.cnrm.cloud.google.com/workloads.uat reconcile skipped
project.resourcemanager.cnrm.cloud.google.com/audit-prj-id-gz1 reconcile skipped
project.resourcemanager.cnrm.cloud.google.com/guardrails-project-gz1 reconcile skipped
project.resourcemanager.cnrm.cloud.google.com/net-host-prj-nonprod-gz1 reconcile skipped
project.resourcemanager.cnrm.cloud.google.com/net-host-prj-prod-gz1 reconcile skipped
project.resourcemanager.cnrm.cloud.google.com/net-perimeter-prj-common-gz1 reconcile skipped
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/allowed-contact-domains reconcile skipped
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/allowed-policy-member-domain reconcile skipped
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/disable-guest-attribute-access reconcile skipped
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/disable-nested-virtualization reconcile skipped
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/disable-serial-port-access reconcile skipped
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/disable-serviceaccount-key-creation reconcile skipped
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/disable-vpc-external-ipv6 reconcile skipped
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/require-shielded-vm reconcile skipped
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/require-trusted-images reconcile skipped
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/restrict-loadbalancer-creation-types reconcile skipped
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/restrict-os-login reconcile skipped
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/restrict-resource-locations reconcile skipped
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/restrict-sql-public-ip reconcile skipped
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/restrict-vm-external-access reconcile skipped
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/restrict-vpc-lien-removal reconcile skipped
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/restrict-vpc-peering reconcile skipped
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/skip-default-network-creation reconcile skipped
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/storage-public-access-prevention reconcile skipped
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/uniform-bucket-level-access reconcile skipped
resourcemanagerpolicy.resourcemanager.cnrm.cloud.google.com/vm-can-ip-forward reconcile skipped
service.serviceusage.cnrm.cloud.google.com/common-nethost-service-compute reconcile skipped
service.serviceusage.cnrm.cloud.google.com/common-nethost-service-logging reconcile skipped
service.serviceusage.cnrm.cloud.google.com/nonprod-nethost-service-compute reconcile skipped
service.serviceusage.cnrm.cloud.google.com/nonprod-nethost-service-dns reconcile skipped
service.serviceusage.cnrm.cloud.google.com/nonprod-nethost-service-logging reconcile skipped
service.serviceusage.cnrm.cloud.google.com/prod-nethost-service-compute reconcile skipped
service.serviceusage.cnrm.cloud.google.com/prod-nethost-service-logging reconcile skipped
storagebucket.storage.cnrm.cloud.google.com/audit-audit-prj-id-gz1 reconcile skipped
storagebucket.storage.cnrm.cloud.google.com/log-bucket-audit-prj-id-gz1 reconcile skipped
iampolicymember.iam.cnrm.cloud.google.com/audit-sink-writer unchanged
iampolicymember.iam.cnrm.cloud.google.com/log-sink-writer unchanged
90 resource(s) applied. 0 created, 89 unchanged, 0 configured, 1 failed
iampolicymember.iam.cnrm.cloud.google.com/audit-sink-writer reconcile skipped
iampolicymember.iam.cnrm.cloud.google.com/log-sink-writer reconcile skipped
0 resource(s) reconciled, 90 skipped, 0 failed to reconcile, 0 timed out

For example all projects below should be on the obrienlabs-dev billing account - only 2 that were moved previously manually were

Add NIST Firewall Rules

To pass the NIST Report from SCC the following Firewall Rules will need to be implemented

Firewall rules should not allow connections from all IP addresses on TCP ports 7000-7001, 7199, 8888, 9042, 9160, 61620-61621
Firewall rules should not allow connections from all IP addresses on TCP port 9090
Firewall rules should not allow connections from all IP addresses on TCP or UDP port 445
Firewall rules should not allow connections from all IP addresses on TCP or UDP port 53
Firewall rules should not allow connections from all IP addresses on TCP ports 9200, 9300
Firewall rules should not allow connections from all IP addresses on TCP port 21
Firewall rules should not allow connections from all IP addresses on TCP port 80
Firewall rules should not allow connections from all IP addresses on TCP ports 389, 636 or UDP port 389
Firewall rules should not allow connections from all IP addresses on TCP ports 11211, 11214-11215 or UDP ports 11211, 11214-11215
Firewall rules should not allow connections from all IP addresses on TCP ports 27017-27019
Firewall rules should not allow connections from all IP addresses on TCP port 3306
Firewall rules should not allow connections from all IP addresses on TCP or UDP ports 137-139
Firewall rules should not allow connections from all IP addresses on TCP ports 1521, 2483-2484 or UDP ports 2483-2484
Firewall rules should not allow connections from all IP addresses on TCP port 110
Firewall rules should not allow connections from all IP addresses on TCP or UDP port 5432
Firewall rules should not allow connections from all IP addresses on TCP or UDP port 3389
Firewall rules should not allow connections from all IP addresses on TCP port 6379
Firewall rules should not allow connections from all IP addresses on TCP port 25
Firewall rules should not allow connections from all IP addresses on TCP or SCTP port 22
Firewall rules should not allow connections from all IP addresses on TCP port 23

Improve Contribution Documention

We should improve the documentation on how to contribute to the repository.

Topics to Cover

Forking/Branching strategy for PRs
When to use kpt pkg get vs git clone

Flip perimeter prod firewall deny rule for ssh:22 to allow - during dev - or flip name to deny to block

Once we get a range or single tunnel proxy or bastion/IAP access going we can flip back to deny

spec:
  description: Allow SSH Connections from the internet
  direction: INGRESS
  deny:
    - protocol: tcp
      ports: 
      - "22"
  sourceRanges:
    - 0.0.0.0/0

Restructure Packages for Multi-Tenancy

We should restructure the packages to leverage deployment via namespacing as described https://cloud.google.com/anthos-config-management/docs/tutorials/project-namespace-blueprint .

This should help keep deployments structured and avoid dumping all resources into a single namespace, this will also be beneficial for reducing the blast radius of teams having too many permissions or access to more resources than needed.

Compliance: ITSG-22 Network Security Zones and ITSG-38 Network Security Zoning - Design Considerations for Placement of Services

We have been focused on ITSG-33 security controls until 202208 - we need to verify our compliance with Network Zoning and ZIPs (Zone Interface Point)

ITSG-22 https://cyber.gc.ca/en/guidance/baseline-security-requirements-network-security-zones-version-20-itsp80022 and https://cyber.gc.ca/sites/default/files/cyber/publications/itsp80022-e.pdf

ITSG-38 Placement of Services with Zones - https://open.canada.ca/data/en/dataset/7ef76a62-bb53-4e9c-b2a4-03e5c53570a1 and
https://cyber.gc.ca/en/guidance/network-security-zoning-design-considerations-placement-services-within-zones-itsg-38
SSC 2020 https://wiki.gccollab.ca/images/9/9d/Network_Security_Zoning_Reference_Architecture.pdf

Flows, firewall demarcation, encryption in transit levels l3/l4 (default internal) + l7

Expand on https://cloud.google.com/architecture/landing-zones/decide-network-design#option-2 in https://cloud.google.com/architecture/landing-zones#what-is-a-google-cloud-landing-zone

Google Front End Service (reverse proxy) - https://cloud.google.com/docs/security/infrastructure/design#google_front_end_service

20220920
Network separation question:
We have a couple options to start -
1 - shared VPC (host) - with service subnets - this works best for PaaS workloads (with some sharing - where separation is at the k8s namespace and/or service level) - (aka transit gateway)
2 - workload (non-shared) VPC's peered to the shared perimeter (1 fortigate cluster per gc-cap - with flow separation)
3 - workload (non-shared) VPC's peered to their own perimeter (usually prod/stg/dev fortigate separation but we can expand) - I have only see 2 above though
For all 1-3 we can separate using explicit and implicit routing/firewall-rules separation
Usually 2-3 are for both free-form sandbox and specific prod workloads (one team does bucket downloads for example) - where the rest of the workloads are ok with sharing the paas in 1

see prototyping that needs to be moved to kcc
org a- gcp.obrien.services https://console.cloud.google.com/networking/peering/list?orgonly=true&project=ospe-obs-obsprd-obspubper&supportedpurview=project

Existing peering to be updated as per
https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/dev/solutions/landing-zone/architecture.md#di-20-separate-vpc-per-cloud-profile-356-workloads

uncomment and KPT render each peer pair in
https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/main/solutions/landing-zone/environments/common/network/network-peering.yaml#L15

Create Documentation static site

Create a documentation site that can be deployed to GitHub instead of expanding the README.

Bastion Host

Create a service that will create a bastion host that uses security best practices.

[ENHANCEMENT] full workaround for #103 where we .krmignore the policy folders that block the landing-zone solution

Describe the bug
A better fix/enhancement for getting policy deployments working in the landing-zone solution

undo adding

environments/common/guardrails-policies
environments/common/general-policies/naming-rules

in https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/blob/main/solutions/landing-zone/.krmignore#L1

to avoid

michael@cloudshell:~/github/GoogleCloudPlatform/pubsec-declarative-toolkit (landing-zone-controller-1z583)$ kpt live apply landing-zone --reconcile-timeout=2m --output=table
Error: 4 resource types could not be found in the cluster or as CRDs among the applied resources.

Resource types:
constraints.gatekeeper.sh/v1beta1, Kind=NamingPolicy
constraints.gatekeeper.sh/v1beta1, Kind=DataLocation
constraints.gatekeeper.sh/v1beta1, Kind=LimitEgressTraffic
constraints.gatekeeper.sh/v1beta1, Kind=CloudMarketPlaceConfig

See linking to existing #103 and the PR #107

To Reproduce
Steps to reproduce the behavior:

Go to '...'
Click on '....'
Scroll down to '....'
See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

Multi-Cluster Solution with Anthos

Create multi-cluster anthos deployment that leverages multi-cluster services, Anthos Service Mesh and Anthos Config Controller.

Create Cloud Shell integration w/ tutorial

Create the Open w/ Cloud Shell GitHub button that clones and launches a tutorial in GCP Cloud Shell for bootstrapping the sandbox.

dev-exp: LZ section 2: kpt pkg get - why repackage from remote instead of using existing local landing-zone folder

section 2 https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/tree/main/solutions/landing-zone#usage

it is not clear where to pull the kpt fetch (inside the repo? and why are we not using the folder already in the repo?)
Went ahead with the default at the repo root - but why are we not just using the local instead of the remote

michael@cloudshell:~/github/GoogleCloudPlatform/pubsec-declarative-toolkit (landing-zone-controller-1z583)$ kpt pkg get https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit.git/solutions/landing-zone landing-zone
Package "landing-zone":
Fetching https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit@main
From https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit
 * branch            main       -> FETCH_HEAD
 * [new branch]      main       -> origin/main
Adding package "solutions/landing-zone".
Fetched 1 package(s).

for example - both are identical as expected 
michael@cloudshell:~/github/GoogleCloudPlatform/pubsec-declarative-toolkit (landing-zone-controller-1z583)$ ls landing-zone/
cicd-examples  environments  img  Kptfile  README.md  setters.yaml  solution.yaml
michael@cloudshell:~/github/GoogleCloudPlatform/pubsec-declarative-toolkit (landing-zone-controller-1z583)$ ls solutions/landing-zone/
cicd-examples  environments  img  Kptfile  README.md  setters.yaml  solution.yaml
michael@cloudshell:~/github/GoogleCloudPlatform/pubsec-declarative-toolkit (landing-zone-controller-1z583)$ git status
On branch main
Your branch is up to date with 'origin/main'.
Untracked files:
  (use "git add <file>..." to include in what will be committed)
        landing-zone/

[BUG] add cd .. between kpt fn render and kpt live commands during resource inventory update - or take out the landing-zone folder reference - landing-zone solution

Describe the bug
running the landing-zone kpt (non-gitops section) directly will fail on the kpt resource inventory update (step 2 of 3)

The 3 commands in the landing-zone solution section 5 are not run in the same directory - you must .. up a dir when running the 2 kpt live commands or take out the landing-zone folder reference from the kpt resource inventory update and kpt apply of the package to the cluster

Additional context

kpt fn render
kpt live init landing-zone --namespace config-control
kpt live apply landing-zone --reconcile-timeout=2m --output=table

to
cd landing-zone
kpt fn render
cd ..
kpt live init landing-zone --namespace config-control
kpt live apply landing-zone --reconcile-timeout=2m --output=table

or take out the folder
michael@cloudshell:~/wse_github/GoogleCloudPlatform/landing-zone (landing-zone-controller-e4g7d)$ kpt live init landing-zone --namespace config-control
Error: invalid directory argument: landing-zone
michael@cloudshell:~/wse_github/GoogleCloudPlatform/landing-zone (landing-zone-controller-e4g7d)$ kpt live init  --namespace config-control
initializing Kptfile inventory info (namespace: config-control)...failed
Error: Inventory information has already been added to the package Kptfile. Changing it after a package has been applied to the cluster can lead to undesired results. Use the --force flag to suppress this error.
michael@cloudshell:~/wse_github/GoogleCloudPlatform/landing-zone (landing-zone-controller-e4g7d)$ kpt live apply  --reconcile-timeout=2m
installing inventory ResourceGroup CRD.
namespace/config-control apply failed: can't adopt an object without the annotation config.k8s.io/owning-inventory
namespace/config-control reconcile skipped
configmap/setters unchanged
accesscontextmanageraccesslevel.accesscontextmanager.cnrm.cloud.google.com/commonaccesslevels unchanged

**Full run results**
michael@cloudshell:~/wse_github/GoogleCloudPlatform (landing-zone-controller-e4g7d)$ ls
landing-zone  pubsec-declarative-toolkit
michael@cloudshell:~/wse_github/GoogleCloudPlatform (landing-zone-controller-e4g7d)$ cd landing-zone/
michael@cloudshell:~/wse_github/GoogleCloudPlatform/landing-zone (landing-zone-controller-e4g7d)$ kpt fn render
Package "landing-zone/environments/common/guardrails-policies":
Package "landing-zone/environments/common":
....
[PASS] "gcr.io/kpt-fn/kubeval:v0.3.0" in 21s
Successfully executed 9 function(s) in 5 package(s).


michael@cloudshell:~/wse_github/GoogleCloudPlatform/landing-zone (landing-zone-controller-e4g7d)$ kpt live init landing-zone --namespace config-control
Error: invalid directory argument: landing-zone
michael@cloudshell:~/wse_github/GoogleCloudPlatform/landing-zone (landing-zone-controller-e4g7d)$ cd ..
michael@cloudshell:~/wse_github/GoogleCloudPlatform (landing-zone-controller-e4g7d)$ kpt live init landing-zone --namespace config-control
initializing Kptfile inventory info (namespace: config-control)...failed
Error: Inventory information has already been added to the package Kptfile. Changing it after a package has been applied to the cluster can lead to undesired results. Use the --force flag to suppress this error.

or see the alternate landing zone code at

where we leave out the folder name

kpt live init --namespace ${NAMESPACE}"

googlecloudplatform / pubsec-declarative-toolkit Goto Github PK

pubsec-declarative-toolkit's Introduction

GCP PubSec Declarative Toolkit

Current Solutions

Quickstart

Additional Documentation

Additional Resources

Disclaimer

pubsec-declarative-toolkit's People

Contributors

Stargazers

Watchers

Forkers

pubsec-declarative-toolkit's Issues

adjust readme instructions to pass in the region when standing up the landing zone solution

GKE cluster will not come up in northamerica-northeast1 with options set due to missing: Secure Boot is not enabled in the 'shielded_instance_config' field - due to a policy override

Working on

Queued prioritized

Queued non-prioritized

Deferred

closed

removing the solution via arete

removing the solution via kubernetes cli (thanks Chris for reminding me to think granular k8s again)

removing the solution via kpt cli

removing the solution via gcloud cli

removing the solution manually

Rerun the CC cluster creation and LZ solution

deletion using anthos delete

Temporary CC cluster shutdown and restart

use case: landing zone shared infrastructure needs an upgrade that meets the threashold of workload eviction - we need an HA configuration that uses a staging or standby perimeter/management/

Recommend Projects

Recommend Topics

Recommend Org