zaba505 / infra Goto Github PK

Description

As a sys admin, I want a config which the boot image will replace itself with, so that I can make changes to the runtime config and never really have to touch the boot config.

Acceptance Criteria

• Boot config replaces itself with runtime config
• Ignition file is built in Actions
• Ignition file is added to releases

Related Issues

No response

Description

As a dev ops eng, I want to automate the creation of DNS A and AAAA records in Cloudflare for the application load balancer, so that the APIs can be publicly exposed under a domain.

Acceptance Criteria

• A record is created using ipv4 address from gateway module
• AAAA record is created using ipv6 address from gateway module

Related Issues

No response

Description

As a devops engineer, I want to abstract over deploying the machine image service in Google Cloud, so that future deployments can leverage a reusable module to manage the entire machine image system.

Acceptance Criteria

• Service Account is used by instances to access boot image storage bucket
• Can scale down to zero to optimize costs
• Monitoring and logs capture is configured to go to Google Cloud monitoring

Related Issues

No response

Description

As a maintainer, I want to onboard renovate, so that all dependencies can be automatically updated.

Acceptance Criteria

• Onboarded

Related Issues

No response

Description

As sec ops, I want to ensure that Cloudflare is the only client able to connect to the application load balancer, so the attack surface is minimized.

Acceptance Criteria

• Certificate is created
• Certs are published to both Cloudflare and GCP
• Per domain support

Related Issues

No response

Description

As a machine manager, I want to write a bootstrap iPXE image for machines to run and chainload to a more specific image based on device parameters like the mac address of the machine.

Acceptance Criteria

• Custom built iPXE image with an embedded script that chain loads another image
• Custom script should call machine mgmt service to get specific image

Related Issues

No response

Description

As a sys admin, I want to make the default load balancer accept insecure requests, so that requests actually make it to it and it replies with the proper 503 response.

Acceptance Criteria

• add insecure option to cloud_run module
• mark default service as insecure

Related Issues

No response

Description

As a maintainer, I want to split the main terraform module into separate modules, so that the entire system can be deployed via smaller terraform applies.

Acceptance Criteria

• Main module is split into multiple modules

Related Issues

No response

Description

As a dev, I want to implement startup and liveness endpoints, so that orchestration tools like Kubernetes can automatically monitor and maintain my application.

Acceptance Criteria

• Startup check is implemented
• Liveness check is implemented

Related Issues

No response

Description

As a maintainer, I want to enable all the package rules, so that Renovate bot will open PRs.

Acceptance Criteria

• Each package rule is enabled

Related Issues

No response

Description

As a sys admin, I want my servers to network boot via iPXE, so that I don't have to manually plug stuff into them.

Acceptance Criteria

• iPXE script is written
• iPXE script is added to releases
• iPXE script builds to PXE image
• iPXE script calls out to http server endpoint

Related Issues

No response

Description

As a dev ops engineer, I want to be able to network boot machines using iPXE scripts fetched from an HTTP service, so that I can more dynamically configures machines.

Acceptance Criteria

• Should build iPXE -> PXE image which chain loads to iPXE image from a HTTP service

Related Issues

No response

Description

As a maintainer, I want to fully specify the go version in both CodeQL and go.mod, so that CodeQL no longer warns about the go version not following 1.N.P.

Acceptance Criteria

• CodeQL warning goes away

Related Issues

No response

Description

As a dev ops, I want to implement a actions workflow for scanning and updating dependencies, so that the services will stay up-to-date

Acceptance Criteria

• Deps are scanned regularly
• Deps are automatically updated and merged

Related Issues

No response

Description

As sec ops, I want to make sure all Cloud Run services are deployed to a private network, so that they can not be publicly accessed.

Acceptance Criteria

• Private network is created
• cloud_run_service module is updated to deploy services to private network

Related Issues

No response

Description

As a project owner, I want to document the network boot architecture, so that it can be analyzed and referenced later during development.

Acceptance Criteria

• DHCP step is documented
• Bootstrap image phase is documented
• Runtime image fetching is documented
• Deployment of boot images to "prod" is documented

Related Issues

No response

Description

As a dev, I want to package my deployable terraform modules into container images that output their plan, so that I can standardize a CD pipeline around simply running a tagged container image.

Acceptance Criteria

• Container wraps around terraform validate and plan commands
• Output plan to stdout
• Accept a vars file
• Accept a backend config file

Related Issues

No response

Description

As a maintainer, I want to add a codeql config, so that it can be maintained moving forward via commits.

Acceptance Criteria

• codeql.yaml is added

Related Issues

No response

Description

As a dev, I want to implement an API endpoint for fetching a bootstrap image, so that servers can bootstrap into iPXE from PXE.

Acceptance Criteria

• Fetch bootstrap image from object storage
• Validate image from object storage
• Return object signature in Etag header field using the following format "<HASH_ALG>/"
• Leverage "Accept-Encoding" for response body encoding per client

Related Issues

No response

Description

As a maintainer, I want to fix a bug introduced by #104 which moved the template funcs from being built in to a functional option.

Acceptance Criteria

• Template functions work once more

Related Issues

No response

Description

As a machine manager, I want my machines to chainload a machine specific image over the network, so that my machines can be as stateless as possible and replicated physical machines can reuse boot scripts/images.

Acceptance Criteria

• Add /machine/image to machine mgmt service
• Accept query params
  • mac address
• Query machines images based on query params

Related Issues

No response

Description

Build for gcp

Acceptance Criteria

It works

Related Issues

No response

Description

As a dev, I want to deploy a storage bucket, so that my application can use it to store and retrieve pre-built ipxe/pxe images for devices.

Acceptance Criteria

• bucket name can be passed as a variable
• action workflow is used to run terraform
• bucket has default encryption enabled
• object retention is enabled

Related Issues

No response

Description

As a dev ops eng, I want to deploy an external load balancer, so that traffic from the internet can reach the deployed services.

Acceptance Criteria

• Global IP is allocated
• Optional multi-region support

Related Issues

No response

Description

As a maintainer, I want to optimize the CI/CD workflow by skipping building services when only terraform files have been changed.

This would also optimize the package storage capacity.

Acceptance Criteria

• Only build services if Go or bazel related files are touched
• Do not build services if Terraform is touched

Related Issues

No response

Description

As a dev, I want to refactor the repo to not place infra under the terraform folder, so that infra architecture components can be found easier.

Acceptance Criteria

• All terraform sub folders are moved to top of repo
• Modules are refactored to be more gitops friendly

Related Issues

No response

Description

As a maintainer, I want the build workflow to get triggered when dependency files (go.mod, go.sum, MODULE.bazel, etc.) are changed, so that changes made by bots and others are also built and tested automatically.

Acceptance Criteria

• Dependabot changes are ran on build workflow

Related Issues

No response

Description

As a maintainer, I want to rename/migrate the gcp terraform module to one named after the service it deploys, so that the module is more maintainable and easily understandable over time.

Acceptance Criteria

• Rename gcp terraform module to machinemgmt

Related Issues

No response

Description

As a sys admin, I want a Fedora CoreOS config which boots into CoreOS, so that I can quickly stand up new machines.

Acceptance Criteria

• Boots
• SSH
• Base ISO is built with Actions

Related Issues

No response

Description

As a dev ops engineer, I want to fetch ipxe scripts via HTTP from Github releases, so that I can dynamically configure machines.

Acceptance Criteria

• it works

Related Issues

No response

Description

As a dev ops eng, I want to be able to deploy the machine mgmt service to multiple regions, so that it can hopefully be more resilient and fault tolerant.

Acceptance Criteria

• Update location variable to accept list of locations
• Deploy service to all locations provided in variable

Related Issues

No response

Description

As a sys admin, I want to install K8s in Fedora CoreOS, so that devs can deploy containers instead of having to worry about the low level stuff.

Acceptance Criteria

• K8s is installed on initiation

Related Issues

No response

Description

As a maintainer, I want to refactor the existing services to use the latest version of z5labs/app http runtime, so that a lot of the boilerplate code in this repo can be removed.

Acceptance Criteria

• lbsink service is updated
• machinemgmt service is updated
• unused packages are removed

Related Issues

No response

Description

As a maintainer, I want to enable Dependabot, so that my dependencies can be automatically updated in a timely manner.

Acceptance Criteria

• Scan go deps
• Scan github actions deps

Related Issues

No response

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Rate-Limited

These updates are currently rate-limited. Click on a checkbox below to force their creation now.

• chore(deps): update docker/login-action digest to 0d4c9c5
• chore(deps): update golang.org/x/exp digest to 7f521ea
• chore(deps): update google.golang.org/genproto/googleapis/rpc digest to 68d350f
• chore(deps): update dependency aspect_bazel_lib to v2.7.7
• chore(deps): update dependency rules_oci to v1.7.6
• chore(deps): update module github.com/go-logr/logr to v1.4.2
• chore(deps): update module github.com/hashicorp/go-retryablehttp to v0.7.7
• chore(deps): update module github.com/spf13/cobra to v1.8.1
• chore(deps): update module google.golang.org/protobuf to v1.34.2
• fix(deps): update module github.com/fclairamb/ftpserverlib to v0.24.1
• fix(deps): update module github.com/googleapis/gax-go/v2 to v2.12.5
• chore(deps): update dependency bazel to v7.2.0
• chore(deps): update dependency gazelle to v0.37.0
• chore(deps): update dependency rules_go to v0.48.1
• chore(deps): update module cloud.google.com/go to v0.115.0
• chore(deps): update module github.com/googlecloudplatform/opentelemetry-operations-go/detectors/gcp to v1.24.0
• chore(deps): update module github.com/googlecloudplatform/opentelemetry-operations-go/exporter/trace to v1.24.0
• chore(deps): update module github.com/googlecloudplatform/opentelemetry-operations-go/internal/resourcemapping to v0.48.0
• chore(deps): update module github.com/sagikazarmark/locafero to v0.6.0
• chore(deps): update module github.com/spf13/viper to v1.19.0
• chore(deps): update module go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc to v0.52.0
• chore(deps): update module go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp to v0.52.0
• chore(deps): update module go.opentelemetry.io/proto/otlp to v1.3.1
• chore(deps): update module golang.org/x/crypto to v0.24.0
• chore(deps): update module golang.org/x/net to v0.26.0
• chore(deps): update module golang.org/x/oauth2 to v0.21.0
• chore(deps): update module golang.org/x/sys to v0.21.0
• chore(deps): update module golang.org/x/text to v0.16.0
• fix(deps): update module go.opentelemetry.io/contrib/detectors/gcp to v1.27.0
• fix(deps): update opentelemetry-go monorepo to v1.27.0 (go.opentelemetry.io/otel, go.opentelemetry.io/otel/exporters/otlp/otlptrace, go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc, go.opentelemetry.io/otel/exporters/stdout/stdouttrace, go.opentelemetry.io/otel/metric, go.opentelemetry.io/otel/sdk, go.opentelemetry.io/otel/trace)
• 🔐 Create all rate-limited PRs at once 🔐

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• chore(deps): update actions/checkout digest to 692973e
• chore(deps): update github/codeql-action digest to 23acc5c
• chore(deps): update google.golang.org/genproto digest to 68d350f
• chore(deps): update google.golang.org/genproto/googleapis/api digest to 68d350f
• fix(deps): update module github.com/z5labs/bedrock to v0.4.9
• chore(deps): update module cloud.google.com/go/auth to v0.5.1
• chore(deps): update module google.golang.org/api to v0.185.0
• fix(deps): update module cloud.google.com/go/storage to v1.42.0
• chore(deps): update module github.com/hashicorp/hcl to v2
• chore(deps): update module github.com/sony/gobreaker to v2
• Click on this checkbox to rebase all open PRs at once

Detected dependencies

bazel-module

MODULE.bazel

• gazelle 0.36.0
• rules_go 0.47.1
• rules_oci 1.7.5
• rules_pkg 0.10.1
• aspect_bazel_lib 2.7.3

bazelisk

.bazelversion

• bazel 7.1.2

github-actions

.github/workflows/ci.yml

• actions/checkout v4@0ad4b8fadaa221de15dcec353f45205ec38ea70b
• actions/cache v4@0c45773b623bea8c8e75f6c82b208c3cf94ea4f9
• docker/login-action v3@e92390c5fb421da1463c202d546fed0ec5c39f20

.github/workflows/codeql.yaml

• actions/checkout v4@0ad4b8fadaa221de15dcec353f45205ec38ea70b
• actions/setup-go v5@cdcb36043654635271a94b9a6d1392de5bb323a7
• github/codeql-action v3@b7cec7526559c32f1616476ff32d17ba4c59b2d6
• github/codeql-action v3@b7cec7526559c32f1616476ff32d17ba4c59b2d6

.github/workflows/terraform.yml

• actions/checkout v4@0ad4b8fadaa221de15dcec353f45205ec38ea70b
• hashicorp/setup-terraform v3@651471c36a6092792c552e8b1bef71e592b462d8

gomod

go.mod

• go 1.22.2
• cloud.google.com/go/storage v1.40.0
• github.com/fclairamb/ftpserverlib v0.24.0
• github.com/googleapis/gax-go/v2 v2.12.4
• github.com/spf13/afero v1.11.0
• github.com/stretchr/testify v1.9.0
• github.com/z5labs/bedrock v0.4.8
• go.opentelemetry.io/contrib/detectors/gcp v1.26.0
• go.opentelemetry.io/otel v1.26.0
• go.opentelemetry.io/otel/sdk v1.26.0
• go.opentelemetry.io/otel/trace v1.26.0
• golang.org/x/sync v0.7.0
• cloud.google.com/go v0.113.0
• cloud.google.com/go/auth v0.4.1
• cloud.google.com/go/auth/oauth2adapt v0.2.2
• cloud.google.com/go/compute/metadata v0.3.0
• cloud.google.com/go/iam v1.1.8
• cloud.google.com/go/trace v1.10.7
• github.com/GoogleCloudPlatform/opentelemetry-operations-go/detectors/gcp v1.23.0
• github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/trace v1.23.0
• github.com/GoogleCloudPlatform/opentelemetry-operations-go/internal/resourcemapping v0.47.0
• github.com/cenkalti/backoff/v4 v4.3.0
• github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc@d8f796af33cc
• github.com/fclairamb/go-log v0.5.0
• github.com/felixge/httpsnoop v1.0.4
• github.com/fsnotify/fsnotify v1.7.0
• github.com/go-logr/logr v1.4.1
• github.com/go-logr/stdr v1.2.2
• github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da@41bb18bfe9da
• github.com/golang/protobuf v1.5.4
• github.com/google/s2a-go v0.1.7
• github.com/google/uuid v1.6.0
• github.com/googleapis/enterprise-certificate-proxy v0.3.2
• github.com/grpc-ecosystem/grpc-gateway/v2 v2.20.0
• github.com/hashicorp/go-cleanhttp v0.5.2
• github.com/hashicorp/go-retryablehttp v0.7.6
• github.com/hashicorp/hcl v1.0.0
• github.com/inconshreveable/mousetrap v1.1.0
• github.com/magiconair/properties v1.8.7
• github.com/mitchellh/mapstructure v1.5.0
• github.com/pelletier/go-toml/v2 v2.2.2
• github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2@5d4384ee4fb2
• github.com/sagikazarmark/locafero v0.4.0
• github.com/sagikazarmark/slog-shim v0.1.0
• github.com/sony/gobreaker v1.0.0
• github.com/sourcegraph/conc v0.3.0
• github.com/spf13/cast v1.6.0
• github.com/spf13/cobra v1.8.0
• github.com/spf13/pflag v1.0.5
• github.com/spf13/viper v1.18.2
• github.com/subosito/gotenv v1.6.0
• go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.51.0
• go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.51.0
• go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.26.0
• go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.26.0
• go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.26.0
• go.opentelemetry.io/otel/metric v1.26.0
• go.opentelemetry.io/proto/otlp v1.2.0
• go.uber.org/multierr v1.11.0
• golang.org/x/crypto v0.23.0
• golang.org/x/exp v0.0.0-20240506185415-9bf2ced13842@9bf2ced13842
• golang.org/x/net v0.25.0
• golang.org/x/oauth2 v0.20.0
• golang.org/x/sys v0.20.0
• golang.org/x/text v0.15.0
• golang.org/x/time v0.5.0
• google.golang.org/api v0.180.0
• google.golang.org/genproto v0.0.0-20240515191416-fc5f0ca64291@fc5f0ca64291
• google.golang.org/genproto/googleapis/api v0.0.0-20240515191416-fc5f0ca64291@fc5f0ca64291
• google.golang.org/genproto/googleapis/rpc v0.0.0-20240515191416-fc5f0ca64291@fc5f0ca64291
• google.golang.org/grpc v1.64.0
• google.golang.org/protobuf v1.34.1
• gopkg.in/ini.v1 v1.67.0
• gopkg.in/yaml.v3 v3.0.1

terraform

terraform/artifact-registry/main.tf

• google >= 5.6.0

terraform/dns/main.tf

• cloudflare >= 4.0

terraform/load-balancer/main.tf

• cloudflare >= 4.0
• docker >= 3.0.2
• google >= 5.6.0
• google-beta >= 5.6.0
• tls >= 4.0.5

terraform/machine-mgmt-service/main.tf

• docker >= 3.0.2
• google >= 5.6.0
• random >= 3.6.0

terraform/modules/copy_container_image/main.tf

• docker 3.0.2

terraform/modules/gcp/cloud_run/main.tf

• google >= 5.6.0

terraform/modules/gcp/service_account/main.tf

• google >= 5.6.0

---

• Check this box to trigger a request for Renovate to run again on this repository

Description

As a machine manager, I want to implement an iPXE boot script/image for my 1u servers, so that I can easily boot the servers at anytime.

Acceptance Criteria

• Build custom iPXE image with embedded script
• Deploy image to machine image bucket
• Index image based on machine mac address

Related Issues

No response

Description

As a sys admin, I want to deploy an API which can return IPXE bootstrap PXE images, so that I can boot servers using a IPXE bootstrap image which would most likely redirect to another image tailored to the specific machine.

Acceptance Criteria

• GET request
• Endpoint = /machine/ipxe/bootstrap
• Response body is the boot image
• Signature should be included as ETAG response header
• Fetch from Github Release

Related Issues

No response