Giter VIP home page Giter VIP logo

helmper's Introduction

Helmper logo

A little helper that pushes Helm Charts and images to your registries, easily configured with a declarative spec.
Report bug · Request feature · Releases · Latest release

Go Report Card License

What is Helmper?

DISCLAIMER: helmper is in beta, so stuff may change.

Helmper logo

helmper is a go program that reads Helm Charts from remote OCI registries and pushes the charts container images to your registries with optional OS level vulnerability patching.

helmper is built with Helm, Oras, Trivy, Copacetic (Buildkit) and Cosign.

Helmper connects via gRPC to Trivy and Buildkit so you can run helmper without root privileges wherever you want.

helmper demonstrates exceptional proficiency in operating within controlled environments that might require Change Management and/or air-gapped networks. This expertise is particularly beneficial in industries subject to stringent regulations, such as Medical and Banking. helmper aims to ensure binary reproducibility of Helm Charts by storing all necessary artifacts in your registries.

helmper provides an interface to reduce the maintenance burden associated with managing a large collection of Helm Charts by:

  • automatically detecting all enabled container images in charts
  • providing an easy way to stay up to date on new chart releases
  • providing option to only import new images, or all images
  • enabling quick patching(and re-patching) of all images
  • enabling signing of images as an integrated part of the process
  • providing a mechanism to check requirements/dependencies before deploying charts with fx GitOps

how?

Core

Simply tell helmper which charts to analyze and registries to use by creating a helmper.yaml file and run helmper from the same folder.

k8s_version: 1.27.9
import:
  enabled: true
charts:
- name: prometheus
  version: 25.8.0
  valuesFile: /workspace/in/values/prometheus/values.yaml # (Optional)
  repo:
    name: prometheus-community
    url: https://prometheus-community.github.io/helm-charts/
registries:
- name: registry
  url: 0.0.0.0:5000
  insecure: true
  plainHTTP: true

Helmper will import the charts, the charts listed as dependencies including all images specified through the Helm values.yaml file.

Note Authentication

Helmper utilizes well known configuration options to interact with registries.

When using the Helm SDK, Helmper will utilize the file defined by HELM_REGISTRY_CONFIG for picking up authentication credentials for registries.

When Helmper is using Oras for interacting with OCI artifacts, Oras utilizes the Docker credentials helper, which will look in the system keychain, $DOCKER_CONFIG/config.json (if set) or $HOME/.docker/config.json file for picking up authentication credentials for all registries.

If your registries requires authentication, simply login with the services own login command.

fx for Docker:

docker login -u user -p pass

Azure:

az acr login -n myregistry

Extended

In this example Helmper will also scan with Trivy, patch with Copacetic and sign with Cosign all identified images before pushing with Oras to all registries.

k8s_version: 1.27.9
charts:
- name: prometheus
  version: 25.8.0
  valuesFile: /workspace/in/values/prometheus/values.yaml # (Optional)
  repo:
    name: prometheus-community
    url: https://prometheus-community.github.io/helm-charts/
registries:
- name: registry # `Helmper` picks up authentication from the environment automatically.
  url: 0.0.0.0:5000
  insecure: true
  plainHTTP: true
import:
  enabled: true
  copacetic:
    enabled: true
    ignoreErrors: true
    buildkitd:
      addr: tcp://0.0.0.0:8888
    trivy:
      addr: http://0.0.0.0:8887
      insecure: true
      ignoreUnfixed: true
    output:
      tars:
        folder: /workspace/.out/tars
        clean: true
      reports:
        folder: /workspace/.out/reports
        clean: true
  cosign:
    enabled: true
    keyRef: /workspace/.devcontainer/cosign.key
    KeyRefPass: ""
    allowInsecure: true
    allowHTTPRegistry: true

Documentation

The full documentation for Helmper can be found at christoffernissen.github.io/helmper.

Compatibility

Helmper utilizes the Helm SDK to maintain full compatibility with both Helm Repositories and OCI registries for storing Helm Charts.

In practice, Helmper currently pushes charts and images to the same destination registry, so it must be OCI compliant.

Helmper utilizes oras-go to push OCI artifacts. Helmper utilizes the Helm SDK to push Helm Charts, as the Helm SDK sets the correct metadata attributes.

Oras and Helm state support all registries with OCI support, for example:

Sources: Helm Oras

For testing, Helmper is using the CNCF Distribution registry.

Install

Simply pick the binary for your platform from the Release section on GitHub.

Linux

VERSION=$(curl -Lso /dev/null -w %{url_effective} https://github.com/christoffernissen/helmper/releases/latest | grep -o '[^/]*$')
curl -LO https://github.com/christoffernissen/helmper/releases/download/$VERSION/helmper-linux-amd64
chmod +x helmper-linux-amd64
sudo mv helmper-linux-amd64 /usr/local/bin/helmper

Mac OS

VERSION=$(curl -Lso /dev/null -w %{url_effective} https://github.com/christoffernissen/helmper/releases/latest | grep -o '[^/]*$')
curl -LO https://github.com/christoffernissen/helmper/releases/download/$VERSION/helmper-darwin-amd64
chmod +x helmper-darwin-amd64
sudo mv helmper-darwin-amd64 /usr/local/bin/helmper

Windows

Extract the tar and launch the exe file.

Scope

In scope

  • Helmper operates with OCI compliant artifacts and OCI compliant registries.
  • Helmper must remain without dependency on a container runtime daemon to work in containers without root privileges.

Out of scope

  • Helmper does not work with other Kubernetes package formats
  • Helmper authenticates with registries with the docker config. Therefore, Helmper will not have any proprietary libraries to facilitate authentication for any cloud providers. Simply use docker login or equivalent before running Helmper, and you should be authenticated for 3 hours for each registry.

Roadmap

  • Operator Framework to enable using Helmper with GitOps in management clusters
  • Add option to import to registries via pipeline for compliance audit trail retention
  • SBOM
  • OpenTelemetry

Code of Conduct

This project has adopted the CNCF Code of Conduct. See CODE_OF_CONDUCT.md for further details.

Credits

Helmper logo and banner have been kindly donated to the project by María Ruiz Garrido ❤️

The gopher's logo of Helmper is licensed under the Creative Commons 3.0 Attributions license.

The original Go gopher was designed by Renee French.

helmper's People

Contributors

christoffernissen avatar dependabot[bot] avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

helmper's Issues

Feature: docker image diff

a bit of a braindump here - to be restructured :)

It would be really cool if helmper could facilitate showing a diff (output from 'diff -bduNr' current-docker/ new-docker/ - and just export the current and "to be updated to" docker image - so one can see what actual changes are being introduced - and maybe spot "nefarious changes" if they happen in your supply chain.
Same with helm charts.

Often supply chain attacks are not at the code - but at the "mirrors"/repos of libs and dependencies
https://arstechnica.com/information-technology/2020/04/725-bitcoin-stealing-apps-snuck-into-ruby-repository/

Helmper should help identify "unexpected changes".. generally. f.ex. not enough use the sha256 of a docker image - so you could change what a tag points to (simply push to same tag again - as supply chain attack) - and most would download the newimage - as they were not actually pointing to the sha256 of the releasetag - but just the tag.

Feature request: consider sub commands like export, import, and copy

I was looking for a tool that could help me easily analyze, export all artifacts from a helm, together with all images used, and then import to a desired (private) registry. This is a very common use case when we work with airgapped env.

Unfortunately, I couldn't find one that could best fit into such simple use case.
So I built an incomplete tool for fun: https://github.com/brightzheng100/helm-packager, which is a toy tool and is far away from complete.

I think your tool is a good starting point and I'd suggest to have sub commands, instead of a giant command with configuration file.

For example:

# 1. To export the Helm Charts and their sub Charts to be a local gz file
helmper export [-c <config file>] [-f <exported gz file>] [--dry-run]

# 2. To import the exported gz file to a registry
helmper import [-c <config file>] [-f <exported gz file>] [--dry-run]

# 3. Copy the Helm Charts and their sub Charts to the desired registry -- this is what current helmper does
# But with the --dry-run flag, we can analyze the Helm Charts and their sub Charts without coping the actual artifacts 
helmper copy [-c <config file>] [--dry-run]

Feature: Retrying failed connections

Hello!

While running helmper it tries to communicate with various sources, services and registries. Sometimes they don't respond in a timely fashion, thus rendering a context deadline exceeded error and exiting abruptly halfway through the execution. See example here:

{"time":"2024-05-29T12:56:01.142365493Z","level":"ERROR","msg":"signing digest: Head \"https://xyz.azurecr.io/v2/openebs/provisioner-nfs/manifests/sha256-f8a913f70026e86dae967050b6cd3790b81dd7c9c580aab306c3a0f36ca49b18.sig\": context deadline exceeded"}

It would be very useful to be able to adjust the timeouts through configuration, but also that helmper uses an exponential backoff mechanism for retrying the failed connection.

Feature: Implement exit codes

Hello 👋

When running helmper as part of a CI pipeline it would be beneficial that helmper returns 0 when it runs without any errors and other exit codes if there's errors. This way it will be automatically detected by the CI pipeline and reported as such.

ECR not working

When using an Amazon Container Registry (ECR) it doesn't work and helmper exits with the following error:

{"time":"2024-05-14T11:48:28.747110569Z","level":"ERROR","msg":"unexpected status from HEAD request to https://number.dkr.ecr.eu-north-1.amazonaws.com/v2/charts/openebs/blobs/sha256:0139e8a13fb95b14f23f49cf437c3027f749b7e1e0d451f66e041c92ea45586c: 403 Forbidden"}

I suspect it's connected with this bug report: helm/helm#12491 which was reverted here though: helm/helm#12527

There's no problem when I try helmper towards an Azure Container Registry (ACR).

No good way to verify whether the command was successful

Follow the Getting Started guide and ran the command.

It was wrong as the registry was not actually listening on port 5000 -- there was the airplay receiver in the MacBook which was listening on this port and I had to disable it and reran the registry to ensure it worked.

Then I tried it with this output:

$ helmper

 _         _
| |_  ___ | | _____  ___  ___  ___
|   || -_|| ||     || . || -_||  _|
|_|_||___||_||_|_|_||  _||___||_|
                    |_|
version 0.1.3 (commit 7a22623ca5f11659d6ead6e91dedd48b1965e01f, built at 2024-05-29T20:21:22Z)

{"time":"2024-05-30T17:03:45.044219+08:00","level":"INFO","msg":"Updated all Helm repositories ✅"}
+---+----------+------------+---------+----------------+--------+---------+--------------------------+---------+----------------------------------+---------+
| # | TYPE     | CHART      | VERSION | LATEST VERSION | LATEST | VALUES  | SUBCHART                 | VERSION | CONDITION                        | ENABLED |
+---+----------+------------+---------+----------------+--------+---------+--------------------------+---------+----------------------------------+---------+
| 0 | Chart    | prometheus | 25.8.0  | 25.21.0        || default |                          |         |                                  |         |
| 1 | Subchart |            |         |                |        | parent  | alertmanager             | 1.7.*   | alertmanager.enabled             ||
| 2 | Subchart |            |         |                |        | parent  | kube-state-metrics       | 5.15.*  | kube-state-metrics.enabled       ||
| 3 | Subchart |            |         |                |        | parent  | prometheus-node-exporter | 4.24.*  | prometheus-node-exporter.enabled ||
| 4 | Subchart |            |         |                |        | parent  | prometheus-pushgateway   | 2.4.*   | prometheus-pushgateway.enabled   ||
+---+----------+------------+---------+----------------+--------+---------+--------------------------+---------+----------------------------------+---------+
 100% [===============] (5/5) Parsing charts...
+---+--------------------------+---------------+---------------------------------------------+----------------------------------------------------------------+
| # | HELM CHART               | CHART VERSION | HELM VALUE PATH                             | IMAGE                                                          |
+---+--------------------------+---------------+---------------------------------------------+----------------------------------------------------------------+
| 0 | prometheus               | 25.8.0        | server.image.digest                         | quay.io/prometheus/prometheus:v2.48.0                          |
|   |                          |               | server.image.repository                     |                                                                |
|   |                          |               | server.image.tag                            |                                                                |
| 1 | prometheus               | 25.8.0        | configmapReload.prometheus.image.digest     | quay.io/prometheus-operator/prometheus-config-reloader:v0.67.0 |
|   |                          |               | configmapReload.prometheus.image.repository |                                                                |
|   |                          |               | configmapReload.prometheus.image.tag        |                                                                |
| 2 | alertmanager             | 1.7.*         | image.repository                            | quay.io/prometheus/alertmanager:v0.26.0                        |
|   |                          |               | image.tag                                   |                                                                |
| 3 | kube-state-metrics       | 5.15.*        | image.repository                            | registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.10.1  |
|   |                          |               | image.sha                                   |                                                                |
|   |                          |               | image.tag                                   |                                                                |
|   |                          |               | image.registry                              |                                                                |
| 4 | prometheus-node-exporter | 4.24.*        | image.digest                                | quay.io/prometheus/node-exporter:v1.7.0                        |
|   |                          |               | image.registry                              |                                                                |
|   |                          |               | image.repository                            |                                                                |
|   |                          |               | image.tag                                   |                                                                |
| 5 | prometheus-pushgateway   | 2.4.*         | image.repository                            | quay.io/prometheus/pushgateway:v1.6.2                          |
|   |                          |               | image.tag                                   |                                                                |
+---+--------------------------+---------------+---------------------------------------------+----------------------------------------------------------------+
+---+--------------------------+---------------+----------------------------------------------------------------+----------+--------+
| # | HELM CHART               | CHART VERSION | IMAGE                                                          | REGISTRY | IMPORT |
+---+--------------------------+---------------+----------------------------------------------------------------+----------+--------+
| 0 | prometheus               | 25.8.0        | quay.io/prometheus/prometheus:v2.48.0                          |||
| 1 | prometheus               | 25.8.0        | quay.io/prometheus-operator/prometheus-config-reloader:v0.67.0 |||
| 2 | alertmanager             | 1.7.*         | quay.io/prometheus/alertmanager:v0.26.0                        |||
| 3 | kube-state-metrics       | 5.15.*        | registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.10.1  |||
| 4 | prometheus-node-exporter | 4.24.*        | quay.io/prometheus/node-exporter:v1.7.0                        |||
| 5 | prometheus-pushgateway   | 2.4.*         | quay.io/prometheus/pushgateway:v1.6.2                          |||
+---+--------------------------+---------------+----------------------------------------------------------------+----------+--------+
|   |                          |               |                                                                |          | 6      |
+---+--------------------------+---------------+----------------------------------------------------------------+----------+--------+
 100% [===============] (5/5) Pushing charts...
{"time":"2024-05-30T17:11:37.045958+08:00","level":"ERROR","msg":"Put \"http://0.0.0.0:5000/v2/prometheus-operator/prometheus-config-reloader/blobs/uploads/f88757cb-bd66-47c6-8732-8b2ed79f8b0a?_state=sSf7oEYh86CKtqXghVxPogjP1jOsNI0HcYLeyci900t7Ik5hbWUiOiJwcm9tZXRoZXVzLW9wZXJhdG9yL3Byb21ldGhldXMtY29uZmlnLXJlbG9hZGVyIiwiVVVJRCI6ImY4ODc1N2NiLWJkNjYtNDdjNi04NzMyLThiMmVkNzlmOGIwYSIsIk9mZnNldCI6MCwiU3RhcnRlZEF0IjoiMjAyNC0wNS0zMFQwOToxMToxOC4wMjM5NDAyNDZaIn0%3D&digest=sha256%3Aef1ccf1c191dcea79cfb35bfab927099c600056ce1a0ce7675dc5d114ae258f6\": readfrom tcp 127.0.0.1:63733->127.0.0.1:5000: stream error: stream ID 65; INTERNAL_ERROR; received from peer"}

See there was an "ERROR" message, and then I tried to list down the entries:

$ curl http://0.0.0.0:5000/v2/_catalog
{"repositories":["busybox","charts/alertmanager","charts/kube-state-metrics","charts/prometheus","charts/prometheus-node-exporter","charts/prometheus-pushgateway","kube-state-metrics/kube-state-metrics","prometheus/alertmanager","prometheus/node-exporter","prometheus/prometheus","prometheus/pushgateway","prometheus-operator/prometheus-config-reloader"]}

So I guess the charts and images had been successfully imported.

But it feels hard to judge whether the command was successful if purely looking at the command output.

Support authentication from helm config.json

In the helm environment variable HELM_REGISTRY_CONFIG is the full path to the configuration file that helm uses for authentication for registries.

As helmper supports ~/.docker/config.json it would also be nice if it supported the authentication from helm.

Feature: Accept environment variable for cosign key

Hello 👋

As I'm implementing helmper into a CI pipeline I realize it would be nice to not have the password for the cosign.key in clear text in the helmper.yaml file. Could an environment variable be supported as well?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.