yannh / kubeconform Goto Github PK

The current release has a binary for Darwin-386. MacOs has never supported 386 CPUs so this is not necessary.

This can be fixed by added the following to the .goreleaser.yml file

    ignore:
      - goos: darwin
        goarch: 386

Hello!

Please add caching resource schemas to disk. It's typical for such an instrument to be run between different projects and downloading the same schemas every time affects performance. It also become very slow on bad internet connection

I was investigating using this instead of kubeval for a project. Scanning multi-document files I end up with errors that are not present if we split up the file into single document files.

eg:

dev.yaml - Deployment ***** failed validation: error unmarshalling resource: error converting YAML to JSON: yaml: did not find expected key
dev.yaml - Deployment ***** is invalid: kind must be one of the following: "Deployment"
dev.yaml - HorizontalPodAutoscaler ***** failed validation: error unmarshalling resource: error converting YAML to JSON: yaml: line 9: did not find expected '-' indicator
dev.yaml - Deployment ***** failed validation: error unmarshalling resource: error converting YAML to JSON: yaml: line 4: did not find expected key
dev.yaml - Deployment ***** is invalid: apiVersion must be one of the following: "apps/v1" - kind must be one of the following: "Deployment" - selector is required
dev.yaml - failed validation: error while parsing: error unmarshaling JSON: while decoding JSON: json: cannot unmarshal string into Go value of type struct { APIVersion string "yaml:\"apiVersion\""; Kind string "yaml:\"kind\""; Metadata struct { Name string "yaml:\"name\""; Namespace string "yaml:\"namespace\""; GenerateName string "yaml:\"generateName\"" } "yaml:\"Metadata\"" }
dev.yaml - StatefulSet ***** is invalid: apiVersion must be one of the following: "apps/v1" - kind must be one of the following: "StatefulSet" - selector is required - template is required - serviceName is required
dev.yaml - Job ***** is invalid: apiVersion must be one of the following: "batch/v1" - kind must be one of the following: "Job" - template is required
dev.yaml - Job ***** failed validation: error unmarshalling resource: error unmarshaling JSON: while decoding JSON: json: cannot unmarshal string into Go value of type map[string]interface {}

First, thank you for this great project! It is so great to get k8s resources validations fast and with a short output.

On 0.4.3, kubeconform -h worked just fine. Now on 0.4.5, one has to supply a file:

On 0.4.3, process exit with code 0:

$ /kubeconform -h
Usage: kubeconform [OPTION]... [FILE OR FOLDER]...
  -cache string
        cache schemas downloaded via HTTP to this folder
[...]

On 0.4.5, it fails with exit code 1:

$ kubeconform -h
Usage: kubeconform [OPTION]... [FILE OR FOLDER]...
  -cache string
[...]
2021/03/08 15:29:13 failing to read data from stdin

So far, I found a workaround by using kubeconform -h /dev/null.

The context on why using kubconform -h is to install in a docker image and check that the binary executes properly:

RUN set -ex \
    && curl -fsSLO --compressed https://github.com/yannh/kubeconform/releases/download/v0.4.5/kubeconform-linux-amd64.tar.gz \
    && echo "f849385451ec54529a1dc90c4b87969760dd50e74e09c55e126278c9b6c7fccb kubeconform-linux-amd64.tar.gz" | sha256sum -c - \
    && tar -xzf "kubeconform-linux-amd64.tar.gz" -C /usr/local/bin \
    && kubeconform -h /dev/null \
    && rm "kubeconform-linux-amd64.tar.gz"

Hi, I was trying to use kubeconform to validate some resources that are supported and some that would be CustomResources. And everything works perfectly when i pass -schema-location default and -schema-location <pathToJson1>. But when i try to do something like this -schema-location default -schema-location <pathToJson1> -schema-location <pathToJson2> it only looks at first schema passed and when it comes to some yaml that 2nd schema describes i get message "It has to be of kind ". Am I maybe doing something wrong with passing flags, or is this kind of behaviour just not supported?

Tried to do validation against K8s 1.19, which as per my other issue, isn't in the default schema repo. Instead of throwing an error, this seemed to just silently fail and skip validation of all resources.

Steps to reproduce:

$ kubeconform -ignore-missing-schemas -kubernetes-version 1.19.0 -summary
Summary: 964 resources found in 450 files - Valid: 0, Invalid: 0, Errors: 0, Skipped: 964

Expected result:
Some sort of error because the 1.19 schemas don't exist.

Presumably this is because I'm passing -ignore-missing-schemas? However if I don't pass -ignore-missing-schemas, I just get Summary: 964 resources found in 450 files - Valid: 0, Invalid: 0, Errors: 964, Skipped: 0.

Validating Lists does not seem to be supported:

stdin - failed validation: could not find schema for List

kubeconform should recognize the kind and validate items individually.

We mostly use it to validate Jsonnet deployments, this format allows the output to be directly passed to kubectl apply -f - (since JSON does not have an equivalent to YAML document streams). An example out there if that can clarify:
https://github.com/brancz/kubernetes-grafana/blob/master/examples/basic.jsonnet

This is supported in kubeval 0.15.0+. Here is the initial implementation and follow up fixes if that helps:
instrumenta/kubeval#221
instrumenta/kubeval#283
instrumenta/kubeval#284

From a quick look at the code, it seems we can only identify Kind: List from this line:

We would probably need to unmarshall before ValidateResource(), if we want to benefit of the parallel workers:

Testing out the junit xml output format, I see we have empty testcases added to the report where kubeconform encounters a yaml document with no content, including where its just a comment - for example;

# https://raw.githubusercontent.com/vmware-tanzu/helm-charts/velero-2.14.7/charts/velero/crds/restores.yaml
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  labels:
    app.kubernetes.io/name: velero
  annotations:
    controller-gen.kubebuilder.io/version: v0.3.0
  name: restores.velero.io

kubeconform -strict -output junit velero/crds/restores.yaml

produces a testsuite with 2 testcases;

<testsuites name="kubeconform" time="0.3551641" tests="1" failures="0" disabled="0" errors="1">
  <testsuite name="velero/crds/restores.yaml" id="1" tests="2" failures="0" errors="1" disabled="0" skipped="0">
    <properties></properties>
    <testcase name="" classname="@"></testcase>
    <testcase name="restores.velero.io" classname="[email protected]/v1beta1">
      <error message="could not find schema for CustomResourceDefinition" type=""></error>
    </testcase>
  </testsuite>
</testsuites>

Our CI/CD-pipeline uses kubeconform to validate Kubernetes resources and today some builds failed due to a temporary outage in GitHub's CDN. To avoid this in the future we downloaded the schemas that we need and point kubeconform to them locally. However, it seems kubeconform is not then able to resolve which schemas to apply to which resources.

For example, I have a directory called schemas, which contains the files service.json and deployment-apps-v1.json, and a directory called resources, which contains the files service.yaml and deployment.yaml.

Now, when I run

kubeconform \
  -kubernetes-version 1.19.8 \
  -schema-location ./schemas/service.json \
  -schema-location ./schemas/deployment-apps-v1.json \
  -strict \
  ./resources/

I'm given the following error:

resources/deployment.yaml - Deployment foobar is invalid: For field spec: Additional property replicas is not allowed - For field spec: Additional property template is not allowed - For field spec.selector.matchLabels: Invalid type. Expected: [string,null], given: object

It seems like kubeconform is applying the validation schema ./schemas/service.json on ./resources/deployment.yaml.

My current solution, which I am not happy with, is to execute kubeconform once per schema and Kubernetes resource. I would prefer if kubeconform understood, when using local schemas, that it should only apply schemas to their corresponding resources.

Also would be great if -schema-location could accept a directory of schemas.

I'm running version v0.4.7.

I am trying to validate Argo-Workflows with kubeconform. Unfortunately no matter what I always get an error because kubeconform could not find schema.

I specifically try to validate the CronWorkflow and WorkflowTemplate in version v1alpha1. They have an openapi definition which I parsed with openapi2jsonschema

openapi2jsonschema https://raw.githubusercontent.com/argoproj/argo-workflows/master/api/openapi-spec/swagger.json 

This creates files schemas/cronworkflow.json and schemas/workflowtemplate.json with the correct schemas.

From the documentation I inferred that it should just work by specifying the format with a Go templated string that indicates how to search for JSON schemas

❯ kubeconform -schema-location default -schema-location 'schemas/{{ .ResourceKind }}.json' -summary --verbose -output json resources/
{
  "resources": [
    {
      "filename": "stdin",
      "kind": "WorkflowTemplate",
      "name": "redacted",
      "version": "argoproj.io/v1alpha1",
      "status": "statusError",
      "msg": "could not find schema for WorkflowTemplate"
    },
    {
      "filename": "stdin",
      "kind": "CronWorkflow",
      "name": "redacted",
      "version": "argoproj.io/v1alpha1",
      "status": "statusError",
      "msg": "could not find schema for CronWorkflow"
    }
  ],
  "summary": {
    "valid": 0,
    "invalid": 0,
    "errors": 2,
    "skipped": 0
  }
}

Without the verbosity it only says

could not find schema for WorkflowTemplate
could not find schema for CronWorkflow

I actually managed to validate my files with kubeval by renaming them to schemas/master-standalone/cronworkflow-argoproj-v1alpha1.json and schemas/master-standalone/workflowtemplate-argoproj-v1alpha1.json` and running with

kubeval --additional-schema-locations file://./schemas

So I am very certain, that the content of the files themselves is fine.

However the renaming is not very maintainable and I would like to use kubeconform here. Curiously I still get the same error when I shun the templated string and instead use the naming scheme that worked in kubeval,

I also read all the help pages but can not find an option that further increases the verbosity level. Thus I did not manage to analyze the error deeper than this. I essentially checked all thinkable combinations of ResourceKind and KindSuffix but to no avail.

It is also not an option to extract the apiVersion as well because I get an error

❯ openapi2jsonschema https://raw.githubusercontent.com/argoproj/argo-workflows/master/api/openapi-spec/swagger.json  --kubernetes
Downloading schema
Parsing schema
Generating shared definitions
Generating individual schemas
Traceback (most recent call last):
  File "/home/maow/.asdf/installs/python/3.9.10/bin/openapi2jsonschema", line 8, in <module>
    sys.exit(default())
  File "/home/maow/.local/lib/python3.9/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/home/maow/.local/lib/python3.9/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/home/maow/.local/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/maow/.local/lib/python3.9/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/home/maow/.asdf/installs/python/3.9.10/lib/python3.9/site-packages/openapi2jsonschema/command.py", line 132, in default
    group = title.split(".")[-3].lower()
IndexError: list index out of range

Presumably this is cause by an incomplete naming in the openapi specification. This is something that I encountered for other projects as well and I am not sure who's fault this is. I assume the standard is either not restrictive enough not being followed consequently.

I would be really thankful for suggestions how to specify the correct schema location, modify the openapi extraction or even just make kubeconform print the path it searched in so that I can out what is not working here.

Hi,

Thanks for the great work!

I might be wrong but I didn't see any shortened args in the help docs. (e.g. -kubernetes-version -> -v, -verbose -> -V)

Do you think it is worth being added?

Hello! I've experimenting with the tool and I'm still stuck on how to name the .json files using the suggested format.
Context: I have a manifest that contains the following CRDs:

• security group policy
• virtual service
• external secret

I correctly converted the files using the provided script. The files are externalsecret_v1.json, securitygrouppolicy_v1beta1.json, virtualservice_v1alpha3.json and virtualservice_v1beta1.json
I run kubeconform as follows:

kustomize build kubernetes/overlays/dev | kubeconform -schema-location default -schema-location ./Kubernetes/schemas/externalsecret_v1.json -summary -schema-location ./Kubernetes/schemas/virtualservice_v1alpha3.json -schema-location ./Kubernetes/schemas/externalsecret_v1.json

and I get the next result:

stdin - VirtualService transaction-orchestrator is invalid: For field spec: Must validate at least one schema (anyOf) - For field spec: data is required - For field spec: Must validate one and only one schema (oneOf) - For field spec: Additional property gateways is not allowed - For field spec: Additional property hosts is not allowed - For field spec: Additional property http is not allowed
stdin - SecurityGroupPolicy transaction-orchestrator is invalid: For field spec: Must validate at least one schema (anyOf) - For field spec: data is required - For field spec: Must validate one and only one schema (oneOf) - For field spec: Additional property securityGroups is not allowed - For field spec: Additional property podSelector is not allowed
Summary: 7 resources found parsing stdin - Valid: 5, Invalid: 2, Errors: 0, Skipped: 0

It seems it's applying the external secrets schema to virtual service and security group policy.
Could you help me on this and clarify what's the expected name of the json files in my case?
Thanks!

Hello, I'm using kubeconform to validate a CronJob, but kubeconform doesn't seem to have the schema in the default location. Is it planed to add it ? Where can I find it so I can define it myself?
Thanks !

Hello,

I would like to suggest a feature to deal with Group on variable allowed for schema-location option.

When I convert all my CRDs on json schema using openapi2jsonschema i'm using this filename format FILENAME_FORMAT='{kind}-{group}-{version}' because I need to distinguish CRDs with same name but from different group.

An exemple :
With flux we have a CRD named Buckets : buckets.source.toolkit.fluxcd.io
With crossplane-aws we have also a CRD named Buckets : buckets.s3.aws.crossplane.io
If we don't distinguish them using the group we have conflicts between those two CRD.

But using -schema-location I can't set this group variable I would love to have :

kubeconform -verbose -summary -strict -schema-location default -schema-location 'https://vibe-k8s-json-schema.s3.eu-west-1.amazonaws.com/crds-json-schema/master-standalone-strict/{{ .ResourceKind }}-{{ .Group }}-{{ .ResourceAPIVersion }}.json'

Indeed today we have :

NormalizedKubernetesVersion - Kubernetes Version, prefixed by v
StrictSuffix - "-strict" or "" depending on whether validation is running in strict mode or not
ResourceKind - Kind of the Kubernetes Resource
ResourceAPIVersion - Version of API used for the resource - "v1" in "apiVersion: monitoring.coreos.com/v1"
KindSuffix - suffix computed from apiVersion - for compatibility with Kubeval schema registries

It would be nice to have the Group as a variable to help distinguish same resource Kind for different group. Exactly as we can do with openapi2jsonschema

WDYT ?

For validating CRDs I have been trying to set up schemas locally in a directory.
My setup:

1. kustomize fetch openapi > openapi.json
2. From openapi.json, I have extracted value corresponding to com.coreos.monitoring.v1.ServiceMonitor and stored it in /tmp/schemas/servicemonitor.json
3. cat sm.yaml | kubeconform --strict --kubernetes-version 1.19.10 -schema-location '/tmp/schemas/{{ .ResourceKind }}.json'

Output: stdin - ServiceMonitor sm failed validation: Object has no key 'definitions'

Data:

$ cat sm.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: sm
  namespace: observability
  labels:
    app: sm
spec:
  selector:
    matchLabels:
      app: sm
  namespaceSelector:
    any: true
  endpoints:
  - port: http
    path: /actuator/prometheus

Schema file - servicemonitor.json.txt (Uploaded as txt but used as JSON due to GitHub constraints ).

Been stuck here for a long time, Any help would be much appreciated.

Say you have the combination of the following:

apiVersion: monitoring.gke.io/v1alpha1
kind: PodMonitor

and

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor

Currently these are looked for at podmonitor-monitoring-v1alpha1.json and podmonitor-monitoring-v1.json respectively.

If the former were to be promoted to v1, they would become indistinguishable from each-other.

Probably the easiest way to solve this is to offer the complete apiVersion field as template variable as well? Though this would also require a respective change in openapi2jsonschema.

Greetings!

Love this project, been adding it to our build out of kubernetes manifest.

It seems like this tool is not finding any DNS-1035 as defined here [a-z]([-a-z0-9]*[a-z0-9] validation errors.

Example invalid manifest

apiVersion: v1
kind: Service
metadata:
  name: foo_bar
  labels:
    app: foo
spec:
  ports:
  - port: 10000
    targetPort: 6379
  selector:
    app: foo

While it can't apply to the cluster (currently 1.19 for this example)

gerald@gerald-laptop:~/playground/manifests$ kubectl apply -f service.yml 
The Service "foo_bar" is invalid: metadata.name: Invalid value: "foo_bar": a DNS-1035 label must consist of lower case alphanumeric characters or '-', start with an alphabetic character, and end with an alphanumeric character (e.g. 'my-name',  or 'abc-123', regex used for validation is '[a-z]([-a-z0-9]*[a-z0-9])?')
gerald@gerald-laptop:~/playground/manifests$ ~/Downloads/kubeconform -summary -strict -output json -verbose service.yml 
{
  "resources": [
    {
      "filename": "service.yml",
      "kind": "Service",
      "name": "foo_bar",
      "version": "v1",
      "status": "statusValid",
      "msg": ""
    }
  ],
  "summary": {
    "valid": 1,
    "invalid": 0,
    "errors": 0,
    "skipped": 0
  }

I tried to integrate kubeconform into the CI workflow of an internal project. On the intial run

helm template . | kubeconform

it had trouble finding the schemas for

• ExternalSecret
• AlertmanagerConfig
• PrometheusRule
• ServiceMonitors

I followed the advice in #51 (comment) to generate a schema using openapi2jsonschema.py from the OpenAPI-Specs provided for AlertmanagerConfig, PrometheusRule and ServiceMonitors.

It worked for ServiceMonitors and PrometheusRule but failed for AlertmanagerConfig with the following error:

convert-crd_1  | Traceback (most recent call last):
convert-crd_1  |   File "/apps/convert/openapi2jsonschema.py", line 123, in <module>
convert-crd_1  |     for y in yaml.load_all(f, Loader=yaml.SafeLoader):
convert-crd_1  |   File "/usr/local/lib/python3.8/site-packages/yaml/__init__.py", line 93, in load_all
convert-crd_1  |     yield loader.get_data()
convert-crd_1  |   File "/usr/local/lib/python3.8/site-packages/yaml/constructor.py", line 45, in get_data
convert-crd_1  |     return self.construct_document(self.get_node())
convert-crd_1  |   File "/usr/local/lib/python3.8/site-packages/yaml/constructor.py", line 60, in construct_document
convert-crd_1  |     for dummy in generator:
convert-crd_1  |   File "/usr/local/lib/python3.8/site-packages/yaml/constructor.py", line 408, in construct_yaml_seq
convert-crd_1  |     data.extend(self.construct_sequence(node))
convert-crd_1  |   File "/usr/local/lib/python3.8/site-packages/yaml/constructor.py", line 129, in construct_sequence
convert-crd_1  |     return [self.construct_object(child, deep=deep)
convert-crd_1  |   File "/usr/local/lib/python3.8/site-packages/yaml/constructor.py", line 129, in <listcomp>
convert-crd_1  |     return [self.construct_object(child, deep=deep)
convert-crd_1  |   File "/usr/local/lib/python3.8/site-packages/yaml/constructor.py", line 100, in construct_object
convert-crd_1  |     data = constructor(self, node)
convert-crd_1  |   File "/usr/local/lib/python3.8/site-packages/yaml/constructor.py", line 427, in construct_undefined
convert-crd_1  |     raise ConstructorError(None, None,
convert-crd_1  | yaml.constructor.ConstructorError: could not determine a constructor for the tag 'tag:yaml.org,2002:value'
convert-crd_1  |   in "<file>", line 3916, column 29

https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/main/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagerconfigs.yaml
https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/main/example/prometheus-operator-crd/monitoring.coreos.com_prometheusrules.yaml
https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/main/example/prometheus-operator-crd/monitoring.coreos.com_servicemonitors.yaml

Opencontainers labels make it easy for various tools to extract information about the image:

LABEL org.opencontainers.image.authors="[email protected]" \
      org.opencontainers.image.source="https://github.com/yannh/kubeconform/" \
      org.opencontainers.image.description="A Kubernetes manifests validation tool" \
      org.opencontainers.image.documentation="https://github.com/yannh/kubeconform/" \
      org.opencontainers.image.licenses="Apache License 2.0" \
      org.opencontainers.image.title="kubeconform" \
      org.opencontainers.image.url="https://github.com/yannh/kubeconform/" \
      org.opencontainers.image.vendor="Some inc."

e.g. dependabot or renovate-bot extract the release notes from the URL provided in image.source.

What do you think about adding this to kubeconform Dockerfiles?

I saw currently kubeconform provides a way to use in GitHub Actions.

I am wondering is there a Docker way to use so that I can use in other CI?

For example, kubeval provides a Docker way to use at the bottom of https://www.kubeval.com/installation/
So when I want to analyze all files under k8s folder, I can do

docker run --rm --volume `pwd`/k8s:/k8s garethr/kubeval k8s/*

Thanks!

I really like kubeconform and now with it, I'm able to validate CRDs very nicely.

I'm heavily relying on gitops/flux for managing my clusters and it's really nice to be able to do a lot of the validation locally with a pre-commit hook :)

I'm using flux2 and this heavily relies on a special CRD that looks like this

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: foo
  namespace: default
spec:
  interval: 1h
  releaseName: foo
  chart:
    spec:
      chart: foo
      version: 5.0.1
  values:
    ingress:
      main:
        enabled: true
        hosts:
          - host: foo.example.tld
            paths:
              - path: /

Now the CRD has a nice schema for everything except the values block (as this is naturally depending on the chart).

helm3 supports Values.Schema.Json for a helm chart, so I'm wondering if there could be a way to make actually use of this for validating the values section. I'm aware that this might be a very special use-case but I think for other users of flux (or similar systems that depend on a CRD for managing helm releases) this could be useful.

Any idea if kubeconform could be extended to support this usecase? Or would there be better approach to do this?

Kubeconform does not give an error when fed a invalid namespace as below

apiVersion: v1
kind: Namespace
metadata:
  name: .hello

❯ kubeconform -verbose -strict namespace.yaml
namespace.yaml - Namespace .hello is valid

Version: 0.4.12

This is same as in KubeVal instrumenta/kubeval#321

I took this YAML example for creating a CRD from the official K8s doc:

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: crontabs.stable.example.com
spec:
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                cronSpec:
                  type: string
                image:
                  type: string
                replicas:
                  type: integer
  scope: Namespaced
  names:
    plural: crontabs
    singular: crontab
    kind: CronTab
    shortNames:
    - ct

When validating this YAML with kubeconform I'm getting the following error:

...failed validation: could not find schema for CustomResourceDefinition

Although the schema exists on the kubernetes-json-schema repo:
https://github.com/yannh/kubernetes-json-schema/blob/master/v1.18.0/customresourcedefinition-apiextensions-v1.json

Copying the schema from the repo and passing it with the -schema-location flag works fine, so I guess the issue is when trying to parse the kind type...

Hi, thanks for the project and for picking up the work of maintaining the schema updates for newer versions of Kubernetes.

It would be nice if you published a GitHub Action to make it easier to use in GitHub's CI system. If you publish Docker images for each release, it should be pretty straightforward to set up.

This is happening when using the -kubernetes-version 1.19.8 flag on kubeconform. Looking through the docs this appears to be a problem because the ConfigMap and VirtualService ( and possibly a bunch more ) resources no longer contain a file named "{{ .ResourceKind }}-v1.json"

https://github.com/yannh/kubernetes-json-schema/tree/master/v1.19.8-standalone-strict

https://github.com/yannh/kubernetes-json-schema/tree/master/v1.18.0-standalone-strict

Here's an example of an error output from docker-compose:

helm-kubeconform_1  | /tmp/helm/test/dev/default/helm/cd950a24.compiled.yml - ConfigMap ihca-carrier-profile-history-es-materializer-configmap-env failed validation: could not find schema for ConfigMap

Here's the example 'compiled' helm chart

# Source: Helm Chart/templates/configmap-env.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: myapp-configmap-env
data:
  ENV1: a_value
  ENV2: b_value
  ENV3: c_value

Here's the command

kubeconform -strict -summary -kubernetes-version 1.19.8 /tmp/helm/test/dev/default/helm/cd950a24.compiled.yml

Note:

• This is probably an issue for more than Just 1.19.8
• Validation is successful without the -kubernetes-version 1.19.8 flag, because it defaults to 1.18.0.
• This helm chart (which contains other resources such as Deployment) is working in k8s 1.19.8.
• Kubeconform will pass, but this is the workaround by removing "{{ .KindSuffix }}" from the location:

  kubeconform -strict -summary -kubernetes-version 1.19.8 \
  -schema-location default \
  -schema-location "https://raw.githubusercontent.com/yannh/kubernetes-json-schema/master/{{ .NormalizedKubernetesVersion }}-standalone{{ .StrictSuffix }}/{{ .ResourceKind }}.json" \
  /tmp/helm/test/dev/default/helm/cd950a24.compiled.yml
  

New Macbooks are based on Apple's M1 ARM CPU. Kubeconform releases should generate a binary for this platform. goreleaser has support for this as I installed a new binary today (v 0.157.0) and got these by default on my own personal go project. I'm not sure why they are not being generated for kubeconform.

I'd love to use the docker image to validate helm charts in our projects, but we use a lot of CRDs. Some recent issues we had with our charts could have been caught much earlier if we had validated the charts against CRD schemas.

Those CRDs do have OpenAPI yaml definitions online, converting these (and caching the result) in a GH action for use with kubeconform would be ideal; your existing openapi2jsonschema.py script would allow us to do so all in one place if only it was included in the docker image.

Would you be able to update your Dockerfile to include it? Either as /openapi2jsonschema or /scripts/openapi2jsonschema.py, I don't much care. It would require that your alpine image includes Python 3 and pyyaml, of course.

My alternative right now is to use

curl -s https://raw.githubusercontent.com/yannh/kubeconform/v0.4.12/scripts/openapi2jsonschema.py > openapi2jsonschema
chmod +x openapi2jsonschema
python3 -m pip install pyyaml
./openapi2jsonschema ...

which is rather fragile as it either requires a version pin, or risks breaking if you ever moved the script or changed the requirements.

I am unable to integrate kubeconform into our Gitlab CI pipeline using the Alpine docker image. The relevant part of my Gitlab CI configuration is

lint-kubeconform:
  stage: validate
  image: ghcr.io/yannh/kubeconform:latest-alpine
  script:
    - kubeconform

The gitlab-runner fails to find a sh:

Runtime platform                                    arch=amd64 os=linux pid=40134 revision=f188edd7 version=14.9.1
Running with gitlab-runner 14.9.1 (f188edd7)
Preparing the "docker" executor
Using Docker executor with image ghcr.io/yannh/kubeconform:latest-alpine ...
Pulling docker image ghcr.io/yannh/kubeconform:latest-alpine ...
Using docker image sha256:48581c23a24fb25e270c6a6900b5fe9c4bd0095b7fe8779cdb45b94217686e6d for ghcr.io/yannh/kubeconform:latest-alpine with digest ghcr.io/yannh/kubeconform@sha256:a46a016956f6f91de40e1a635c1c7f75748720db592540b28a1c80f08c8d5991 ...
Preparing environment
Running on runner--project-0-concurrent-0 via LX-02014465...
Getting source from Git repository
Fetching changes...
Initialized empty Git repository in /builds/project-0/.git/
Created fresh repository.
Checking out f1afcbf4 as feature/kubeconform...

Skipping Git submodules setup
Executing "step_script" stage of the job script
Using docker image sha256:48581c23a24fb25e270c6a6900b5fe9c4bd0095b7fe8779cdb45b94217686e6d for ghcr.io/yannh/kubeconform:latest-alpine with digest ghcr.io/yannh/kubeconform@sha256:a46a016956f6f91de40e1a635c1c7f75748720db592540b28a1c80f08c8d5991 ...
sh - failed validation: lstat sh: no such file or directory
-c - failed validation: lstat -c: no such file or directory
ERROR: Job failed: exit code 1

FATAL: exit code 1  

It looks like the error described in #47. Is there something wrong with my configuration or is this a regression?

I am using fluxv2 and following their multi-tenancy example which has a script that uses kubeconform to validate files. I am also using Mozilla SOPS to encrypt secrets to store in git. sops will add a additional field sops to the secret which is not conforming with the v1/Secret API spec which fails the validation with Additional property sops is not allowed. A few possible solutions I could do:

• remove -strict flag but it won't be able to catch potential legitimate issues
• move sops encrypted secrets somewhere that's not being validated by kubeconform
• create my own v1/Secrets json schema but then I won't be able to use the default schema for the other default k8s schemas since mine won't able to override what's provided in default, thus I need to maintain all the schemas to have a custom v1/Secret.

Any of the above is a workaround to be able to get my test pass. Any suggestions on a better solution? Thanks for the great tool!

it would be great to be able to install this with brew install kubeconform. It's pretty simple to create the scripts for this and I'd be willing to take a first pass at this.

Hi @yannh !

I want to introduce asdf-kubeconform in my blog, and use this nice hero image.
Could you tell me License policy to use this image?

Hi!

I'm seeing some issues with -ignore-missing-schemas when supplying a JSON schema file ending with .json. There will be many custom resources that we need to be ignored, and listing them under the -skip flag would make it painful to maintain.

$ kubeconform -ignore-missing-schemas -schema-location default -schema-location "https://raw.githubusercontent.com/argoproj/argo-workflows/master/api/jsonschema/schema.json" app.yaml
 
app.yaml - VaultSecret sample-secret is invalid: For field (root): Must validate one and only one schema (oneOf) - For field apiVersion: apiVersion does not match: "argoproj.io/v1alpha1" - For field kind: kind does not match: "ClusterWorkflowTemplate"

Expected behavior: ignore resources where schemas could not be found.

Came from instrumenta/kubeval#301 because I got the same issue there 😅

Then I found kubeconform!

I have a file hm-ingress.yaml:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: hm-ingress
  namespace: hm
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
    - http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: web-service
                port:
                  number: 80

Currently when I use kubeconform to validate it, I got error

Ingress hm-ingress failed validation: could not find schema for Ingress

Any idea? Thanks!

When additional http schema locations return a successful (as in 200 status code) response with other than the expected schema, the validation wrongly fails in at least two scenarios (assuming the resource is correct):

1. You add an http location with CRDs and the schema for your resource is not there but you have -ignore-missing-schemas
   This should succeed because of the -ignore-missing-schemas flag

2. You add 2 (or more) http locations with CRDs and your resource exists in the 2nd location
   This should succeed because the schema does exists in the 2nd location

The issue is due to assuming that a successful http response must be a schema, since it is only possible to add schema locations ending with json, which is a fair assumption especially with github hosted content.

When we try a url for a non-existing schema github hosted, we get:

curl -v --location https://raw.githubusercontent.com/xeipuuv/gojsonschema/master/testdata/draft7/THIS_DOES_NOT_EXIST.json
> /* IRRELEVANT HTTP REQUEST HEADERS */
> 
< HTTP/1.1 404 Not Found
< Connection: keep-alive
< Content-Length: 14
< /* MORE HTTP RESPONSE HEADERS */

The status code response is 404 correctly handled by kubeconform, no issues there.
When we try for a location hosted in bitbucket, we get:

curl -v --location https://bitbucket.org/REDACTED/raw/THIS_DOES_NOT_EXIST.json
> /* IRRELEVANT HTTP REQUEST HEADERS */
> 
< HTTP/1.1 302 Found
< /* MORE HTTP RESPONSE HEADERS  */
>
> 
< HTTP/1.1 200 OK
< Content-Type: text/html
< /* MORE HTTP RESPONSE HEADERS  */

it is redirecting to a human friendly error page saying that the resource is not there. Because the status code is 200, kubeconform stops looking in the registries and later fails parsing the schema when it is already too late.

Probably this could be fixed by some bitbucket server configuration, but I can see this easily happening with any other server exposing the same behavior and for which you don't have any control.

The issue is in the downloadSchema function returning prematurely without validating the schema parsing result:

I will submit a PR for your consideration to augment the test to reproduce the issue and of course, to fix it.

I just noticed today I am no longer able to pull container images.

Trying to pull ghcr.io/yannh/kubeconform:v0.4.13...
Error: parsing image configuration: unable to retrieve auth token: invalid username/password: unauthorized: unauthenticated: User cannot be authenticated with the token provided.

When validating apiextensions.k8s.io/v1 CustomResourceDefinition resources I get this error:

<file> - CustomResourceDefinition servicemonitors.monitoring.coreos.com failed validation: could not find schema for CustomResourceDefinition

Why is that? I thought this API was part of the Kubernetes API?

Hello,

is it possible to check whole folder with more schema locations for CRDs?
Here is example when i check one helm release and one standard k8s yaml in one folder:
PS C:\tools> .\kubeconform.exe .\ingress.yaml
.\ingress.yaml - HelmRelease nginx failed validation: could not find schema for HelmRelease
PS C:\tools> .\kubeconform.exe -schema-location .\schema\helmrelease_v2beta1.json .\ingress.yaml
PS C:\tools> .\kubeconform.exe -schema-location .\schema\helmrelease_v2beta1.json .\deploy.yaml
.\deploy.yaml - Deployment infra-safe is invalid: For field spec: chart is required - For field spec: interval is required - For field spec: Additional property selector is not allowed - For field spec: Additional property template is not allowed - For field spec: Additional property replicas is not allowed
PS C:\tools> .\kubeconform.exe .\deploy.yaml

I want to put kubeconform to our CI pipeline to check our manifests, but i don't want to get false positive alerts which they block our pipeline.

Thank you

Hi all,
When I tried to manage my CRDs and validate them using Kubeconform, I got a problem with parsing the correct Kind from my yaml file to match the new schema generated file.

Process

1. I dumped all my CRDs from my Cluster
2. Dumped the python file openapi2jsonschema (from https://raw.githubusercontent.com/yannh/kubeconform/master/scripts/openapi2jsonschema.py) to do the conversion work
3. Validation with kubeconform -schema-location default --schema-location '{{ .ResourceKind }}_{{ .ResourceAPIVersion }}.json' <crd-name-file.yaml>

Input

Here some of my CRDs ready to be converted, with the format: {crd-kind}-{apiversion}

clusterissuer_v1beta1.json
clusterissuer_v1.json
kafkauser_v1alpha1.json
kafkauser_v1beta1.json
...

My Kubeconfrom version

kubeconform -v
v0.4.13

Expected

By this, I expected to have all my yaml files converted and checked with kubeconform

Output

kubeconform -summary -output json  -schema-location default -schema-location './schemas/{{ .ResourceKind }}_{{ .ResourceAPIVersion }}.json' ./input/prom-crd.yaml
{
  "resources": [
    {
      "filename": "./input/prom-crd.yaml",
      "kind": "CustomResourceDefinition",
      "name": "prometheuses.monitoring.coreos.com",
      "version": "apiextensions.k8s.io/v1",
      "status": "statusError",
      "msg": "could not find schema for CustomResourceDefinition"
    }
  ],
  "summary": {
    "valid": 0,
    "invalid": 0,
    "errors": 1,
    "skipped": 0
  }
}

With

.
├── input
│   └── prom-crd.yaml
├── schemas
│   └── prometheus_v1.json
└── script-crd.sh

2 directories, 3 files

AND

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
...
spec:
  conversion:
    strategy: None
  group: monitoring.coreos.com
  names:
    categories:
    - prometheus-operator
    kind: Prometheus
    listKind: PrometheusList
    plural: prometheuses
    singular: prometheus
  scope: Namespaced
  versions:
  - additionalPrinterColumns:
    - description: The version of Prometheus
      jsonPath: .spec.version
      name: Version
      type: string
    - description: The desired replicas number of Prometheuses
      jsonPath: .spec.replicas
      name: Replicas
      type: integer
    - jsonPath: .metadata.creationTimestamp
      name: Age
      type: date
    name: v1
...

Troubleshooting

For instance and as the error msg shows, I found that {{ .ResourceKind }} variable is containing the CustomResourceDefinition value
While I was expecting to have prometheus value
I checked that by passing a static correct ResourceKind

kubeconform -summary -output json  -schema-location default -schema-location './schemas/prometheus_{{ .ResourceAPIVersion }}.json' ./input/prom-crd.yaml
{
  "resources": [
    {
      "filename": "./input/prom-crd.yaml",
      "kind": "CustomResourceDefinition",
      "name": "prometheuses.monitoring.coreos.com",
      "version": "apiextensions.k8s.io/v1",
      "status": "statusInvalid",
      "msg": "For field spec: Additional property group is not allowed - For field spec: Additional property names is not allowed - For field spec: Additional property scope is not allowed - For field spec: Additional property versions is not allowed - For field spec: Additional property conversion is not allowed - For field status: availableReplicas is required - For field status: paused is required - For field status: replicas is required - For field status: unavailableReplicas is required - For field status: updatedReplicas is required - For field status: Additional property storedVersions is not allowed - For field status: Additional property acceptedNames is not allowed - For field status: Additional property conditions is not allowed"
    }
  ],
  "summary": {
    "valid": 0,
    "invalid": 1,
    "errors": 0,
    "skipped": 0
  }
}

Thank you !!

Using the Open API to JSON utility script provided, I've pulled down the schemas for all the relevant third party CRDs my org uses in our K8S clusters. The -schema-location option can be used multiple times, but for multiple JSON schema files it becomes a bit unwieldy if YAML files containing mixed resources and CRDs is validated against.

Ideally the -schema-location option would support being pointed to a directory rather than expect a single JSON file per usage where it would pick up any JSON files located therein.

As an example, I have a YAML file which contains an Ingress, Service, SealedSecret (bitnami sealed-secrets), and Receiver (fluxcd). In order to validate this particular file I'd need to do something like kubeconform -summary -verbose -schema-location default -schema-location path/to/sealedsecret.json -schema-location path/to/flux/receiver.json foo.yaml. A simpler approach would be to have a directory locally - call it crds/ containing the respective JSON schemas. Then I could do something like kubeconform -summary -verbose -schema-location default -schema-location path/to/crds foo.yaml and not concern myself with the name of a particular CRD's JSON schema.

As an exercise for giggles, I tried feeding in a large number of CRD JSON files doing something like export SCHEMAS="" ; for FILE in *; do SCHEMAS="$SCHEMAS -schema-location $FILE"; done && kubeconform -summary -verbose -schema-location default $SCHEMAS foo.yaml. The result was "file name too long", so there is a limit to how much you can try to feed to it with multiple/many -schema-location flags.

Hi, partly question, partly an issue.

There are reports over at https://github.com/instrumenta/kubernetes-json-schema that the repo is unmaintained. Apparently PRs for K8s 1.19 and 1.20 haven't been merged.

A partial workaround is to use schemas in this repo: https://github.com/awslabs/cdk8s/tree/master/kubernetes-schemas/. However these seem aligned to EKS versions, so there's no 1.20 yet.

Is there a better default location that we can use for up to date schemas?

Most of commonly used linters can now output SARIF format

Would it be also possible to have it for kubeconform ?

It would improve a lot its integration in the next version of MegaLinter :)

Note: you can see how it has been managed in golang for revive: https://github.com/mgechev/revive/pull/508/files , using garif package

Best regards

When trying to convert the prometheus CRDs to json schema, I get this error from the python script:

$ python3 ~/kubeconform/scripts/openapi2jsonschema.py https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/main/bundle.yaml
Traceback (most recent call last):
  File "/home/ubuntu/kubeconform/scripts/openapi2jsonschema.py", line 123, in <module>
    for y in yaml.load_all(f, Loader=yaml.SafeLoader):
  File "/usr/lib/python3/dist-packages/yaml/__init__.py", line 130, in load_all
    yield loader.get_data()
  File "/usr/lib/python3/dist-packages/yaml/constructor.py", line 45, in get_data
    return self.construct_document(self.get_node())
  File "/usr/lib/python3/dist-packages/yaml/constructor.py", line 60, in construct_document
    for dummy in generator:
  File "/usr/lib/python3/dist-packages/yaml/constructor.py", line 408, in construct_yaml_seq
    data.extend(self.construct_sequence(node))
  File "/usr/lib/python3/dist-packages/yaml/constructor.py", line 129, in construct_sequence
    return [self.construct_object(child, deep=deep)
  File "/usr/lib/python3/dist-packages/yaml/constructor.py", line 129, in <listcomp>
    return [self.construct_object(child, deep=deep)
  File "/usr/lib/python3/dist-packages/yaml/constructor.py", line 100, in construct_object
    data = constructor(self, node)
  File "/usr/lib/python3/dist-packages/yaml/constructor.py", line 427, in construct_undefined
    raise ConstructorError(None, None,
yaml.constructor.ConstructorError: could not determine a constructor for the tag 'tag:yaml.org,2002:value'
  in "<file>", line 2789, column 29

I messed around a bit trying to figure it out, but to no avail. Has anyone seen this error before?

This is an interesting one so bear with me :)

First, this is the error I stumbled when validation FluxCD CRDs. The creationTimestamp can't be null and must be a string according to the spec:

# wget https://github.com/fluxcd/source-controller/releases/download/v0.22.5/source-controller.crds.yaml
# kubeconform -schema-location default -schema-location "https://raw.githubusercontent.com/yannh/kubernetes-json-schema/master/master/customresourcedefinition.json" source-controller.crds.yaml
source-controller.crds.yaml - CustomResourceDefinition gitrepositories.source.toolkit.fluxcd.io is invalid: For field metadata.creationTimestamp: Invalid type. Expected: string, given: null
source-controller.crds.yaml - CustomResourceDefinition buckets.source.toolkit.fluxcd.io is invalid: For field metadata.creationTimestamp: Invalid type. Expected: string, given: null
source-controller.crds.yaml - CustomResourceDefinition helmrepositories.source.toolkit.fluxcd.io is invalid: For field metadata.creationTimestamp: Invalid type. Expected: string, given: null
source-controller.crds.yaml - CustomResourceDefinition helmcharts.source.toolkit.fluxcd.io is invalid: For field metadata.creationTimestamp: Invalid type. Expected: string, given: null

So I raised an issue at fluxcd/flux2#2623 since they seemed to be the culprit.
But received a response that they use controller-gen to generate the CRDs. So I looked into controller-gen and they also do not set the creationTimestamp they import k8s.io/apimachinery/pkg/apis/meta/v1 for the generation of the metadata (where the creationTimestamp is part of)(source).

So now the issue, in the package documentation (link) of k8s.io/apimachinery/pkg/apis/meta/v1 it says the following about the creationTimestamp

// CreationTimestamp is a timestamp representing the server time when this object was
// created. It is not guaranteed to be set in happens-before order across separate operations.
// Clients may not set this value. It is represented in RFC3339 form and is in UTC.
//
// Populated by the system.
// Read-only.
// Null for lists.
// More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#metadata
// +optional
Creation[Time](https://pkg.go.dev/k8s.io/apimachinery/pkg/apis/meta/v1#Time)stamp Time `json:"creationTimestamp,omitempty" protobuf:"bytes,8,opt,name=creationTimestamp"`

So there is something in the docblock about Null for lists(?) but in the openapi spec of Kubernetes is this Time resource marked as string:

"io.k8s.apimachinery.pkg.apis.meta.v1.Time": {
      "description": "Time is a wrapper around time.Time which supports correct marshaling to YAML and JSON.  Wrappers are provided for many of the factory methods that the time package offers.",
      "format": "date-time",
      "type": "string"
},

https://raw.githubusercontent.com/kubernetes/kubernetes/master/api/openapi-spec/swagger.json

So now, what is wrong here? Is it indeed allowed to be a string or null and should the openapi spec be an enum? Or is there something ink8s.io/apimachinery/pkg/apis/meta/v1 wrong? Or should controller-gen explictly set the creationTimestamp to an string?
The last one seems kinda wrong to me since it seems to me that the k8s.io/apimachinery/pkg/apis/meta/v1 should return data valid to the api spec.

I am sorry for all the text and if I did not understand something correctly :)
I don't develop in golang (or at all really) and the relation between the package and the openapi spec is quite vague for me.

The Dockerfile currently doesnt seem to work on GitLab CI for example, as GL-CI expects to be able to oppen up a shell in the container and execute the cmd that way.
Just specifying entrypoint and args isnt allowed by gitlab and treatet as an invalid config.
For the dokcer image to work properly with Gitlab CI there would need to be a shell inside the container, although the entrypoint could just remain as it is right now.

Hey all, I had an idea.
What if we added a diagram of the project's architecture in the Readme file, this will help people grasp how the validation works, I thought we can do something similar to the datree project

any thoughts?

Righ now files are limited to 4MB

For example, I have a 17MB output for our Grafana deployment, mainly dozens of dashboards in ConfigMaps: 😛

Summary: 0 resource found parsing stdin - Valid: 0, Invalid: 0, Errors: 0, Skipped: 0

kubeval had the same issue and solved it with a default buffer (expend itself dynamically) and io.Copy() like in instrumenta/kubeval#220. Not sure how it affects the overall performance.

If you want to reproduce, here's a couple quick and dirty scripts to generate large files:

package main

import "fmt"

const (
	N = 20000
)

func main() {
	fmt.Println(
		`{
    "apiVersion": "v1",
    "kind": "List",
    "items": [`,
	)

	for i := 0; i < N; i++ {
		fmt.Printf(
			`        {
            "apiVersion": "v1",
            "kind": "ConfigMap",
            "metadata": {
                "name": "yet-another-confimap-%d"
            },
            "data": {
                "key": "value"
            }
        }`,
			i,
		)
		if i == N - 1 {
			fmt.Println()
		} else {
			fmt.Println(",")
		}
	}

	fmt.Println(`    ]
}
`)
}

package main

import "fmt"

func main() {
	fmt.Println(
		`apiVersion: v1
kind: List
items:`,
	)

	for i := 0; i < 50000; i++ {
		fmt.Printf(
			`- apiVersion: v1
  kind: ConfigMap
  metadata:
    name: yet-another-confimap-%d
  data:
    key: value
`,
			i,
		)
	}
}

Hi,

I'm a bit confused because I'm currently working on validation of schemas (k8s, istio, ...) in pipelines gitlab.

The problem:

• I'm working with this schema : https://kubernetesjsonschema.dev/v1.18.0/limitrange.json
• I have a simple LimitRange configuration file with an error on Kind name "LimitRangeeee"
• In local, kubeconform returns "Could not find schema for "LimitRangeee" (this is the expected behavior)
• In my pipeline (through an image docker) kubeconform returns no error whereas this is the same configuration file

Did you have any idea why in the pipeline the behavior is different compared to le local one?

Same file, same binary of kubeconform and same schema (the remote one at https://kubernetesjsonschema.dev/v1.18.0/limitrange.json)

Thank you in advance
Alexis

BAD:

GOOD:

Empty files aren't causing errors while verified with kubeconform, but are failing while used with k8s.

Expected behaviour:

the command line tool should print an error if checking files without valid k8s configurations.

Follow up #58

Kubeconform currently uses 1.18.0 as default Kubernetes version.
It would be great to use latest unless people need use old version. In that case, people can use -kubernetes-version to use the old version.
And currently latest version is 1.21.0 already. 😀

We curently have folders which contain kustomize overlays.

dev/kustomization.yaml - failed validation: error while parsing: missing 'kind' key
dev/notification-controller_flux-system.yaml - Deployment notification-controller is invalid: For field spec: selector is required - For field spec.template.spec: containers is required

As you can see validation of those resources fails and ignoring based on regex isnt really possible, as i would need the regex feature to work conditionally (e.g. only exclude .yaml when a kustomize.yaml ist present in the folder)