open-policy-agent / conftest Goto Github PK

I've been super happy to see the interest in Conftest, both from folks wanting to use it and from several folks who have already contributed fixes and features. One of the reasons to start this under an org rather than my own name was to make it a touch easier to add other folks if the project proved interesting.

With that, I thought it would be easiest to open an issue to discuss.

Pinging @brendanjryan, @jpreesem, @Blokje5, @xchapter7x and @Proplex just in case that is something of interest?

Hey y'all :)

I think a great feature extension to #5 would also be the ability to do this cross-referencing across actual files on a disk, as well. For example, Terraform files in a module. It would be cool to be able to write a policy that allows for comparing forwarding rule resources with their respective load balancers or firewall rules.

To expand the potential places where Conftest could be used, it would be good to support CUE natively as an input. Conftest currently supports JSON or YAML only as input, but should work with any format we can unmarshall.

We have CUE examples, but would be useful to not require pipeline commands together.

This should just be about either using something more generic, or adding some logic around the current use of the YAML module around this line.

Recently open policy agent gatekeeper has implemented changes in rego code. Now they use violation instead of deny. We use conftest to verify our PRs before merge. That simple change has caused the conftest to skip all the checks which worked before.

We kindly ask you to extend deny rule name to a list of (deny|rule|violation).

here you have the patch.

commit cf0c59d92802f5b53bdabc452295b296baef59b8
Author: Oleksandr DUDCHENKO <[email protected]>
Date:   Fri Sep 6 17:21:30 2019 +0300

    extended list for possible deny rule names

diff --git a/pkg/commands/test/test.go b/pkg/commands/test/test.go
index f5ed225..acf9d0d 100644
--- a/pkg/commands/test/test.go
+++ b/pkg/commands/test/test.go
@@ -26,7 +26,7 @@ import (
 )
 
 var (
-	denyQ = regexp.MustCompile("^deny(_[a-zA-Z]+)*$")
+	denyQ = regexp.MustCompile("^(deny|rule|violation)(_[a-zA-Z]+)*$")
 	warnQ = regexp.MustCompile("^warn(_[a-zA-Z]+)*$")
 )

I would like to test if the containerPort in the Deployment config and targetPort in service config are equal. For this scenario, is it possible to compare values between multiple inputs data(both service and deployment manifest are in the same file)

I noticed in #48 that CI is not running on pull requests - although it is configured to run on master.

Is this intentional? If not i'd be more than happy to get this integrated on every build!

I build a proof of concept kubectl plugin for conftest, which can be found here https://github.com/instrumenta/conftest/tree/master/plugin. I'll let folks read the one liner in there to understand why I say proof of concept :)

It works as follows:

$ kubectl conftest -h
A Kubectl plugin for using Conftest to test objects in Kubernetes using Open Policy Agent

See https://github.com/instrumenta/conftest for more information

Usage:
   kubectl conftest (TYPE[.VERSION][.GROUP] [NAME] | TYPE[.VERSION][.GROUP]/NAME)

You can use the same syntax as kubectl get to grab lists or individual resources, and then have conftest apply policy against them.

$ kubectl conftest service
   Found service hello-kubernetes but services are not allowed
   Found service kubernetes but services are not allowed

I reason this is useful for auditing an existing cluster. Especially useful if you have opinions encoded in Rego, and someone has just given you a new cluster to manage. It's also useful when writing policies, as you can easily test it against your real cluster and not just mocked data.

I proposed this to the Krew index, kubernetes-sigs/krew-index#146, but that raised a larger conversation about what the Krew folks want in the index.

Opening this issue to track taking the PoC and making a more robust plugin. I'm imagining that as another standalone Go binary, reusing much of the Conftest code. This is a probably a good place to look at the public Conftest library interface too.

With the work started in #55 we are soon to be able to support multiple output formats. Some discussion in #26 and #42.

Opening this issue to discuss TAP output support. I think the best we can do is:

1..3
not ok 1 - deployment.yaml - Containers must not run as root
not ok 2 - deployment.yaml - Deployments are not allowed
# Warnings
not ok 3 - deployment.yaml - Services not allowed

1. We can currently only show failing rules, so no ok responses at present
2. I think we should include Warnings in the output. The # is a diagnostic information, it's not machine readable be is at least human readable. The alternative would be to not include the warnings which I feel is worse
3. Each line includes the file and the message, similar to #58

The TAP specification can be found at http://testanything.org/tap-specification.html

TAPs advantage is its simplicity, so a number of tools can consume it and display the results in whatever way they choose.

Another proof of concept, I build a Helm plugin for Conftest https://github.com/instrumenta/helm-conftest

helm conftest <path-to-chart>

Currently this simply allows accessing conftest from Helm. Helm runs helm template under the hood to render the template, and conftest then tests that document against the defined policies.

Not supported in the above, but probably desirable, would be to be able to pass a number of the helm template arguments, in particular --set and --set-file.

The initial proof of concept is a very simple Bash script, but it would probably be good to create a standalone binary which uses the relevant packages from Helm. Similar to #52, this would be another case for testing the public interface for Conftest as a library.

Hoping that we can get a new release cut soon. Particularly interested in using the feature built in by @Blokje5 in PR #83 .

@garethr ping :)

Hi,
could you provide an example with a policy to check Kubernetes recommanded labels ?
Thanks.

Here: https://kubernetes.io/docs/concepts/overview/working-with-objects/common-labels/#labels

Although Conftest can be used for testing any configuration, there is certainly lots of interest in testing Kubernetes configuration. I don't see this as much about Conftest, as much as about content. ie. it's about a standard library of Rego code which provides a useful DSL for common Kubernetes assertions.

Conftest has the push and pull commands that allow for saving OPA bundles to a OCI registry. This is somewhat experimental, but serves as a nice way of sharing Rego.

Opening an issue here to discuss, although my first instinct is to open a new repository for the project if folks like the idea.

I could imagine things like:

• Kind assertions
• Common security gotchas (eg. runAsNonRoot, or implementing the kubesec rules in Rego)
• Labels
• Patterns for use with Gatekeeper
• Image assertions (whitelisting registries, prohibiting latest tags, etc.)

I'm thinking of starting with Kubernetes, but obviously the same pattern would hold for other common formats.

Assume we got a dummy.yaml and contains:

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: xapi
spec:
  replicas: 15
  template:
    metadata:
      labels:
        app: xapi
    spec:
      hostAliases:
      - ip: "127.0.0.1"
        hostnames:
        - "foo.org"
      - ip: "127.0.0.1"
        hostnames:
        - "bar.org"
      containers:
      - name: nginx
        image: nginx:alpine
        ports:
        - containerPort: 80        
        resources:
          requests:
            cpu: "1000m"

and a deny.rego file in policy folder,

package main

deny[msg] {
  output := sprintf("%s", [input.spec.template.spec])
  not contains(output, "{\"hostnames\": [\"foo.org\"], \"ip\": \"127.0.0.1\"}")
  msg = output
}

And when we execute test commands several times manually or

while true; do echo "conftest test dummy.yaml"; conftest test dummy.yaml;  done

$ conftest test dummy.yaml
$ conftest test dummy.yaml
$ conftest test dummy.yaml
FAIL - dummy.yaml - {"containers": [{"image": "nginx:alpine", "name": "nginx", "ports": [{"containerPort": 80}], "resources": {"requests": {"cpu": "1000m"}}}], "hostAliases": [{"ip": "127.0.0.1", "hostnames": ["foo.org"]}, {"hostnames": ["bar.org"], "ip": "127.0.0.1"}]}
$ conftest test dummy.yaml
$ conftest test dummy.yaml
FAIL - dummy.yaml - {"containers": [{"image": "nginx:alpine", "name": "nginx", "ports": [{"containerPort": 80}], "resources": {"requests": {"cpu": "1000m"}}}], "hostAliases": [{"ip": "127.0.0.1", "hostnames": ["foo.org"]}, {"hostnames": ["bar.org"], "ip": "127.0.0.1"}]}

We sometimes fail, sometimes pass. This bug is probably caused by Rego but can affect tests.
Do we have to open this issue to OPA and close this one?

Any thoughts?

Conftest is silent, even on failing policy, with sprintf errors. For example:

deny["unable to find a gke container resource"] {
  not check_for_container_resources
}

check_for_container_resources {
    containers := [
      container |
      some i
      name := input.Resources[i].Name
      type := input.Resources[i].Type
      startswith(type, "google_container")
      container := type
    ]
    count(containers) == 2
}

results in exit 1

package main

deny[sprintf("unable to find a gke container resource %s %s", [input.Resources[i].Type, input.Resources[i].Name])] {
  not check_for_container_resources
}

check_for_container_resources {
    containers := [
      container |
      some i
      name := input.Resources[i].Name
      type := input.Resources[i].Type
      startswith(type, "google_container")
      container := type
    ]
    count(containers) == 2
} 

results in exit 0

conftest should support ability to provide output formatting, which would help to integrate it better with other tools especially editors.

Most common linters provide two options:

• disable color output
• allow providing formatting string such as <file-path>:<line>:<column>:<type>:<rule-id>:<message> which allows it to be parsed by other tools, or output format in other standards like JUnit/xUnit

Right now conftest:

• (probably) always produce color output
• does not allow formatting

Useful Links:
https://github.com/golangci/golangci-lint
Pycodestyle https://github.com/PyCQA/pycodestyle#example-usage-and-output
Vim quick formats https://vim.fandom.com/wiki/Errorformats
Yamllint https://yamllint.readthedocs.io/en/stable/quickstart.html#running-yamllint

Checkstyle https://checkstyle.sourceforge.io/
CodeClimate https://docs.codeclimate.com/docs/configuring-test-coverage
Junit format https://github.com/rodrigodiez/go-junit

Has there been any talk on moving some of the private functions like buildCompile to a separate package in the repo so they could be used as a library in other projects? Or would you have any interest in a PR that does this?

The Docker API is not intended to be stable, so our dependency on it is going to cause dependency problems for us. We're already experiencing some.

It would probably be in our best interest to move buildkit's parser functionality to its own module, and own that functionality.

EDIT: Updating to include our go.mod as a whole. Right now we have a lot of replace directives due to our dependencies. I'd like to use this issue as a means to track upstream PRs to fix their go.mod files, so ours becomes cleaner as well.

oras: oras-project/oras#132

Hi, thanks for this awesome project :) I've been trying to use an OCI registry to push and pull the policies and hit a problem.

I have a conftest.toml as follows:

policy = "policy"

[[policies]]
repository = "localhost:5000/conftest-circleci"
tag = "latest"

It's pointing to my local docker registry but fails to pull it on conftest update.

$ conftest --debug update
INFO[0000] Downloading: localhost:5000/conftest-circleci
FATA[0000] Downloading policy failed: object required

It looks like it's missing the tag because the localhost:5000 is containing a : and conftest thinks the string already contains a tag.

https://github.com/instrumenta/conftest/blob/master/conftest.go#L323

This example here https://www.openpolicyagent.org/docs/latest/terraform/ shows how to write rego policy to produce some score out of the json file. Can conftest do something similar ?

Hey there 👋, loving conftest so far.

I'm interesting in picking up #5 and I also want to write tests for it. However, the functions are written in a way that a bulk of the action is within an anonymous function in the Command{} struct.

Would y'all be open to the idea of separating it out as such:

func NewTestCommand() *cobra.Command {

	cmd := &cobra.Command{
		Use:     "test <file> [file...]",
		Short:   "Test your configuration files using Open Policy Agent",
		Version: fmt.Sprintf("Version: %s\nCommit: %s\nDate: %s\n", constants.Version, constants.Commit, constants.Date),

		Run: TestFunction()


func TestFunction(ccmd *cobra.Command, args []string)
ctx := context.Background()

			if len(args) < 1 {
				cmd.SilenceErrors = true
				log.G(ctx).Fatal("The first argument should be a file")
			}
...

Right now we're re-defining a lot of the Cobra command definitions for each test so that we can single out functionality for testing. Changing each command to use a standalone function would reduce the amount of code duplication for testing. This would make testing easier for others, and as far as I'm aware not alter any user functionality.

Would you accept a PR into master with this change? Thanks :)
cc: @Salamandastron1

When testing for two labels being required within a deployment, if either match passes the whole block is accepted.

Given a rule;

deny[msg] {
  input.kind = "Deployment"
  not input.spec.selector.matchLabels.app
  not input.spec.selector.matchLabels.release
  msg = sprintf("Deployment[%s] - Containers must provide app/release labls for pod selectors", [name])
}

See Gist for deployment.yaml test case with 4 deployments.

Expected Behaviour

• All rules should match before accepting

deployment.yaml
   Deployment[app_only] - Containers must provide app/release labls for pod selectors
   Deployment[release_only] - Containers must provide app/release labls for pod selectors
   Deployment[no_labels] - Containers must provide app/release labls for pod selectors

Actual

$ conftest test deployment.yaml
deployment.yaml
   Deployment[no_labels] - Containers must provide app/release labls for pod selectors

Version

conftest --version
Version: 0.6.0
Commit: a27d0739a785fc52c421339d129267772a15662f
Date: 2019-05-20T07:40:07Z

Test Case

See public Gist for configuration and test case:

https://gist.github.com/warmfusion/531aec5da54b1b248b02f40c48199e19

It looks like the Terraform 0.12 syntax isn't supported.

.tf snippet:

resource "aws_vpc" "vpc" {
  cidr_block           = var.cidr
...
}

Error:

FATA[0000] Problem running evaluation: load terraform config failed: Error parsing JSON document as HCL: At 2:26: Unknown token: 2:26 IDENT var.cidr

To expand the potential places where Conftest could be used, it would be good to support TOML as an input. Conftest currently supports JSON or YAML only as input, but should work with any format we can unmarshall.

This should just be about either using something more generic, or adding some logic around the current use of the YAML module around this line.

I'm a big fan of OPA's "table" output largely due to its readability, which you can see at work in this blog post that I wrote. It'd be nice to add that to the supported output options for Conftest.

The implementation seems pretty trivial so I'll attempt it myself.

The current command lint output of conftest is difficult to parse by CI or other command line tooling - mostly due to the fact that it does use show absolute file paths or distinguish between errors and warnings aside from color.

I propose changing the "output" to be of the form <msg_type>: <file> - <msg> (still colorized)

For example:

WARNING: /Users/brendanjryan/projects/conftest/example/test.yaml - nginx-deployment should not be configured to live in the default namespace
FAILURE: /Users/brendanjryan/projects/conftest/example/test.yaml - nginx-deployment must not include any Horizontal Pod AutoScalers

Reference of existing format:

conftest test deployment.yaml
deployment.yaml
   Containers must not run as root
   Deployments are not allowed

Is this something we are interested in supporting? While this may break existing integrations - it will make future ones much easier to write.

Currently releasing a new version of conftest, and then building the Docker images and pushing to Docker Hub, are done manually offline.

Configuration is all done with goreleaser (GitHub release) and Make (Docker).

It should be possible to have CircleCI do this when new tags are added to master.

Basic introductions in https://goreleaser.com/ci/

While the help documentation shows that the -p argument is meant to provide a directory path, the error message when giving an explicit filename is not clear;

$ conftest test -p specific_policy.rego file.yaml
FATA[0000] Problem building rego compiler: readdirent: invalid argument

Would it be possible either accept an explicit policy document as the argument, or make it clearer that the -p argument expects a directory rather than a file (or other type)

Version

MacOS - 10.13.6

$ conftest --version
Version: 0.6.0
Commit: a27d0739a785fc52c421339d129267772a15662f
Date: 2019-05-20T07:40:07Z

To expand the potential places where Conftest could be used, it would be good to support EDN as an input. Conftest currently supports JSON or YAML only as input, but should work with any format we can unmarshall.

This should just be about either using something more generic, or adding some logic around the current use of the YAML module around this line.

I would like to have multiple policy directories being used in single run.

Use case :
I would like to have CI image with some common rules being there, but allow users to add their own rules to some other directory, and in CI run to use conftest to apply all rego rules

Hi,
To expand the potential places where Conftest could be used, it would be good to support dockerfile as an input.
Thanks

Right now if I want to execute both conftest and evaluate the policies using Rego unit tests, I need to install both conftest and OPA. Ideally we would be able to use conftest for this purpose as well, as this would simplify my CI/CD pipeline.

Hello,

I saw that OPA was in the inspiration list and I am currently looking for tools that would help me test my terraform code.

I didn't try both tools a lot for the moment, but what would you think would be an advantage of contest compared to using open policy agent ?

Both seems very close in term of features for my use case.

Thanks

To expand the potential places where Conftest could be used, it would be good to support HCL as used in Terraform and GitHub Actions as an input. Conftest currently supports JSON or YAML only as input, but should work with any format we can unmarshall.

This should just be about either using something more generic, or adding some logic around the current use of the YAML module around this line.

While looking at Gatekeeper (https://github.com/open-policy-agent/gatekeeper) I noticed their policies allow you to attach structured data to any violation, like this snippet from the Gatekeeper Readme:

        violation[{"msg": msg, "details": {"missing_labels": missing}}] {
          provided := {label | input.review.object.metadata.labels[label]}
          required := {label | label := input.parameters.labels[_]}
          missing := required - provided
          count(missing) > 0
          msg := sprintf("you must provide labels: %v", [missing])
        }

I think something like this could potentially be a really powerful addition to conftest, for example, right now when I try to write tests for conftest policies I often end up using something like:

test_some_policy {
    deny[msg] with input as {...}
    contains(msg, "Some error raised")
}

which feels somewhat fragile as changing the wording of the deny breaks the test. Having the ability to attach arbitrary data allows you to add things like error codes, urls with more information etc.

Of course implementing something like this would be a major change as it'd affect both how you write policies and the format of the output. It could theoretically be made backwards compatible by keeping the signature of warn and deny the same and adding a new third field (like for example the violation from the Gatekeeper example) which has the new signature. It could even have a separate value indicating whether it is a warn or fail based on one of the structured outputs, something like:

violation[{"msg": msg, "level": "warn", "details": {}] {
}

The presence of a violation record could affect the format of the output, or an explicit flag could yield the alternate output format.

Would a PR adding some or all of this functionality be welcome?

To expand the usage of conftest with stdin or unknown file extensions, we need a flag for allowing injection of requested parser type.
After this feature enabled, we will support the commands below:

cat examples/ini/grafana.ini | ./conftest test -p examples/ini/policy/ --flag=ini -
./conftest test -p examples/some/policy/ --flag=toml  examples/some/unknown.xyz

I would like to test if the namespace of the resource is the same as folder name where my configuration files are present. In this case, I would like to get the parent folder name.

This used to work - looks like the code in question has been re-worked lately to check for file-extension types yaml/toml.

Given a valid yaml file deploy.yaml ...

cat deploy.yaml | conftest test - 
Unable to find available parser for extension 

Where as the following works fine

conftest test deploy.yaml 

Just tracking here for visibility, and probably for linking into the documentation. See the PR tektoncd/catalog#60 for the Task itself.

When upgrading from 0.11 to 0.13, conftest went silent. No warnings or failed tests.

# previously

$ brew switch conftest 0.11.0
$ conftest test deployment.yaml
FAIL - prometheus-adapter is running as root

# now

$ brew switch conftest 0.13.0
$ conftest test deployment.yaml

# silent...

I expected at least some output, not just silently passing all tests.

$ conftest --version
Version: 0.13.0
Commit: 8340065
Date: 2019-09-27T22:40:00Z

First and foremost, thanks for putting this tool together. I saw your talk and got super excited about the possibilities of linting Kubernetes YML with OPA. I am still digging through everything, but I am really concerned about something, and hope I am just missing the whole picture.

Let's just take the really simple example of not allowing Services.

In a CI pipeline, someone is going to submit a PR with just their Kubernetes YML. I would like to use conftest on that YML against my policies. In this case, no services allowed. If that passed, the CI will pass.

Then, using (hopefully) the exact same policy, I'd want to have that policy running on my Kubernetes cluster, denying services.

Unfortunately I'm seeing a lot of tests, packages, etc that seem to treat these very differently. E.g. these tests are quite different than the examples you have here

https://www.openpolicyagent.org/docs/edge/guides-kubernetes-admission-control/

Hello,

I am wondering if the rules being names deny and warn is mandatory ?
I've been first writing my policies with opa using the test framework available with it. It allows a TDD approach when writing the policies.

When testing conftest (which have a better output for a CI integration), I have been kind of disappointed to see that all the rules should be named deny or warn, It forced me to rename my rules, and I am no more able to use opa tests anymore :/

Did I miss something ?
Would it be possible for conftest to look for rules through a regex ?
It would allow to write rules named deny_some_explicit_name or warn_some_other_name so that conftest finds the rules and one can still test the policies through opa test framework.

As a Conftest user, I want to know that how the inputs transformed into structured data. This feature especially needed for when we are writing tests against Dockerfile, hcl2, or ini like files, that needs to be parsed.

The JSON output for the parsing process would be nice, in order to understand exactly how the files are transformed and how we can write tests against them:
For instance, to understand which one is correct, input.x.y["z.t"] = foo or input["x"].y["z.t"] = foo, I have to take a look at the parsers.

I think this can be achieved with a flag like --reveal-structure but other suggested ideas are welcomed. btw, I'm not sure if this feature is needed for the users, so let's discuss it!

Any thoughts?

Version 0.6.0 worked great, but with 0.7.0 and 0.9.0, conftest seems to be rerunning my policies multiple times when I run conftest test. Is there some change I need to make? My policies don't differ structurally from the kubernetes example in the repo.

To expand the potential places where Conftest could be used, it would be good to support INIT as an input. Conftest currently supports JSON, YAML, HCL, TOML and CUE as input, but should work with any format we can unmarshall.

This is a matter of implementing the parser interface based on the existing examples.

Error with some in rego:

tags_valid_pascal_case(tags) {
    some key
    # Foreach key and value, validate regex 
    val := tags[key]
    re_match(`^([A-Z][a-z0-9]+)+`, key)
    re_match(`^([A-Z][a-z0-9]+)+`, val)
}

results in the following error:

time="2019-06-19T13:13:16Z" level=fatal msg="Problem building rego compiler: 1 error occurred: main.rego:7: rego_parse_error: no match found\n\t    some key\n\t         ^"

Is this related to the interal OPA version?

Right now conftest compiles the Rego policies at the given directory. However, it does not recursively search that directory, which makes it hard to structure a set of policies.

For example I was trying the following directory structure for policies:

policy/
  terraform/
    aws/
      aws.rego
  kubernetes/
main.rego

surprisingly it did not error. However my policies just did not work as expected.

To reproduce try the following:

aws.rego:

package aws 

minimum_tags = {"ApplicationRole", "Owner", "Project"}

tags_contain_proper_keys(tags) {
    keys := {key | tags[key]}
    leftover := minimum_tags - keys
    leftover == set()
}

tags_contain_minimum_set[i] = resources {
    changeset := input.resource_changes[i]
    tags := changeset.change.after.tags
    resources := [resource | resource := changeset.address; not tags_contain_proper_keys(changeset.change.after.tags)]
}

main.rego

package main

import data.aws

deny[msg] {
    resources := aws.tags_contain_minimum_set[_]
    resources != []
    msg := sprintf("Invalid tags (missing minimum required tags) for the following resources: %v", [resources])
}

input.json

{
    "format_version": "0.1",
    "terraform_version": "0.12.4",
    "planned_values": {
        "root_module": {
            "resources": [
                {
                    "address": "aws_s3_bucket.profile_picture_storage",
                    "mode": "managed",
                    "type": "aws_s3_bucket",
                    "name": "profile_picture_storage",
                    "provider_name": "aws",
                    "schema_version": 0,
                    "values": {
                        "acl": "private",
                        "bucket_prefix": "profile-picture-storage",
                        "cors_rule": [],
                        "force_destroy": false,
                        "lifecycle_rule": [],
                        "logging": [],
                        "object_lock_configuration": [],
                        "policy": null,
                        "replication_configuration": [],
                        "server_side_encryption_configuration": [
                            {
                                "rule": [
                                    {
                                        "apply_server_side_encryption_by_default": [
                                            {
                                                "kms_master_key_id": null,
                                                "sse_algorithm": "AES256"
                                            }
                                        ]
                                    }
                                ]
                            }
                        ],
                        "tags": null,
                        "versioning": [
                            {
                                "enabled": true,
                                "mfa_delete": false
                            }
                        ],
                        "website": []
                    }
                },
                {
                    "address": "aws_s3_bucket_public_access_block.profile_picture_storage_access_rules",
                    "mode": "managed",
                    "type": "aws_s3_bucket_public_access_block",
                    "name": "profile_picture_storage_access_rules",
                    "provider_name": "aws",
                    "schema_version": 0,
                    "values": {
                        "block_public_acls": true,
                        "block_public_policy": true,
                        "ignore_public_acls": true,
                        "restrict_public_buckets": true
                    }
                }
            ]
        }
    },
    "resource_changes": [
        {
            "address": "aws_s3_bucket.profile_picture_storage",
            "mode": "managed",
            "type": "aws_s3_bucket",
            "name": "profile_picture_storage",
            "provider_name": "aws",
            "change": {
                "actions": [
                    "create"
                ],
                "before": null,
                "after": {
                    "acl": "private",
                    "bucket_prefix": "profile-picture-storage",
                    "cors_rule": [],
                    "force_destroy": false,
                    "lifecycle_rule": [],
                    "logging": [],
                    "object_lock_configuration": [],
                    "policy": null,
                    "replication_configuration": [],
                    "server_side_encryption_configuration": [
                        {
                            "rule": [
                                {
                                    "apply_server_side_encryption_by_default": [
                                        {
                                            "kms_master_key_id": null,
                                            "sse_algorithm": "AES256"
                                        }
                                    ]
                                }
                            ]
                        }
                    ],
                    "tags": null,
                    "versioning": [
                        {
                            "enabled": true,
                            "mfa_delete": false
                        }
                    ],
                    "website": []
                },
                "after_unknown": {
                    "acceleration_status": true,
                    "arn": true,
                    "bucket": true,
                    "bucket_domain_name": true,
                    "bucket_regional_domain_name": true,
                    "cors_rule": [],
                    "hosted_zone_id": true,
                    "id": true,
                    "lifecycle_rule": [],
                    "logging": [],
                    "object_lock_configuration": [],
                    "region": true,
                    "replication_configuration": [],
                    "request_payer": true,
                    "server_side_encryption_configuration": [
                        {
                            "rule": [
                                {
                                    "apply_server_side_encryption_by_default": [
                                        {}
                                    ]
                                }
                            ]
                        }
                    ],
                    "versioning": [
                        {}
                    ],
                    "website": [],
                    "website_domain": true,
                    "website_endpoint": true
                }
            }
        },
        {
            "address": "aws_s3_bucket_public_access_block.profile_picture_storage_access_rules",
            "mode": "managed",
            "type": "aws_s3_bucket_public_access_block",
            "name": "profile_picture_storage_access_rules",
            "provider_name": "aws",
            "change": {
                "actions": [
                    "create"
                ],
                "before": null,
                "after": {
                    "block_public_acls": true,
                    "block_public_policy": true,
                    "ignore_public_acls": true,
                    "restrict_public_buckets": true
                },
                "after_unknown": {
                    "bucket": true,
                    "id": true
                }
            }
        }
    ],
    "configuration": {
        "root_module": {
            "resources": [
                {
                    "address": "aws_s3_bucket.profile_picture_storage",
                    "mode": "managed",
                    "type": "aws_s3_bucket",
                    "name": "profile_picture_storage",
                    "provider_config_key": "aws",
                    "expressions": {
                        "acl": {
                            "constant_value": "private"
                        },
                        "bucket_prefix": {
                            "constant_value": "profile-picture-storage"
                        },
                        "server_side_encryption_configuration": [
                            {
                                "rule": [
                                    {
                                        "apply_server_side_encryption_by_default": [
                                            {
                                                "sse_algorithm": {
                                                    "constant_value": "AES256"
                                                }
                                            }
                                        ]
                                    }
                                ]
                            }
                        ],
                        "versioning": [
                            {
                                "enabled": {
                                    "constant_value": true
                                }
                            }
                        ]
                    },
                    "schema_version": 0
                },
                {
                    "address": "aws_s3_bucket_public_access_block.profile_picture_storage_access_rules",
                    "mode": "managed",
                    "type": "aws_s3_bucket_public_access_block",
                    "name": "profile_picture_storage_access_rules",
                    "provider_config_key": "aws",
                    "expressions": {
                        "block_public_acls": {
                            "constant_value": true
                        },
                        "block_public_policy": {
                            "constant_value": true
                        },
                        "bucket": {
                            "references": [
                                "aws_s3_bucket.profile_picture_storage"
                            ]
                        },
                        "ignore_public_acls": {
                            "constant_value": true
                        },
                        "restrict_public_buckets": {
                            "constant_value": true
                        }
                    },
                    "schema_version": 0
                }
            ]
        }
    }
}

execute the following command with the policies stored in a policy dir:

conftest test input.json

When the policies are stored in the same directory this input will return an error. Otherwise it does not return anything (Not even an error).

The buildCompiler method in the test command should be updated to recursively search for subdirectories when compiling the rego policies.

Hi,

All moderns CI tools seems to take the "modular path" where each job step is an OCI image and an overrided entrypoint to customize the command.

Personally I like it and I think Conftest, which as a lot of CI use-cases, make it difficult for remote policies.

Today you have to setup a 2 steps job:

• conftest pull instrumenta.azurecr.io/test then conftest test ...
• echo "[[policies]]..." > conftest.toml then conftest test --update ...
• oras pull bundle.bar/opa/instrumenta/kubernetes:latest -a then conftest test -p kubernetes.rego ...

I haven't UX suggestion atm but I think a one-liner like conftest test -p oci://bundle.bar/opa/instrumenta/kubernetes:latest ... could be a great feature

Reproduction steps:

1. Start a local docker registry:

docker run -it --rm -p 5000:5000 registry

2. push a set of policies with Conftest (version 0.11)

conftest push localhost:5000/policies:latest

results:

conftest push localhost:5000/policies:latest
time="2019-08-10T14:19:31Z" level=fatal msg="failed to do request: Head http://localhost:5000/v2/policies/blobs/sha256:086ee78899cc6be63d638028055085630d199198424063098fee3c47383928e4: dial tcp 127.0.0.1:5000: connect: connection refused"

With ORAS it does succeed:

oras push localhost:5000/policies:latest ./policy

Uploading e05080fb451c policy
Pushed localhost:5000/policies:latest
Digest: sha256:28d7f27c0158b6df69ce40176ba9da081c6c49be9c450324e06e80d5bf3251c8

oras pull localhost:5000/policies:latest  --output ./tmp

Downloaded e05080fb451c policy
Pulled localhost:5000/policies:latest
Digest: sha256:28d7f27c0158b6df69ce40176ba9da081c6c49be9c450324e06e80d5bf3251c8

Hi! We've started using conftest for evaluating our K8S YAML files last week, and it's been a nice way to encode certain conventions we expect our configurations to follow.

One thing that we immediately noticed we wanted but was not possible was to be able to verify fields across documents. For example, a GCE Ingress's service port needs to align with the backing ports in the service/deployment. Or, a deployment that pulls in values from a configmap needs to match the metadata names.

I took a quick skim of the code, and in general, it appears that it would require restructuring a bit to support this (the input object may need to be more akin to a map or an array), but doing so would unlock some serious configuration checking super powers.

Thanks!
Brian

As we both add more features, more examples and more integrations (Helm, GitHub Actions, etc.) the README won't be the best place to put all the documentation.

It would be good to create a separate docs site with multiple pages.

I've had good luck with MkDocs (https://www.mkdocs.org/) before, for instance with
https://github.com/instrumenta/kubeval. This just involves a simple config file and a directory of markdown files managed in the same repo.