optiopay / klar Goto Github PK

View Code? Open in Web Editor NEW

506.0 23.0 140.0 1.82 MB

Integration of Clair and Docker Registry

License: MIT License

Go 99.42% Dockerfile 0.58%

security clair severity-vulnerabilities docker-registry docker-image security-audit

klar's Introduction

Klar

Integration of Clair and Docker Registry (supports both Clair API v1 and v3)

Klar is a simple tool to analyze images stored in a private or public Docker registry for security vulnerabilities using Clair https://github.com/coreos/clair. Klar is designed to be used as an integration tool so it relies on enviroment variables. It's a single binary which requires no dependencies.

Klar serves as a client which coordinates the image checks between the Docker registry and Clair.

Binary installation

The simplest way is to download the latest release (for OSX and Linux) from https://github.com/optiopay/klar/releases/ and put the binary in a folder in your PATH (make sure it has execute permission).

Installation from source code

Make sure you have Go language compiler installed and configured https://golang.org/doc/install

Then run

go get github.com/optiopay/klar

make sure your Go binary folder is in your PATH (e.g. export PATH=$PATH:/usr/local/go/bin)

Usage

Klar process returns 0 if the number of detected high severity vulnerabilities in an image is less than or equal to a threshold (see below) and 1 if there were more. It will return 2 if an error has prevented the image from being analyzed.

Klar can be configured via the following environment variables:

CLAIR_ADDR - address of Clair server. It has a form of protocol://host:port - protocol and port default to http and 6060 respectively and may be omitted. You can also specify basic authentication in the URL: protocol://login:password@host:port.
CLAIR_OUTPUT - severity level threshold, vulnerabilities with severity level higher than or equal to this threshold will be outputted. Supported levels are Unknown, Negligible, Low, Medium, High, Critical, Defcon1. Default is Unknown.
CLAIR_THRESHOLD - how many outputted vulnerabilities Klar can tolerate before returning 1. Default is 0.
CLAIR_TIMEOUT - timeout in minutes before Klar cancels the image scanning. Default is 1
DOCKER_USER - Docker registry account name.
DOCKER_PASSWORD - Docker registry account password.
DOCKER_TOKEN - Docker registry account token. (Can be used in place of DOCKER_USER and DOCKER_PASSWORD)
DOCKER_INSECURE - Allow Klar to access registries with bad SSL certificates. Default is false. Clair will need to be booted with -insecure-tls for this to work.
DOCKER_TIMEOUT - timeout in minutes when trying to fetch layers from a docker registry
DOCKER_PLATFORM_OS - The operating system of the Docker image. Default is linux. This only needs to be set if the image specified references a Docker ManifestList instead of a usual manifest.
DOCKER_PLATFORM_ARCH - The architecture the Docker image is optimized for. Default is amd64. This only needs to be set if the image specified references a Docker ManifestList instead of a usual manifest.
REGISTRY_INSECURE - Allow Klar to access insecure registries (HTTP only). Default is false.
JSON_OUTPUT - Output JSON, not plain text. Default is false.
FORMAT_OUTPUT - Output format of the vulnerabilities. Supported formats are standard, json, table. Default is standard. If JSON_OUTPUT is set to true, this option is ignored.
WHITELIST_FILE - Path to the YAML file with the CVE whitelist. Look at whitelist-example.yaml for the file format.
IGNORE_UNFIXED - Do not count vulnerabilities without a fix towards the threshold

Usage:

CLAIR_ADDR=localhost CLAIR_OUTPUT=High CLAIR_THRESHOLD=10 DOCKER_USER=docker DOCKER_PASSWORD=secret klar postgres:9.5.1

Debug Output

You can enable more verbose output but setting KLAR_TRACE to true.

run export KLAR_TRACE=true to persist between runs.

Dockerized version

Klar can be dockerized. Go to $GOPATH/src/github.com/optiopay/klar and build Klar in project root. If you are on Linux:

CGO_ENABLED=0 go build -a -installsuffix cgo .

If you are on Mac don't forget to build it for Linux:

GOOS=linux go build .

To build Docker image run in the project root (replace klar with fully qualified name if you like):

docker build -t klar .

Then pass env vars as separate --env arguments, or create an env file and pass it as --env-file argument. For example save env vars as my-klar.env:

CLAIR_ADDR=localhost
CLAIR_OUTPUT=High
CLAIR_THRESHOLD=10
DOCKER_USER=docker
DOCKER_PASSWORD=secret

Then run

docker run --env-file=my-klar.env klar postgres:9.5.1

Amazon ECR support

There is no permanent username/password for Amazon ECR, the credentials must be retrived using aws ecr get-login and they are valid for 12 hours. Here is a sample script which may be used to provide Klar with ECR credentials:

DOCKER_LOGIN=`aws ecr get-login --no-include-email`
PASSWORD=`echo $DOCKER_LOGIN | cut -d' ' -f6`
REGISTRY=`echo $DOCKER_LOGIN | cut -d' ' -f7 | sed "s/https:\/\///"`
DOCKER_USER=AWS DOCKER_PASSWORD=${PASSWORD} ./klar ${REGISTRY}/my-image

Google GCR support

For authentication against GCR (Google Cloud Registry), the easiest way is to use the application default credentials. These only work when running Klar from GCP. The only requirement is the Google Cloud SDK.

DOCKER_USER=oauth2accesstoken
DOCKER_PASSWORD="$(gcloud auth application-default print-access-token)"

With Docker:

DOCKER_USER=oauth2accesstoken
DOCKER_PASSWORD="$(docker run --rm google/cloud-sdk:alpine gcloud auth application-default print-access-token)"

klar's People

Stargazers

Watchers

Forkers

alpe pombredanne supereagle nivlaaa solidnerd dieterdemeyer furuholm sublimino millidavids caruccio electric-it yash069 xueshanf wtlecit1 owms flix-tech bougar stakater joakimlofgren tbarnier lli-hiya antxxxx r0610205 agius bunnyyiu vadim-codefresh codefresh-io austbot cplee ccsalway jaumann tamalsaha adamnoll kinihun atsushi-ishibashi quorauk johannestegner thereverendtom vivek-sixt opt9 mosoriob xtal0 leahnp mbode dangerousben dustinvanbuskirk smsantana tsaarni cafuego victoriza neumayer vitaliibalashov spicysouvlaki pnordh sudhirpandey nabadger sriddell jeff-knurek qiq scopej leslieattwell chessracer khj0651 rererecursive naibaf0 thredup richahasija karthiknannapaneni ibrokethecloud rleavengood-scwx ayoneeyee cleveritcz rkhozinov heww bluelabsio the-smooth-operator rox-wolf intento nanosecco vickatte dcvasi ericrpowers mmerrill3 faust64 greut aminueza gonfva-bcl jepsenwan jmferrer ethiansjh hemanthg781 orlando-jamie portshift-admin https-github-com-tigareksaperdana-co-id rafiportshift vanko951 intfrr amchau zdiyax 5l1v3r1

klar's Issues

Getting "Can't pull image"

We installed klar but unable to run it successfully. We have local ubuntu image that we wanted to scan. Our clair is running on Docker container (172.17.0.3). We tried following:

CLAIR_ADDR=http://172.17.0.3:6060 CLAIR_THRESHOLD=1 klar ubuntu:latest

and received following:
Can't pull image: Token request returned 401

We also tried to scan public image:
klar ubuntu:latest
and received the same error: Can't pull image: Token request returned 401

Are we missing some config parameters or env variables?
How would we run klar to scan images in public and/or private registry?

Error 500 while testing images in private registry(gitlab)

I am getting the following errors while analysing an image with the docker images built. Also I am getting the similar error in standalone installation. Any idea about the errors?

Analysing 10 layers
Push layer 0 failed: Can't even read an error message: invalid character '<' looking for beginning of value
Push layer 1 failed: Can't even read an error message: invalid character '<' looking for beginning of value
Push layer 2 failed: Can't even read an error message: invalid character '<' looking for beginning of value
Push layer 3 failed: Can't even read an error message: invalid character '<' looking for beginning of value
Push layer 4 failed: Can't even read an error message: invalid character '<' looking for beginning of value
Push layer 5 failed: Can't even read an error message: invalid character '<' looking for beginning of value
Push layer 6 failed: Can't even read an error message: invalid character '<' looking for beginning of value
Push layer 7 failed: Can't even read an error message: invalid character '<' looking for beginning of value
Push layer 8 failed: Can't even read an error message: invalid character '<' looking for beginning of value
Push layer 9 failed: Can't even read an error message: invalid character '<' looking for beginning of value
Analyse image https://images.myreg.com/v2/group/project:apache2.2-00 failed: Analyze error 500:

<title>Error 500 Server Error</title>

HTTP ERROR 500

Problem accessing /v1/layers/0feb49bb03b8e8456fcab2f9f589a408421081619f8ea41dc9deb6a96b740a36af63486e1262fdb56303221b45ec4167699cd7e533ad5a8bc5abff693bd09828. Reason:

    Server Error

Caused by:

javax.servlet.ServletException: javax.servlet.ServletException: An error javax.ws.rs.NotFoundException occurred. The original cause was: RESTEASY003210: Could not find resource for full path: https://images.myreg.com/v1/layers/0feb49bb03b8e8456fcab2f9f589a408421081619f8ea41dc9deb6a96b740a36af63486e1262fdb56303221b45ec4167699cd7e533ad5a8bc5abff693bd09828?vulnerabilities

Support for Image Manifest v2

As stated in PR #21 there have been braking changes to the manifest format when moving from "Image Manifest V 2, Schema 1" [1] to "Image Manifest V 2, Schema 2" [2].

The PR addressed requesting the new schema version, but there are more changes needed. This is probably causing the bug in #22 (sorry!).

The following changes to the manifest fields are relevant for Klar:

name has been removed
tag has been removed
fsLayers has been renamed layers
fsLayers.blobSum has been renamed layers.digest

PR #21 handles 1 and 2, but not 3 and 4.

The decoding of the manifest needs to be treated differently btw v1 and v2. The question is if the v1 or v2 terminology should be used. I can provide a new PR once this has been decided.

[1] - https://docs.docker.com/registry/spec/manifest-v2-1/#manifest-field-descriptions
[2] - https://docs.docker.com/registry/spec/manifest-v2-2/#image-manifest-field-descriptions

ECR Failed to analyze using API v1 and v3 errors

It's certainly not really clear whats happening but

used helm from contrib of clair to deploy into a kubernetes cluster... pods are up and a public service endpoint ELB is up via AWS (postgres backend for clair)

assuming one follows directions on klar setup for ECR (which by the way could use some help in clarity - ie: its missing the fact you need to define CLAIR_ADDR) ..

i run the below:

CLAIR_ADDR=SOME_ELB_THING.us-east-1.elb.amazonaws.com DOCKER_USER=AWS DOCKER_PASSWORD=${PASSWORD} /usr/bin/klar  ${REGISTRY}/myapp:12345

output:

Analysing 12 layers
Failed to analyze using API v1: push image https://AWSACCOUNTID.dkr.ecr.us-east-1.amazonaws.com/v2/myapp:12345 to Clair failed: can't even read an error message: invalid character 'N' looking for beginning of value

Failed to analyze using API v3: push image https://AWSACCOUNTID.dkr.ecr.us-east-1.amazonaws.com/v2/myapp:12345 to Clair failed: rpc error: code = Unavailable desc = all SubConns are in TransientFailure

Failed to analyze, exiting

all clair sees according to logs is:

{"Event":"Handled HTTP request","Level":"info","Location":"server.go:105","Time":"2017-12-19 18:24:49.860861","elapsed time (ms)":0.018253,"method":"POST","remote addr":"172.20.7.43:9931","request uri":"/v1/layers","status":"404"}

cannot build code with go in linux

Cannot build code with go in linux environment.

go build .
main.go:10:2: cannot find package "github.com/optiopay/klar/clair" in any of:
/usr/local/go/src/github.com/optiopay/klar/clair (from $GOROOT)
/root/go/src/github.com/optiopay/klar/clair (from $GOPATH)
main.go:11:2: cannot find package "github.com/optiopay/klar/docker" in any of:
/usr/local/go/src/github.com/optiopay/klar/docker (from $GOROOT)
/root/go/src/github.com/optiopay/klar/docker (from $GOPATH)

Filter out unstable Debian issues

A lot of errors are unfixable because the fixes are in unreleased versions of Debian. Please allow filtering out of those results. Thanks.

Docker run failing

Docker run on klar is failing.
The docker file content :
FROM scratch

ENV DOMAIN=skydns.local
ENV RELEASE 0.1

Required by golang's time pkg

ENV ZONE_INFO /zoneinfo.zip
COPY assets/zoneinfo.zip /

Required for SSL

COPY assets/ca-bundle.crt /etc/ssl/certs/ca-certificates.crt

COPY klar /

ENTRYPOINT ["/klar"]
CMD [""]

=======

and output for this is:-
docker run --env-file=my-klar.env klar postgres:9.5.1
standard_init_linux.go:178: exec user process caused "no such file or directory"

False positive

I've detected that when clair has a certain problem and can't pull the image layers, klar doesn't detect it as a fail:

DOCKER_USER=AWS DOCKER_PASSWORD=${PASSWORD} CLAIR_ADDR=http://clair.whatever.nope klar $private_registry/$image
Analysing 8 layers
Push layer 0 failed: Can't push layer to Clair: Post http://clair.whatever.nope:6060/v1/layers: dial tcp 10.10.10.10:6060: i/o timeout
Push layer 1 failed: Can't push layer to Clair: Post http://clair.whatever.nope:6060/v1/layers: dial tcp 10.10.10.10:6060: i/o timeout
Push layer 2 failed: Can't push layer to Clair: Post http://clair.whatever.nope:6060/v1/layers: dial tcp 10.10.10.10:6060: i/o timeout
Push layer 3 failed: Can't push layer to Clair: Post http://clair.whatever.nope:6060/v1/layers: dial tcp 10.10.10.10:6060: i/o timeout
Push layer 4 failed: Can't push layer to Clair: Post http://clair.whatever.nope:6060/v1/layers: dial tcp 10.10.10.10:6060: i/o timeout
Push layer 5 failed: Can't push layer to Clair: Post http://clair.whatever.nope:6060/v1/layers: dial tcp 10.10.10.10:6060: i/o timeout
Push layer 6 failed: Can't push layer to Clair: Post http://clair.whatever.nope:6060/v1/layers: dial tcp 10.10.10.10:6060: i/o timeout
Push layer 7 failed: Can't push layer to Clair: Post http://clair.whatever.nope:6060/v1/layers: dial tcp 10.10.10.10:6060: i/o timeout
Analyse image https:/whatever-registry:latest failed: Get http://clair.whatever.nope:6060/v1/layers/7ad37577aa1ca7a2b30e1bb6ffffaasd1236ccee1cce45ff8f33ed1fa2659bc9cabf089e1591dec74d10c3128c063a798d4fc990583a7ab86ff766e11ff599ec4?vulnerabilities: dial tcp 10.10.10.10:6060: i/o timeout
Found 0 vulnerabilities

Support basic authentication

It would be nice to support basic authentication, since the default behavior is to be open to everybody.

Help understanding ECR command

DOCKER_LOGIN=aws ecr get-login
PASSWORD=echo $DOCKER_LOGIN | cut -d' ' -f6
REGISTRY=echo $DOCKER_LOGIN | cut -d' ' -f9 | sed "s/https:\/\///"
DOCKER_USER=AWS DOCKER_PASSWORD=${PASSWORD} ./klar ${REGISTRY}/my-image

What is sed doing here? Is it possible to scan all images in the registry at once instead of just a specific image?

ECR Role integration

Currently utilizing klar, and would love to run it hand and hand with something such as amazon-ecr-credential-helper. With this you can utilize a role associated with your node that gives it permissions to access ECR without having to refederate with ecr get-login for example. However, you currently have to use docker user aws/docker password. Is there any way to bypass this?

Klar does not give layers globally unique names

Originally posted this issue as a Clair issue but per the discussion on quay/clair#439 it appears this is an issue with Klar so I am submitting it here.

When using Klar to scan public images from Docker Hub I often receive a 422 error when POSTing layers to Clair. Some examples of popular and public images I cannot scan are tomcat:7-jre8, postgres:9.5.3 (versions prior to 9.5.3 work but later versions do not), python:3.5, nginx:1.12-alpine, redis:3, nginx:3.5, and openjdk:7.

Clair will log a warning about a different feature name and version each time when the offending layer is POSTed in. For example, running klar tomcat:7-jre8
I will see the following line in the Clair logs

{"Event":"Namespace unknown","Level":"warning","Location":"worker.go:211","Time":"2017-08-28 20:13:36.031713","feature name":"init-system-helpers","feature version":"1.48","layer":"sha256:eb9b7457396558f37d230d8cb10dc083fb304895030d40ce8b54598e176546ec"}
{"Event":"Handled HTTP request","Level":"info","Location":"router.go:57","Time":"2017-08-28 20:13:36.031956","elapsed time":316097842,"method":"POST","remote addr":"10.124.3.72:23672","request uri":"/v1/layers","status":"422"}

Per this comment, this appears to be caused by Klar naming layers using the sha hash instead of using globally unique names as required by the Clair V2 API.

json output

Since in clair.go the function analyse returns a json, I would like to ask if it is possible to extend main.go from line 60 to 67 in order to let the user decide if she wants a json output or a human readable text.

A json output will integrate klar in many solutions since the results can be showed in any fashion you like.

Push layer n failed: Can't even read an error message: invalid character 'N' looking for beginning of value

I been trying to scan local images and I get;

$ docker images | grep debian
debian                                      wheezy              0ec46eb38976        4 days ago          85.1MB

$ CLAIR_ADDR=192.168.99.100 CLAIR_OUTPUT=High CLAIR_THRESHOLD=10 klar debian         
Analysing 1 layers
Push layer 0 failed: Can't even read an error message: invalid character 'N' looking for beginning of value
Analyse image https://registry-1.docker.io/v2/library/debian:latest failed: Analyze error 404: Not Found

Found 0 vulnerabilities

$

Same for remote images:

$ CLAIR_ADDR=192.168.99.100 CLAIR_OUTPUT=High CLAIR_THRESHOLD=10 klar postgres:latest
Analysing 12 layers
Push layer 11 failed: Can't even read an error message: invalid character 'N' looking for beginning of value
Push layer 10 failed: Can't even read an error message: invalid character 'N' looking for beginning of value
Push layer 9 failed: Can't even read an error message: invalid character 'N' looking for beginning of value
Push layer 8 failed: Can't even read an error message: invalid character 'N' looking for beginning of value
Push layer 7 failed: Can't even read an error message: invalid character 'N' looking for beginning of value
Push layer 6 failed: Can't even read an error message: invalid character 'N' looking for beginning of value
Push layer 5 failed: Can't even read an error message: invalid character 'N' looking for beginning of value
Push layer 4 failed: Can't even read an error message: invalid character 'N' looking for beginning of value
Push layer 3 failed: Can't even read an error message: invalid character 'N' looking for beginning of value
Push layer 2 failed: Can't even read an error message: invalid character 'N' looking for beginning of value
Push layer 1 failed: Can't even read an error message: invalid character 'N' looking for beginning of value
Push layer 0 failed: Can't even read an error message: invalid character 'N' looking for beginning of value
Analyse image https://registry-1.docker.io/v2/library/postgres:latest failed: Analyze error 404: Not Found

Found 0 vulnerabilities

$ docker pull postgres:latest
latest: Pulling from library/postgres
aa18ad1a0d33: Pull complete 
986b6272a22e: Pull complete 
a23afadd4a20: Pull complete 
b6de223fb1ca: Pull complete 
9656638a0c77: Pull complete 
46a1c4f90191: Pull complete 
ac765cd48f50: Pull complete 
c48f2447ba76: Pull complete 
affdbd879518: Pull complete 
d9cad1b8a255: Pull complete 
72d207c99115: Pull complete 
0fea6738d3c5: Pull complete 
Digest: sha256:2f8080b9910a8b4f38ff5a55a82e77cb43d88bdbb16d723c71d18493590832e9
Status: Downloaded newer image for postgres:latest

$ CLAIR_ADDR=192.168.99.100 CLAIR_OUTPUT=High CLAIR_THRESHOLD=10 klar postgres:latest
Analysing 12 layers
Push layer 11 failed: Can't even read an error message: invalid character 'N' looking for beginning of value
Push layer 10 failed: Can't even read an error message: invalid character 'N' looking for beginning of value
Push layer 9 failed: Can't even read an error message: invalid character 'N' looking for beginning of value
Push layer 8 failed: Can't even read an error message: invalid character 'N' looking for beginning of value
Push layer 7 failed: Can't even read an error message: invalid character 'N' looking for beginning of value
Push layer 6 failed: Can't even read an error message: invalid character 'N' looking for beginning of value
Push layer 5 failed: Can't even read an error message: invalid character 'N' looking for beginning of value
Push layer 4 failed: Can't even read an error message: invalid character 'N' looking for beginning of value
Push layer 3 failed: Can't even read an error message: invalid character 'N' looking for beginning of value
Push layer 2 failed: Can't even read an error message: invalid character 'N' looking for beginning of value
Push layer 1 failed: Can't even read an error message: invalid character 'N' looking for beginning of value
Push layer 0 failed: Can't even read an error message: invalid character 'N' looking for beginning of value
Analyse image https://registry-1.docker.io/v2/library/postgres:latest failed: Analyze error 404: Not Found

Found 0 vulnerabilities
$

Is this a bug or am I doing something wrong?

Talking to ECR times out.

I have been trying to get Klar to pull images from our ECR in Amazon, but it simply times out. We can perform a docker pull just fine, but Klar doesn't work.

[ec2-user@ip-??-??-123-123 ~]$ CLAIR_ADDR=$REGISTRY CLAIR_OUTPUT=High CLAIR_THRESHOLD=10 DOCKER_USER=AWS DOCKER_PASSWORD=$PASSWORD ./klar-1.5-linux-amd64 123456789.dkr.ecr.eu-west-1.amazonaws.com/postgres
Get error
Can't pull image: Get https://123456789.dkr.ecr.eu-west-1.amazonaws.com/v2/postgres/manifests/latest: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Is this a problem with Klar, are are we using it in the wrong way? Should we instead be building our images locally, and then scanning using a local repo BEFORE we push into ECR?

Proposal: Get the vulnerabilities of the image according to its non-empty top layer

Now the vulnerabilities of the image is got layer by layer, but there maybe are two problems:

If one feature is added in the parent layer, but removed in the children layer, the vulnerabilities of this feature will be in the result.
Some vulnerabilities of the parent layer are duplicated in the result as they will be got again by their children.

The Get Layer api of Clair will list of features indexed in this layer and all of its parents. It will merge this layer's and its parents' vulnerabilities, considering the above two problems. So we can get the full vulnerabilities of the image just according to its non-empty top layer rather than layer by layer.
As the empty layers in the image manifest will break the relationship between the parent layers and their children, so they should be skipped when post layers to Clair. At the same time, the full vulnerabilities of the image should be got according to its non-empty top layer.

@hashmap Any ideas about this proposal? If you think it is ok, I am glad to contribute this enhancement.

Create binaries for version 1.5

Currently release 1.5 only has source code but not the binaries.
The previous version have source as well as binaries.
Can you make the binaires for 1.5 available for download.

Default port not appended for API V3

Port 6060 is appended to the url if no port has been specified for API V1. This is not the case for API V3.

localhost:5000 registry

We run private registry on localhost:5000, without username/password. Can Klar be used in this case?

Introduce integration tests

This is important part of the test suite, unfortunately, I'm not sure how to make it opensource. My current test is a shell script:

source my-klar-env
./klar mysql
./klar postgres:9.6
./klar postgres@sha256:1364924c753d5ff7e2260cd34dc4ba05ebd40ee8193391220be0f9901d4e1651
./klar skynetservices/skydns
./klar skynetservices/skydns:2.5.3a
./klar some.private.registry/user/image
./klar some.private.registry/user/image:tag

The problem is in my-klar-env which contains docker registry creds. It could be a local file which each developer must customize. However it would be nice to make it a part of Travis CI build, so user account must be exposed.

We need it to run tests twice with different env files - with and without docker creds, scanning only public images in the second case.

Can't pull fsLayers Error on 1.3

After upgrading to 1.3 I'm getting "Can't pull fsLayers" error on some images

1.2.1 works fine for these same images.

What information do you need from me?

/opt/klar-1.2.1 <ecr registry / image >
Analysing 28 layers
Found 11 vulnerabilities
...

/opt/klar-1.3 <ecr registry / image >
Can't pull fsLayers

Are there any plans to support proxy?

We are trying to run Klar against docker images stored in AWS ECR. However, all our outbound traffic needs to go out via Proxy server which Klar currently does not recognize. Are there any plans to support proxy capabilities?

Analyzing a local image fails with `Can't pull image: Token request returned 401`

I tried to analyze a local Docker image on macOS using Docker for Mac and it failed with the following error: Can't pull image: Token request returned 401

In the near future we would like to use klar inside a dind container, after building the Docker image and before pushing it into our internal registry.

Can't pull fsLayers with public GCR image

I've deployed Clair into a Kubernetes cluster from the released quay.io/coreos/clair:v2.0.1 Docker image. I'm trying to use Klar 2.0.1 to scan a public image hosted on gcr.io, Google's container registry. But when I do that, I'm getting the Can't pull fsLayers error:

Here's the command I'm running:

CLAIR_ADDR=http://localhost:6060 ./klar-2.0.1-linux-amd64 gcr.io/google_containers/addon-resizer:1.0

I get this error for any public gcr.io image, while images hosted elsewhere scan just fine. Note that I'm not running any of this on GCP and I'm not using a private image, so I'm not authenticating with the instructions here: https://github.com/optiopay/klar#google-gcr-support

JSON output invalid

When trying to utilise the JSON output I have issues.

$ CLAIR_ADDR=localhost JSON_OUTPUT=true ./klar-2.0.1-linux-amd64 httpd | jq .
parse error: Invalid numeric literal at line 1, column 4

Looking at the output it's a non-json line at the top:

$ CLAIR_ADDR=localhost JSON_OUTPUT=true ./klar-2.0.1-linux-amd64 httpd | head -n1
Got results from Clair API v1

Probably would be good to omit this message or include it into the JSON stream.

Examples for citicality

I am using clairctl, but the output is not as nice structured as klar. In clairctl, I get the overview of ciritcalities like this:
Unknown: x
Negligible: x
Low: x
Medium: x
High: x
Critical: x
Defcon1: x

I asume that klar useses the same ciriticalities? To integrate klar into my Jenkins with the logparser plugin, I need to know what to grep for.

Scan GCR public container images

I am wondering how to scan public images hosted on GCR, e.g. gcr.io/google-containers/busybox. No authentication is needed in this case, but klar would give this error:

docker run --env-file=envs my-klar gcr.io/google-containers/busybox
Can't pull image: Can't parse Www-Authenticate: Basic realm=""

My envs contains no DOCKER_USER and DOCKER_PASSWORD:

CLAIR_ADDR=https://clair.example.com

Thanks!

klar getting 404 from clair?

Not sure if this is something I might be doing wrong.

Below I'm attempting to run a test against public centos container image.

Clair/Klar/Docker/Registry are all running locally in a single linux VM with --net=host to simplify it all.

I'm using docker registry:2 (28525f9a6e46), klar (tried git:1e20efb and 1.5-rc2), and clair:latest (be223c092e09).

----> HTTP REQUEST:
GET /v2/centos/manifests/latest HTTP/1.1
Host: localhost:5000
Accept: application/vnd.docker.distribution.manifest.v2+json


<---- HTTP RESPONSE:
HTTP/1.1 200 OK
Content-Length: 529
Content-Type: application/vnd.docker.distribution.manifest.v2+json
Date: Wed, 11 Oct 2017 01:59:46 GMT
Docker-Content-Digest: sha256:822de5245dc5b659df56dd32795b08ae42db4cc901f3462fc509e91e97132dc0
Docker-Distribution-Api-Version: registry/2.0
Etag: "sha256:822de5245dc5b659df56dd32795b08ae42db4cc901f3462fc509e91e97132dc0"
X-Content-Type-Options: nosniff

{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
   "config": {
      "mediaType": "application/vnd.docker.container.image.v1+json",
      "size": 1863,
      "digest": "sha256:196e0ce0c9fbb31da595b893dd39bc9fd4aa78a474bbdc21459a3ebe855b7768"
   },
   "layers": [
      {
         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
         "size": 73386947,
         "digest": "sha256:d9aaf4d82f249dc101a6638ff5177fe926cdebfa6c42d874dfa5029533da0e72"
      }
   ]
}
Analysing 1 layers
----> HTTP REQUEST:
POST /v1/layers HTTP/1.1
Host: localhost:6060
Content-Type: application/json

{"Layer":{"Name":"196e0ce0c9fbb31da595b893dd39bc9fd4aa78a474bbdc21459a3ebe855b7768d9aaf4d82f249dc101a6638ff5177fe926cdebfa6c42d874dfa5029533da0e72","Path":"http://localhost:5000/v2/centos/blobs/sha256:d9aaf4d82f249dc101a6638ff5177fe926cdebfa6c42d874dfa5029533da0e72","ParentName":"","Format":"Docker","Features":null,"Headers":{"Authorization":""}}}
<---- HTTP RESPONSE:
HTTP/1.1 404 Not Found
Content-Length: 10
Content-Type: text/plain; charset=utf-8
Date: Wed, 11 Oct 2017 01:59:46 GMT
X-Content-Type-Options: nosniff

Not Found

Push layer 0 failed: Can't even read an error message: invalid character 'N' looking for beginning of value
----> HTTP REQUEST:
GET /v1/layers/196e0ce0c9fbb31da595b893dd39bc9fd4aa78a474bbdc21459a3ebe855b7768d9aaf4d82f249dc101a6638ff5177fe926cdebfa6c42d874dfa5029533da0e72?vulnerabilities HTTP/1.1
Host: localhost:6060


<---- HTTP RESPONSE:
HTTP/1.1 404 Not Found
Content-Length: 10
Content-Type: text/plain; charset=utf-8
Date: Wed, 11 Oct 2017 01:59:46 GMT
X-Content-Type-Options: nosniff

Not Found

Analyse image http://localhost:5000/v2/centos:latest failed: Analyze error 404: Not Found

Found 0 vulnerabilities

Output klar version?

I don't see support for a klar version CLI command of any kind. It would be nice.

Header Authorization for request to clair: No token is set

Hello developers,

if i use klaras desrcibed:

CLAIR_ADDR=http://localhost CLAIR_THRESHOLD=10 DOCKER_USER=me DOCKER_PASSWORD=secret klar postgres:9.5.1

with user and password these parameters are sue to get the dockerImages.
But if each layer is analyzed by clair it usese the token which is never initialized.

I have set the token in the docker.go by

if i.Token == "" { req.SetBasicAuth(i.user, i.password) i.Token = req.Header.Get("Authorization") }

Did i use the klar wrong or is there an other way to call klar ?

Thank you for your help.

Greatings
Sascha

Possible to scan more than one image at a time?

Is Klar able to scan all images in my EC2 Repository at once? Or does each image have to be scanned individually?

Add tests for functionality in main.go

It has been grown and should be refactored and covered by tests. Major part of the code may be moved to klar.go

Default port documentation is not consistent when using https

Hi,

First of all, thanks for the development of this CLI tool, it makes clair much easier to integrated inside a CI/CD workflow :)

The current documentation for CLAIR_ADDR in the usage section of the README indicates that the port 6060 is used by default if it is not set.

This is not the current behavior of the code (despite similar comments), as port 6060 will be set by default only for http scheme, as this condition is false for https due to the fact that the : character will be on 5th position due to the extra character.

Do you prefer to

Fix the code to match the documentation (by transforming the < into <=)? This would break the current behavior and force people to specify port 443 in their configuration if they update their version of klar. I can make the PR if that's the chosen solution.

Update the documentation to match the current version of the code? No breaking changes, but documentation will be harder to read since the behavior will look a little inconsistent between http and https (why would one set a default port and not the other?)

Regards,
Michael

dial tcp [::1]:6060: getsockopt: connection refused

I have a clair container running locally and this works:

curl http://localhost:6060/v1/namespaces/debian%3A8/vulnerabilities?limit=2
{
  "Vulnerabilities": [
    {
      "Name": "CVE-2011-3887",
      "NamespaceName": "debian:8",
      "Description": "Google Chrome before 15.0.874.102 does not properly handle javascript: URLs, which allows remote attackers to bypass intended access restrictions and read cookies via unspecified vectors.",
      "Link": "https://security-tracker.debian.org/tracker/CVE-2011-3887",
      "Severity": "Medium",
      "Metadata": {
        "NVD": {
          "CVSSv2": {
            "Score": 5,
            "Vectors": "AV:N/AC:L/Au:N/C:P/I:N"
          }
        }
      }
    },

...

However, when I run klar container:

CLAIR_ADDR=http://localhost:6060
CLAIR_OUTPUT=High
CLAIR_THRESHOLD=10
DOCKER_USER=xxxxxxx
DOCKER_PASSWORD=xxxxxx

docker run --env-file=envvars my-klar registry
Analysing 10 layers
Push layer 4 failed: Can't push layer to Clair: Post http://localhost:6060/v1/layers: dial tcp [::1]:6060: getsockopt: connection refused
Push layer 3 failed: Can't push layer to Clair: Post http://localhost:6060/v1/layers: dial tcp [::1]:6060: getsockopt: connection refused
Push layer 2 failed: Can't push layer to Clair: Post http://localhost:6060/v1/layers: dial tcp [::1]:6060: getsockopt: connection refused
Push layer 1 failed: Can't push layer to Clair: Post http://localhost:6060/v1/layers: dial tcp [::1]:6060: getsockopt: connection refused
Push layer 0 failed: Can't push layer to Clair: Post http://localhost:6060/v1/layers: dial tcp [::1]:6060: getsockopt: connection refused
Analyse image https://registry-1.docker.io/v2/library/registry:latest failed: Get http://localhost:6060/v1/layers/sha256:154ef19ddee64edd702ab722c0bee283cf34eab32dde16243bc98adef65b2b52?vulnerabilities: dial tcp [::1]:6060: getsockopt: connection refused
Found 0 vulnerabilities

The clair container log doesn't show any connections. Any ideas?

Centos vulnerability still failing with fixed by solution implemented

We have a centos image that we've updated with the fixed by in the below block. The clair scan via klar is still failing. We've done a lot to debug, including running the new images and printing out their packages and versions. Again, the fixed by produced by our scans is the version of the packages we have implemented. Any suggestions on how to navigate this scenario?
{ "LayerCount": 2, "Vulnerabilities": { "High": [ { "Name": "RHSA-2017:2832", "NamespaceName": "centos:7", "Description": "Network Security Services (NSS) is a set of libraries designed to support the cross-platform development of security-enabled client and server applications. Security Fix(es): * A use-after-free flaw was found in the TLS 1.2 implementation in the NSS library when client authentication was used. A malicious client could use this flaw to cause an application compiled against NSS to crash or, potentially, execute arbitrary code with the permission of the user running the application. (CVE-2017-7805) Red Hat would like to thank the Mozilla project for reporting this issue. Upstream acknowledges Martin Thomson as the original reporter.", "Link": "https://access.redhat.com/errata/RHSA-2017:2832", "Severity": "High", "FixedBy": "0:3.28.4-12.el7_4" }, { "Name": "RHSA-2017:2832", "NamespaceName": "centos:7", "Description": "Network Security Services (NSS) is a set of libraries designed to support the cross-platform development of security-enabled client and server applications. Security Fix(es): * A use-after-free flaw was found in the TLS 1.2 implementation in the NSS library when client authentication was used. A malicious client could use this flaw to cause an application compiled against NSS to crash or, potentially, execute arbitrary code with the permission of the user running the application. (CVE-2017-7805) Red Hat would like to thank the Mozilla project for reporting this issue. Upstream acknowledges Martin Thomson as the original reporter.", "Link": "https://access.redhat.com/errata/RHSA-2017:2832", "Severity": "High", "FixedBy": "0:3.28.4-12.el7_4" }, { "Name": "RHSA-2017:2832", "NamespaceName": "centos:7", "Description": "Network Security Services (NSS) is a set of libraries designed to support the cross-platform development of security-enabled client and server applications. Security Fix(es): * A use-after-free flaw was found in the TLS 1.2 implementation in the NSS library when client authentication was used. A malicious client could use this flaw to cause an application compiled against NSS to crash or, potentially, execute arbitrary code with the permission of the user running the application. (CVE-2017-7805) Red Hat would like to thank the Mozilla project for reporting this issue. Upstream acknowledges Martin Thomson as the original reporter.", "Link": "https://access.redhat.com/errata/RHSA-2017:2832", "Severity": "High", "FixedBy": "0:3.28.4-12.el7_4" }, { "Name": "RHSA-2017:1680", "NamespaceName": "centos:7", "Description": "The Berkeley Internet Name Domain (BIND) is an implementation of the Domain Name System (DNS) protocols. BIND includes a DNS server (named); a resolver library (routines for applications to use when interfacing with DNS); and tools for verifying that the DNS server is operating correctly. Security Fix(es): * A flaw was found in the way BIND handled TSIG authentication for dynamic updates. A remote attacker able to communicate with an authoritative BIND server could use this flaw to manipulate the contents of a zone, by forging a valid TSIG or SIG(0) signature for a dynamic update request. (CVE-2017-3143) * A flaw was found in the way BIND handled TSIG authentication of AXFR requests. A remote attacker, able to communicate with an authoritative BIND server, could use this flaw to view the entire contents of a zone by sending a specially constructed request packet. (CVE-2017-3142) Red Hat would like to thank Internet Systems Consortium for reporting these issues. Upstream acknowledges Clement Berthaux (Synacktiv) as the original reporter of these issues. Bug Fix(es): * ICANN is planning to perform a Root Zone DNSSEC Key Signing Key (KSK) rollover during October 2017. Maintaining an up-to-date KSK, by adding the new root zone KSK, is essential for ensuring that validating DNS resolvers continue to function following the rollover. (BZ#1459649)", "Link": "https://access.redhat.com/errata/RHSA-2017:1680", "Severity": "High", "FixedBy": "32:9.9.4-50.el7_3.1" } ] } } ERROR: Job failed: exit code 1

Can't scan images via Clair

I'll start by saying that I'm not sure if this is an issue with Klar or with Clair itself. I've setup two different instances of Klar and Clair (local on my laptop and on a remote system) and I am seeing errors when trying to scan any Docker images.

→ ./klar my_team/centos7
Can't pull image: Token request returned 401%                                                                                                    
→ ./klar registry.domain.com:31832/my_team/centos7
Decode error
Can't pull image: json: cannot unmarshal number into Go value of type docker.Image% ```

When I look at the logs from the Clair container, I see no errors and I've set CLAIR_ADDR to the clair instance. Any idea what's happening and what I can do to fix it?

Dockerfile build fails

Hi, first of all congratulations for your work.

I have tried to create the Docker image of Klar using the latest Dockerfile.

This is the result:

Sending build context to Docker daemon 1.123 MB
Step 1 : FROM scratch
 ---> 
Step 2 : ENV DOMAIN skydns.local
 ---> Using cache
 ---> bd6f5e0205fc
Step 3 : ENV RELEASE 0.1
 ---> Using cache
 ---> f95d6f8a50b4
Step 4 : ENV ZONE_INFO /zoneinfo.zip
 ---> Using cache
 ---> 1cbaffced8a0
Step 5 : COPY assets/zoneinfo.zip /
 ---> Using cache
 ---> a7c974a97a0d
Step 6 : COPY assets/ca-bundle.crt /etc/ssl/certs/ca-certificates.crt
 ---> Using cache
 ---> a5ed1769c440
Step 7 : COPY klar /
lstat klar: no such file or directory

I have changed "klar" with "clair" (since no directory named klar exists) and it compile but can't start due a missing file too.

Can you please confirm if the current Dockerfile is able to build using "sudo docker build -t klar ." ?

Thank you very much in advance!

Severity threshold

Klar currently allows to filter output by severity and has a threshold by high severity vulnerabilities count.

It would be nice to allow the customization of the threshold level, for example to include Medium vulnerabilities in the threshold count, and return 1 if any.

Allow Response to be limited by Severity Level

It seems that the Klar just outputs all vulnerabilities no matter what the severity level. It would be nice to have an option for the Klar command to limit the response. I do like the summary portion of it though.

Something like this:
<ENV_VARS> klar --out_level="High" <image>

The output would be something similar to this:

CVE-2016-1252: [High]
A man-in-the-middle attacker could circumvent the InRelease signature of a repository, leading to a malicious package being installed and, therefore, remote arbitrary code execution.
http://people.ubuntu.com/~ubuntu-security/cve/CVE-2016-1252
-----------------------------------------
Negligible: 10
Low: 69
Medium: 86
High: 1

How does ECR integration work?

It seems there is no explanation on usage with ECR.

With AWS ECR you typically use IAM Roles with EC2 instance so all we need is aws ecr get-login

There is no need to pass in DOCKER_USER and DOCKER_PASSWORD and I'm getting

Can't pull image: Can't parse Www-Authenticate: Basic realm="https://1234567890.dkr.ecr.ap-southeast-2.amazonaws.com/",service="ecr.amazonaws.com"

Any work around here?

Klar produces duplicate entries

I am doing vulnerability analysis of the docker image in our Artifactory. I don't know if this is caused by my personal mistake, but I've noticed that Klar gives wrong result.

What I mean is:

After I run command like CLAIR_ADDR=http://localhost CLAIR_OUTPUT=High CLAIR_THRESHOLD=10 DOCKER_USER=me DOCKER_PASSWORD=secret klar <Image>, it prints out the result showing number of vulnerabilities in different severities. However, if shows a lot of duplicate CVE entries and also it DOES NOT show all the vulnerabilities compared to what I can get using CLAIR API curl http://192.168.99.100:30060/v1/layers/<LAYER_ID>\?vulnerabilities.

Can someone explain this?

Can't specify clair external port

Hi,

I'm exposing Clair behind port 80 instead of 6060, If I specify the port the APIv1 fails with a parsing error and the APIv3 gives an transport is closing error

CLAIR_ADDR=http://{{ clair_address}}:80 klar debian 
Analysing 1 layers
Failed to analyze using API v1: push image https://registry-1.docker.io/v2/library/debian:latest to Clair failed: can't even read an error message: invalid character '<' looking for beginning of value

Failed to analyze using API v3: push image https://registry-1.docker.io/v2/library/debian:latest to Clair failed: rpc error: code = Unavailable desc = transport is closing

Failed to analyze, exiting

The port is open but Clair logs doesn't show any activity

Klar version: 2.0
Clair version/image: quay.io/coreos/clair-git:latest
Kubernetes version: v1.7.8-gke.0

Thanks

CLAIR_OUTPUT filter doesn't seem to work

I am using the latest klar I believe (used go get github.com/optiopay/klar to install) which should supports CLAIR_OUTPUT. However my scan output still shows all levels of severity.

CLAIR_ADDR=https://clair.example.com CLAIR_OUTPUT=High CLAIR_THRESHOLD=10  DOCKER_USER=username DOCKER_PASSWORD=password JSON_OUTPUT=true klar jenkins | jq '.' > output.json
grep Severity output.json  | sort -u
      "Severity": "High",
      "Severity": "Low"
      "Severity": "Low",
      "Severity": "Medium",
      "Severity": "Negligible"
      "Severity": "Negligible",
      "Severity": "Unknown"
      "Severity": "Unknown",

Did I miss anything? Thanks!

Make Docker registry configurable

At the moment the used Docker registry is hardcoded (const dockerHub = "registry-1.docker.io"). The Docker registry should be configurable to be able to use a private Docker registry.

Support for image digest

Is it possible to also support image digests? Possible like:

klar postgres@sha256:1364924c753d5ff7e2260cd34dc4ba05ebd40ee8193391220be0f9901d4e1651

(should be the same as postgres:9.6)

This would help testing specific running images in Docker/Kubernetes, for example if XXXX:latest is used or the tag is reused for a newer image, the version from the registry could be updated/not affected but your running image version is affected.

"parent layer is unknown" + "resource cannot be found"

Hi,
I am running klar on an image in our registry and receive the following error:

Push layer 1 failed: Push error 422: {"Error":{"Message":"worker: OS and/or package manager are not supported"}}

Push layer 0 failed: Push error 400: {"Error":{"Message":"worker: parent layer is unknown, it must be processed first"}}

Analyse image http://localhost/v2/image/name:0.0.1 failed: Analyze error 404: {"Error":{"Message":"the resource cannot be found"}}

And in the clair logs this appears:

{"Event":"Handled HTTP request","Level":"info","Location":"router.go:57","Time":"2017-09-19 09:11:49.755765","elapsed time":59881969,"method":"GET","remote addr":"[::1]:46938","request uri":"/v1/layers/sha256:<redacted-256-hash>?vulnerabilities","status":"404"}

Running docker pull image/name:0.0.1 is working, this is why I am suspecting klar.
Do you have any idea what the issue could be?

Versions:
klar: v1.4.1
clair: quay.io/coreos/clair:v2.0.1

Use existing registry token

Currently Klar relies on docker registry login and password provided as env vars to generate an auth token. As result klar may be used without docker client. However in some cases usage of existing token may be desirable. For example integration with Google Container Registry is much simpler in that case.

Initially a user generates a toke using docker login command and potentially vendor-specific tools (aws, gcloud cli tools). This command creates the file $HOME/.docker/config.json:

{
    "auths": {
        "https://index.docker.io/v2/": {
            "auth": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx="
        },
        "https://registry.example.com/v2/": {
            "auth": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx="
        }
    }
}

If login and password are not provided Klar should parse this file and use existing token. In case if file does not exist or the token is not found the process must terminate with an error.

Filter out accepted findings

As a vulnerability manager I would like to filter out risks from the output which I accepted. Is there a way in klar now or planned in the near by future? As I am not good in golang, I will create a simple bash script to filter out CVEs in case there is now way.

Difference between Klar and Claire

Would somebody be able to explain the benefit of using Klar vs. Claire to me?

Client.Timeout exceeded while awaiting headers

I'm seeing the following an awful lot: Failed to analyze using API v1: push image <nameofimage> to Clair failed: can't push layer to Clair: Post <clairurl>/v1/layers: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

Presumably because clair is initially empty and having to download all the layers afresh, therefore taking more than whatever default timeout is set.

Is there a configuration option or preferably an environment variable I can use to increase the timeout?