Secure your Kubernetes cluster with the most good practices from CIS in a automated way using Ansible.

Common kubernetes administrators do not care about security concerns of their On Premise clusters. They use to think that the cluster is securized itself when running on an internal network, or maybe they delegate security concerns to the company security team (CISO). Also, cloud providers offers a kubernetes solution as a service (EKS, AKS, GKE...) but you do not know how these providers implement security features on the cluster you are using. So this solution applies automated checks to measure the security level of your cluster. You will see that it is perfect for running on an On Premise infrastructure due the analogies between deploying a On Premise k8s cluster ant the Vagrant environment of this repo. In this way you can use the Ansible playbooks not only to benchamrk docker and k8s but to configure the kubernetes into a machines network (masters and nodes).

Based on:

• docker-bench-security

• kube-bench

You can deploy a Kubernetes cluster completely functional for testing apps/services using Vagrant (increase VM resources in the Vagrantfile for custom needs). There are some manifests on config/k8s/services that you can use to provision apps and services in your cluster. Be sure to add your installation logic in tasks/services-k8s.yaml. You can control this installation with corresponding variables in vars/k8s.yaml.

Possible services you can install

• Cert-Manager (to review/update)

• Nginx Ingress Controller (to review/update)

• ArgoCD

TODO

• Harden k8s cluster. This repo only run tests but not harden the cluster. Securize the k8s masters may be complicated to automate cause parameters variables between environments. Also, these manifests are very important so you may break the cluster if you make a mistake. You can take manifests examples located on config/k8s/etc/.

• Add script for creating TLS Docker certs automatically for being able to connect to the servers Docker daemon.

• Set variables with envsub

• Add more services for testing and hack.

New features for docker-bench-security that you can see on benchmark_docker

• Ansible configuration.

• New argument -v for benchmark/docker-bench-security.sh. You can pass the Docker official version and check if corresponds with yours.

$ sudo bash benchmark/docker-bench-security.sh -v 20.10.5

• New colours styles:

  • [WARN] in red for critical issues that you have to fix.
  • [INFO] in yellow for external (manually) actions (checks) you have to do in order to follow CIS recommendation. In general this check does not have a critical impact in your Docker environment nor may depend on external factors.
  • [NOTE] in blue for just a recommendations/suggestion or information to fix the point.

• config/docker/daemon.json file configuration provided.

{
    "cgroup-parent": "",          # CIS 2.9 SET your /foobar/path
    "log-level": "info",          # CIS 2.2
    "icc": false,                 # CIS 2.1
    "live-restore": true,         # CIS 2.13
    "userland-proxy": false,      # CIS 2.14
    "no-new-privileges": true,    # CIS 2.17
    "selinux-enabled": true,      # CIS 5.2
    "userns-remap": "default",    # CIS 2.8 BUG see troubleshooting
    "storage-driver": ""          # CIS 2.5 NOT "aufs"
}

• docker/docker-compose.yaml file provided with examples and security options.

• Vagrant: v2.4.0
• Vagrant images: bento/ubuntu-18.04
• Ansible: core v2.16.1
• Kubernetes: v1.29
• Docker: v24.0.2 (build cb74dfc)
• Docker Compose: v1.17.1 (build unknown)
• Docker bench: v1.3.5 (modified by me)
• Kube bench: v0.7.0

Vagrantfile provided for testing Ansible playbooks.

• Install vagrant.

• Set required variables on vars/ files.

Kubernetes cluster based on vagrant uses flannel or calico as network plugin. See Troubleshooting to handle with possible errors.

You can deploy the configuration for hardening your machines

• Install ansible on local machine (where you are going to deploy from).

• Generate ssh keys and copy them on remote machines. See keys/README.md and create_user_ansible.sh for further information.

• Configure your hosts in inventory/hosts which ansible will connect to.

• Set required variables on vars/ files. If your machines already has installed Docker then set docker_configuration to false.

Run $ ansible all -m ping for testing your configuration and check if ansible can connect to your machines.

If you want to test manually if your cluster is securized...

• Copy benchmark_docker folder to your remote machines.

• Copy the config/docker/daemon.json to the Docker Daemon config path (by default /etc/docker/daemon.json) on your remote machines. You can add more options from config/docker/daemon-template.json. NOTE: care about "userns-remap" option (see Troubleshooting part for further information).

• If you want to test the docker/docker-compose.yaml you have to copy it to another path. See docker/README.md

• Install vagrant and run $ vagrant up. Now you have a Kubernetes cluster running in virtualbox.

$ vagrant status
$ vagrant ssh k8s-master
$ kubectl get nodes
$ journalctl -u kubelet
$ kubectl get po -n kube-system

Just run the following script run_playbook.sh for configuring and hardening Docker in your hosts.

$ bash run_playbook.sh playbooks/docker-k8s.yaml

Check benchmark logs on benchmark_docker/results and benchmark_k8s/results

Please go to docker-bench-security and kube-bench for further information

docker-bench-security

The CIS based checks are named check_<section>_<number>, e.g. check_2_6 and community contributed checks are named check_c_<number>.

A complete list of checks is present in functions_lib.sh.

$ bash docker-bench-security.sh -l /tmp/docker-bench-security.sh.log -v 20.10.5 will run all checks and compare the Docker official version with your Docker enviroment.

$ bash docker-bench-security.sh -l /tmp/docker-bench-security.sh.log -c check_2_2 will only run check 2.2 Ensure the logging level is set to 'info'.

$ bash docker-bench-security.sh -l /tmp/docker-bench-security.sh.log -e check_2_2 will run all available checks except 2.2 Ensure the logging level is set to 'info'.

$ bash docker-bench-security.sh -l /tmp/docker-bench-security.sh.log -e docker_enterprise_configuration will run all available checks except the docker_enterprise_configuration group

$ bash docker-bench-security.sh -l /tmp/docker-bench-security.sh.log -e docker_enterprise_configuration,check_2_2 will run all available checks except the docker_enterprise_configuration group and 2.2 Ensure the logging level is set to 'info'

$ bash docker-bench-security.sh -l /tmp/docker-bench-security.sh.log -c container_images -e check_4_5 will run just the container_images checks except 4.5 Ensure Content trust for Docker is Enabled

kube-bench

This proejct automate k8s benchmark using a go binary (kube-bench from benchmark-k8s). You can run

$ ./kube-bench --help
$ ./kube-bench --benchmark cis-1.6

For instance, with benchmark-k8s/job.yaml you will deploy a pod that bench the cluster

$ kubectl apply -f job.yaml
$ kubectl logs $pod_name

• You should make persistant every new rule you add with auditctl (see CIS 1). Check it after restart your environment. Ansible makes these default rules persistent but there may be an error if the paths/files do not exist.

• NOT to restart the Docker Daemon!! First stop it, make your config changes and finally start it.

$ systemctl stop docker # Stopping docker.service, but it can still be activated by docker.socket
$ systemctl stop docker.socket

• Fail to start a container with following ERROR

ERROR: for python  Cannot start service python: OCI runtime create failed: container_linux.go:367: starting container process caused: process_linux.go:495: container init caused: write sysctl key kernel.domainname: open /proc/sys/kernel/domainname: permission denied: unknown

bug: domainname denied if userns enabled

explanation: you can not set the domainname just the hostname.

remmediation: delete "userns-remap": "default" from config/docker/daemon`

related to: CIS 2.8, even /etc/subuid /etc/subgid are created.

• Sometimes kubeadm can not start and report the following error:

[ERROR CRI]: container runtime is not running. Validate service connection: validate CRI v1 runtime API for endpoint unix:///var/run/containerd/containerd.sock

remmediation: It is necessary to delete the installation config for containerd. the task "Copy containerd configuration" is implementd on tasks/configure-docker.yaml fix the bug. Also review config/containerd/README.md

1. Basic troubleshooting

$ kubectl get nodes -o wide
$ kubectl get po -n kube-system

• Kubernetes Legacy Package Repositories Will Be Frozen On September 13, 2023

2. Network Plugin

• The default CIDR range for flannel is 10.244.0.0/16. If you are using kubeadm init, make sure to use -–pod-network-cidr=10.244.0.0/16. NOTE: this project implement vagrant tests with --pod-network-cidr=10.244.0.0/16 and the option --iface=eth1 in config/k8s/plugins/flannel/kube-flannel.yaml. Be sure to change this options if you want to modify the pod network.

• Nodes are in status Ready but flannel pods are on Error or CrashLoopBackOff

Error registering network: failed to acquire lease: node "node1" pod cidr not assigned

remmediation: unless the cluster is initiallized with the --pod-network-cidr argument, sometimes it fails (issue). So you have to run

$ sudo cat /etc/kubernetes/manifests/kube-controller-manager.yaml | grep -i cluster-cidr
    - --cluster-cidr=10.244.0.0/16

Which results is the subnet taken from vars/k8s.yaml and Vagrantfile. Then copy that CIDR and paste in the following command

$ for i in $(kubectl get nodes | grep node | awk '{print $1}'); do kubectl patch node $i -p '{"spec":{"podCIDR":"10.244.0.0/16"}}'; done

ERROR! [DEPRECATED]: ansible.builtin.include has been removed. Use include_tasks or import_tasks instead. This feature was removed from ansible-core in a release after 2023-05-16. Please update your playbooks.
Ansible failed to complete successfully. Any error output should be
visible above. Please fix these errors and try again.

remmediation: check your ansible version.

Docker

Docker Security (official github)

Docker Security Options

Docker Compose Configuration

Containers Security toolkit

Kubernetes

Kubeadm

Installing kubeadm

Creating a cluster with kubeadm

Cluster Networking

Calico network plugin

Flannel network plugin

Extra

Security features from CIS 5.

1. AppArmor

AppArmor (Application Armor) is a Linux Security Module (LSM). It protects the operating system by applying profiles to individual applications or containers. In contrast to managing capabilities with CAP_DROP and syscalls with seccomp, AppArmor allows for much finer-grained control. For example, AppArmor can restrict file operations on specified paths.

AppArmor security profiles for Docker

Protege contenedores con AppArmor

2. SElinux

The Docker daemon relies on a OCI compliant runtime (invoked via the containerd daemon) as its interface to the Linux kernel namespaces, cgroups, and SELinux

Secure your containers with SELinux

3. Seccomp

Seccomp is a sandboxing facility in the Linux kernel that acts like a firewall for system calls (syscalls). It uses Berkeley Packet Filter (BPF) rules to filter syscalls and control how they are handled. These filters can significantly limit a containers access to the Docker Host's Linux kernel - especially for simple containers/applications.

The docker/no-chmod.json file is a profile with the chmod(), fchmod(), and chmodat() syscalls removed from its whitelist.