Security test: Are any containers running in privileged mode? about testsuite HOT 15 CLOSED

Moved some items to Spec coverage. See lfn-cnti/certification#55 (comment)

Remaining A/C moved to Acceptance Criteria: lfn-cnti/certification#55 (comment)

from testsuite.

Level of effort in points: 0, 1, 2, 3, 5, 8, 13, 21, infinity: =
8 pts

from testsuite.

Use kubectl get deployments to see a list of all deployments
Use kubectl get pods --all-namespaces -o jsonpath='{.items[*].spec.containers[?(@.securityContext.privileged==true)].name}' to see a list of containers that have privileged mode enabled in the current cluster
Play with

helm_chart_container_name: coredns
white_list_helm_chart_container_names: [falco, nginx, coredns, calico-node, kube-proxy, nginx-proxy]

in cnf-conformance/sample-cnfs/sample-coredns-cnf/cnf-conformance.yml
to enable and disable the white list or the main container that should not be in privileged mode when running deployment_name: privileged-coredns or deployment_name: privileged-coredns verbose

from testsuite.

Spec test implementation tasks:

Setup and deploy the non-privileged target CNF

• Run crystal src/cnf-conformance.cr sample_coredns_setup verbose
  • This will Copy over a sample CNF folder for CoreDNS to target in the /cnfs/ folder
  • It will also install the CNF into K8s
• The cnf-conformance.yml should be in the coredns CNF subfolder of the cnfs folder
• There should name of the CoreDNS deployment in cnf-conformance.yml configuration
• Run kubectl get deployments to check that the CoreDNS deployment is listed

Setup and deploy the non-whitelisted privileged target CNF

• Run crystal src/cnf-conformance.cr sample_privileged_cnf_non_whitelisted_setup verbose
  • This will Copy over a sample CNF folder to target in the /cnfs/ folder
  • It will also install the CNF into K8s
• The cnf-conformance.yml should be in the CNF subfolder of the cnfs folder
• There should name of the CNF deployment in cnf-conformance.yml configuration
• Run kubectl get deployments to check that the CNF deployment name is listed

Setup and deploy the whitelisted privileged target CNF

• Run crystal src/cnf-conformance.cr sample_privileged_cnf_whitelisted_setup verbose
  • This will Copy over a sample CNF folder to target in the /cnfs/ folder
  • It will also install the CNF into K8s
• The cnf-conformance.yml should be in the CNF subfolder of the cnfs folder
• There should name of the CNF deployment in cnf-conformance.yml configuration
• There should a allowed value in the whiteliste section of the cnf-conformance.yml configuration
• Run kubectl get deployments to check that the CNF deployment name is listed

Steps to To check if any containers are running in privileged mode for a non-privileged CNF

• Follow common steps above
• Follow the Setup and deploy the non-privileged target CNF steps above to deploy the target CNF
• Run conformance command crystal src/cnf-conformance.cr privileged
• I would expect to see summary output which says something like this pseudo example:
  • FAIL: CNF_NAME is running in privileged mode
  • PASS: CNF_NAME is NOT running in privileged mode
• There will be output showing if the test PASSED, colored green, or FAILED, colored red
• I would expect the output to say PASSED in green'

Steps to To check if any containers are running in privileged mode in verbose mode for a non-privileged CNF

• Follow common steps above
• Follow the Setup and deploy the non-privileged target CNF steps above to deploy the target CNF
• Run conformance command crystal src/cnf-conformance.cr privileged verbose
• Presumption: CNF already deployed to K8s
• There will be output on the screen showing the list of deployments
  • eg. Test runs kubectl get deployments
• I would expect to see the target CNF in the list of deployments
• There will be output on the screen showing the list of privileged deployments
  • eg. Test runs kubectl to show privileged deployments
• I would expect to NOT see the target CNF in the list of deployments
• I would expect to see summary output which says something like this pseudo example:
  • FAIL: CNF_NAME is running in privileged mode
  • PASS: CNF_NAME is NOT running in privileged mode
• There will be output showing if the test PASSED, colored green, or FAILED, colored red
• I would expect the output to say PASSED in green'

Steps to To check a that a non-whitelisted K8s deployment FAILS the privileged mode test

• Follow common steps above
• Follow the Setup and deploy the non-whitelisted privileged target CNF steps above to deploy the target CNF
• Configure the target CNF to be a K8s service
  • Create sample configuration which uses the name of a privileged K8s service as the deployment_name
  • Do not whitelist the "CNF" for privilege mode
  • Pseudo configuration example:

  deployment_name: privileged-cnf
  

• Run conformance command crystal src/cnf-conformance.cr privileged
• I would expect to see summary output which says something like this pseudo example:
  • FAIL: CNF_NAME is running in privileged mode
• I would expect the output to say FAILED in red'

Steps to To check a that a whitelisted K8s deployment PASSES the privileged mode test

• Follow common steps above
• Follow the Setup and deploy the whitelisted privileged target CNF steps above to deploy the target CNF
• Run conformance command crystal src/cnf-conformance.cr privileged
• I would expect to see summary output which says something like this pseudo example:
  • FAIL: CNF_NAME is running in privileged mode
• I would expect the output to say FAILED in red'
• I would expect the output to say PASSED in green'

from testsuite.

Acceptance Criteria:

Prerequisites:

• Find or create a K8s cluster to use
  • Steps for creating will be in another ticket
  • Talk with team for access to an existing cluster
• Set the KUBECONFIG environment variable for accessing the target K8s cluster
• Pull down latest version of CNF conformance test code from develop branch
• Run shard install

Steps to use automated integration specs with a sample CNF using privileged mode tests:

crystal spec --tag privileged

Steps to test a sample cnf using all privileged mode tests:

• Run all privileged mode tests

crystal src/cnf-conformance.cr privileged

Steps to test a sample CNF against all tests in the test category:

• Run all security tests

crystal src/cnf-conformance.cr security

• I expect to see the output and results from the all test
• I expect to see PASSED for the all test

Steps to test a sample CNF against all tests in the CNF Conformance test suite:

• Run all of the CNF Conformance tests

crystal src/cnf-conformance.cr all

• I expect to see the output and results from the all test
• I expect to see PASSED for the all test

from testsuite.

@wavell @denverwilliams cleanup has errors

Error: uninstall: Release not loaded: bad-helm-coredns: release: not found
Error: uninstall: Release not loaded: privileged-coredns: release: not found
Error: uninstall: Release not loaded: privileged-coredns: release: not found```

from testsuite.

@denverwilliams the A/C for the priv CNF fails with a task not found error. Please update the A/C

The A/C does not match the task name. It appears the task name is: sample_privileged_cnf_non_whitelisted_setup

I tried the A/C command and the name of the sample file before finding the task in the code.

lib/sam/src/sam.cr:41:5 in 'invoke'
lib/sam/src/sam.cr:77:7 in 'process_tasks'
lib/sam/src/sam.cr:56:7 in 'help'
src/cnf-conformance.cr:8:1 in '__crystal_main'
/usr/lib/crystal/crystal/main.cr:106:5 in 'main_user_code'
/usr/lib/crystal/crystal/main.cr:92:7 in 'main'
/usr/lib/crystal/crystal/main.cr:115:3 in 'main'
__libc_start_main
_start
???
Task sample_privileged_cnf_setup_coredns was not found
rosenberg% ls -ld sample-cnfs/sample_privileged_cnf_setup_coredns
drwxr-xr-x 3 taylor taylor 4096 Mar 11 16:09 sample-cnfs/sample_privileged_cnf_setup_coredns

from testsuite.

@wavell @denverwilliams the whitelisted CNF setup is deploying the non-whitelisted CNF

osenberg% crystal src/cnf-conformance.cr sample_privileged_cnf_whitelisted_setup
Successfully created directories for cnf-conformance
NAME                         DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
privileged-coredns-coredns   1         1         1            0           1s
second_count = 0
NAME                         DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
privileged-coredns-coredns   1         1         1            0           1s
second_count = 1
NAME                         DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
privileged-coredns-coredns   1         1         1            0           2s
second_count = 2
NAME                         DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
privileged-coredns-coredns   1         1         1            0           3s
second_count = 3
NAME                         DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
privileged-coredns-coredns   1         1         1            0           5s
second_count = 4
NAME                         DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
privileged-coredns-coredns   1         1         1            0           6s
second_count = 5
NAME                         DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
privileged-coredns-coredns   1         1         1            0           7s
second_count = 6
NAME                         DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
privileged-coredns-coredns   1         1         1            0           9s
second_count = 7
NAME                         DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
privileged-coredns-coredns   1         1         1            0           10s
second_count = 8
NAME                         DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
privileged-coredns-coredns   1         1         1            0           11s
Successfully setup sample_privileged_cnf_non_whitelisted
rosenberg% kubectl get deployment
NAME                         DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
privileged-coredns-coredns  

from testsuite.

@wavell @denverwilliams cleanup has errors

Error: uninstall: Release not loaded: bad-helm-coredns: release: not found
Error: uninstall: Release not loaded: privileged-coredns: release: not found
Error: uninstall: Release not loaded: privileged-coredns: release: not found```

@taylor
These warnings are expected at the moment. The main reason this is happening is because we are now using multiple CNFs, and having to do multiple cleanups, which sometimes results in the cleanup attempting to delete CNFs that are no longer there. Will work on a patch for this in the next few days.

from testsuite.

@denverwilliams the A/C for the priv CNF fails with a task not found error. Please update the A/C

The A/C does not match the task name. It appears the task name is: sample_privileged_cnf_non_whitelisted_setup

I tried the A/C command and the name of the sample file before finding the task in the code.

lib/sam/src/sam.cr:41:5 in 'invoke'
lib/sam/src/sam.cr:77:7 in 'process_tasks'
lib/sam/src/sam.cr:56:7 in 'help'
src/cnf-conformance.cr:8:1 in '__crystal_main'
/usr/lib/crystal/crystal/main.cr:106:5 in 'main_user_code'
/usr/lib/crystal/crystal/main.cr:92:7 in 'main'
/usr/lib/crystal/crystal/main.cr:115:3 in 'main'
__libc_start_main
_start
???
Task sample_privileged_cnf_setup_coredns was not found
rosenberg% ls -ld sample-cnfs/sample_privileged_cnf_setup_coredns
drwxr-xr-x 3 taylor taylor 4096 Mar 11 16:09 sample-cnfs/sample_privileged_cnf_setup_coredns

@taylor A/C has been updated

from testsuite.

@wavell @denverwilliams the whitelisted CNF setup is deploying the non-whitelisted CNF

osenberg% crystal src/cnf-conformance.cr sample_privileged_cnf_whitelisted_setup
Successfully created directories for cnf-conformance
NAME                         DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
privileged-coredns-coredns   1         1         1            0           1s
second_count = 0
NAME                         DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
privileged-coredns-coredns   1         1         1            0           1s
second_count = 1
NAME                         DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
privileged-coredns-coredns   1         1         1            0           2s
second_count = 2
NAME                         DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
privileged-coredns-coredns   1         1         1            0           3s
second_count = 3
NAME                         DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
privileged-coredns-coredns   1         1         1            0           5s
second_count = 4
NAME                         DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
privileged-coredns-coredns   1         1         1            0           6s
second_count = 5
NAME                         DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
privileged-coredns-coredns   1         1         1            0           7s
second_count = 6
NAME                         DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
privileged-coredns-coredns   1         1         1            0           9s
second_count = 7
NAME                         DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
privileged-coredns-coredns   1         1         1            0           10s
second_count = 8
NAME                         DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
privileged-coredns-coredns   1         1         1            0           11s
Successfully setup sample_privileged_cnf_non_whitelisted
rosenberg% kubectl get deployment
NAME                         DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
privileged-coredns-coredns  

@taylor
Patch has been pushed up here: lfn-cnti/certification@2259eb6

from testsuite.

@wavell @denverwilliams cleanup has errors

Error: uninstall: Release not loaded: bad-helm-coredns: release: not found
Error: uninstall: Release not loaded: privileged-coredns: release: not found
Error: uninstall: Release not loaded: privileged-coredns: release: not found```

@taylor
These warnings are expected at the moment. The main reason this is happening is because we are now using multiple CNFs, and having to do multiple cleanups, which sometimes results in the cleanup attempting to delete CNFs that are no longer there. Will work on a patch for this in the next few days.

@denverwilliams sounds good. we can handle this in a new ticket. @williscool and I were talking about working on the UX for various end users. This would be addressed for one of the end users.

from testsuite.

@wavell @denverwilliams the whitelisted CNF setup is deploying the non-whitelisted CNF

osenberg% crystal src/cnf-conformance.cr sample_privileged_cnf_whitelisted_setup

...

@taylor
Patch has been pushed up here: 2259eb6

@denverwilliams this passes A/C for me now on develop. Will check on master when it's fully merged

from testsuite.

All A/C pass on master.

from testsuite.

https://github.com/cncf/cnf-conformance/blob/master/USAGE.md#security-tests looks good -

• crystal src/cnf-conformance.cr privileged is marked as ✔️ (Recreatable/passes integration tests)

Closing issue as complete

from testsuite.