Compatibility Test: check for the use of alpha K8s API endpoints about testsuite HOT 12 CLOSED

Spoke with Akash, since this is a POC, to get the test to fail using the sample requires more indepth setup on the cluster. No reason to keep this hung up in progress status, this can be review approved at this time as Akash vetted the FAIL status during his setup.

from testsuite.

Create test for "To see if the cnf is using k8s alpha endpoints"

from testsuite.

Acceptance Criteria

Common steps

• Find or create a K8s cluster to use
  • Talk with team for access to an existing cluster
  • Kind K8s clusters will work as well
• Set the KUBECONFIG environment variable for accessing the target K8s cluster
• As a developer, Pull down latest version of CNF conformance test code from {FEATURE} branch
  • Check that {FEATURE} Branch passes all A/C
  • Check that {FEATURE} Branch passes tests in GitHub Actions
• As a peer reviewer, check that PR passes in GitHub Actions
  • if yes, merge PR to master
  • if no, make a comment and request that developer make changes to pass tests
• As a peer reviewer, Pull down latest version of CNF conformance test code from main branch
• Run shards install

Update Prerequisite checker

• NA

Added to overall points:

• I see the test in the embedded points.yml as valid test score

When I run install_apisnoop, I should see that apisnoop is installed on the cluster:

• Run ./cnf-testsuite install_apisnoop
• I can see apisnoop is installed

When I run alpha_k8s_apis, I should see PASS results that alpha apis are not used:

• Run ./cnf-testsuite cnf_setup cnf-path=./sample-cnfs/sample_coredns
• Run ./cnf-testsuite alpha_k8s_apis poc
• I can see results for alpha_k8s_apis.

When I run alpha_k8s_apis, I should see FAIL results that alpha apis are not used:

• Run ./cnf-testsuite cnf_setup cnf-path=sample-cnfs/sample-alpha-apis/cnf-testsuite.yml
• Run ./cnf-testsuite alpha_k8s_apis poc
• I can see results for alpha_k8s_apis.

from testsuite.

@agentpoyo Thank you for helping validate the test. I found an issue with the test when I created a negative CNF (a CNF that uses alpha k8s APIs). I've attached a PDF with the notes from discussions.

Notes on negative test for apisnoop.pdf

from testsuite.

@HashNuke I'm only getting SKIPPED on results using your new sample and or using CoreDNS sample CNF.

It claims it's installed as well when I do a install_apisnoop after uninstall and reinstall to make sure it's installed:

root@cnf-testsuite-test1:~/cnf-testsuite# LOG_LEVEL=info ./cnf-testsuite install_apisnoop
I, [2022-01-20 21:05:13 +00:00 #296634]  INFO -- cnf-testsuite-verbose: install_apisnoop
I, [2022-01-20 21:05:13 +00:00 #296634]  INFO -- cnf-testsuite: apisnoop already installed. Skipping git clone for apisnoop.

from testsuite.

@HashNuke @agentpoyo What is the status of this issue? Is this test still in progress?

from testsuite.

@HashNuke @agentpoyo I'd recommend closing this issue if it was included in v0.24.0 and open new issues for bugs or enhancements to be made going forward.

from testsuite.

Adding some notes for expected behaviour below for the alpha_k8s_apis test

SCENARIO-1

• CNF uses alpha Kubernetes API.
• The API -REQUIRES- a k8s feature flag or release flag to be enabled.

Then running the test should result in a failure with the message below:

CNF failed to install on apisnoop cluster

Sample CNF to test: sample-alpha-apis

SCENARIO-2

• CNF uses alpha any Kubernetes API.
• The API does -NOT REQUIRE- a k8s feature flag or release flag to be enabled.

Then running the test should result in a failure with the message below:

FAILED: CNF uses Kubernetes alpha APIs

Sample CNF to test: Do not have a sample CNF for this yet since most of the alpha APIs are covered by Scenario-1 mentioned above.

SCENARIO-3

CNF does not use any alpha Kubernetes API.

Sample CNF to test: sample_coredns

PASSED: CNF does not use Kubernetes alpha APIs

from testsuite.

from testsuite.

@HashNuke I cannot get the sample-alpha-apis to fail, passes everytime:

from testsuite.

Moving back to "ready for Dev" to investigate further. @HashNuke

from testsuite.

This POC was added in v0.24.0. Please open a new issue for any additional effort on this test, if needed.

from testsuite.