crystal src/cnf-conformance.cr rolling_deployment_update

Pre-req:

• Slide content updated
• Create outline with all categories #18
• Add content for Installable and Upgradeable section > https://github.com/cncf/cnf-conformance/blob/master/TEST-CATEGORIES.md#installable-and-upgradeable

Tasks:

• optionally, add pseudocode to markdown > https://github.com/cncf/cnf-conformance/blob/master/PSEUDO-CODE.md#to-test-if-the-cnf-can-perform-a-rolling-update-1

• Can the CNF perform a rolling update (i.e. kubectl rolling update?

• code test

Possibly use K8s policy enforcement tool k-rail to validate that CNFs continue to carry traffic when appropriate policies are enforced

Goal: Review Privileged Pods and add more content

Tasks:

• Review Privileged Pods
• Create copy of slides
• add more content
• Create an outline
• Create a rough draft
• Peer review of draft
• Iterate as needed, peer review (optional)
• Add content to https://docs.google.com/presentation/d/1K81kQgqs3HMp-P1TVoeI6D2kIz7Qbz9JjTFohfTb9yA/edit#slide=id.g7d9866f192_0_0

ref: s32

Acceptance Criteria: lfn-cnti/certification#55 (comment)

Security test: Are any containers are running in privileged mode?

EPIC: CNF Conformance Test Suite

Release: v0.0.1

Test Category: Security

Type of test (static or runtime) = runtime

• example code: #23

Environment set up tasks:

Upstream tool set up tasks: (conformance suite + upstream tools)

• N/A

CNF setup Tasks

Sample CNF tasks:

Create non-whitelisted privileged sample CNF

• Download the CoreDNS helm chart
• Modify to enable privileged mode
• Create sample cnf-conformance.yml configuration which uses the new sample CNF
  • Pseudo configuration:

  deployment_name: privileged-coredns
  

Create whitelisted privileged sample CNF

• Copy the privileged sample CNF to whitelisted_privileged_cnf
• Create sample cnf-conformance.yml configuration which whitelists the CNF for privileged mode
  • Pseudo configuration:

  deployment_name: privileged-coredns
  privileged_mode: allowed
  

Code implementation tasks:

• Create code which checks if a target CNF is whitelisted for privilege mode
• Create test which checks privilege mode of a deployment
  • If the CNF is privileged mode and not whitelisted the test should show fail
  • If the CNF is privileged mode and whitelisted the test should pass
  • If the CNF is not privileged the test should pass
• Create code to show verbose output for the deployment and privilege information in the K8s cluster
• Spec integration coverage -> lfn-cnti/certification#55 (comment)

Documentation tasks:

• Update Test Categories md if needed
• Update Pseudo Code md if needed
• Update USAGE md if needed
• Update installation instructions if needed

QA tasks

Dev Review:

• walk through A/C
• do you get the expected result?
• if yes, move to Needs Peer Review column
• if no, document what additional tasks will be needed

Peer review:

• walk through A/C
• do you get the expected result?
• if yes, move to Reviewer Approved column
• if no, document what did not go as expected, including error messages and screenshots (if possible)

Goal: Review Hardware support tests and add more content

Tasks:

• Add missing tests from older slides to https://docs.google.com/presentation/d/1K81kQgqs3HMp-P1TVoeI6D2kIz7Qbz9JjTFohfTb9yA/edit#slide=id.g7d332a6c45_0_15
• Review Hardware support Tests
• research upstream tests
• add more content, if possible
• Create an outline
• Create a rough draft
• Peer review of draft
• Iterate as needed, peer review (optional)
• Add content to https://docs.google.com/presentation/d/1K81kQgqs3HMp-P1TVoeI6D2kIz7Qbz9JjTFohfTb9yA/edit#slide=id.g7d332a6c45_0_57

ref: s43

[Documentation] create a CONTRIBUTING.md

Short description:

• to provide potential project contributors with a short guide to how they can help with the CNF Conformance Test Suite

Location:

• https://github.com/cncf/cnf-conformance

Tasks:

• Reference upstream contributor guides such as https://github.com/kubernetes/community/tree/master/contributors/guide
• Create an outline
• Create a rough draft
• Peer review of draft
• Iterate as needed, peer review (optional)
• Add content to https://github.com/cncf/cnf-conformance/blob/master/CONTRIBUTING.md

Goal: Review using OPA Gatekeeper validate that CNFs continue to carry traffic when appropriate policies are enforced

Tasks:

• Research OPA Gatekeeper
• Create an outline
• Create a rough draft
• Peer review of draft
• Iterate as needed, peer review (optional)
• Add content to markdown

ref: s43

[Documentation] rename "Stateless Tests" to "Statelessness Tests"

Short description:

• fix typo
• rename Stateless Tests to Statelessness Tests

Tasks:

• Update USAGE https://github.com/cncf/cnf-conformance/blob/master/USAGE.md#scaling-tests
• Update PSEUDO-CODE https://github.com/cncf/cnf-conformance/blob/master/PSEUDO-CODE.md

Pre-req:

• Slide content updated
• Create outline with all categories #18
• Add content for Scalability section of README

Tasks:

• optionally, add pseudocode to markdown > https://github.com/cncf/cnf-conformance/blob/master/PSEUDO-CODE.md#scaling-tests

• code test > lfn-cnti/certification@6a72243

• Peer review of commit > lfn-cnti/certification@6a72243

Acceptable Criteria

Compatibility Test: check for the use of alpha K8s API endpoints

EPIC: CNF Conformance Test Suite

Release: v0.0.N

Test Category: Compatibility

Type of test (static or runtime) = TBD

Similar tests:

• (WIP) To run K8s API testing for checking for the use of beta endpoints

crystal src/cnf-conformance.cr api_snoop_beta

• (WIP) To run K8s API testing for ensuring the use of generally available endpoints

crystal src/cnf-conformance.cr api_snoop_general_apis

Code Implementation Tasks:

• Create test for "To run K8s API testing for checking for the use of alpha endpoints"

Documentation tasks:

• Update Test Categories md if needed
• Update Pseudo Code md if needed
• Update USAGE md if needed
• Update installation instructions if needed

Goal: Review Configuration and Lifecycle and add more content

Tasks:

• Add missing tests from older slides to https://docs.google.com/presentation/d/1K81kQgqs3HMp-P1TVoeI6D2kIz7Qbz9JjTFohfTb9yA/edit#slide=id.g7d332a6c45_0_15
• Review Configuration and Lifecycle Tests
• research upstream tests
• add more content, if possible
• Create an outline
• Create a rough draft
• Peer review of draft
• Iterate as needed, peer review (optional)
• Add content to https://docs.google.com/presentation/d/1K81kQgqs3HMp-P1TVoeI6D2kIz7Qbz9JjTFohfTb9yA/edit#slide=id.g7d332a6c45_0_39

ref: s39

Title: [Workload] Microservices test: specialized_init_system

Description of problem

There are proper init systems and sophisticated supervisors that can be run inside of a container. Both of these systems properly reap and pass signals. Sophisticated supervisors are considered overkill because they take up too many resources and are sometimes too complicated. Some examples of sophisticated supervisors are: supervisord, monit, and runit. Proper init systems are smaller than sophisticated supervisors and therefore suitable for containers. Some of the proper container init systems are tini, dumb-init, and s6-overlay.

Description of solution

The test should fails if PID 1 is supervisord, monit, or runit, and passes if PID 1 is tini, dumb-init, or s6-overlay
The test should fail if PID 1 is anything else.

https://cloud.google.com/architecture/best-practices-for-building-containers

Acceptance Criteria:

Microservice Test: Microservices test: specialized_init_system

EPIC: CNF Conformance Test Suite

Release: vN.N.N

Test Category: Configuration

Type of test (static or runtime) = ?

• example code: N/A

Environment set up tasks:

Upstream tool set up tasks: (conformance suite + upstream tools)

• N/A

CNF setup Tasks

Sample CNF tasks:

Code implementation tasks:

• [ ]

Documentation tasks:

• Update Test Categories md if needed
• Update Pseudo Code md if needed
• Update USAGE md if needed
• Update installation instructions if needed

QA tasks

Dev Review:

• walk through A/C
• do you get the expected result?
• if yes,
  • move to Needs Peer Review column
  • create Pull Request and follow check list
  • Assign 1 or more people for peer review
• if no, document what additional tasks will be needed

Peer review:

• walk through A/C
• do you get the expected result?
• if yes,
  • move to Reviewer Approved column
  • Approve pull request
• if no,
  • document what did not go as expected, including error messages and screenshots (if possible)
  • Add comment to pull request
  • request changes to pull request

Reference:

Tools:

• Falco
• sysdig-inspect

[Documentation] Re-name "Scaling Tests" to "Scalability Tests" in USAGE.md

Short description:

• fix typo
• rename Scaling Tests to Scalability Tests

Tasks:

• Update USAGE https://github.com/cncf/cnf-conformance/blob/master/USAGE.md#scaling-tests
• Update PSEUDO-CODE https://github.com/cncf/cnf-conformance/blob/master/PSEUDO-CODE.md

[Documentation] update TEST-CATEGORIES.md to reflect updated content on slides v0.3

Short description:

• update the Test-Categories markdown with up-to-date conformance tests
• remove platform related tests and add to #32 to be deferred but documented

Tasks:

• review v0.3 deck
• make adjustments to https://github.com/cncf/cnf-conformance/edit/master/TEST-CATEGORIES.md
• submit pull request with edits
• peer review of pull request
• merge pull request into master branch

[Documentation] Create MAINTAINERS.md

Short description:

• A markdown file containing information about maintainers and contributors.
• Moving content from main README.md

Acceptance Criteria

Configuration test: is there a liveness entry in the Helm chart?

Short Description:

• to check for a liveness entry in the deployment file of the Helm chart

EPIC: CNF Conformance Test Suite

Release: v0.0.1

Test Category: Configuration and Lifecycle

Type of test (static or runtime) = Runtime

• example code: #72

Environment set up tasks:

Upstream tool set up tasks: (conformance suite + upstream tools)

• N/A

CNF setup Tasks

Sample CNF tasks:

Code implementation tasks:

Documentation tasks:

• Update Test Categories md if needed
• Update Pseudo Code md if needed
• Update USAGE md if needed
• Update installation instructions if needed

QA tasks

Dev Review:

• walk through A/C
• do you get the expected result?
• if yes, move to Needs Peer Review column
• if no, document what additional tasks will be needed

Peer review:

• walk through A/C
• do you get the expected result?
• if yes, move to Reviewer Approved column
• if no, document what did not go as expected, including error messages and screenshots (if possible)

Acceptance Criteria

Scalability Test: Can CNF increase and decrease capacity?

EPIC: CNF Conformance Test Suite

Release: v0.0.1

Test Category: Scalability

Type of test (static or runtime) = Runtime

• example code: lfn-cnti/certification@6a72243

Environment set up tasks:

• Copy admin.conf (the kubeconfig) to new file name $HOME/shared/conformance01.kubeconfig (for accessing the shared conformance cluster)
• Get SSH access to one of the shared dev machines to use as your test workstation
  • Ask @denverwilliams, @wavell, @nupejosh or @taylor
• Talk with @denverwilliams, @wavell to get access to the KUBECONFIG file
• Set the KUBECONFIG environment variable to point to the shared kubeconfig file for accessing the target K8s cluster export KUBECONFIG=~/shared/conformance01.kubeconfig)
• Follow the installation instructions to set up the the conformance suite for development and testing
• Update installation instructions to cover new setup and existing workstation updates. Example:
  • You should have a working conformance suite that compiles and runs correctly. (eg. crystal-lang installed)
  • If you have it installed, pull down latest version of CNF conformance test code from develop branch (eg. git fetch ; get rebase from the cnf-conformance directory)
  • Run shard install to get dependencies

Upstream tool set up tasks: (conformance suite + upstream tools)

• N/A

CNF setup Tasks

• Use the https://github.com/cncf/cnf-conformance/tree/master/cnfs for adding target CNFs to be tested
• Create example conformance configuration file for configuring the use of the target CNF
• Create example environment configuration that has the CNF container name list
• Create code to use the conformance CNF configuration and target folder for testing the CNF
• Add documentation for testing a target CNFs with the cnformance suite (eg. add to usage md). Uncludes
  • Path to Helm chart under cnfs folder
  • Options to pass helm chart
• Add documentation for using target CNFs with the cnformance suite (eg. add to usage md), Include
  • The name of the containers needs to be configured

Sample CNF tasks:

• Create a sample-cnf folder
• Create a subfolder for the sample cnf (eg. dns-cnf1)
• Create a sample CNF using a minimal container which requires no host or vanilla K8s modifications (no extensions) --> eg. CoreDNS container
• Create sample CNF conformance configuration file for using the sample CNF
• Create sample environment configuration that has the sample CNF container name list
• Add TL;DR doc on using the sample in the sample folder

Code implementation tasks:

• If one test blows up (eg. CNF code not found for static) the suite should continue to run
• Add util task which can deploy a CNF that has "valid conformance deploy configuration"
  • helm chart v3.0
  • helm chart path
  • containers name
  • NOTE: does not run when tests run (include all)
• Create new increase test which increases the replica set
  • Check and save current replica count
  • At start set replica to 1 (overriding CNF defaults)
  • Show data from cluster about container
  • Do increase
  • Show data from cluster about container
  • Reset replica back to original saved replica count
  • Show pass or fail for doing increase
  • Show green for pass and red for fail
• Create new decrease test which decreases the repica set
  • Check and save current replica count
  • At start set replica to more than 1 (Eg. 3; overriding CNF defaults)
  • Show data from cluster about container
  • Do decrease
  • Show data from cluster about container
  • Reset replica back to original saved replica count
  • Show pass or fail for doing decrease
  • Show green for pass and red for fail
• Update existing increase_decrease test to call the individual increase and decrease
• Think about and document how we can roll up a summary for sets of tests => lfn-cnti/certification#30
• Scalability category should call the increase and decrease test set
• The all test command should run the increase and decrease tests
• Add the increase test to the "runtime test set"
• Add the decrease test to the "runtime test set"
• Optionally, Add the "increase+decrease test set" to the "runtime test set"

Documentation tasks:

• Update Test Categories md if needed
• Update Pseudo Code md if needed
• Update USAGE md if needed
• Update installation instructions if needed

QA tasks

Dev Review:

• walk through A/C
• do you get the expected result?

Steps after A/C Passes for developer:

If all A/C passes for developer

• open a Pull Request to master branch and mark as "developer approved"
• Tag team members for peer review

Peer review:

• walk through A/C
• do you get the expected result?
• if yes, move to Reviewer Approved column
• if no, document what did not go as expected, including error messages and screenshots (if possible)

Steps after A/C fails

If all A/C fails for peer reviewer

• Add a comment to the PR summarizing the problem tagging the developer
• Attach any console output or logs that can help the developer fix the issue

Steps after A/C Passes for peer reviewer

• Mark the PR as approved and merge the PR into master

[Implementation] rename command to Run all hardware resources and scheduling tests

What I see now in https://github.com/cncf/cnf-conformance/edit/master/USAGE.md:

Hardware Resources and Scheduling Tests

✔️ Run all hardware resources and scheduling tests

crystal src/cnf-conformance.cr hardware_affinity

What I would like to see:

crystal src/cnf-conformance.cr hardware_and_scheduling

[Documentation] Create an EXAMPLES.md

Short description:

• Create an EXAMPLES.md

• Describe 4-5 open source CNFs that could run in the CNF Conformance Test Suite > ie. CoreDNS Sample CNF

• Describe pros and cons to using each CNF in the evolution from VNF to CNF (Gold)

Tasks:

• Create an EXAMPLES.md
• Create an outline
• Create a rough draft
• Peer review of draft
• Iterate as needed, peer review (optional)
• Add content to EXAMPLES.md

(WIP) Performing K8s conformance testing by running Sonobuoy on the cluster

Part of: Compatability Tests (WIP)

Tasks:

• Follow-up to #20
• Write code to implement Sonobuoy tests
• TBD

Goal: Review Stateless Tests and add more content

Tasks:

• Add missing tests from older slides to https://docs.google.com/presentation/d/1K81kQgqs3HMp-P1TVoeI6D2kIz7Qbz9JjTFohfTb9yA/edit#slide=id.g7d332a6c45_0_15
• Review Stateless Tests
• research upstream tests
• add more content, if possible
• Create an outline
• Create a rough draft
• Peer review of draft
• Iterate as needed, peer review (optional)
• Add content to https://docs.google.com/presentation/d/1K81kQgqs3HMp-P1TVoeI6D2kIz7Qbz9JjTFohfTb9yA/edit#slide=id.g7d332a6c45_0_21

ref: s36

[Documentation] update USAGE.md to reflect updated content on slides v0.3

Short description:

• update the Usage markdown with up-to-date conformance tests
• remove platform related tests and add to #32 to be deferred but documented

Tasks:

• review v0.3 deck
• make adjustments to ### [Documentation] update PSEUDO-CODE.md to reflect updated content on slides v0.3
• submit pull request with edits
• peer review of pull request
• merge pull request into master branch

Goal: Review Installable and Upgradeable tests and add more content

Tasks:

• Add missing tests from older slides to https://docs.google.com/presentation/d/1K81kQgqs3HMp-P1TVoeI6D2kIz7Qbz9JjTFohfTb9yA/edit#slide=id.g7d332a6c45_0_15
• Review Installable and Upgradeable Tests
• research upstream tests
• add more content, if possible
• Create an outline
• Create a rough draft
• Peer review of draft
• Iterate as needed, peer review (optional)
• Add content to https://docs.google.com/presentation/d/1K81kQgqs3HMp-P1TVoeI6D2kIz7Qbz9JjTFohfTb9yA/edit#slide=id.g7d332a6c45_0_51

ref: s41

[Ownership] Please rename and transfer to cncf/cnf-conformance

• Rename from cnf_conformance to cnf-conformance
• Transfer from vulk to cncf

Pre-Req:

• Slide content updated
• Add outline for test to categories markdown
• Investigate potential tools for implementation and document findings > https://hackmd.io/8rGSATfsQg63fXW4VO16PA?view
• select a tool to use to check privileged mode, minimal/least effort

Tasks:

• optionally, add pseudocode to markdown
• Add new POC test code, to Test if a CNF is running in Privileged mode

Goal: Review Compatibility Tests and add more content

Tasks:

• Add missing tests from older slides to https://docs.google.com/presentation/d/1K81kQgqs3HMp-P1TVoeI6D2kIz7Qbz9JjTFohfTb9yA/edit#slide=id.g7d332a6c45_0_15
• Review Compatibility Tests
• research upstream tests
• add more content, if possible
• Create an outline
• Create a rough draft
• Peer review of draft
• Iterate as needed, peer review (optional)
• Add content to https://docs.google.com/presentation/d/1K81kQgqs3HMp-P1TVoeI6D2kIz7Qbz9JjTFohfTb9yA/edit#slide=id.g7d332a6c45_0_15

ref: s35

Goal: Review Scalability Tests and add more content

Tasks:

• Add missing tests from older slides to https://docs.google.com/presentation/d/1K81kQgqs3HMp-P1TVoeI6D2kIz7Qbz9JjTFohfTb9yA/edit#slide=id.g7d332a6c45_0_15
• Review Scalability Tests
• research upstream tests
• add more content, if possible
• Create an outline
• Create a rough draft
• Peer review of draft
• Iterate as needed, peer review (optional)
• Add content to https://docs.google.com/presentation/d/1K81kQgqs3HMp-P1TVoeI6D2kIz7Qbz9JjTFohfTb9yA/edit#slide=id.g7d332a6c45_0_33

ref: s37

Goal: Update high-level README.md for CNF Conformance test categories

Tasks:

• Update high-level README.md
• Add what we are aiming to do w/ CNF Conformance Tests
• Add high-level CNF Conformance test categories
• Create an outline based on https://docs.google.com/document/d/1Ncg_Z8Yq-hKixruHfutG-ASi3Qm0Bi1DaRyXF80EJ0g/edit
• Peer review of outline
• Create a rough draft
• Peer review of draft
• Iterate as needed, peer review (optional)
• Add content to README.md

Acceptance Criteria

Configuration test: is there a readiness entry in the Helm chart?

EPIC: CNF Conformance Test Suite

Release: v0.0.1

Test Category: Configuration and Lifecycle

Type of test (static or runtime) =

• example code: #72

Environment set up tasks:

Upstream tool set up tasks: (conformance suite + upstream tools)

• N/A

CNF setup Tasks

Sample CNF tasks:

Documentation tasks:

• Update Test Categories md if needed
• Update Pseudo Code md if needed
• Update [USAGE md]Code implementation tasks:
  (https://github.com/cncf/cnf-conformance/blob/master/USAGE.md) if needed
• Update installation instructions if needed

QA tasks

Dev Review:

• walk through A/C
• do you get the expected result?
• if yes, move to Needs Peer Review column
• if no, document what additional tasks will be needed

Peer review:

• walk through A/C
• do you get the expected result?
• if yes, move to Reviewer Approved column
• if no, document what did not go as expected, including error messages and screenshots (if possible)

[Acceptance Criteria]

[Documentation] Create slides to show running all of the CNF Conformance tests

Short description:

• to create example slides to show screenshots of running all tests (as of Feb 26)

EPIC: CNF Conformance Test Suite

Release: v0.0.1

Test Category: all

Type of test (static or runtime) = both

Tasks:

• follow steps in https://github.com/cncf/cnf-conformance/blob/master/USAGE.md
• run

crystal src/cnf-conformance.cr all

• capture screen grabs of process
• create slide deck
• add images to slides

[Implementation] rename command to Test if the Helm chart is valid

What I see now in https://github.com/cncf/cnf-conformance/edit/master/USAGE.md:

(WIP) Test if the Helm chart is valid)

crystal src/cnf-conformance.cr helm_chard_valid

What I would like to see instead:

(WIP) Test if the Helm chart is valid)

crystal src/cnf-conformance.cr helm_`chart`_valid

Proof of Concept: Scoring results from tests

Test Category: Scoring

Type of test (static or runtime) = N/A

Tasks:

• Add a quick overview of the type of test you are researching/creating a POC for
• Investigate potential tools for implementation and document findings, https://hackmd.io/new (add link to ticket)
• Select a tool to use, minimal/least effort, and add selection to ticket
• Add new POC test code
• Add comment suggesting updates as needed for:
  • the test categories markdown
  • the pseudo code markdown
  • slide content updates, LINK_TO_UPDATES
  • the README
• Tag 1 or more people to peer review

Reference:

https://docs.google.com/presentation/d/1Bci-rzXnVqlvT2hhSXCBawtsKf8HURC2K4kJSlnB43Q/edit#slide=id.g7c55edb53f_0_535

If the CNF Conformance Test Suite proves useful, it would be natural to build out a certification program for CNFs
This could replicate the self-certification process used by the Certified Kubernetes program
It can be done in conjunction with LF Networking’s OPNFV Verification Program (OVP) and the Common NFVI Telco Task Force (CNTT)
One option would to be to emulate LEED Certification and the Core Infrastructure Initiative Best Practices Badge by having the test suite specify scoring ranges as bronze, silver, etc.
We also want probably want to explicitly include a “not-so-great” result meaning that that the CNF does not implement some base level of best practices
It is not clear that there is any need for platform conformance and certification beyond what is already provided by Certified Kubernetes

[Documentation] update PSEUDO-CODE.md to reflect updated content on slides v0.3

Short description:

• update the Pseudo-Code markdown with up-to-date conformance tests
• remove platform related tests and add to #32 to be deferred but documented

Tasks:

• review v0.3 deck
• make adjustments to https://github.com/cncf/cnf-conformance/blob/master/PSEUDO-CODE.md
• submit pull request with edits
• peer review of pull request
• merge pull request into master branch

Goal: Review Security Tests and add more content

Tasks:

• Add missing tests from older slides to https://docs.google.com/presentation/d/1K81kQgqs3HMp-P1TVoeI6D2kIz7Qbz9JjTFohfTb9yA/edit#slide=id.g7d332a6c45_0_15
• Review Security Tests
• research upstream tests
• add more content, if possible
• Create an outline
• Create a rough draft
• Peer review of draft
• Iterate as needed, peer review (optional)
• Add content to https://docs.google.com/presentation/d/1K81kQgqs3HMp-P1TVoeI6D2kIz7Qbz9JjTFohfTb9yA/edit#slide=id.g7d332a6c45_0_27

ref: s38

Research and Documentation: OPNFV SampleVNF for creating example CNFs

Research and POC spike for using OPNFV SampleVNF to create example CNFs

Tasks:

• review https://01.org/intel-data-plane-performance-demonstrators/prox-overview
• review https://git.opnfv.org/samplevnf
• Create https://hackmd.io/ md file to document if OPNFV SampleVNF could be used for creating example CNFs => https://hackmd.io/@nickolaev/SJXuPeV48
• Peer review

[Documentation] Re-name "Hardware and Affinity support" to "Hardware Resources and Scheduling Tests" in USAGE.md, Pseudo Code and Test Categories

Short description:

• fix typo
• rename Hardware and Affinity support Tests to Hardware Resources and Scheduling Tests

Tasks:

• Update USAGE https://github.com/cncf/cnf-conformance/blob/master/USAGE.md#scaling-tests
• Update PSEUDO-CODE https://github.com/cncf/cnf-conformance/blob/master/PSEUDO-CODE.md
• Update TEST-CATEGORIES.md

Please change license to Apache

• Apache License
• Version 2.0, January 2004
• http://www.apache.org/licenses/

WIP Acceptance Criteria

Installable test: Is the CNF's helm chart valid / Does the CNF's helm chart pass linting?

EPIC: CNF Conformance Test Suite

Release: v0.0.1

Test Category: Installable + Upgradeable

Type of test (static or runtime) = static

Environment set up tasks:

• Follow the installation instructions to set up the the conformance suite for development and testing
• Update installation instructions to cover new setup and existing workstation updates. Example:
  • You should have a working conformance suite that compiles and runs correctly. (eg. crystal-lang installed)
  • If you have it installed, pull down latest version of CNF conformance test code from develop branch (eg. git fetch ; get rebase from the cnf-conformance directory)
  • Run shard install to get dependencies

Upstream tool set up tasks: (conformance suite + upstream tools)

• N/A linter is part of Helm, Helm is installed already

Sample CNF tasks: (eg. Vendor CNF)

• update sample cnf-conformance.yml for using CoreDNS as a valid CNF in the sample coredns folder
  • Example config:

  ---
  cnf_name: coredns
  deployment_name: coredns # coming in issue #24
  chart_path: cnfs/coredns 
  

• Create a bad helm chart CNF Sample based on CoreDNS working chart
• Create a sample configuration for testing a CNF
  • cnf folder name (that's a subdir of the /cnfs/ folder)
  • chart folder name

  ---
  cnf_name: badhelmchart
  deployment_name: badhelmchart # coming in issue #24
  chart_path: cnfs/sample-bad-helm-chart
  

Target CNF Tasks (eg. How do we use the sample CNF)

DEFERRED See lfn-cnti/certification#6 (comment)

• Check /cnfs folder for CNFs.
  • For each folder check inside for a cnf-conformance.yml. If the file is found run conformance tests for that CNF
• Update documentation to say that each CNF should be in a subfolder of the CNF

Code implementation tasks:

• If one test blows up (eg. CNF code not found for static) the suite should continue to run
• Add code to run helm linter
• Add code for checking results in output from the linter
• Add code for specifying the helm chart path
  • Support finding helm chart path from target CNF configuration file. Example pseudo configuration:

  ---
  cnf_name: coredns
  deployment_name: coredns # coming in issue #24
  chart_path: cnfs/coredns 
  

  • Support finding helm chart path from CLI option. Example pseudo code: --cnf-chart-path cnfs/coredns/cnf-conformance.yml
• Print the name of the chart that was tested
• Print if the test was PASS or FAIL
• Make PASS green and RED fail
• Add the this new test to as a dependent test to the appropriate category so that it runs when that category runs

Documentation tasks:

• Update Test Categories md if needed
• Update Pseudo Code md if needed
• Update USAGE md if needed
• Update installation instructions if needed

QA tasks

Dev Review:

• walk through A/C
• do you get the expected result?
• if yes, move to Needs Peer Review column
• if no, document what additional tasks will be needed

Peer review:

• walk through A/C
• do you get the expected result?
• if yes, move to Reviewer Approved column
• if no, document what did not go as expected, including error messages and screenshots (if possible)

(WIP) Performing K8s conformance testing by running Sonobuoy on the cluster

Part of: Compatability Tests (WIP)

Tasks:

• Create markdown to document Sonobuoy tests
• TBD

To help with developing and continually testing the CNF Conformance suite it would be nice to have a set of sample CNFs to use.

General requirements or guidelines for the samples:

• Easy to install (eg. helm v3.0)
• No host modifications
• Easy to install K8s extensions (if required), with no host reboot required
• Information on how the CNF is used in a real world use case (so it's purpose can be communicated)

Feel free to ideas for CNFs

• directly to the ticket
• more extended info to the hackmd shared markdown (or a new one if needed)

Goal: Review Planned Implementation and add more content

Tasks:

• Review Planned Implementation
• Create copy of slides
• add more content
• Add more details about static tests
• Add more details about runtime tests
• Add url to github repo
• Create an outline
• Create a rough draft
• Peer review of draft
• Iterate as needed, peer review (optional)
• Add content to https://docs.google.com/presentation/d/1K81kQgqs3HMp-P1TVoeI6D2kIz7Qbz9JjTFohfTb9yA/edit#slide=id.g47ab1bdd76_0_54

ref: s25

Goal: Review Observability and add more content

Tasks:

• Add missing tests from older slides to https://docs.google.com/presentation/d/1K81kQgqs3HMp-P1TVoeI6D2kIz7Qbz9JjTFohfTb9yA/edit#slide=id.g7d332a6c45_0_15
• Review Observability Tests
• research upstream tests
• add more content, if possible
• Create an outline
• Create a rough draft
• Peer review of draft
• Iterate as needed, peer review (optional)
• Add content to https://docs.google.com/presentation/d/1K81kQgqs3HMp-P1TVoeI6D2kIz7Qbz9JjTFohfTb9yA/edit#slide=id.g7d332a6c45_0_45

ref: s40

What are open source tools and projects which can be used in testing CNF Compliance.

Some examples could be of tools and areas to test

• APIsnoop: Use Kubernetes audit logs to track every K8s API call used by a CNF and therefore specify what version of K8s is required and any beta or alpha API usage
• Helm v3 chart testing (ct): linting and testing
• Helm v3 installation: Functional test on CNF Testbed
• Validate Prometheus metrics using promtool
• Validate OpenTelemetry metrics
• Validate Fluentd-compatible logging

Acceptance Criteria

[Documentation] create an INSTALL.md

Short description:

• to provide potential project contributors with an install guide to how they can set up the CNF Conformance Test Suite on their device

Location:

• https://github.com/cncf/cnf-conformance

Tasks:

• Reference upstream install guides, as needed
• Create an outline
• Create a rough draft
• Peer review of draft
• Iterate as needed, peer review (optional)
• Add content to https://github.com/cncf/cnf-conformance/blob/master/INSTALL.md

[Documentation] Create a .md to share research on tooling options

Short description:

• to document which tools we investigated
• to document which tools we implemented
• to document which tools we did not implement

Tasks:

• create rough draft
• new file in https://github.com/cncf/cnf-conformance/tree/master/tools for Tooling-Research
• add content to Tooling-Research.md
• peer review
• merge

[Bugfix] rename command to run all of the scalability tests from "scaling" to "scalability"

Scalability Tests

What we see now on https://github.com/cncf/cnf-conformance/edit/master/USAGE.md,

To run all of the scalability tests

crystal src/cnf-conformance.cr scaling

What I would like to see:

To run all of the scalability tests

crystal src/cnf-conformance.cr scalability

[Documentation] add QUICK START to main README