Comments (13)
[WIP] Acceptance Criteria for peer review:
Acceptance Criteria:
Steps to test a CNF w/ a valid Helm chart:
- Pull down latest version of CNF conformance test code from
develop
branch - Run
shards install
- Run
crystal src/cnf-conformance.cr sample_coredns_setup
- Run
crystal src/cnf-conformance.cr helm_chart_valid
- Result: I would expect the output to be Pass with the CoreDNS CNF sample
Steps to test a CNF w/ an invalid Helm chart:
- Pull down latest version of CNF conformance test code from
develop
branch - Run shard install
- Run
crystal src/cnf-conformance.cr sample_coredns_cleanup
- Run
crystal src/cnf-conformance.cr bad_helm_cnf_setup
- Run
crystal src/cnf-conformance.cr helm_chart_valid
- Result: I would expect the output to be Failing
Steps to test a sample CNF w/ a valid Helm chart against all tests in the test category:
- Run
crystal src/cnf-conformance.cr cleanup
- Run
crystal src/cnf-conformance.cr sample_coredns_setup
- Run
crystal src/cnf-conformance.cr installability
- I expect to see PASSED for the helm valid test
Steps to test the entire CNF Conformance test suite with the sample CNF w/ a valid Helm chart:
- Follow Steps to test a CNF w/ a valid Helm chart from above
- Run
crystal src/cnf-conformance.cr all
- I expect to see PASSED for the helm valid test
from testsuite.
Level of effort in pts: 0, 1, 2, 3, 5, 8, 13, 21, infinity: 5 pts
@wavell 8 pts
@williscool 5 pts
@lixuna 5 pts
@denverwilliams 5 pts
@taylor 5 pts
ashleigh 5 pts
nupejosh 8 pts
from testsuite.
Added new task above:
- If one test blows up (eg. CNF code not found for static) the suite should continue to run
from testsuite.
Added to A/C:
Steps to test a sample CNF w/ a valid Helm chart against all tests in the test category:
- Follow Steps to test a CNF w/ a valid Helm chart from above
- Run crystal src/cnf-conformance.cr installability
- I expect to see PASSED for the helm valid test
from testsuite.
Deferring support for testing multiple CNFs which are in the /cnfs folder.
Target CNF Tasks (eg. How do we use the sample CNF)
- Check /cnfs folder for CNFs.
- For each folder check inside for a cnf-conformance.yml. If the file is found run conformance tests for that CNF
- Update documentation to say that each CNF should be in a subfolder of the CNF
from testsuite.
@williscool please add a sample invalid CNF helm chart and test that the linter fails. The CoreDNS working helm chart can be modified to be invalid. See A/C for testing.
I tested on develop
branch
rosenberg% git status
On branch develop
Your branch is up to date with 'origin/develop'.
I do not see a sample invalid cnf
rosenberg% ls sample-cnfs
sample-coredns-cnf
from testsuite.
@williscool, it looks like the setup task uses a different directory lfn-cnti/certification@89d81ce#diff-d8bebbf3115bd02eeb3a3e284a0535f7R75
Please match the cnf-conformance.yml to the cnf directory from setup
from testsuite.
@williscool Please update the CNF README to match the new sample cnf name, directory, etc
from testsuite.
@williscool It looks like some steps are not needed in this sample cnf including the helm chart install https://github.com/cncf/cnf-conformance/blob/develop/sample-cnfs/sample-bad_helm_coredns-cnf/README.md#core-dns-installation
Please review and remove the steps that are invalid for this sample
from testsuite.
@williscool the cnf-conformance.yml points to the coredns folder instead of a the "cnfs/bad_helm_chart" folder created in setup
From what I can tell it looks like the setup will copy that cnf-conformance.yml over and use it when the test runs.
from testsuite.
@wharris3, please add a comment with updated A/C steps for testing Steps to test a CNF w/ an invalid Helm chart:
It appears a new setup command is needed for that to run. Something like
rosenberg% crystal src/cnf-conformance.cr bad_helm_cnf_setup
Successfully created directories for cnf-conformance
Cloning into '/home/taylor/p/w/cncf/test/cnf-conformance/cnfs/bad_helm_coredns'...
remote: Enumerating objects: 58, done.
remote: Counting objects: 100% (58/58), done.
remote: Compressing objects: 100% (39/39), done.
remote: Total 55328 (delta 35), reused 30 (delta 19), pack-reused 55270
Receiving objects: 100% (55328/55328), 94.72 MiB | 3.04 MiB/s, done.
Resolving deltas: 100% (29285/29285), done.
Then run the test
rosenberg% crystal src/cnf-conformance.cr helm_chart_valid
FAILURE: Helm Chart stable/coredns Lint Failed
from testsuite.
@denverwilliams @wavell @williscool with the latest updates (PR 86, PR 87) is this ready for peer review again? If so please move to the peer review column and tag someone.
from testsuite.
A/C passes for helm chart. (PASS and FAIL scenarios)
from testsuite.
Related Issues (20)
- [Improvement] Optimize runtime of single_process_type test
- [Improvement] Refactor K8sTshark module
- Document TSC Decision for PR Approval
- [Feature] Request to support multiple Helm Charts in CNTI Test Catalog HOT 4
- [Feature] Drop or rework k8s_netstat library
- [BUG] Problem with cluster-tools using TTY as input HOT 5
- [Improvement] hardcoded_ip_addresses_in_k8s_runtime_configuration should give more information on failure
- [Improvement] Ignore bash/bash-script processes in single_process_type check
- [MAINTENACE] Clarify usage and behavior of testing multiple CNFs HOT 4
- Global timer makes litmus create verdict sooner than the AUT is ready causing the node drain test to fail HOT 3
- [BUG] CNF Cert test cases not working in offline mode. HOT 2
- [Feature] Warning for running specs with cnfs present
- [Documentation] Incorporate Lost Information into Revised Documentation
- [Feature] Support multiple container engines
- [MAINTENANCE] Remove AIRGAP mode from code HOT 1
- [Feature] Separate manifest installation mode from helm
- [Feature] Join installation modes HelmChart and HelmDirectory
- [Feature] Redesign oran_e2_connection test HOT 1
- [Feature] redesign observability tracing test
- [Feature] Redesign workload_resource_test function HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from testsuite.