opencontainers / oci-conformance Goto Github PK

@stephenrwalli has done some great work here:
https://github.com/stephenrwalli/OCI_testing

We should all review

Keppel by SAP: https://github.com/sapcc/keppel

Shall we add one note that "no company can review its own results" to the following clause if "peer verification" doesn't indicate it?

(4) Peer Verification

One or more members of the OCI Certification Program Working Group (Cert WG) may attempt to reproduce the test results for your product and may contact you with questions.

So that people running tests have a clear indication of not just YES/Pass or NO/Fail, we may want to also include a "not implemented" value if a particular functionality is simply not implemented.

(This issue was raised in an OCI Runtime ConCall the week of 5/8 - 5/12/2017)

The OCI runtime validation suite is able to test runtimes that expose the OCI Runtime interface to users (operations create, start, etc. and the config.json file). It should be possible to test software like runc, bwrap-oci, crun, railcar.

What about software that use a OCI runtime internally but don't directly expose the OCI Bundle to users, such as containerd, Docker, CRI-O? Should they be able to be certified in some ways?

I am not sure it would make much sense but I have not seen discussion whether it should be in scope of the certification or not.

Technically, if the software does not follow the OCI Runtime Command Line Interface, the validation suite cannot test it. Exposing such a CLI interface in containerd or CRI-O does not make sense to me, since their goal is to expose a gRPC interface which is semantically different to the OCI Runtime CLI.

/cc @caniszczyk

Regarding the conformance dashboard located here: https://oci.bloodorange.io

I think the source for that github pages site + CI should be migrated to this repo, and some domain setup such as conformance.opencontainers.org.

However, it's not clear which results should be included. What if, as part of submission for certification via PR, the submitter can set a field in PRODUCT.yaml:

ci: true

or something similar, which indicates that there is an associated CI definition located in this repo, which will be built on some timer, updating the dashboard results page.

As part of this, commercial vendors would need to supply secrets/credentials with access to some test registry that we can use for the tests.

In the example script, the following line exists:

cd tmp && docker build -t conformance:latest -f Dockerfile.conformance .

For the distribution spec, the Dockerfile is called Dockerfile, not Dockerfile.conformance. I'm not sure if this should be changed in the script or if it would be better to rename the Dockerfile in the distribution-spec repo.

A few ideas mentioned:

• Part of applying for certification includes committing to do peer verification for one (or two?) other products
• Scoreboard of peer verification participation

We should consider opening this repo to the public for wider feedback, especially from the OCI TDC and dev community.

Not sure should it be a program or a committee, e.g., the Trademark Board or Certification Working Group, to issue an official certificate of approval?

(5) Certify

If your product passes testing and peer review, the OCI Certification Program will issue an official certificate of approval.

Hey there! Django-oci is a bit different than the registries I see here in that it's not an official service, it's something that a user can implement as a plugin in their own Django app. That said, I do run the complete conformance testing suite: https://vsoch.github.io/django-oci/conformance/. Is there a way that we can include it here? Maybe a different kind of group that is for plugins and similar that implement the OCI distribution spec?

What privacy restrictions would we put in-place for certification of products before they are publicly announced?

Assigning to @caniszczyk since he has the next action of getting input from Linux Foundation colleagues.

Last year I'd floated Open Badges as a user-verifiable way for certified implementations to show their certification status. I think that should be part of the regular process, for a few reasons:

• It has space for an account of the achievement (e.g. “OCI Certified Runtime v1.0 on amd64 Linux. {link to opencontainers/certification issue with results}”). That makes it easy to show a pretty, tiny graphic (nice for the certified implementation) while still allowing access to the technical details (nice for a curious user).
• The issuer can revoke badges unilaterally, instead of the current policy of asking the certified implementation to remove their stale marks (which may be difficult if the certified implementation has become abandonware).

Thoughts?

I see different kinds of software and services who could be interested by a certification related to the image-spec:

1. any containerized software, to be able to say "this image of e.g. Redis is an OCI Certified Image".
2. container runtimes, to be able to say "the containerd runtime can generate a OCI bundle from an OCI image as per the OCI Image Spec"
3. registry services, to be able to say "all images provided by quay.io/Docker Hub respect the OCI image spec"
4. builders, like "docker build", or buildah

If I understand the description in #26, the focus of the OCI Certified Image program is on my point 1.

But parts of the image-spec are not only about the image format but about specifying the actions that a container runtime could do (my point 2). See image-spec/conversion.md:

• Extraction of the root filesystem from the set of filesystem layers.
• Conversion of the image configuration blob to an OCI Runtime configuration blob.

Shouldn't we have a certification program for this as well? The rules about the filesystem layers with the whiteout files can be tricky, so it seems useful to me to have a series of tests that container runtimes could run.

/cc @opencontainers/image-tools-maintainers @opencontainers/image-spec-maintainers

Certification program version deprecation should align with product cycles.

Ex: Products with software on disk may be easier to update than products with software burned into silicon.

I will continue to contribute to the runtime and image projects as a member of the technical community, but I need to bow out of certification work.

Migrate the contents of this repo:
https://github.com/bloodorangeio/oci-distribution-conformance-results

Add "verified" column to show whether or not registries have signed off on results

OCI-conformnace tests for all registry service providers are done via GitHub actions locating at .github/workflows/.

By looking through all the yaml files, all 4 tests against a single registry are conducted at the same time with the same repositories.
Taking ACR as an example,

• acr_1.yml is scheduled at 0 6 * * *, running against ocitest.
• acr_2.yml is scheduled at 0 6 * * *, running against ocitest.
• acr_3.yml is scheduled at 0 6 * * *, running against ocitest.
• acr_4.yml is scheduled at 0 6 * * *, running against ocitest.

It is possible that acr_2.yml is running in the middle but some tags or blobs are removed by the tear down phase of other tests, causing false positive results.

Here are two options on the solutions.

1. Run tests against different OCI_NAMESPACE per test scenario.
2. Run tests at different times like
   • 0 6 * * * for *_1.yml
   • 10 6 * * * for *_2.yml
   • 20 6 * * * for *_3.yml
   • 30 6 * * * for *_4.yml

Maybe we can fix it by simply renaming slug to azure-container-registry, but this may be bigger issue

cc @tianon

We should have a MAINTAINERS file and list the main maintainers that vote

@RobDolinMS @stephenrwalli

For test reports, publish and link to results in OCI AWS account

There's some mixed views on whether an OCI membership should be required for certification or not. We should have a discussion to come up with a final decision here, there are pros and cons, but I generally like to require membership with certification as it's a membership benefit and helps sustain the project over the long term.

Hi,
I'm thinking of making the certification program more automatically. The demo is like this: https://github.com/liangchenye/oci-cert-demo, a runtime project could add the certification result icon to its README.md.

The detailed explaination is here:
opencontainers/runtime-tools#527