containerd / project Goto Github PK

Cross-project utilities, scripts, etc.

License: Apache License 2.0

Shell 100.00%

project's Introduction

containerd common project repository

This cross-project repository holds utilities, scripts, and common files used across the containerd core projects and many sub-projects within the containerd organization.

release-tool utility

The release-tool utility, previously maintained here in the common project repository, has its own repository now. It is now located in the release-tool repo.

Common scripts

To reduce the need for several copies of tools which perform DCO checks, file license header checks, and Go vendor compliance, a set of scripts is maintained here and used within the CI configuration of various projects within the containerd organization.

The simplest way to see the integration of this common project into a Travis CI configuration is to look at the example in the containerd/continuity project here.

Common tools are retrieved during install: with go get, and then this containerd/project repository is cloned in the before_script:. Finally, tools from the project repository are then used for checking DCO signoff, license file headers, and vendoring matching between the commit and vendor config file.

Project core documents

Project governance, maintainer list, and contributing guidelines are all maintained as single copies within this repository and linked to from all containerd projects to reduce duplication and maintenance across all repos.

You can see each of these core documents here:

For an example of how to include these in a project's README.md file see the following markdown:

## Project details

{Some-project} is a containerd sub-project, licensed under the [Apache 2.0 license](./LICENSE).
As a containerd sub-project, you will find the:
 * [Project governance](https://github.com/containerd/project/blob/main/GOVERNANCE.md),
 * [Maintainers](https://github.com/containerd/project/blob/main/MAINTAINERS),
 * and [Contributing guidelines](https://github.com/containerd/project/blob/main/CONTRIBUTING.md)

information in our [`containerd/project`](https://github.com/containerd/project) repository.

Non-core project documents

If your project is a non-core addition to the containerd GitHub organization, please make the following changes to your project once approved and added:

Clearly state in an opening sentence within your project README.md that "Project X is a non-core subproject of containerd"
Add the project details boilerplate provided above with the following two changes:
1. The first line should be modified to state: {Some-project} is a non-core containerd subproject
2. Do not link to the core MAINTAINERS file in containerd/project. That link should be modified to point to your existing non-core project MAINTAINERS file.

project's People

Contributors

Stargazers

Watchers

project's Issues

Proposal for taking a non-core project "nydus-snapshotter" in

Containerd is awesome. Containerd remote snapshotter design is awesome too. Nydus is a container image acceleration solution. It supports container image lazy loading and a few other nice features. It comes with a remote snapshotter to work smoothly with containerd. We want to propose to add nydus-snapshotter as a non-core subproject to the containerd community.

We started the nydus project from Alibaba Cloud and Ant Group about 3 years ago and it is now deployed in Ant Group in a large scale and provides stable container image services to thousands of container images and dozens of customers varying in business fields. It benefits our internal users and customers by significantly reducing container startup time. Alibaba Cloud Function Compute also utilizes nydus as its code package solution.

Nydus offers a container image oriented and optimized filesystem format which provides the following key features,

Chunk level data de-duplication among layers in a single repository to reduce storage, transport and memory cost
Deleted(whiteout) files in certain layer aren't packed into nydus image, therefore, image size may be reduced
E2E image data integrity check. So security issues like "Supply Chain Attack" can be avoided and detected at runtime
Integrated with CNCF incubating project Dragonfly to distribute container images in P2P fashion and mitigate the pressure on container registries
Different container image storage backends are supported. For example, Registry, NAS, Aliyun/OSS and applying other remote storage backend like AWS S3 is also possible.
Record files access pattern during runtime gathering access trace/log, by which user's abnormal behaviors are easily caught. So we can ensure the image can be trusted

Beyond above essential features, nydus can be flexibly configured as a FUSE-base userspace filesystem or in-kernel EROFS with an on-demand loader userspace daemon and integrating nydus with VM-based container runtime is much easier.

Lightweight integration with VM-based containers runtime like KataContainers. In fact, KataContainers is considering supporting nydus as a native image acceleration solution. The ongoing PR is add nydusd support to introduce lazy load ability
Nydus closely cooperates with Linux in-kernel disk filesystem EROFS. Containers' rootfs can directly be set up by EROFS with lazy pulling capability. The corresponding changes had been merged into Linux kernel since v5.16 EROFS_What_Are_We_Doing_Now_For_Containers.pdf

We already gave a presentation on KubeCon NA 2021 about nydus' value Faster Container Image Distribution on
a Variety of Tools with Lazy Pulling

Nydus cooperates with Harbor to develop Harbor Acceleration Service to commodiously convert OCI image to nydus format.

To help to boom the containerd community's snapshotter, provide more choices to adapt various use cases for users and attract more developers. I am asking the containerd community to take nydus-snapshotter in as a non-core project and @changweige and @eryugey will be the initial maintainers

If I misunderstood the process of proposing a non-core project, please correct me.

9 maintainers' LGTMs are required (2/3)

[governance] discuss Moby TSC umbrella vs. in-project governance only

Per the CNCF TOC graduation vote discussion, this issue reflects our commitment to revisit project governance with external connection to the Moby TSC.

Add non-core project "remote-snapshotter" to containerd organization

Pull operation is one of the time-consuming steps in container's lifecycle. To address this issue, we've discussed "Remote Snapshotter" plugin (containerd/containerd#3731, containerd/containerd#2968 and containerd/containerd#2943, etc.). It allows containerd to prepare snapshots by directly mounting them from remote stores, which is much faster than downloading and unpacking entire image contents.

ktock/remote-snapshotter is a Remote Snapshotter implementation based on the discussions. It's been used as a blueprint in containerd/containerd#3731 also used for testing some Remote Snapshotter related patches containerd/containerd#3911, containerd/containerd#3846 and containerd/containerd#3793). It supports standard-compatible remote snapshots with stargz image format by default and CermVM-FS is also under the integration work leveraging the remote-snapshotter's pluggable architecture. For more details about the functionality of remote-snapshotter, please see our documentation.

We still need further discussion inside/outside of containerd including client tools like CRI plugin and Buildkit for improving and leveraging Remote Snapshotter functionality. Adding remote-snapshotter to containerd project umbrella is valuable to encourage such cross-project discussions based on the official implementation and also to gather more collaborators for this movement.

Kohei Tokunaga(@ktock) have prepared the ktock/remote-snapshotter repository for migration to the containerd organization. This is specifically proposed as a non-core project per the recent project modes we added to containerd's governance. Kohei Tokunaga(@ktock) will be an initial maintainer of the repository. According to the non-core document requirement, we will add the boilerplate to the project (containerd/stargz-snapshotter#23).

9 maintainer's LGTM required (2/3)

CI hygiene tracking issue

Due to various deprecations in GitHub Actions, we need to walk through all our sub-projects and get the following updates in flight. Since it's a fairly long list a tracking issue here in the overall project repo might help coordinate/track finishing this over time.

Deprecations

Ubuntu 18.04 OS runners: 18.04 is being deprecated and only 20.04 and 22.04 are supported. For many sub-projects for which we produce no binaries this should be an easy upgrade. For our main branch we have already made the change and our release process is "protected" by using containers in which we build the release binaries against the 18.04 LTS glibc/libraries. For release/1.5 and release/1.6 we may need to figure out how to continue to make releases without having 18.04 runners.
Node12 as an actions runtime: affects any built-in actions we use which use Node12 as a runtime; looking at CI output, for us this looks to be concentrated on actions/checkout and actions/setup-go which should both be updated to use "@v3" at a minimum.

While we are at it, we should be cleaning up any uses of old Go releases and moving all sub-project CI to Go 1.19.x.

Core project branches

main already complete
release/1.6
release/1.5 (EOL on 28 Feb so not going to update)

Sub-project checklist

Non-core subprojects

Maintainers of these projects may already be handling these issues, but maybe we can check/help if they want help.

Add non-core project "ttrpc-rust" to containerd organization

The current containerd/ttrpc project is designed to be the GRPC for low-memory environments. The ttrpc is implemented with Golang and its serialization mechanism is protobuf, which is cross-language. As a result it is reasonable to have ttrpc implementation in other language. In the new guest agent of kata-containers project, we implemented a rust version of the agent and are planning to replace the RPC protocol with ttrpc. The initial version of ttrpc-rust has been put at alipay/ttrpc-rust and been tested in kata environment. We think containerd is the best place to host ttrpc-rust because here is the upstream of ttrpc, and this have discussed in #33.

@teawater and @lifupan have prepared the alipay/ttrpc-rust repository for migration to the containerd organization.

This is specifically proposed as a non-core project per the recent project modes we added to containerd's governance. The maintainers of the repository will be Hui Zhu and Fupan Li. According to the non-core document requirement, we will add the boilerplate to the project ( containerd/ttrpc-rust#4 ).

9 maintainer's LGTM required (2/3)

Validate all containerd subprojects have linkage to project documents

Need to go project by project and either replace maintainers, governance, and contributing guidelines copies (where they exist) or simply add the boilerplate to README.md of each project with links here to containerd/project:

Enable OpenSSF Scorecard Github Action

Hello, I am Joyce and I'm working on behalf of Google and the OpenSSF to help essential open-source projects improve their supply-chain security. The OpenSSF is a non-profit foundation dedicated to improving the security of the open-source community. It counts GitHub as a founding member.

The Scorecard system combines dozens of automated checks to let maintainers better understand their project's supply-chain security posture. It is developed by the OpenSSF, with direct support from GitHub.

Since containerd is widely used and has a important role in countless projects, it's been included in the OpenSSF's list of the 100 most critical open-source projects. Adopting the Scorecard Github Action will help you to increase the overall security of the repository and garantee that the repository and contribution process is safe from malicious sabotage.

However, the OpenSSF has also developed the Scorecard GitHub Action, which adds the results of its checks to the project's security dashboard, as well as suggestions on how to solve any issues (see examples below). This Action has been adopted by 1600+ projects already.

Would you be interested in a PR which adds this Action? Optionally, it can also publish your results to the OpenSSF REST API, which allows a badge with the project's score to be added to its README.

Any doubts or concerns please feel free to reach me out.

Governance process for removing under-maintained projects

Discussed during #34: we should document a simple process for removing a contributed non-core project given there is a higher chance they may end up unmaintained at some point in the future.

// cc: @yujuhong

Security Slam 2023 Participation

Hello containerd community!

In preparation for this year's Cloud Native Security Slam, we've completed a survey of CNCF end users across multiple industries, including Construction, Cybersecurity, Aerospace & Defense, Game Development & Consumer Services, Consulting, and Nuclear.

Through this survey, end users have identified their interest in seeing security improvements to the projects they use. We've asked them to share which Security Slam goals are most interesting to them— and we've compiled the results in a hope that this will help your prioritization during the upcoming event.

While some users have not authorized us to share their name, we've still included their responses in our calculation for you. We CAN tell you that the containerd responses included Procore, the US Space Force, Epic Games and Infosys.

After calculating the responses according to the interest-weight, we've found these to be the most interesting things that containerd end users would like to see, from the five possible Security Slam badges.

The Mechanizer
The Inspector
The Defender, The Chronicler (tie)

More information will be announced in the event kickoff webinar on October 10th, including how to register for cash & swag prizes, details about how success is measured, and resources to help achieve each of the badge goals.

If you can't make it to the webinar, a recording will be made available within 24hrs. It will be sent out to the community newsletter with any essential details you may have missed.

Join the community & sign up for the webinar here: https://community.cncf.io/cloud-native-security-slam/

A quick look at the 2023 Event Badges

The Chronicler

Ensure that security documentation has properly formatted data relating to software supply chain security decisions, including instructions for end users seeking to validate provenance artifacts.

The Inspector

Ensure that a security self-assessment has been completed according to TAG-Security documented standards.

The Cleaner

Bring all CLOMonitor non-security scores to 100% for the project, indirectly increasing overall supply chain security (Best Practices, Documentation, License, Legal).

The Defender

Ensure each project repo is accounted for within CLOMonitor; Ensure proper check set is assigned to each project repo; Bring security score to 100% for the project (This statistically decreases the future likelihood of vulnerabilities).

The Mechanizer

Ensure that every release has an automated mechanism to supply SBOM and provenance artifacts.

Add non-core project "imgcrypt" to containerd organization

Over the past year there were many PRs and discussions about the encrypted container image support proposed across OCI, Kubernetes, and the runtimes including containerd. We finally agreed on the stream processor implementation, available in the 1.3 release, to allow media types to have an external processor binary which can handle actions on that stream of bytes on behalf of containerd. This allows containerd's core to not include special case code or implementation details for this and any future special media types, modes of compression, and so on.

@stefanberger and @lumjjb have prepared the stefanberger/imgcrypt repository for migration to the containerd organization.

This is specifically proposed as a non-core project per the recent project modes we added to containerd's governance. The maintainers of the repository will be Stefan Berger and Brandon Lum.

For a broader view of this work across many projects and repositories, a recent comment by @lumjjb is extremely helpful to see the full scope and status of the work: opencontainers/image-spec#775 (comment)

Having this repository as part of the containerd organization is valuable as it allows users/operators to assemble and deliver the required stream processors which can handle the encrypted layer media types as part of the containerd project umbrella. This repository currently includes features for ctr which were not accepted in the containerd core, but are extremely helpful for those trying to handle the current "chicken and egg" issue of not having a tool to test the encryption and decryption of layers. As this work matures and these capabilities are available across tools like buildkit and various other container build pipelines, it will be less necessary to have a special ctr tool for testing.

9 maintainer's LGTM required (2/3)

Share some dev tools installation script among projects.

One example is golangci-lint. It is useful to make sure all projects are using the same version to keep the code convention identical.

Shutdown `[email protected]` mailing list and migrate to GitHub Security Advisories

The [email protected] mailing list is full of spams and almost completely useless.

Can we shutdown the list and just migrate to GitHub Security Advisories (https://github.com/containerd/containerd/security/advisories/new)?

People who strongly refuse to (or who are not allowed to) create an account on GitHub may still directly reach out to the Core Committers via email or other communication methods to report vulnerabilities.

v1/stats: Missing memory.oom_control fields

Hi,

I am playing with containerd package in our environment which is configured with cgroupv1. In our environment, we would like to have oom_kill metric from memory.oom_control to understand the # of times a process under a given cgroup has been killed due to oom. I notice we do have memory.events fields of the memory controller already taken care of in v2/stats. However, if someone is still using v1/stats, then they would not find those fields related to oom. Cgroupv1 has different name of this file (memory.oom_control). Would be great to include those metrics for v1 (although it has gone old now).

Thanks,
Vishal

Add non-core project "runwasi" to containerd organization

runwasi (https://github.com/deislabs/runwasi) is a Rust based containerd shim for running WASI workloads using a Wasm / WASI runtime. The current runtime is wasmtime, but could be extended to use other Wasm runtimes as well.

runwasi is also the basis for the spin and slight Wasm shims which live in deislabs/containerd-wasm-shims repo, and is also the basis for Docker + Wasm Preview using the wasmedge runtime.

I'd like to propose adding runwasi to @containerd organization as a non-core project. This proposal has no impact on the scope of the core project.

10 maintainer LGTM required (2/3 of 15 maintainers)

/cc @cpuguy83

Add non-core project "nerdctl" to containerd organization

nerdctl (https://github.com/AkihiroSuda/nerdctl) is a Docker-compatible CLI for containerd.

Unlike ctr tool, nerdctl has human-friendly and stable CLI based on Docker UI/UX.

nerdctl also supports complex operations that are not possible with ctr, e.g.,

Exposing ports, using CNI portmap (and RootlessKit for rootless mode): nerdctl run -p <PORT> ...
Creating restartable containers, using OCI hook: nerdctl run --restart=always ...
Pulling images (nerdctl pull) with ~/.docker/config.json and credential helper binaries such as docker-credential-ecr-login
Showing logs of detached containers, using binary log driver: nerdctl logs ...
Lazy-pulling using stargz snapshotter
Decrypting ocicrypt images

I'd like to propose adding nerdctl to @containerd organization as a non-core project.
This proposal has no impact on the scope of the core project.

10 maintainer LGTM required (2/3 of 14 maintainers)

add gometalinter.json?

Can we add gometalinter.json to this repo so that we can catch lint/fmt issues like containerd/cgroups#62 ?

cc @estesp

Add non-core project "rust-extensions" to containerd org

There has been growing interest in Rust language (ttrpc-rust is a good example) and it's a de-facto language for microVMs nowadays. I've been working on crates that allow extending various parts of containerd. Potentially this can open doors for new runtime areas (especially with projects like rust-vmm) written in Rust.

I'd like to propose adding rust extensions to @containerd organization as a non-core project to push it further (will be available at containerd/rust-extensions)

Currently there are 4 crates:

containerd-client - GRPC client to query containerd daemon APIs.
containerd-shim - shim library (similar to shim.Run in Go) that simplifies shim development.
containerd-shim-logging - shim binary logging.
containerd-shim-client - TTRPC protos and shim client.

The repo with code is temporarily available here for review.

This proposal has no impact on the scope of the core project.

9 maintainer's LGTM required (2/3 of 13 maintainers)

Clarify required LGTM counts for merging pull requests

We have been requiring 2 LGTMs for core and 1 LGTM for non-core, but it seems we haven't documented this.

Add non-core project "alibaba/accelerated-container-image", "alibaba/overlaybd" to containerd organization

The existing lazy pulling image formats are filesystem-based. However, implementing a POSIX-complaint file system interface and exposing it via the OS kernel is complex. And underlay filesystem support is restricted in current file-system-based image services.

We consider a block device to support lazy-pulling image which has fully posix-compliant and multiple filesystem supported as a supplement.

Accelerated container image is an open-source implementation of paper "DADI: Block-Level Image Service for Agile and Elastic Application Deployment. USENIX ATC'20". It contains two components:

overlaybd-snapshotter alibaba/accelerated-container-image, a containerd snapshotter plugin.
overlaybd alibaba/overlaybd, the storage backend which provides an implementation of overlaybd by TCMU

And several features show as below:

High Performance It's a block-device-based storage of OCI image, which has much lower complexity than filesystem-based implementations
High Reliability overlaybd outputs virtual block devices through TCMU (TCM in Userspace), which is widely used and supported in most operating systems. Overlaybd backstore can recover from failures or crashes, which is difficult for FUSE-based image formats.
Efficiency virtualization supported passing a block-device from host to microVM (like kata-container) via virtio-blk is usually less performance cost. On the other hand, passing 'rootfs' from host to microVM via virtio-fs is also supported after overlaybd has been mounted.
Native Support for Writable overlaybd can be used as a writable/container layer. The end-users can build their overlaybd images naturally without conversion.
Multiple File System Supported overlaybd provide a virtual block device, which is independent of the underlay filesystem. It's convenient for users to choose their ideal image filesystem, such as xfs, btrfs, zfs even NTFS, and makes it possible to run Windows container on Linux host, or vice versa.

By now, overlaybd has been deployed at scale in the production environment of Alibaba and Alibaba Cloud Function Compute (FaaSNet: Scalable and Fast Provisioning of Custom Serverless Container Runtimes at Alibaba Cloud Function Compute. USENIX ATC'21), serving one of the world’s largest ecommerce platforms.

Furthermore, Microsoft is interested in using overlaybd to build their container ecosystem (Azure). Intel will also take some Linux kernel development work based on overlaybd.

We have prepared the 'alibaba/accelerated-container-image' and 'alibaba/overlaybd' 2 repositories for migration to the containerd organization. It will be more beneficial to other developers. This is proposed as a non-core project, @BigVan and @liulanzheng will be the initial maintainers of the repositories.
9 maintainer's LGTM required (2/3)

[Proposal] Templates for Pull Requests

Nerdctl contributors were recently talking about having pull request templates / good commit messages. One thing that was proposed was Github PR templates

https://docs.github.com/en/communities/using-templates-to-encourage-useful-issues-an[…]requests/creating-a-pull-request-template-for-your-repository

The PR template can ask the following questions:

What is the change?
Why is the change important (can be fixes something)?
Any gotchas to be aware of?

cc @containerd/nerdctl-committers & @containerd/nerdctl-maintainers

Add non-core project "nri" to containerd organization

See containerd/containerd#4411

NRI is a proposed new interface for managing resources. While this proposal is very new, I am proposing we do the work under a new sub-project rather than adding directly in containerd/containerd or an outside namespace. Inside the organization is a better place to collaborate and has proven to be a good place for other early and experimental work. As non-core projects do not have any pre-defined end state, the project could become a core project, go to another organization, or be removed if we decide on something else.

The repository will be created and setup in accordance with non-core project layouts as github.com/containerd/nri

9 maintainer's LGTM required (2/3)

GOVERNANCE.md: replace "mailing list" with "pull requests"

We have not been using the mailing list, so we should replace "the mailing list" in GOVERNANCE.md to "pull requests".

https://github.com/containerd/project/blob/b73a06ec6945bc2788c3ce8ac3ac40d8fd4eaa8b/GOVERNANCE.md

Adding maintainers

...
Periodically, the existing maintainers curate a list of contributors that have shown regular activity on the project over the prior months. From this list, maintainer candidates are selected and proposed on 🔴 the maintainers mailing list.

After a candidate has been announced on 🔴 the maintainers mailing list, the existing maintainers are given five business days to discuss the candidate, raise objections and cast their vote. Votes may take place on 🔴 the mailing list or via pull request comment. Candidates must be approved by at least 66% of the current maintainers by adding their vote on 🔴 the mailing list. The reviewer role has the same process but only requires 33% of current maintainers. Only maintainers of the repository that the candidate is proposed for are allowed to vote.
...

Removal of inactive maintainers

...
If the maintainer wants to remain a maintainer, but is unable to perform the required duties they can be removed with a vote of at least 66% of the current maintainers. An 🔴 e-mail is sent to 🔴 the mailing list, inviting maintainers of the project to vote. The voting period is five business days. Issues related to a maintainer's performance should be discussed with them among the other maintainers so that they are not surprised by a pull request removing them.

Adding ttrpc-rust as sub-project

We just open source our ttrpc-rust implementation in https://github.com/alipay/ttrpc-rust and https://crates.io/crates/ttrpc .
We would like add it as sub-project of containerd.

Please help us about the process. Thanks!

Add non-core project "stargz-snapshotter" to containerd organization

Pull operation is one of the time-consuming steps in container's lifecycle. To address this issue, we've been discussing "Remote Snapshotter" plugin. Please refer to the previous proposal(#41) for the general context in containerd community.

"stargz-snapshotter"(ktock/stargz-snapshotter) is a remote snapshotter implementation which has the following features:

standard compatible remote snapshots leveraging stargz format by CRFS, which we can use with standard-compliant registry
generic remote snapshotter package which is importable as Go packages and configurable for other remote snapshotter developers(based on the previous discussion #41)

Adding stargz-snapshotter to containerd project umbrella as the first implementation of remote snapshotters is valuable to push the remote snapshotter's movement forward. Following-up remote snapshotters can use the generic(and official) parts to easily create their implementation with filesystems they like. There is an ongoing work of implementing remote snapshotter in CernVM-FS community(ktock/stargz-snapshotter#27), but we propose only stargz-based snapshotter here.

Kohei Tokunaga(@ktock) have prepared the ktock/stargz-snapshotter repository for migration to the containerd organization. This is proposed as a non-core project and Kohei Tokunaga(@ktock) will be an initial maintainer of the repository. According to the non-core document requirement, we will add the boilerplate to the project (ktock/stargz-snapshotter#23).

9 maintainer's LGTM required (2/3)

Enable OpenSSF allstar for the entire org (in addition to scorecard in the core repo)

OpenSSF scorecard was enabled for the core repo in containerd/containerd#7404 but probably we don't want to maintain this for all the repos under the containerd org.

Can we enable allstar for the entire org, in addition to the scorecard in the core repo?

Migrate common content to the "project" repo

The expectation is that we have a mix of content across the various containerd sub-project repos, some of it duplicated, and some of it that is unevenly applied (e.g. DCO checks).

This project repo will be a collection point for common content/scripts that can then be integrated into subproject CI (e.g. git clone https://github.com/containerd/project; ...) and used with uniformity across all projects that need a specific functionality.

If it makes sense we can also add some common documentation to this repo (general contributing guidance, license, and other common info), and potentially move the containerd website content to this repo as well.

containerd / project Goto Github PK

project's Introduction

containerd common project repository

release-tool utility

Common scripts

Project core documents

Non-core project documents

project's People

Contributors

Stargazers

Watchers

Forkers

project's Issues

Deprecations

Core project branches

Sub-project checklist

Non-core subprojects

A quick look at the 2023 Event Badges

The Chronicler

The Inspector

The Cleaner

The Defender

The Mechanizer

Adding maintainers

Removal of inactive maintainers

Recommend Projects

Recommend Topics

Recommend Org