mic92 / cntr Goto Github PK

Hi, I'm trying out cntr to debug a nix package, but I get the following error when trying to connect:

# cntr attach -t command cntr-/nix/store/cv9dl6yr37a3c0iqqjmz76ba5bsda01n-notmuch-0.35
unable to move container mounts to new mountpoint: EOPNOTSUPP: Operation not supported on transport endpoint

My system information:

• OS: NixOS unstable (my current NixOS configuration)
• cntr version: 1.5.1
• Filesystem info: /nix and / are two separate zfs datasets, /tmp is tmpfs

I have tried the following things, without change of error message:

• reboot
• run cntr as root with sudo
• run cntras root with su
• run cntr as root with doas
• run nix-build as root with sudo

It seems like some kernel options/capabilities/modules may be missing, but I don't know which one that would be.

The release workflow failed and the release does not include pre-compiled binaries.

To reproduce run umask 077 in your shell and then attach to any container running non-root user inside. After that try to run cntr exec. It fails, because /.cntr/pid file was created with permissions 600 and is not readable.

I've tested with zsh and bash.

Also with umask set to 000 the pid file is created with permissions 666 which means it's writable by anyone, which could potentially be a security issue?

To fix this the correct permissions should be set when this file is created:

Reported here: https://discuss.linuxcontainers.org/t/cntr-a-container-debugging-tool/1370/9

$ ./target/debug/cntr attach 5e020cea
no suitable container found, got the following errors:
  - ProcessId: not a valid pid: `5e020cea`: invalid digit found in string
  - Rkt: could not find container process belonging to rkt container '5e020cea': no child process found for pid 20849
  - Docker: Failed to list containers. 'docker inspect --format '{.State.Running};{.State.Pid}' 5e020cea' exited with exit code: 1: Error: No such object: 5e020cea
  - Nspawn: Failed to list containers. 'machinectl show --property=Leader 5e020cea' exited with exit code: 1: Could not get path to machine: No machine '5e020cea' known
  - Lxc: Failed to list containers. 'lxc-info --no-humanize --pid --name 5e020cea' exited with exit code: 1: 5e020cea doesn't exist

currently the error message is not too helpful.

I tested cntr the other day and it failed to attach to a container. I tried running the command as a non-root user in the docker group. It was installed via cargo install cntr.

Do you have any idea what is wrong here and how to fix it?

$ cntr attach 0531f16fe281
failed to setup /.cntr, failed to create setcap executable, Failed set file capability CAP_SYS_CHROOT on /tmp/cntr.p60d0d/cntr-exec, setxattr failed, EPERM: Operation not permitted

$ TMPDIR=/home/[redacted]/.cache cntr attach 0531f16fe281
failed to setup /.cntr, failed to create setcap executable, Failed set file capability CAP_SYS_CHROOT on /home/[redacted]/.cache/cntr.pw6ZLa/cntr-exec, setxattr failed, EPERM: Operation not permitted

hi~
After reading your paper, I have some questions, can you explain it ? thanks a lot, sorry for my poor english.

1. the paper said there's a way to configure where the CntrFS Server stay in host or fat container? but I didn't find the server in host and didn't see a new fat container? but I guess the default is that it run in the host. can you explain it? and how to configure it ?
2. the cntr needs fuse support, but I did't install the fuse kernel driver before, is the cntr dose the job?
3. the binary command such as vim/htop/top after attached the application container, where those located? in CntrFS Server or download from network when launch the CntrFS server?

thanks for your reply.

https://github.com/cberner/fuser now supports building without libfuse.
This way we don't need to maintain a fuse library ourself.

https://discuss.linuxcontainers.org/t/cntr-a-container-debugging-tool/1370/2?u=mic92

At the moment we have a relatively high FUSE api level: 7.26 and require ambient capabilities.
Both can be made optional and backwards compatible

If I have user namespaces enabled on a Linux box, the nix daemon can run as a "nobody" user with sandboxing enabled and this "just works" without issue.

However, if I then try to use cntr with an interactive breakpoint, I see this:

To attach install cntr and run the following command as root:

   cntr attach -t command cntr-/nix/store/q1l80p0z1v7gc2q7zri8xlkx5s2ap86k-debug

and indeed, it fails if not run as root, as advertised.

$ cntr attach -t command cntr-/nix/store/q1l80p0z1v7gc2q7zri8xlkx5s2ap86k-debug
    failed to get status of target process: Could not find namespace pid in /proc/221016/status

Is this a fundamental limitation or is there a way that I can find the nobody daemon's namespace pid (either as another user, or as the nobody daemon itself)?

As someone who manages their system using home-manager, there are no tools in the system wide config, the shell cntr gives me is barebones. Is it possible to get the shell with all the tools installed for my user when running cntr?

I suspect, I'd have to circumvent the sudo call somewhat? Maybe assign my user a certain group?

I tried out this tool coming from the nixpkgs documentation, I installed to my system configurations however it seems that when I try to attatch, via eg:
sudo cntr attach -t command cntr-/nix/store/hsi0wjy93c1cv98kr7qgpyaaw3fa2fd3-asm-lsp-0.5.0, the command just hangs and nothing happens. Doing a Ctrl+C takes me back to the prompt, however cntr keeps running in the background:

$ ps -aux | grep "cntr"
root       12215  0.0  0.0   3472  1536 ?        D    16:19   0:00 cntr attach -t command cntr-/nix/store/hsi0wjy93c1cv98kr7qgpyaaw3fa2fd3-asm-lsp-0.5.0
root       13466  0.0  0.0   3472  1408 ?        D    16:22   0:00 cntr attach -t command cntr-/nix/store/hsi0wjy93c1cv98kr7qgpyaaw3fa2fd3-asm-lsp-0.5.0
root       13530  0.0  0.0   3472  1536 ?        D    16:22   0:00 cntr attach -t command cntr-/nix/store/hsi0wjy93c1cv98kr7qgpyaaw3fa2fd3-asm-lsp-0.5.0 bash
root       13749  0.0  0.0   3472  1664 ?        D    16:22   0:00 cntr attach -t command cntr-/nix/store/hsi0wjy93c1cv98kr7qgpyaaw3fa2fd3-asm-lsp-0.5.0 bash
root       13929  0.0  0.0   3492  1408 pts/1    D    16:23   0:00 cntr attach -t command cntr-/nix/store/hsi0wjy93c1cv98kr7qgpyaaw3fa2fd3-asm-lsp-0.5.0
root       14519  0.0  0.0 222736  2688 pts/1    S+   16:27   0:00 grep cntr

(I might have tried a few times to attach).

I am also unable to sudo kill -9 13466, as in it doesn't get killed...

I am not sure if the root of the problem is some misconfiguration on my system, nixos, or cntr.

It shows:

When I try to use the socket-proxy branch of cntr I get the following error most of the time that I try to attach with any command:

sudo cntr attach -t command cntr-/nix/store/5shlk963b0ca6s7pd27rb7gpxp577c8v-java-debugme -- somecommand
failed to setup socket proxy: failed to bind socket '/tmp/cntrfs-sockets.wAcUADoChfPH/0': EACCES: Permission denied

Sometimes it works.

I want to get a ZIP file of a directory found in a cntr container - a logs directory of a failing Nix Build. I tried:

sudo cntr attach -t command cntr-/nix/store/...-${name} -- zip -r - build/source/tests/_logs > tests_logs.zip

But it doesn't work - the ZIP file generated is a text file with the text:

zip error: Invalid command arguments (cannot write zip file to terminal)

The -- convention cannot be used to differentiate arguments for cntr vs arguments for the command
For example,
cntr attach -t command cntr-/nix/store/xyzzy bash --rcfile foo.sh
fails. The issue is that cntr considers --rcfile to be a unknown option rather than an argument for bash.
If the -- convention were supported, then this probably would have worked:
cntr attach -t command cntr-/nix/store/xyzzy -- bash --rcfile foo.sh

See a description of -- here: https://www.gnu.org/software/libc/manual/html_node/Argument-Syntax.html

Steps to reproduce

1. Install Nix
2. Mount an NFS to /media/nas: mount -t nfs nas.lan:/ /media/nas
3. Bind mount /nix/store: mount -o bind /media/nas/nix-store /nix/store
4. Follow instructions listed here to create a debug sandbox.
5. Observe that executing /.cntr/cntr-exec gives the error "Operation not supported". And nothing in nix-store or my regular user is accessible(with the same error), even if I set --effective-user to root or my regular user.
6. Unmount /nix/store, rm -rf /nix/store, and reinstall Nix
7. Observe that the same nix-build stops during buildPhase with a container whose /.cntr/cntr-exec is executable, and that the expect /nix/store tools are available and execute fine.

I speculate that two issues are relevant:

1. cntr sets CAP_SYS_CHROOT on a copy of itself, but this is likely not persisted across NFS.
2. My NFSv4 is configured to squash all user ids to the same user, but in a multi-user Nix daemon installation there are 32 nixbld1 users. So it is possible that permissions given to those are being lost during the user id squash.

It seems reasonable to throw an explicit error if the permissions responsible cannot be set, rather than build an unusable container.

Does it provide completions for common shells like https://docs.rs/clap_complete/latest/clap_complete/?

Otherwise mount move from ctnr-temp to ctnrfs/var/lib/cntr fails with ENOENT

Ran into this error, I could resolve it by patching cntr to create /var/lib/cntr in the parent attach process right after mountns receive.

Sorry for the sketchy details.. let me know if you need more info

Not familiar with cntr, so I don't have much to add other than what I said in the workaround Nixpkgs pr… the fix was to make output not be a tty, and when I straced the pre-fix cntr invocation in the NixOS test, it hung right after some tty-related ioctls and I believe it had just received SIGWINCH. Didn't catch the actual call it was stuck on unfortunately.

1. why we need fuse support? if I created a new mount namespace, add I mount the slim container filesystem in /var/lib/cntr and mount debugging tool (whatever located in host or container) in /usr/bin/, everything seems ok, why need fuse? why redirect request between them?
2. furthermore, I did't know why UnionFS is not good? I think it can also expand the slim container at run-time, but the paper says "However, in practice, users still end up with multiple base images due to the use of different base image distributions across different containers", the question is that with using cntr, this question is solved? in production environment, different application (java php...) of course have different base images?
3. int step#3, why at begins mount at TMP/ and chroot into /? Because it was in a new nested namespace, what happens if I mount at / at begins?
4. How to with IPC namespace? in step#3, after you bind mount /proc, you set this process in application ipc namespace? in this paper, the term 'namespace' only refer to the mount namespace? This also confused me a lot.
5. when I use cntr attach pid , I saw the cntr forks two child process, can you explain the purpose of them? I know one of them attach to application containers, create nested namespace, finally chroot /.

Thanks for your reply.

https://discuss.linuxcontainers.org/t/cntr-a-container-debugging-tool/1370/4

I know that there is a version that works with Nix. If it merged into master yet?

Step 8/14 : RUN cargo build --release --target=x86_64-unknown-linux-musl
 ---> Running in c3fa8844dd31
    Updating crates.io index
error: failed to select a version for the requirement `clap = "=4.0.15"`
candidate versions found which didn't match: 3.2.23, 3.2.22, 3.2.21, ...
location searched: crates.io index
required by package `cntr v1.5.1 (/)`
The command '/bin/sh -c cargo build --release --target=x86_64-unknown-linux-musl' returned a non-zero code: 101

Tried with Cargo.lock from the pending PR as well and still the same issue.

Step 8/14 : RUN cargo build --release --target=x86_64-unknown-linux-musl
 ---> Running in fe6714ef2e60
    Updating crates.io index
error: failed to select a version for the requirement `clap = "=4.0.18"`
candidate versions found which didn't match: 3.2.23, 3.2.22, 3.2.21, ...
location searched: crates.io index
required by package `cntr v1.5.1 (/)`
The command '/bin/sh -c cargo build --release --target=x86_64-unknown-linux-musl' returned a non-zero code: 101

Will it be possible to install cntr on macOS?

The time zone for viewing date in cntr and docker exec is abnormal. How can I adjust it? Thanks!

I'm using cntr from nixpkgs 19.09 (on NixOS), doesn't matter of which package build I'm trying to debug, it just fails ans locks completely the terminal

sudo RUST_BACKTRACE=full cntr attach -t command cntr-/nix/store/ih6pz06jrn806lx1315gc1djklxh6zfc-terraform-provider-keycloak-1.13.2
[sudo] password for eon: 
WARNING: the per-user profile dir /nix/var/nix/profiles/per-user/root should belong to user id 1000
thread '<unnamed>' panicked at 'BUG! got unknown d_entry type received from d_type', src/fs.rs:602:14
stack backtrace:
0:     0x55c51067844c - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hd5cf01a6458823ce
1:     0x55c51065f72d - core::fmt::write::h239984b34be80164
2:     0x55c510677151 - std::io::Write::write_fmt::h71b3597bde52e201
3:     0x55c51067a490 - std::panicking::default_hook::{{closure}}::h22b55119be64c15e
4:     0x55c510679f65 - std::panicking::rust_panic_with_hook::hc671c362d83d7750
5:     0x55c510658684 - std::panicking::begin_panic::hbfd0096d822b4d01
6:     0x55c51060d92a - cntr::fs::CntrFs::generic_readdir::h62575c5ba9c58ab2
7:     0x55c51061733f - fuse::request::Request::dispatch::h596ab4aba69542f6
8:     0x55c510651452 - std::sys_common::backtrace::__rust_begin_short_backtrace::h800c518d8b197517
9:     0x55c51063e146 - core::ops::function::FnOnce::call_once{{vtable.shim}}::h2f3283e15ccecb31
10:     0x55c510676c5f - <alloc::boxed::Box<F> as core::ops::function::FnOnce<A>>::call_once::h36570ddedd92a948
11:     0x55c510677dac - std::sys::unix::thread::Thread::new::thread_start::hac7b969fab3af94c
12:     0x7f46fce81ef7 - start_thread
13:     0x7f46fcba122f - __GI___clone
14:                0x0 - <unknown>

I have tried to compile with both ways and in both I get compilation errors.

stacktrace cntr.txt

• I use an ark linux manjaro kde plasma 18.1.5
• I installed cargo 1.41.0
• i tried to install cntr but got rust specific error messages
• anyone knows what goes wrong ?

[rene@thinky api-platform]$ cargo install cntr
Updating crates.io index
Installing cntr v1.1.2
Compiling libc v0.2.66
Compiling maybe-uninit v2.0.0
Compiling cfg-if v0.1.10
Compiling log v0.4.8
Compiling bitflags v1.2.1
Compiling pkg-config v0.3.17
Compiling stable_deref_trait v1.1.1
Compiling spin v0.4.10
Compiling void v1.0.2
Compiling thread-scoped v1.0.2
Compiling remove_dir_all v0.5.2
Compiling argparse v0.2.2
Compiling owning_ref v0.3.3
Compiling concurrent-hashmap v0.2.2
Compiling log v0.3.9
Compiling smallvec v0.6.13
Compiling cntr-fuse v0.3.6
Compiling rand v0.4.6
Compiling time v0.1.42
Compiling num_cpus v1.12.0
Compiling cntr-nix v0.10.0-pre1
Compiling parking_lot_core v0.2.14
Compiling tempdir v0.3.7
error[E0308]: mismatched types
--> /home/rene/.cargo/registry/src/github.com-1ecc6299db9ec823/cntr-nix-0.10.0-pre1/src/macros.rs:197:26
|
197 | $entry = libc::$entry,
| ^^^^ expected i32, found u32
|
::: /home/rene/.cargo/registry/src/github.com-1ecc6299db9ec823/cntr-nix-0.10.0-pre1/src/sys/resource.rs:6:1
|
6 | / libc_enum!{
7 | | #[repr(i32)]
8 | | pub enum Resource {
9 | | RLIMIT_AS,
... |
35 | | }
36 | | }
| |_- in this macro invocation
|
help: you can convert an u32 to i32 and panic if the converted value wouldn't fit
|
197 | $entry = libc.try_into().unwrap()::$entry,
| ^^^^^^^^^^^^^^^^^^^^^^^^

error[E0308]: mismatched types
--> /home/rene/.cargo/registry/src/github.com-1ecc6299db9ec823/cntr-nix-0.10.0-pre1/src/macros.rs:197:26
|
197 | $entry = libc::$entry,
| ^^^^ expected i32, found u32
|
::: /home/rene/.cargo/registry/src/github.com-1ecc6299db9ec823/cntr-nix-0.10.0-pre1/src/sys/resource.rs:6:1
|
6 | / libc_enum!{
7 | | #[repr(i32)]
8 | | pub enum Resource {
9 | | RLIMIT_AS,
... |
35 | | }
36 | | }
| |_- in this macro invocation
|
help: you can convert an u32 to i32 and panic if the converted value wouldn't fit
|
197 | $entry = libc.try_into().unwrap()::$entry,
| ^^^^^^^^^^^^^^^^^^^^^^^^

error[E0308]: mismatched types
--> /home/rene/.cargo/registry/src/github.com-1ecc6299db9ec823/cntr-nix-0.10.0-pre1/src/macros.rs:197:26
|
197 | $entry = libc::$entry,
| ^^^^ expected i32, found u32
|
::: /home/rene/.cargo/registry/src/github.com-1ecc6299db9ec823/cntr-nix-0.10.0-pre1/src/sys/resource.rs:6:1
|
6 | / libc_enum!{
7 | | #[repr(i32)]
8 | | pub enum Resource {
9 | | RLIMIT_AS,
... |
35 | | }
36 | | }
| |_- in this macro invocation
|
help: you can convert an u32 to i32 and panic if the converted value wouldn't fit
|
197 | $entry = libc.try_into().unwrap()::$entry,
| ^^^^^^^^^^^^^^^^^^^^^^^^

error[E0308]: mismatched types
--> /home/rene/.cargo/registry/src/github.com-1ecc6299db9ec823/cntr-nix-0.10.0-pre1/src/macros.rs:197:26
|
197 | $entry = libc::$entry,
| ^^^^ expected i32, found u32
|
::: /home/rene/.cargo/registry/src/github.com-1ecc6299db9ec823/cntr-nix-0.10.0-pre1/src/sys/resource.rs:6:1
|
6 | / libc_enum!{
7 | | #[repr(i32)]
8 | | pub enum Resource {
9 | | RLIMIT_AS,
... |
35 | | }
36 | | }
| |_- in this macro invocation
|
help: you can convert an u32 to i32 and panic if the converted value wouldn't fit
|
197 | $entry = libc.try_into().unwrap()::$entry,
| ^^^^^^^^^^^^^^^^^^^^^^^^

error[E0308]: mismatched types
--> /home/rene/.cargo/registry/src/github.com-1ecc6299db9ec823/cntr-nix-0.10.0-pre1/src/macros.rs:197:26
|
197 | $entry = libc::$entry,
| ^^^^ expected i32, found u32
|
::: /home/rene/.cargo/registry/src/github.com-1ecc6299db9ec823/cntr-nix-0.10.0-pre1/src/sys/resource.rs:6:1
|
6 | / libc_enum!{
7 | | #[repr(i32)]
8 | | pub enum Resource {
9 | | RLIMIT_AS,
... |
35 | | }
36 | | }
| |_- in this macro invocation
|
help: you can convert an u32 to i32 and panic if the converted value wouldn't fit
|
197 | $entry = libc.try_into().unwrap()::$entry,
| ^^^^^^^^^^^^^^^^^^^^^^^^

error[E0308]: mismatched types
--> /home/rene/.cargo/registry/src/github.com-1ecc6299db9ec823/cntr-nix-0.10.0-pre1/src/macros.rs:197:26
|
197 | $entry = libc::$entry,
| ^^^^ expected i32, found u32
|
::: /home/rene/.cargo/registry/src/github.com-1ecc6299db9ec823/cntr-nix-0.10.0-pre1/src/sys/resource.rs:6:1
|
6 | / libc_enum!{
7 | | #[repr(i32)]
8 | | pub enum Resource {
9 | | RLIMIT_AS,
... |
35 | | }
36 | | }
| |_- in this macro invocation
|
help: you can convert an u32 to i32 and panic if the converted value wouldn't fit
|
197 | $entry = libc.try_into().unwrap()::$entry,
| ^^^^^^^^^^^^^^^^^^^^^^^^

error[E0308]: mismatched types
--> /home/rene/.cargo/registry/src/github.com-1ecc6299db9ec823/cntr-nix-0.10.0-pre1/src/macros.rs:197:26
|
197 | $entry = libc::$entry,
| ^^^^ expected i32, found u32
|
::: /home/rene/.cargo/registry/src/github.com-1ecc6299db9ec823/cntr-nix-0.10.0-pre1/src/sys/resource.rs:6:1
|
6 | / libc_enum!{
7 | | #[repr(i32)]
8 | | pub enum Resource {
9 | | RLIMIT_AS,
... |
35 | | }
36 | | }
| |_- in this macro invocation
|
help: you can convert an u32 to i32 and panic if the converted value wouldn't fit
|
197 | $entry = libc.try_into().unwrap()::$entry,
| ^^^^^^^^^^^^^^^^^^^^^^^^

error: aborting due to 7 previous errors

For more information about this error, try rustc --explain E0308.
Compiling parking_lot v0.5.5
error: could not compile cntr-nix.
warning: build failed, waiting for other jobs to finish...
error: failed to compile cntr v1.1.2, intermediate artifacts can be found at /tmp/cargo-install5r9ZE3

Caused by:
build failed

Suppose you start an unprivileged podman container like this:

podman run -it debian /bin/sh

Now, running sudo cntr attach $(podman ps | tail -n1 | awk '{ print $1 }') fails with the following error:

no suitable container found, got the following errors:
  - ProcessId: not a valid pid: `eb77cae38179`: invalid digit found in string
  - Podman: Failed to list containers. 'podman inspect --format {{.State.Running}};{{.State.Pid}} eb77cae38179' exited with exit status: 125: Error: inspecting object: no such object: "eb77cae38179"
  - Docker: Failed to list containers. 'docker inspect --format {{.State.Running}};{{.State.Pid}} eb77cae38179' exited with exit status: 125: Error: inspecting object: no such object: "eb77cae38179"
  - Nspawn: Failed to list containers. 'machinectl show --property=Leader eb77cae38179' exited with exit status: 1: Could not get path to machine: No machine 'eb77cae38179' known

This is because running podman inspect as root doesn't give you information about a container started as another user. Obviously, running cntr without sudo won't work because the user has insufficient permissions.

Another thing I tried is to obtain the PID of the process in the container and run cntr attach against it:

$ sudo cntr attach 1396643
[sudo] password for ma27:
cannot create container mountpoint /var/lib/cntr: Permission denied (os error 13)

^C
$

Is it even possible nowadays to achieve that? If yes, how?

Currently using cntr 1.5.1 from NixOS 22.11.

I added cntr and I'm trying to run use it with a breakpointHook in nixos, but I get

sudo cntr attach -t command cntr-/nix/store/02p9zkkq9mvxvdrnmviwdl0ifiq0ms7h-sudachi-vedd7896                      !
cannot create container mountpoint /var/lib/cntr: No such file or directory (os error 2) 

no idea why because the /var/lib/cntr directory exists.