Discussed in #68

Instead of creating a new directory inside the jails directory, automatically create a new ZFS dataset when creating a new jail. This allows for creating snapshots if individual jails, with custom retention rules. Implement it in a backwards compatible way, so jlmkr remove will continue to work with jails which were created as normal directories (instead of datasets) in the jails dir.

I'm opening this issue to keep track of feature requests/improvements. I have no plans to implement this a.t.m. but contributions are welcome.

I have updated TrueNAS to TrueNAS-SCALE-22.12.4.2.
Latest script version from main branch can't fix systemd-nspawn link with error:

File "/mnt/pool/jailmaker/jlmkr.py", line 669
End":36938,"fullyQualifiedName":"startup_jails","identUtf16":{"start":{"lineNumber":1035,"utf16Col":4},"end":{"lineNumber":1035,"utf16Col":17}},"extentUtf16":{"start":{"lineNumber":1035,"utf16Col":0},"end":{"lineNumber":1038,"utf16Col":35}}},{"name":"main","kind":"function","identStart":36945,"identEnd":36949,"extentStart":36941,"extentEnd":41024,"fullyQualifiedName":"main","identUtf16":{"start":{"lineNumber":1041,"utf16Col":4},"end":{"lineNumber":1041,"utf16Col":8}},"extentUtf16":{"start":{"lineNumber":1041,"utf16Col":0},"end":{"lineNumber":1159,"utf16Col":32}}}]}},"copilotInfo":null,"copilotAccessAllowed":false,"csrf_tokens":{"/Jip-Hop/jailmaker/branches":{"post":"Q-CcRBfYYIvyxarS7APvCfojLo3ZlKWOxsmuKl83rlslK8xEawmean6ru_GvKdqjZ-ulRpin8icXPN6s2tHvsA"},"/repos/preferences":{"post":"38jXX8F5Qzmo_mCvsxcfYtk3XE1StJUxTCc8dfbDnm8rzpmg-iRSbiO_ktXHJZ_7DwoXfKhA-_afKK9oeb1cBw"}}},"title":"jailmaker/jlmkr.py at main · Jip-Hop/jailmaker"}
^
IndentationError: unexpected indent

Version from release 1.0.0 works ok for me.

When I enter in shell with jlmkr shell archLinux does not run it as a login-shell and therefore does not run the shell user's environment files (e.g., in my case, ~/.bashrc).
Is this intentional?
How can I solve it?

Discussed in #42

Consider implementing jlmkr clone or jlmkr duplicate method. Keep in mind #80 when implementing this. Also look out for hostname conflicts, machine-id conflicts and possibly other files which should be unique to each jail.

I'm opening this issue to keep track of feature requests/improvements. I have no plans to implement this a.t.m. but contributions are welcome.

It can take about 1 minute for jails using --network-bridge to be assigned an IPv4 address via DHCP. Is this normal systemd-nspawn behavior or is this a jailmaker bug? What can we do to speed up the IPv4 assignment?

Getting this error:

-- WARNING, the following logs are for debugging purposes only --

I0227 16:30:43.055366 3314 nvc.c:376] initializing library context (version=1.12.0, build=7678e1af094d865441d0bc1b97c3e72d15fcab50)
I0227 16:30:43.055432 3314 nvc.c:350] using root /
I0227 16:30:43.055437 3314 nvc.c:351] using ldcache /etc/ld.so.cache
I0227 16:30:43.055442 3314 nvc.c:352] using unprivileged user 65534:65534
I0227 16:30:43.055460 3314 nvc.c:393] attempting to load dxcore to see if we are running under Windows Subsystem for Linux (WSL)
I0227 16:30:43.055577 3314 nvc.c:395] dxcore initialization failed, continuing assuming a non-WSL environment
I0227 16:30:43.057645 3315 nvc.c:278] loading kernel module nvidia
I0227 16:30:43.057787 3315 nvc.c:282] running mknod for /dev/nvidiactl
I0227 16:30:43.057820 3315 nvc.c:286] running mknod for /dev/nvidia0
I0227 16:30:43.057840 3315 nvc.c:290] running mknod for all nvcaps in /dev/nvidia-caps
I0227 16:30:43.063197 3315 nvc.c:218] running mknod for /dev/nvidia-caps/nvidia-cap1 from /proc/driver/nvidia/capabilities/mig/config
I0227 16:30:43.063256 3315 nvc.c:218] running mknod for /dev/nvidia-caps/nvidia-cap2 from /proc/driver/nvidia/capabilities/mig/monitor
I0227 16:30:43.064371 3315 nvc.c:296] loading kernel module nvidia_uvm
I0227 16:30:43.064395 3315 nvc.c:300] running mknod for /dev/nvidia-uvm
I0227 16:30:43.064434 3315 nvc.c:305] loading kernel module nvidia_modeset
I0227 16:30:43.064464 3315 nvc.c:309] running mknod for /dev/nvidia-modeset
I0227 16:30:43.064644 3316 rpc.c:71] starting driver rpc service
I0227 16:30:43.064985 3314 rpc.c:135] driver rpc service terminated with signal 15
nvidia-container-cli: initialization error: load library failed: libnvidia-ml.so.1: cannot open shared object file: no such file or directory
I0227 16:30:43.065009 3314 nvc.c:434] shutting down library context

Looks like everything might not be getting passed through.

Discussed in #21

I'm working on templates for new jails. It's implemented by allowing the user to provide a config file during jlmkr create or by executing jlmkr create mydockerjail /mnt/tank/path/to/docker/config. The new initial_setup config value can contain commands to run in the jail before the jail is started for the first time.

This feature is available for testing on the develop branch. See this docker template for example. I hope to release a new version of jailmaker with this feature included in the upcoming weeks.

Consider implementing a jlmkr snapshot and jlmkr rollback method.

Blocked by: #80.

If you run jlmkr start jailname it wont throw an error but running jlmkr list will show the jail is not running. Machinectl confirms it is not running. The journalctl error is Too many open files.

Currently the rootfs image is downloaded from linuxcontainers. Support additional rootfs image sources (at least when creating a jail from a template config file).

Currently the image is specified in the config template like so:

distro=debian
release=bookworm

Alternative sources could be from a URL or local filesystem, starting with file:// (could be dir or tar) or http(s)://. E.g.

image=https://mirror.serverion.com/fedora/releases/39/Container/x86_64/images/Fedora-Container-Base-39-1.5.x86_64.tar.xz

Perhaps machinectl could be used to download and extract the rootfs.

Confusingly IX Systems are saying that if you use their recommended installation folder / dataset it will upgrade to future versions nicely, then point us to this page which says to use a different folder / dataset.
Specifically this link say to use /mnt/tank/sandboxes/ or /mnt/tank/jails and your instructions say to use /mnt/tank/jailmaker

I note that when I run the script with no options in the folder named sandboxes, the script complains that it is not in /mnt/tank/jailmaker. I also tried changing the JAILS_DIR_PATH, but that seemed to present an error also. Regardless of it being configurable, I think the instructions need to be in alignment. It's very confusing being pointed here from there and finding out their instructions are wrong. Probably...

it's further confusing that this site links me back to that page as instructions....

I'm going to change my dataset to jailmaker, but I'm not convinced that it will survive future version upgrades if I do, due to the way that page is worded.

I'm not sure yet, but possibly this only applies if you run the script without arguments, I just haven't got that far as I'm not sure yet what is the correct way to do it. I guess I'll progress and see what goes wrong :(

Jailmaker cant start containers. Already reverted so don't have the specific output. I think it may have to do with this commit, but not sure: truenas/middleware@1d7c9fa

Hi there,

I just rebooted Scale and found that starting a jail also brings up the last run docker compose.

How does that work?

I expected to manually need to run docker compose up ...

Do I miss something?

hi. is possible to use wireguard as a client in systemd-nspawn containers? jailmaker seems to be passing --capability=all but default, but using wireguard inside does not seem to work (e.g. wg-quick up wg0). would it work with a different type of networking mode besides the default, host networking? thx

Is there a way to bind /proc/sys to be RW inside of the jails? I am not able to start k3s because it keeps telling me that /proc/sys/vm is read only even though I —bind=/proc

Oct 16 16:57:02 k1 k3s[7490]: I1016 16:57:02.806728 7490 state_mem.go:35] "Initializing new in-memory state store"
Oct 16 16:57:02 k1 k3s[7490]: I1016 16:57:02.806970 7490 state_mem.go:75] "Updated machine memory state"
Oct 16 16:57:02 k1 k3s[7490]: E1016 16:57:02.807649 7490 kubelet.go:1480] "Failed to start ContainerManager" err="open /proc/sys/vm/overcommit_memory: read-only file system"
Oct 16 16:57:02 k1 systemd[1]: k3s.service: Main process exited, code=exited, status=1/FAILURE
Oct 16 16:57:02 k1 systemd[1]: k3s.service: Failed with result 'exit-code'.
Oct 16 16:57:02 k1 systemd[1]: k3s.service: Consumed 7.181s CPU time.

Discussed in #40

jailmaker create --name='JAILNAME' --release=debian12 --macvlan --intel  --nvidia=0 --bind='/mnt/tank:/home/' --autostart --start  

Allow for commands such as jlmkr create and jlmkr remove to be executed from scripts in a non-interactive way. Currently these commands may ask for confirmation or more information. This is desired for normal usage. But it would be nice if there's a way to force removal without confirmation and to create a new jail without being asked to start the jail afterwards (or to ignore the warning when creating a jail without systemd).

#84 is a step into this direction, as it greatly reduces the amount of interaction required to create a jail, but is not enough to cover this request.

I'm opening this issue to keep track of feature requests/improvements. I have no plans to implement this a.t.m. but contributions are welcome.

You have hyperlinks for most things - I tried to google it and it came up with other stuff. Then when I installed the default container, it showed me the link - perhaps it's specific to the container type, but I'd say it isn't really.

https://manpages.debian.org/bookworm/systemd-container/systemd-nspawn.1.en.html

It would have been helpful to me, so I'm suggesting it here.

Thanks.

Improve jlmkr edit

1. copy config to tmp file
2. edit the tmp file
3. after save, parse and validate new config
4. if invalid, don't apply
5. else move original config to config.bak (may be valid syntax, but could still cause jail not to start)
6. move tmp file over original config
7. check if contents have changed after edit
8. only show hint to restart the jail if content has changed

Alternatively to moving files around and creating .bak files, perhaps creating a snapshot of the jails dir (or jail subdir in case #80 and #82 are done) would be even better.

I have a TrueNAS Scale - on the latest release (Bluefin) version and have been using jailmaker as a test for a while.
I have a single jail running portainer

When the jail is not running I do not have the issue shown here - but when I start the jail i get spammed with the following messages in the console log. [apologies for formatting - it looks good when I am editing the issue - so I have added some extra spaces]

`Sep 28 21:45:03 newnas.sendarian.co.uk kernel: br0: port 3(vb-docker-jail) entered blocking state

Sep 28 21:45:03 newnas.sendarian.co.uk kernel: br0: port 3(vb-docker-jail) entered disabled state

Sep 28 21:45:03 newnas.sendarian.co.uk kernel: device vb-docker-jail entered promiscuous mode

Sep 28 21:45:03 newnas.sendarian.co.uk kernel: br0: port 3(vb-docker-jail) entered blocking state

Sep 28 21:45:03 newnas.sendarian.co.uk kernel: br0: port 3(vb-docker-jail) entered listening state

Sep 28 21:45:03 newnas.sendarian.co.uk kernel: br0: port 3(vb-docker-jail) entered disabled state

Sep 28 21:45:08 newnas.sendarian.co.uk kernel: IPv6: ADDRCONF(NETDEV_CHANGE): host0: link becomes ready

Sep 28 21:45:08 newnas.sendarian.co.uk kernel: br0: port 3(vb-docker-jail) entered blocking state

Sep 28 21:45:08 newnas.sendarian.co.uk kernel: br0: port 3(vb-docker-jail) entered listening state

Sep 28 21:45:23 newnas.sendarian.co.uk kernel: br0: port 3(vb-docker-jail) entered learning state

Sep 28 21:45:25 newnas.sendarian.co.uk kernel: docker0: port 1(veth9aba5b5) entered blocking state

Sep 28 21:45:25 newnas.sendarian.co.uk kernel: docker0: port 1(veth9aba5b5) entered disabled state

Sep 28 21:45:25 newnas.sendarian.co.uk kernel: device veth9aba5b5 entered promiscuous mode

Sep 28 21:45:25 newnas.sendarian.co.uk kernel: docker0: port 1(veth9aba5b5) entered blocking state

Sep 28 21:45:25 newnas.sendarian.co.uk kernel: docker0: port 1(veth9aba5b5) entered forwarding state

Sep 28 21:45:25 newnas.sendarian.co.uk kernel: IPv6: ADDRCONF(NETDEV_CHANGE): docker0: link becomes ready

Sep 28 21:45:25 newnas.sendarian.co.uk kernel: docker0: port 1(veth9aba5b5) entered disabled state

Sep 28 21:45:27 newnas.sendarian.co.uk kernel: eth0: renamed from vethe8b166a

Sep 28 21:45:27 newnas.sendarian.co.uk kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth9aba5b5: link becomes ready

Sep 28 21:45:27 newnas.sendarian.co.uk kernel: docker0: port 1(veth9aba5b5) entered blocking state

Sep 28 21:45:27 newnas.sendarian.co.uk kernel: docker0: port 1(veth9aba5b5) entered forwarding state

Sep 28 21:45:38 newnas.sendarian.co.uk kernel: br0: port 3(vb-docker-jail) entered forwarding state

Sep 28 21:45:38 newnas.sendarian.co.uk kernel: br0: topology change detected, sending tcn bpdu

Sep 28 21:46:52 newnas.sendarian.co.uk kernel: br-be2b02c1ea85: port 1(veth9390273) entered blocking state

Sep 28 21:46:52 newnas.sendarian.co.uk kernel: br-be2b02c1ea85: port 1(veth9390273) entered disabled state

Sep 28 21:46:52 newnas.sendarian.co.uk kernel: device veth9390273 entered promiscuous mode

Sep 28 21:46:53 newnas.sendarian.co.uk kernel: eth0: renamed from veth989bad2

Sep 28 21:46:53 newnas.sendarian.co.uk kernel: IPv6: ADDRCONF(NETDEV_CHANGE): veth9390273: link becomes ready

Sep 28 21:46:53 newnas.sendarian.co.uk kernel: br-be2b02c1ea85: port 1(veth9390273) entered blocking state

Sep 28 21:46:53 newnas.sendarian.co.uk kernel: br-be2b02c1ea85: port 1(veth9390273) entered forwarding state

Sep 28 21:46:53 newnas.sendarian.co.uk kernel: IPv6: ADDRCONF(NETDEV_CHANGE): br-be2b02c1ea85: link becomes ready

Sep 28 21:46:53 newnas.sendarian.co.uk kernel: br-be2b02c1ea85: port 1(veth9390273) entered disabled state

Sep 28 21:46:53 newnas.sendarian.co.uk kernel: veth989bad2: renamed from eth0

Sep 28 21:46:53 newnas.sendarian.co.uk kernel: br-be2b02c1ea85: port 1(veth9390273) entered disabled state

Sep 28 21:46:53 newnas.sendarian.co.uk kernel: device veth9390273 left promiscuous mode

Sep 28 21:46:53 newnas.sendarian.co.uk kernel: br-be2b02c1ea85: port 1(veth9390273) entered disabled state`

I start docker-jail with the following command (actually "./jlmakr.py start docker-jail")
systemd-run --property=KillMode=mixed --property=Type=notify --property=RestartForceExitStatus=133 --property=SuccessExitStatus=133 --property=Delegate=yes --property=TasksMax=infinity --collect --setenv=SYSTEMD_NSPAWN_LOCK=0 --unit=jlmkr-docker-jail --working-directory=./jails/docker-jail '--description=My nspawn jail docker-jail [created with jailmaker]' --setenv=SYSTEMD_SECCOMP=0 --property=DevicePolicy=auto -- systemd-nspawn --keep-unit --quiet --boot --machine=docker-jail --directory=rootfs --capability=all '--system-call-filter=add_key keyctl bpf' --network-bridge=br0 --resolv-conf=bind-host

The jail has its own IP address on my network. I am using advanced networking (from the manual) to give the jail its own IP seperate from the host

docker-jail config file:
`docker_compatible=1
gpu_passthrough_intel=0
gpu_passthrough_nvidia=0
systemd_nspawn_user_args=--network-bridge=br0 --resolv-conf=bind-host

You generally will not need to change the options below

systemd_run_default_args=--property=KillMode=mixed --property=Type=notify --property=RestartForceExitStatus=133 --property=SuccessExitStatus=133 --property=Delegate=yes --property=TasksMax=infinity --collect --setenv=SYSTEMD_NSPAWN_LOCK=0
systemd_nspawn_default_args=--keep-unit --quiet --boot`

80-container-host0.network file in container

`[Match]
Virtualization=container
Name=host0

[Network]
DHCP=false
Address=192.168.38.191/24
Gateway=192.168.38.15
LinkLocalAddressing=no
LLDP=yes
EmitLLDP=customer-bridge

[DHCP]
UseTimezone=yes
`

Anything else I can provide?

thanks for your great support.
I installed the jail with debian bullseye

During the steps I am not sure what to use I did a bind and bind-ro to some dataset storage I created.
but I don't know if more is needed for proper run.
I did not install docker compatible stuff since on the docker official site it says to remove all kinda packages.

After installing the docker engine at the offical site for debian. I won't start the hello world thing.
Docker doesn't seem to start and won't start.

With systemctl it gives error
systemctl status docker.service
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Thu 2023-08-17 10:54:37 CEST; 3min 35s ago
TriggeredBy: ● docker.socket
Docs: https://docs.docker.com
Process: 5506 ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock (code=exited, status=1/FAILURE)
Main PID: 5506 (code=exited, status=1/FAILURE)
CPU: 68ms

Aug 17 10:54:37 debian systemd[1]: docker.service: Scheduled restart job, restart counter is at 3.
Aug 17 10:54:37 debian systemd[1]: Stopped Docker Application Container Engine.
Aug 17 10:54:37 debian systemd[1]: docker.service: Start request repeated too quickly.
Aug 17 10:54:37 debian systemd[1]: docker.service: Failed with result 'exit-code'.
Aug 17 10:54:37 debian systemd[1]: Failed to start Docker Application Container Engine.

With dockerd it gives error.

INFO[2023-08-17T11:06:01.419074950+02:00] Starting up
INFO[2023-08-17T11:06:06.430785140+02:00] [graphdriver] using prior storage driver: overlay2
INFO[2023-08-17T11:06:06.430914543+02:00] Loading containers: start.
INFO[2023-08-17T11:06:06.432392912+02:00] unable to detect if iptables supports xlock: 'iptables --wait -L -n': iptables v1.8.7 (legacy): can't initialize iptables table filter': Permission denied (you must be root)
Perhaps iptables or your kernel needs to be upgraded. error="exit status 3" INFO[2023-08-17T11:06:06.439589634+02:00] stopping event stream following graceful shutdown error="<nil>" module=libcontainerd namespace=moby failed to start daemon: Error initializing network controller: error obtaining controller instance: failed to create NAT chain DOCKER: iptables failed: iptables -t nat -N DOCKER: iptables v1.8.7 (legacy): can't initialize iptables tablenat': Permission denied (you must be root)
Perhaps iptables or your kernel needs to be upgraded.
(exit status 3)
This is the journal
INFO[2023-08-17T11:06:01.419074950+02:00] Starting up
INFO[2023-08-17T11:06:06.430785140+02:00] [graphdriver] using prior storage driver: overlay2
INFO[2023-08-17T11:06:06.430914543+02:00] Loading containers: start.
INFO[2023-08-17T11:06:06.432392912+02:00] unable to detect if iptables supports xlock: 'iptables --wait -L -n': iptables v1.8.7 (legacy): can't initialize iptables table filter': Permission denied (you must be root)
Perhaps iptables or your kernel needs to be upgraded. error="exit status 3" INFO[2023-08-17T11:06:06.439589634+02:00] stopping event stream following graceful shutdown error="<nil>" module=libcontainerd namespace=moby failed to start daemon: Error initializing network controller: error obtaining controller instance: failed to create NAT chain DOCKER: iptables failed: iptables -t nat -N DOCKER: iptables v1.8.7 (legacy): can't initialize iptables tablenat': Permission denied (you must be root)
Perhaps iptables or your kernel needs to be upgraded.
(exit status 3)

Not sure what to use.
I just want to run this thing and learning on the way to get this native stuff running.

Hi,
seems you have made it: https://www.truenas.com/docs/scale/scaletutorials/apps/sandboxes/ ❤️
Will you stop working on jailmaker?

thx

Discussed in #38

I've implemented jlmkr restart. It's available for testing on the develop branch. I hope to release a new version of jailmaker with this feature included in the upcoming weeks.

nvm I was a bit stupid with the curl command

It's great to have intel and nvidia support, but I'm missing the AMD option.

Thank you!

Arch Linux in jail.
I have installed qbittorrent-nox from Arch official repositories in jail.
I've changed last line of this service from:

[Install]
WantedBy=multi-user.target

to

[Install]
WantedBy=default.target

and I enabled the service systemctl enable qbittorrent-nox@apps.

But when I restart the virtual machine, it doesn't start the service. What is wrong?

• TrueNAS-SCALE-23.10.1
• jailmaker - latest
• jail client: debian
• network: host mode

ping is ok, but can not apt update

PING mirrors.tuna.tsinghua.edu.cn(2402:f000:1:400::2 (2402:f000:1:400::2)) 56 data bytes
64 bytes from 2402:f000:1:400::2 (2402:f000:1:400::2): icmp_seq=1 ttl=45 time=57.4 ms
64 bytes from 2402:f000:1:400::2 (2402:f000:1:400::2): icmp_seq=2 ttl=45 time=56.3 ms
^C
--- mirrors.tuna.tsinghua.edu.cn ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 56.317/56.846/57.375/0.529 ms
root@docker:~# apt update
Ign:1 https://mirrors.tuna.tsinghua.edu.cn/debian bookworm InRelease
Ign:2 https://mirrors.tuna.tsinghua.edu.cn/debian bookworm-updates InRelease
Ign:3 https://mirrors.tuna.tsinghua.edu.cn/debian bookworm-backports InRelease
Ign:4 https://mirrors.tuna.tsinghua.edu.cn/debian-security bookworm-security InRelease
Ign:1 https://mirrors.tuna.tsinghua.edu.cn/debian bookworm InRelease
Ign:2 https://mirrors.tuna.tsinghua.edu.cn/debian bookworm-updates InRelease
Ign:3 https://mirrors.tuna.tsinghua.edu.cn/debian bookworm-backports InRelease
Ign:4 https://mirrors.tuna.tsinghua.edu.cn/debian-security bookworm-security InRelease
Ign:1 https://mirrors.tuna.tsinghua.edu.cn/debian bookworm InRelease
Ign:2 https://mirrors.tuna.tsinghua.edu.cn/debian bookworm-updates InRelease
Ign:3 https://mirrors.tuna.tsinghua.edu.cn/debian bookworm-backports InRelease
Ign:4 https://mirrors.tuna.tsinghua.edu.cn/debian-security bookworm-security InRelease
Err:1 https://mirrors.tuna.tsinghua.edu.cn/debian bookworm InRelease
  Temporary failure resolving 'mirrors.tuna.tsinghua.edu.cn'
Err:2 https://mirrors.tuna.tsinghua.edu.cn/debian bookworm-updates InRelease
  Temporary failure resolving 'mirrors.tuna.tsinghua.edu.cn'
Err:3 https://mirrors.tuna.tsinghua.edu.cn/debian bookworm-backports InRelease
  Temporary failure resolving 'mirrors.tuna.tsinghua.edu.cn'
Err:4 https://mirrors.tuna.tsinghua.edu.cn/debian-security bookworm-security InRelease
  Temporary failure resolving 'mirrors.tuna.tsinghua.edu.cn'
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
All packages are up to date.
W: Failed to fetch https://mirrors.tuna.tsinghua.edu.cn/debian/dists/bookworm/InRelease  Temporary failure resolving 'mirrors.tuna.tsinghua.edu.cn'
W: Failed to fetch https://mirrors.tuna.tsinghua.edu.cn/debian/dists/bookworm-updates/InRelease  Temporary failure resolving 'mirrors.tuna.tsinghua.edu.cn'
W: Failed to fetch https://mirrors.tuna.tsinghua.edu.cn/debian/dists/bookworm-backports/InRelease  Temporary failure resolving 'mirrors.tuna.tsinghua.edu.cn'
W: Failed to fetch https://mirrors.tuna.tsinghua.edu.cn/debian-security/dists/bookworm-security/InRelease  Temporary failure resolving 'mirrors.tuna.tsinghua.edu.cn'
W: Some index files failed to download. They have been ignored, or old ones used instead.

systemd' log

root@docker:~# systemctl status systemd-resolved 
● systemd-resolved.service - Network Name Resolution
     Loaded: loaded (/lib/systemd/system/systemd-resolved.service; enabled; preset: 
enabled)
     Active: active (running) since Tue 2023-12-26 13:39:38 CST; 17min ago
       Docs: man:systemd-resolved.service(8)
             man:org.freedesktop.resolve1(5)
             https://www.freedesktop.org/wiki/Software/systemd/writing-ne
twork-configuration-managers
             https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients

   Main PID: 41 (systemd-resolve)
     Status: "Processing requests..."
      Tasks: 1 (limit: 37926)
     Memory: 3.8M
        CPU: 148ms
     CGroup: /system.slice/systemd-resolved.service
             └─41 /lib/systemd/systemd-resolved

Dec 26 13:39:38 docker systemd[1]: Starting systemd-resolved.service - Network Name Resolution...
Dec 26 13:39:38 docker systemd-resolved[41]: Positive Trust Anchors:
Dec 26 13:39:38 docker systemd-resolved[41]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d
Dec 26 13:39:38 docker systemd-resolved[41]: Negative trust anchors: home.arpa 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-
addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-addr.arpa 23.172.in-addr.arpa 24.172.in-addr.arpa 25.172.in-addr.arpa 26.172.in-addr.arpa 27.172.in-addr.
arpa 28.172.in-addr.arpa 29.172.in-addr.arpa 30.172.in-addr.arpa 31.172.in-addr.arpa 168.192.in-addr.arpa d.f.ip6.arpa corp home internal intranet lan local private 
test
Dec 26 13:39:38 docker systemd-resolved[41]: Using system hostname 'docker'.
Dec 26 13:39:38 docker systemd-resolved[41]: Failed to open /etc/resolv.conf: Permission denied
Dec 26 13:39:38 docker systemd[1]: Started systemd-resolved.service - Network Name Resolution.

so, i do this

root@docker:~# ls -l /etc/resolv.conf 
-rwxrwx--- 1 root root 119 Dec 26 13:47 /etc/resolv.conf
root@docker:~# chmod +r /etc/resolv.conf 
root@docker:~# ls -l /etc/resolv.conf 
-rwxrwxr-- 1 root root 119 Dec 26 13:47 /etc/resolv.conf

now, it's work

root@docker:~# apt update
Hit:1 http://deb.debian.org/debian bookworm InRelease
Hit:2 http://deb.debian.org/debian bookworm-updates InRelease
Hit:3 http://deb.debian.org/debian-security bookworm-security InRelease
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
All packages are up to date.

I've installed docker a few different ways (via apt, via apt and docker.io, and via the convenience script) within a jail that I've created. They all seem to give this bit near the end of install Could not execute systemctl: at /usr/bin/deb-systemd-invoke line 145. which I'm not sure if it is related or not. When I check the status of the service it shows failed:

Jan 14 22:51:17 fireflyiii systemd[1]: docker.service: Start request repeated too quickly.
Jan 14 22:51:17 fireflyiii systemd[1]: docker.service: Failed with result 'exit-code'.
Jan 14 22:51:17 fireflyiii systemd[1]: Failed to start docker.service - Docker Application Container Engine.

When I check journalctl I see this:

Jan 14 22:51:57 fireflyiii systemd[1]: docker.service: Start request repeated too quickly.
Jan 14 22:51:57 fireflyiii systemd[1]: docker.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ The unit docker.service has entered the 'failed' state with result 'exit-code'.
Jan 14 22:51:57 fireflyiii systemd[1]: Failed to start docker.service - Docker Application Container Engine.
░░ Subject: A start job for unit docker.service has failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ A start job for unit docker.service has finished with a failure.
░░ 
░░ The job identifier is 705 and the job result is failed.

I went ahead and enabled debug for the docker daemon and I see this in journalctl now:

Jan 14 23:02:43 fireflyiii dockerd[361]: time="2024-01-14T23:02:43.468550889-08:00" level=warning msg="Your kernel does not support cgroup memory limit"
Jan 14 23:02:43 fireflyiii dockerd[361]: time="2024-01-14T23:02:43.468602867-08:00" level=warning msg="Unable to find cpu cgroup in mounts"
Jan 14 23:02:43 fireflyiii dockerd[361]: time="2024-01-14T23:02:43.468622288-08:00" level=warning msg="Unable to find blkio cgroup in mounts"
Jan 14 23:02:43 fireflyiii dockerd[361]: time="2024-01-14T23:02:43.468639229-08:00" level=warning msg="Unable to find cpuset cgroup in mounts"
Jan 14 23:02:43 fireflyiii dockerd[361]: time="2024-01-14T23:02:43.468656172-08:00" level=warning msg="Unable to find pids cgroup in mounts"
Jan 14 23:02:43 fireflyiii dockerd[361]: time="2024-01-14T23:02:43.469843763-08:00" level=info msg="[core] Channel Connectivity change to SHUTDOWN" module=grpc
Jan 14 23:02:43 fireflyiii dockerd[361]: time="2024-01-14T23:02:43.469896987-08:00" level=info msg="[core] Subchannel Connectivity change to SHUTDOWN" module=grpc
Jan 14 23:02:43 fireflyiii dockerd[361]: time="2024-01-14T23:02:43.469928056-08:00" level=debug msg="Cleaning up old mountid : start."
Jan 14 23:02:43 fireflyiii dockerd[361]: failed to start daemon: Devices cgroup isn't mounted

Here is my jlmkr config file:

startup=1
docker_compatible=1
gpu_passthrough_intel=0
gpu_passthrough_nvidia=0
systemd_nspawn_user_args=--bind='/mnt/sandisk-ssd/fireflyiii:/home' --network-bridge='br0' --resolv-conf='bind-host'
# You generally will not need to change the options below
systemd_run_default_args=--property=KillMode=mixed --property=Type=notify --property=RestartForceExitStatus=133 --property=SuccessExitStatus=133 --property=Delegate=yes --property=TasksMax=infinity --collect --setenv=SYSTEMD_NSPAWN_LOCK=0
systemd_nspawn_default_args=--keep-unit --quiet --boot

I'm running TrueNAS-SCALE-22.12.1 on an Intel i5-6500 CPU

I've check docker on the host and it seems to be running fine.

Create automated unit/integration tests to test jailmaker during development. Ideally these tests would run, bare metal, on TrueNAS SCALE (and on nightly versions) with an NVIDIA GPU (to test passthrough as well) after pushing to this repo.

• Create Actions workflow on a GitHub hosted Ubuntu runner? Do these support ZFS?
• Can the Pull Request Instructions / Jenkins Commands be used for this?

Contributions welcome!

After installing on TrueNAS-SCALE-22.12.3.3 I get this error when I try to create a jail

USE THIS SCRIPT AT YOUR OWN RISK!
IT COMES WITHOUT WARRANTY AND IS NOT SUPPORTED BY IXSYSTEMS.

Install the recommended image (debian bullseye)? [Y/n] y

Enter jail name: portainer

Docker won't be installed by jlmkr.
But it can setup the jail with the capabilities required to run docker.
You can turn DOCKER_COMPATIBLE mode on/off post-install.

Make jail docker compatible right now? [y/N] y

Detected the presence of an intel GPU.

Passthrough the intel GPU? [y/N]
Detected the presence of an nvidia GPU.

Passthrough the nvidia GPU? [y/N] y

WARNING: CHECK SYNTAX

You may pass additional flags to systemd-nspawn.
With incorrect flags the jail may not start.
It is possible to correct/add/remove flags post-install.

Show the man page for systemd-nspawn? [y/N] n
Traceback (most recent call last):
  File "/usr/local/sbin/jlmkr", line 1156, in <module>
    main()
  File "/usr/local/sbin/jlmkr", line 1111, in main
    create_jail(args.name)
  File "/usr/local/sbin/jlmkr", line 828, in create_jail
    raise error
  File "/usr/local/sbin/jlmkr", line 628, in create_jail
    base_os_version = platform.freedesktop_os_release().get('VERSION_CODENAME', release)
AttributeError: module 'platform' has no attribute 'freedesktop_os_release'

python --version
Python 3.9.2

cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

The TrueNAS Scale 24.04 Dragonfish beta appears to have moved from a permissive file system to a read only filesystem. This prohibits creating symlinks, enabling apt, among others.

The jailmaker install outputs

root@truenas[/mnt/hermes/jailmaker]# ./jlmkr.py install
systemd-nspawn is already installed.
Creating symlink /usr/local/sbin/jlmkr to /mnt/hermes/jailmaker/jlmkr.py.
Traceback (most recent call last):
  File "/mnt/hermes/jailmaker/./jlmkr.py", line 1165, in <module>
    main()
  File "/mnt/hermes/jailmaker/./jlmkr.py", line 1117, in main
    install_jailmaker()
  File "/mnt/hermes/jailmaker/./jlmkr.py", line 1027, in install_jailmaker
    os.symlink(SCRIPT_PATH, target)
OSError: [Errno 30] Read-only file system: '/mnt/hermes/jailmaker/jlmkr.py' -> '/usr/local/sbin/jlmkr'
root@truenas[/mnt/hermes/jailmaker]# 

Same outcome when manually attempting to add chmod +x to apt*

Personally it feels like iX is resisting any changes to the "correct" way of using their platform. I think I'm going to rollback, export all my configs and then switch to ProxmoxVE. Although this project has been absolutely incredible to use and I was planning on contributing more, diagnosing issues every release has made me think "why am i even doing this?" Proxmox has better container performance, usability, and I can just add a network share from Proxmox.

Happy to revisit this project in the future. Ideally iX would give up on their horrid, lossy, unperformant k8s server and move to containers natively. If that ever happens I will gladly switch to TrueNAS Scale for the rest of my server's lifetime, but this may be the end of my TrueNAS journey unfortunately.

Eager to try this out but I am logged in as root, created the dataset followed the first steps.
And I get stuck here

Als tried sudo what am I doing wrong?

Hello,

I was testing this with some different distros, and it seems that any distro that has a recent systemd version installed isn't able to run Docker due to problems with cgroups. This is true for the ArchLinux image and also with the Debian Bullseye image when updating the systemd version from the debian-backports repository. The Debian image with the stable version of systemd works correctly. The version of systemd in Debian bullseye stable is 247.3-7+deb11u1 and the version in Debian bullseye backport is 252.5-2~bpo11+1. When running with newer systemd the cgroups version reverts to v1, however in the old systemd in stable Debian it reports the use of cgroupsv2. Maybe there is an argument that can be passed to systemd that will revert the behaviour of the old versions?

The following is the log for an attempt at running the docker daemon in a distro with a newer version of systemd installed:

$> sudo dockerd -D
INFO[2023-03-10T18:15:26.526041305-03:00] Starting up                                  
DEBU[2023-03-10T18:15:26.528674385-03:00] Listener created for HTTP on unix (/var/run/docker.sock) 
DEBU[2023-03-10T18:15:26.529054650-03:00] Golang's threads limit set to 226350         
INFO[2023-03-10T18:15:26.529268808-03:00] [core] [Channel #1] Channel created           module=grpc
INFO[2023-03-10T18:15:26.529282240-03:00] [core] [Channel #1] original dial target is: "unix:///run/containerd/containerd.sock"  module=grpc
INFO[2023-03-10T18:15:26.529302807-03:00] [core] [Channel #1] parsed dial target is: {Scheme:unix Authority: Endpoint:run/containerd/containerd.sock URL:{Scheme:unix Opaque: User: Host: Path:/run/containerd/containerd.sock RawPath: OmitHost:false ForceQuery:false RawQuery: Fragment: RawFragment:}}  module=grpc
INFO[2023-03-10T18:15:26.529321677-03:00] [core] [Channel #1] Channel authority set to "localhost"  module=grpc
INFO[2023-03-10T18:15:26.529392959-03:00] [core] [Channel #1] Resolver state updated: {
  "Addresses": [
    {
      "Addr": "/run/containerd/containerd.sock",
      "ServerName": "",
      "Attributes": {},
      "BalancerAttributes": null,
      "Type": 0,
      "Metadata": null
    }
  ],
  "ServiceConfig": null,
  "Attributes": null
} (resolver returned new addresses)  module=grpc
DEBU[2023-03-10T18:15:26.529505739-03:00] metrics API listening on /var/run/docker/metrics.sock 
INFO[2023-03-10T18:15:26.529533097-03:00] [core] [Channel #1] Channel switches to new LB policy "pick_first"  module=grpc
INFO[2023-03-10T18:15:26.529631573-03:00] [core] [Channel #1 SubChannel #2] Subchannel created  module=grpc
INFO[2023-03-10T18:15:26.529663146-03:00] [core] [Channel #1 SubChannel #2] Subchannel Connectivity change to CONNECTING  module=grpc
INFO[2023-03-10T18:15:26.529689846-03:00] [core] [Channel #1 SubChannel #2] Subchannel picks a new address "/run/containerd/containerd.sock" to connect  module=grpc
INFO[2023-03-10T18:15:26.531175594-03:00] [core] [Channel #1] Channel Connectivity change to CONNECTING  module=grpc
INFO[2023-03-10T18:15:26.531443813-03:00] [core] [Channel #1 SubChannel #2] Subchannel Connectivity change to READY  module=grpc
INFO[2023-03-10T18:15:26.531546598-03:00] [core] [Channel #1] Channel Connectivity change to READY  module=grpc
INFO[2023-03-10T18:15:26.532028197-03:00] [core] [Channel #4] Channel created           module=grpc
INFO[2023-03-10T18:15:26.532044907-03:00] [core] [Channel #4] original dial target is: "unix:///run/containerd/containerd.sock"  module=grpc
INFO[2023-03-10T18:15:26.532071212-03:00] [core] [Channel #4] parsed dial target is: {Scheme:unix Authority: Endpoint:run/containerd/containerd.sock URL:{Scheme:unix Opaque: User: Host: Path:/run/containerd/containerd.sock RawPath: OmitHost:false ForceQuery:false RawQuery: Fragment: RawFragment:}}  module=grpc
INFO[2023-03-10T18:15:26.532084716-03:00] [core] [Channel #4] Channel authority set to "localhost"  module=grpc
INFO[2023-03-10T18:15:26.532116636-03:00] [core] [Channel #4] Resolver state updated: {
  "Addresses": [
    {
      "Addr": "/run/containerd/containerd.sock",
      "ServerName": "",
      "Attributes": {},
      "BalancerAttributes": null,
      "Type": 0,
      "Metadata": null
    }
  ],
  "ServiceConfig": null,
  "Attributes": null
} (resolver returned new addresses)  module=grpc
INFO[2023-03-10T18:15:26.532142755-03:00] [core] [Channel #4] Channel switches to new LB policy "pick_first"  module=grpc
INFO[2023-03-10T18:15:26.532165819-03:00] [core] [Channel #4 SubChannel #5] Subchannel created  module=grpc
INFO[2023-03-10T18:15:26.532194305-03:00] [core] [Channel #4 SubChannel #5] Subchannel Connectivity change to CONNECTING  module=grpc
INFO[2023-03-10T18:15:26.532231747-03:00] [core] [Channel #4 SubChannel #5] Subchannel picks a new address "/run/containerd/containerd.sock" to connect  module=grpc
INFO[2023-03-10T18:15:26.532242588-03:00] [core] [Channel #4] Channel Connectivity change to CONNECTING  module=grpc
INFO[2023-03-10T18:15:26.532396875-03:00] [core] [Channel #4 SubChannel #5] Subchannel Connectivity change to READY  module=grpc
INFO[2023-03-10T18:15:26.532416251-03:00] [core] [Channel #4] Channel Connectivity change to READY  module=grpc
DEBU[2023-03-10T18:15:26.532688762-03:00] Using default logging driver journald        
DEBU[2023-03-10T18:15:26.532752295-03:00] [graphdriver] priority list: [overlay2 fuse-overlayfs btrfs zfs aufs overlay devicemapper vfs] 
DEBU[2023-03-10T18:15:26.532768117-03:00] processing event stream                       module=libcontainerd namespace=plugins.moby
DEBU[2023-03-10T18:15:29.800226698-03:00] successfully detected metacopy status         storage-driver=overlay2 usingMetacopy=false
DEBU[2023-03-10T18:15:29.802434855-03:00] backingFs=zfs, projectQuotaSupported=false, usingMetacopy=false, indexOff="", userxattr=""  storage-driver=overlay2
INFO[2023-03-10T18:15:29.802447535-03:00] [graphdriver] using prior storage driver: overlay2 
DEBU[2023-03-10T18:15:29.802456128-03:00] Initialized graph driver overlay2            
DEBU[2023-03-10T18:15:29.803783506-03:00] No quota support for local volumes in /var/lib/docker/volumes: Filesystem does not support, or has not enabled quotas 
INFO[2023-03-10T18:15:29.809109806-03:00] [core] [Channel #1] Channel Connectivity change to SHUTDOWN  module=grpc
INFO[2023-03-10T18:15:29.809130328-03:00] [core] [Channel #1 SubChannel #2] Subchannel Connectivity change to SHUTDOWN  module=grpc
INFO[2023-03-10T18:15:29.809140869-03:00] [core] [Channel #1 SubChannel #2] Subchannel deleted  module=grpc
INFO[2023-03-10T18:15:29.809147180-03:00] [core] [Channel #1] Channel deleted           module=grpc
DEBU[2023-03-10T18:15:29.809153931-03:00] Cleaning up old mountid : start.             
failed to start daemon: Devices cgroup isn't mounted

Also, when running on old systemd the directory structure for /sys/fs/cgroup/ looks like this:

$> ls /sys/fs/cgroup/
cgroup.controllers      cpu.idle               cpu.weight.nice      memory.low           pids.current
cgroup.events           cpu.max                dev-hugepages.mount  memory.max           pids.events
cgroup.freeze           cpu.max.burst          init.scope           memory.min           pids.max
cgroup.kill             cpu.pressure           io.max               memory.numa_stat     sys-kernel-config.mount
cgroup.max.depth        cpuset.cpus            io.pressure          memory.oom.group     sys-kernel-debug.mount
cgroup.max.descendants  cpuset.cpus.effective  io.stat              memory.pressure      sys-kernel-tracing.mount
cgroup.procs            cpuset.cpus.partition  io.weight            memory.stat          system.slice
cgroup.stat             cpuset.mems            memory.current       memory.swap.current  user.slice
cgroup.subtree_control  cpuset.mems.effective  memory.events        memory.swap.events
cgroup.threads          cpu.stat               memory.events.local  memory.swap.high
cgroup.type             cpu.weight             memory.high          memory.swap.max

But on newer systemd versions, it looks like this:

$>  ls /sys/fs/cgroup/
systemd

As per title; I've just updated the jailmaker script from v1.0.1 to v1.1.0, and the NVIDIA GPU on my system is no longer accessible inside the nspawn jail/container which was working prior to the update.

To recreate the issue with a new jail:

# jlmkr create -gn 1 -gi 0 --docker_compatible 1 --distro ubuntu --release jammy  test
Creating jail test with default config.
Overriding distro config value with ubuntu.
Overriding docker_compatible config value with 1.
Overriding gpu_passthrough_nvidia config value with 1.
Overriding release config value with jammy.
The cached copy has expired, re-downloading...
Downloading the image index
Downloading the rootfs
Downloading the metadata
The image cache is now ready
Unpacking the rootfs

---
You just created an Ubuntu jammy amd64 (20240302_07:42) container.

Starting the container on v1.1.0 looks like:

Starting jail test with the following command:

systemd-run --collect --property=Delegate=yes --property=RestartForceExitStatus=133 --property=SuccessExitStatus=133 --property=TasksMax=infinity --property=Type=notify --setenv=SYSTEMD_NSPAWN_LOCK=0 --property=KillMode=mixed --unit=jlmkr-test --working-directory=./jails/test '--description=My nspawn jail test [created with jailmaker]' --setenv=SYSTEMD_SECCOMP=0 --property=DevicePolicy=auto -- systemd-nspawn --bind-ro=/sys/module --boot --inaccessible=/sys/module/apparmor --quiet --keep-unit --machine=test --directory=rootfs --capability=all '--property=DeviceAllow=char-drm rw'

I don't see any NVIDIA-related mounts here? eg. nvidia-smi is not available in the container either etc. If i mount it in manually, it errors about missing libraries.

Reverting back to v1.0.1 script and running it again, the appropriate NVIDIA mounts are back:

Starting jail test with the following command:

systemd-run --collect --property=Delegate=yes --property=RestartForceExitStatus=133 --property=SuccessExitStatus=133 --property=TasksMax=infinity --property=Type=notify --setenv=SYSTEMD_NSPAWN_LOCK=0 --property=KillMode=mixed --unit=jlmkr-test --working-directory=./jails/test '--description=My nspawn jail test [created with jailmaker]' --setenv=SYSTEMD_SECCOMP=0 --property=DevicePolicy=auto -- systemd-nspawn --bind-ro=/sys/module --bind=/usr/bin/nvidia-smi --boot --inaccessible=/sys/module/apparmor --quiet --keep-unit --machine=test --directory=rootfs --capability=all '--system-call-filter=add_key keyctl bpf' '--property=DeviceAllow=char-drm rw' --bind-ro=/usr/lib/x86_64-linux-gnu/nvidia/current/libnvidia-nvvm.so.535.54.03 --bind-ro=/usr/lib/x86_64-linux-gnu/nvidia/current/libnvidia-ptxjitcompiler.so.535.54.03 --bind-ro=/usr/lib/x86_64-linux-gnu/nvidia/current/libcuda.so.535.54.03 --bind-ro=/usr/lib/x86_64-linux-gnu/nvidia/current/libnvidia-encode.so.535.54.03 --bind-ro=/usr/lib/x86_64-linux-gnu/nvidia/current/libnvcuvid.so.535.54.03 --bind-ro=/usr/bin/nvidia-persistenced --bind=/dev/nvidia0 --bind-ro=/usr/lib/x86_64-linux-gnu/nvidia/current/libnvidia-cfg.so.535.54.03 --bind=/dev/nvidia-uvm-tools --bind=/dev/nvidiactl --bind=/dev/nvidia-uvm --bind-ro=/usr/lib/x86_64-linux-gnu/nvidia/current/libnvidia-ml.so.535.54.03 --bind-ro=/usr/bin/nvidia-smi --bind-ro=/usr/lib/nvidia/current/nvidia-smi

This behaviour is confirmed on TrueNAS Scale 23.10.2 and 23.10.1.1 with an RTX3060 12GB

I've installed Debian 12 with jlmkr, under TrueNAS and I can get it to listen on HTTP, but not on HTTPS.
Not sure if it is something specific to my config, but I can't see any errors in the logs.

Is it some restriction of how jailmaker works?

Feature request: Would be nice to have an option to delete or otherwise clean up jails created by the script rather than having to do it manually. Something like "./jlmkr.py remove myjail" with a confirmation prompt ("Are you sure Y/n")

Hello,
After many hours of troubleshooting strange issues within my jail, I finally pinpointed my issues down to permissions. I resolved my issues by changing the location of the jlmkr.py script to a different dataset. Once I moved the script to a different dataset and created a new jail, everything is working correctly. The dataset with issues has SMB shares and ACLs, which I think may be the cause.

Here is some of the behaviors:

1. apt-get update errors with Temporary failure resolving deb.debian.org

root@testjail:~# apt-get update
Err:1 http://deb.debian.org/debian bullseye InRelease
Temporary failure resolving 'deb.debian.org'
Err:2 http://deb.debian.org/debian bullseye-updates InRelease
Temporary failure resolving 'deb.debian.org'
Err:3 http://deb.debian.org/debian-security bullseye-security InRelease
Temporary failure resolving 'deb.debian.org'
Reading package lists... Done
W: Failed to fetch http://deb.debian.org/debian/dists/bullseye/InRelease Temporary failure resolving 'deb.debian.org'
W: Failed to fetch http://deb.debian.org/debian/dists/bullseye-updates/InRelease Temporary failure resolving 'deb.debian.org'
W: Failed to fetch http://deb.debian.org/debian-security/dists/bullseye-security/InRelease Temporary failure resolving 'deb.debian.org'
W: Some index files failed to download. They have been ignored, or old ones used instead.

Looking at /etc/resolv.conf, has incorrect permissions:

root@testjail:~# ls -l /etc/resolv.conf
-rwx------ 1 root root 37 Apr 30 13:50 /etc/resolv.conf

Running chmod 644 /etc/resolv.conf resolves the issue and running apt-get update works.

2. apt-get install iptables throws error.

root@testjail:~# apt-get install iptables
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Suggested packages:
firewalld kmod
The following NEW packages will be installed:
iptables
0 upgraded, 1 newly installed, 0 to remove and 23 not upgraded.
7 not fully installed or removed.
Need to get 0 B/382 kB of archives.
After this operation, 2,582 kB of additional disk space will be used.
debconf: delaying package configuration, since apt-utils is not installed
(Reading database ... 11726 files and directories currently installed.)
Preparing to unpack .../iptables_1.8.7-1_amd64.deb ...
Unpacking iptables (1.8.7-1) ...
dpkg: error processing archive /var/cache/apt/archives/iptables_1.8.7-1_amd64.deb (--unpack):
unable to open '/usr/lib/x86_64-linux-gnu/xtables/libip6t_hl.so.dpkg-new': No such file or directory
Errors were encountered while processing:
/var/cache/apt/archives/iptables_1.8.7-1_amd64.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)

I was unable to determine the actual cause for the installation of iptables to error out, but once I moved the jailmaker script to a different dataset and created a new jail, everything is working as expected now.

First issue yay! But it's more of a feature request.

I'm currently running a nextcloud-aio docker install inside a jail and have trouble using the built-in backup solution.
It doe not work because there is FUSE requiered. With LXC this has to be explicitly enabled is there any chance on getting this for nspawn to work?

For reference:

nextcloud/all-in-one#273

https://forum.proxmox.com/threads/persistently-use-fuse-inside-of-lxc-container.95810/#post-415631

https://medium.com/myatus/quick-note-fuse-inside-proxmox-lxc-container-481706005a09

I started an Ubuntu Jammy jail on TrueNAS Scale and tried to modify sshd service settings (port).
Upon restarting a service it threw an error:

systemctl restart sshd
Failed to allocate directory watch: Too many open files

Googling suggested increasing the limit by ulimit -n ..., but I suspect it does not address the root cause.

Could you please advise on this case?

Thanks.

Any way to pass through device /dev/kvm ?

Would be great to get this running https://github.com/dockur/windows in jailmaker, or is there another way?

I honestly have no idea if this is an issue or not. I think its mostly a lack of knowledge

With a normal VM, the guest runs, stops and starts in a normal manner. If you just kill the VM its the equivalent of turning the power off. When you turn it on it boots up in x seconds / minutes and all is good. Shutting down takes an equivalent amount of time

Jailmaker / I guess systemd-container or whatever its called doesn't seem to do that.

Startup is, or seems to be effectively immediate
Just as is shutdown. And if I am logged in via SSH to the jail and type shutdown now - the jail just vanishes.

So on to my question - is this data safe? Or am I just hitting the power off button each time?
[Note it does seem that portainer takes a little while to start up - but that may be the IP address taking 20 seconds bit]

Maybe I missed it during setup. But also not sure where to post this question as perhaps this is no issue.

How do I get "dev/serial/by-id/someusb" from truenas scale into the jail? Can I just add a --bind-ro to this folder?
Or is that not accesible as it is not a mount folder?
I'd like to add zwave stick as a device to zwave-js-ui container. I don't see this dev folder in the jail

Hello,

I've been trying to setup docker for a bit inside a jail (all working nicely for the most part!) - but I'm a bit stuck with the current networking setup.

I would like to keep host networking (the deafult), but as you mention TrueNAS will bind to 0.0.0.0 for it's UI, blocking ports 80 and 443.

I tried to do the following from the README:

To workaround this issue when using host networking, you may disable DHCP and add several static IP addresses (Aliases) through the TrueNAS web interface. If you setup the TrueNAS web interface to only listen on one of these IP addresses, the ports on the remaining IP addresses remain available for the jail to listen on.

I disabled DHCP, added a new Alias IP and marked that IP to be the "Web Interface IPv4 Address" in System > General > GUI. Sadly, even though it works (I can mark one or the other, and only that specific IP will show the UI) my docker container can't bind to 0.0.0.0:80.

I was able to bind the container once while the TrueNAS UI was restarting, which broke the TrueNAS UI (essentially, it didn't manage to bind to anything) - so it seems that no matter what IP you set, it keeps binding or occupying 0.0.0.0?

The tooltip in TrueNAS mentions:

The built-in HTTP server binds to the wildcard address of 0.0.0.0 (any address) and issues an alert if the specified address becomes unavailable.

So maybe there's no way to stop TrueNAS anymore? I didn't check how it looks in older versions.

Have you maybe managed to get this working recently, or am I missing some obvious setting change to stop TrueNAS from holding 0.0.0.0:80 hostage, or there's no way to bind a container to 0.0.0.0 and I should specify the IP alias directly in the container? Thanks!

TrueNAS-SCALE-22.02.4
jailmaker - latest
jail client: debian 
network: bridge

I have docker_compatible=1 in the config file. When trying to run docker in a jail, I get error
unable to start container process: unable to apply cgroup configuration: mkdir /sys/fs/cgroup/cpuset/docker: read-only file system

./jlmkr.py install
systemd-nspawn is already installed.
File /usr/local/sbin/jlmkr already exists... Maybe it's a broken symlink from a previous install attempt?
Skipped creating new symlink /usr/local/sbin/jlmkr to /mnt/SSD1/jailmaker/jlmkr.py.
Done installing jailmaker.

How do I fix this?
I had to rename the zpool because it was ridiculously long. But now I can not use the jlmkr command nor can I reïnstall it?

By default, Jailmaker disables link-local addressing for bridge and macvlan networks. The comment says that LinkLocalAddressing=yes interferes with DHCP address assignment.

For IPv4, this isn't a big deal, but for IPv6, link-local addressing is required. Otherwise, the container won't be able to get an IPv6 address via SLAAC or DHCPv6. This effectively breaks IPv6

I would suggest the following changes to jlmkr create:

1. Set LinkLocalAddressing=ipv6 when the container is created. This is actually the default in systemd anyway, and keeps link-local disabled for IPv4 (which seems to have been Jailmaker's original intent), but enables it for IPv6 (fixing IPv6 networking in the container).
2. Set DHCP=yes instead of DHCP=ipv4 since there's no longer any conflict with the LinkLocalAddressing setting. (In any case, SLAAC or DHCPv6 will still be used appropriately based on the router advertisements the container OS receives.)

With this configuration on a Debian Bookworm container in a TrueNAS SCALE Cobia host, everything seems to work fine. The container receives an IPv4 address (from DHCP), an IPv6 link-local address, and an IPv6 global address (from SLAAC).

How to set a static IP address when using --network-macvlan mode. Currently, I do this by entering the container and running the following command:

ip link set mv-enp4s0 up

ip addr add 192.168.1.2/24 dev mv-enp4s0
ip route add default via 192.168.1.1 dev mv-enp4s0

Forgive my limited IT skills to figure out the problem. I followed the instructions and while the jail was created, a machine was not. The log files show a lot of errors:

-- Journal begins at Tue 2023-04-18 03:04:40 CDT, ends at Tue 2023-04-18 20:39:38 CDT. --
Apr 18 20:24:47 truenas systemd[1]: Starting My nspawn jail myjail [created with jailmaker]...
Apr 18 20:24:47 truenas systemd[1]: Started My nspawn jail myjail [created with jailmaker].
Apr 18 20:24:47 truenas systemd-nspawn[1508324]: systemd 247.3-7+deb11u1 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPT>
Apr 18 20:24:47 truenas systemd-nspawn[1508324]: Detected virtualization systemd-nspawn.
Apr 18 20:24:47 truenas systemd-nspawn[1508324]: Detected architecture x86-64.
Apr 18 20:24:47 truenas systemd-nspawn[1508324]: Detected first boot.
Apr 18 20:24:47 truenas systemd-nspawn[1508324]:
Apr 18 20:24:47 truenas systemd-nspawn[1508324]: Welcome to Debian GNU/Linux 11 (bullseye)!
Apr 18 20:24:47 truenas systemd-nspawn[1508324]:
Apr 18 20:24:47 truenas systemd-nspawn[1508324]: Set hostname to .
Apr 18 20:24:47 truenas systemd-nspawn[1508324]: Initializing machine ID from container UUID.
Apr 18 20:24:47 truenas systemd-nspawn[1508324]: Failed to create control group inotify object: Too many open files
Apr 18 20:24:47 truenas systemd-nspawn[1508324]: Failed to allocate manager object: Too many open files
Apr 18 20:24:47 truenas systemd-nspawn[1508324]: [!!!!!!] Failed to allocate manager object.
Apr 18 20:24:47 truenas systemd-nspawn[1508324]: Exiting PID 1...
Apr 18 20:24:47 truenas systemd[1]: jlmkr-myjail.service: Main process exited, code=exited, status=255/EXCEPTION
Apr 18 20:24:47 truenas systemd[1]: jlmkr-myjail.service: Failed with result 'exit-code'.
Apr 18 20:28:39 truenas systemd[1]: Starting My nspawn jail myjail [created with jailmaker]...
Apr 18 20:28:39 truenas systemd[1]: Started My nspawn jail myjail [created with jailmaker].
Apr 18 20:28:39 truenas systemd-nspawn[1533219]: systemd 247.3-7+deb11u1 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPT>
Apr 18 20:28:39 truenas systemd-nspawn[1533219]: Detected virtualization systemd-nspawn.
Apr 18 20:28:39 truenas systemd-nspawn[1533219]: Detected architecture x86-64.
Apr 18 20:28:39 truenas systemd-nspawn[1533219]: Detected first boot.
Apr 18 20:28:39 truenas systemd-nspawn[1533219]:
Apr 18 20:28:39 truenas systemd-nspawn[1533219]: Welcome to Debian GNU/Linux 11 (bullseye)!
Apr 18 20:28:39 truenas systemd-nspawn[1533219]:
Apr 18 20:28:39 truenas systemd-nspawn[1533219]: Set hostname to .
Apr 18 20:28:39 truenas systemd-nspawn[1533219]: Initializing machine ID from container UUID.
Apr 18 20:28:39 truenas systemd-nspawn[1533219]: Failed to create control group inotify object: Too many open files
Apr 18 20:28:39 truenas systemd-nspawn[1533219]: Failed to allocate manager object: Too many open files
Apr 18 20:28:39 truenas systemd-nspawn[1533219]: [!!!!!!] Failed to allocate manager object.
Apr 18 20:28:39 truenas systemd-nspawn[1533219]: Exiting PID 1...
Apr 18 20:28:39 truenas systemd[1]: jlmkr-myjail.service: Main process exited, code=exited, status=255/EXCEPTION
Apr 18 20:28:39 truenas systemd[1]: jlmkr-myjail.service: Failed with result 'exit-code'.
Apr 18 20:31:48 truenas systemd[1]: Starting My nspawn jail myjail [created with jailmaker]...
Apr 18 20:31:48 truenas systemd[1]: Started My nspawn jail myjail [created with jailmaker].
Apr 18 20:31:48 truenas systemd-nspawn[1547125]: systemd 247.3-7+deb11u1 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPT>
Apr 18 20:31:48 truenas systemd-nspawn[1547125]: Detected virtualization systemd-nspawn.
Apr 18 20:31:48 truenas systemd-nspawn[1547125]: Detected architecture x86-64.
Apr 18 20:31:48 truenas systemd-nspawn[1547125]: Detected first boot.
Apr 18 20:31:48 truenas systemd-nspawn[1547125]:
Apr 18 20:31:48 truenas systemd-nspawn[1547125]: Welcome to Debian GNU/Linux 11 (bullseye)!
Apr 18 20:31:48 truenas systemd-nspawn[1547125]:
Apr 18 20:31:48 truenas systemd-nspawn[1547125]: Set hostname to .
Apr 18 20:31:48 truenas systemd-nspawn[1547125]: Initializing machine ID from container UUID.
Apr 18 20:31:48 truenas systemd-nspawn[1547125]: Failed to create control group inotify object: Too many open files

The readme says

[ ! -f /dev/nvidia-uvm ] && modprobe nvidia-current-uvm && /usr/bin/nvidia-modprobe -c0 -u

but doesn't that mean if the file, /dev/nvidia-uvm does NOT exist, try and load the nvidia-current-urm driver? Should it be

[ -f /dev/nvidia-uvm ] && modprobe nvidia-current-uvm && /usr/bin/nvidia-modprobe -c0 -u

?

Having issues setting up bridge networking. I followed these instructions to set up a bridge: https://www.youtube.com/watch?v=7clQw132w58, then added the repsective arguments you list, ensure the right device (br0) is provided. The jail has no internet connectivity.