Giter VIP home page Giter VIP logo

incus's Introduction

Linux Containers logo

LXC

LXC is the well-known and heavily tested low-level Linux container runtime. It is in active development since 2008 and has proven itself in critical production environments world-wide. Some of its core contributors are the same people that helped to implement various well-known containerization features inside the Linux kernel.

Status

Type Service Status
CI (Linux) GitHub Build Status
CI (Linux) Jenkins Build Status
Project status CII Best Practices CII Best Practices
Fuzzing OSS-Fuzz Fuzzing Status
Fuzzing CIFuzz CIFuzz

System Containers

LXC's main focus is system containers. That is, containers which offer an environment as close as possible as the one you'd get from a VM but without the overhead that comes with running a separate kernel and simulating all the hardware.

This is achieved through a combination of kernel security features such as namespaces, mandatory access control and control groups.

Unprivileged Containers

Unprivileged containers are containers that are run without any privilege. This requires support for user namespaces in the kernel that the container is run on. LXC was the first runtime to support unprivileged containers after user namespaces were merged into the mainline kernel.

In essence, user namespaces isolate given sets of UIDs and GIDs. This is achieved by establishing a mapping between a range of UIDs and GIDs on the host to a different (unprivileged) range of UIDs and GIDs in the container. The kernel will translate this mapping in such a way that inside the container all UIDs and GIDs appear as you would expect from the host whereas on the host these UIDs and GIDs are in fact unprivileged. For example, a process running as UID and GID 0 inside the container might appear as UID and GID 100000 on the host. The implementation and working details can be gathered from the corresponding user namespace man page.

Since unprivileged containers are a security enhancement they naturally come with a few restrictions enforced by the kernel. In order to provide a fully functional unprivileged container LXC interacts with 3 pieces of setuid code:

  • lxc-user-nic (setuid helper to create a veth pair and bridge it on the host)
  • newuidmap (from the shadow package, sets up a uid map)
  • newgidmap (from the shadow package, sets up a gid map)

Everything else is run as your own user or as a uid which your user owns.

In general, LXC's goal is to make use of every security feature available in the kernel. This means LXC's configuration management will allow experienced users to intricately tune LXC to their needs.

A more detailed introduction into LXC security can be found under the following link

Removing all Privilege

In principle LXC can be run without any of these tools provided the correct configuration is applied. However, the usefulness of such containers is usually quite restricted. Just to highlight the two most common problems:

  1. Network: Without relying on a setuid helper to setup appropriate network devices for an unprivileged user (see LXC's lxc-user-nic binary) the only option is to share the network namespace with the host. Although this should be secure in principle, sharing the host's network namespace is still one step of isolation less and increases the attack vector. Furthermore, when host and container share the same network namespace the kernel will refuse any sysfs mounts. This usually means that the init binary inside of the container will not be able to boot up correctly.

  2. User Namespaces: As outlined above, user namespaces are a big security enhancement. However, without relying on privileged helpers users who are unprivileged on the host are only permitted to map their own UID into a container. A standard POSIX system however, requires 65536 UIDs and GIDs to be available to guarantee full functionality.

Configuration

LXC is configured via a simple set of keys. For example,

  • lxc.rootfs.path
  • lxc.mount.entry

LXC namespaces configuration keys by using single dots. This means complex configuration keys such as lxc.net.0 expose various subkeys such as lxc.net.0.type, lxc.net.0.link, lxc.net.0.ipv6.address, and others for even more fine-grained configuration.

LXC is used as the default runtime for Incus, a container hypervisor exposing a well-designed and stable REST-api on top of it.

Kernel Requirements

LXC runs on any kernel from 2.6.32 onwards. All it requires is a functional C compiler. LXC works on all architectures that provide the necessary kernel features. This includes (but isn't limited to):

  • i686
  • x86_64
  • ppc, ppc64, ppc64le
  • riscv64
  • s390x
  • armv7l, arm64
  • loongarch64

LXC also supports at least the following C standard libraries:

  • glibc
  • musl
  • bionic (Android's libc)

Backwards Compatibility

LXC has always focused on strong backwards compatibility. In fact, the API hasn't been broken from release 1.0.0 onwards. Main LXC is currently at version 4.*.*.

Reporting Security Issues

The LXC project has a good reputation in handling security issues quickly and efficiently. If you think you've found a potential security issue, please report it by e-mail to all of the following persons:

  • serge (at) hallyn (dot) com
  • stgraber (at) ubuntu (dot) com
  • brauner (at) kernel (dot) org

For further details please have a look at

Becoming Active in LXC development

We always welcome new contributors and are happy to provide guidance when necessary. LXC follows the kernel coding conventions. This means we only require that each commit includes a Signed-off-by line. The coding style we use is identical to the one used by the Linux kernel. You can find a detailed introduction at:

and should also take a look at the CONTRIBUTING file in this repo.

If you want to become more active it is usually also a good idea to show up in the LXC IRC channel #lxc-dev on irc.libera.chat. We try to do all development out in the open and discussion of new features or bugs is done either in appropriate GitHub issues or on IRC.

When thinking about making security critical contributions or substantial changes it is usually a good idea to ping the developers first and ask whether a PR would be accepted.

Semantic Versioning

LXC and its related projects strictly adhere to a semantic versioning scheme.

Downloading the current source code

Source for the latest released version can always be downloaded from

You can browse the up to the minute source code and change history online

Building LXC

Without considering distribution specific details a simple

meson setup -Dprefix=/usr build
meson compile -C build

is usually sufficient.

Getting help

When you find you need help, the LXC projects provides you with several options.

Discuss Forum

We maintain a discuss forum at

where you can get support.

IRC

You can find us in #lxc on irc.libera.chat.

Mailing Lists

You can check out one of the two LXC mailing list archives and register if interested:

incus's People

Contributors

adamcstephens avatar adglkh avatar awalvie avatar brauner avatar chrisglass avatar ctrlrsf avatar dnegreira avatar freeekanayaka avatar gabrielmougard avatar hallyn avatar hnakamur avatar jochumdev avatar juneezee avatar markylaing avatar masnax avatar mihalicyn avatar mikemccracken avatar monstermunchkin avatar musicdin avatar presztak avatar roosterfish avatar ru-fu avatar simondeziel avatar stgraber avatar tarruda avatar techtonik avatar tenforward avatar tomponline avatar tych0 avatar weblate avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

incus's Issues

GPU not released on VM stop

Following lxc/lxc#4332 (comment) I'm opening the issue here.

Required information

Click to see full
  • Distribution: Ubuntu
  • Distribution version: 22.04
  • The output of
    • lxc-start --version
5.0.0~git2209-g5a7b9ce67
  • lxc-checkconfig
LXC version 5.0.0~git2209-g5a7b9ce67
Kernel configuration not found at /proc/config.gz; searching...
Kernel configuration found at /boot/config-6.2.0-26-generic
--- Namespaces ---
Namespaces: enabled
Utsname namespace: enabled
Ipc namespace: enabled
Pid namespace: enabled
User namespace: enabled
Network namespace: enabled

--- Control groups ---
Cgroups: enabled
Cgroup namespace: enabled

Cgroup v1 mount points: 


Cgroup v2 mount points: 
/sys/fs/cgroup

Cgroup v1 systemd controller: missing
Cgroup v1 freezer controller: missing
Cgroup ns_cgroup: required
Cgroup device: enabled
Cgroup sched: enabled
Cgroup cpu account: enabled
Cgroup memory controller: enabled
Cgroup cpuset: enabled

--- Misc ---
Veth pair device: enabled, not loaded
Macvlan: enabled, not loaded
Vlan: enabled, not loaded
Bridges: enabled, loaded
Advanced netfilter: enabled, loaded
CONFIG_IP_NF_TARGET_MASQUERADE: enabled, not loaded
CONFIG_IP6_NF_TARGET_MASQUERADE: enabled, not loaded
CONFIG_NETFILTER_XT_TARGET_CHECKSUM: enabled, not loaded
CONFIG_NETFILTER_XT_MATCH_COMMENT: enabled, not loaded
FUSE (for use with lxcfs): enabled, not loaded

--- Checkpoint/Restore ---
checkpoint restore: enabled
CONFIG_FHANDLE: enabled
CONFIG_EVENTFD: enabled
CONFIG_EPOLL: enabled
CONFIG_UNIX_DIAG: enabled
CONFIG_INET_DIAG: enabled
CONFIG_PACKET_DIAG: enabled
CONFIG_NETLINK_DIAG: enabled
File capabilities: 

Note : Before booting a new kernel, you can check its configuration
usage : CONFIG=/path/to/config /usr/bin/lxc-checkconfig
  • uname -a
Linux q1 6.2.0-26-generic #26~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Jul 13 16:27:29 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
  • cat /proc/self/cgroup
0::/user.slice/user-1000.slice/session-3.scope
  • cat /proc/1/mounts
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
udev /dev devtmpfs rw,nosuid,relatime,size=263874368k,nr_inodes=65968592,mode=755,inode64 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=52797032k,mode=755,inode64 0 0
efivarfs /sys/firmware/efi/efivars efivarfs rw,nosuid,nodev,noexec,relatime 0 0
/dev/nvme0n1p2 / ext4 rw,relatime,stripe=32 0 0
securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0
tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0
cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime 0 0
pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0
bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0
systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=92734 0 0
hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0
mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0
debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0
tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0
fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0
configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0
ramfs /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0
/dev/nvme0n1p1 /boot/efi vfat rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 0 0
/dev/loop0 /snap/core20/1852 squashfs ro,nodev,relatime,errors=continue,threads=single 0 0
/dev/loop1 /snap/core20/1974 squashfs ro,nodev,relatime,errors=continue,threads=single 0 0
/dev/loop2 /snap/core22/858 squashfs ro,nodev,relatime,errors=continue,threads=single 0 0
/dev/loop4 /snap/snapd/18596 squashfs ro,nodev,relatime,errors=continue,threads=single 0 0
/dev/loop3 /snap/lxd/25112 squashfs ro,nodev,relatime,errors=continue,threads=single 0 0
/dev/loop5 /snap/snapd/19457 squashfs ro,nodev,relatime,errors=continue,threads=single 0 0
binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /run/snapd/ns tmpfs rw,nosuid,nodev,noexec,relatime,size=52797032k,mode=755,inode64 0 0
nsfs /run/snapd/ns/lxd.mnt nsfs rw 0 0
tmpfs /var/snap/lxd/common/ns tmpfs rw,relatime,size=1024k,mode=700,inode64 0 0
nsfs /var/snap/lxd/common/ns/shmounts nsfs rw 0 0
nsfs /var/snap/lxd/common/ns/mntns nsfs rw 0 0
tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=52797028k,nr_inodes=13199257,mode=700,uid=1000,gid=1000,inode64 0 0
lxcfs /var/lib/lxcfs fuse.lxcfs rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other 0 0

Issue description

I have an Ubuntu:22.04 system with multiple GPUs, NVIDIA drivers 535 server installed, persistence mode is off. I pass individual GPUs to a VM as PCI passthrough. When I pass a single GPU to the VM, start it and then stop, the GPU is not returned to the host system (i.e. nvidia-smi does not show it anymore). When I pass multiple GPUs to the VM, start it and then stop, the GPU with the lowest PCI address on the host is not returned to the host system (i.e. nvidia-smi does not show it anymore), but the other GPUs get returned just fine.

Restarting the VMs again the GPUs are visible inside the VM, but if I start a container with nvidia-driver passthrough, only the GPUs that are currently visible on the host (i.e. all installed minus those that were not returned from the VMs earlier) are visible in the container. The only info I can find is that syslog says "Failed to stop device".

Steps to reproduce

  1. run nvidia-smi -L on host
  2. create VM with single GPU via passthrough
  3. start VM
  4. stop VM
  5. run nvidia-smi -L on host (the GPU that was passthrough to the VM will not be listed)
  6. create VM with multiple GPUs via passthrough
  7. start VM
  8. stop VM
  9. run nvidia-smi -L on host (the GPU will the lowest PCI address on the host that was passthrough to the VM will also not be listed)
  10. run container with nvidia-driver passthrough (same status as on the host)

Information to attach

Click to see full
  • VM log (lxc info --show-log vm2)
Name: vm2
Status: STOPPED
Type: virtual-machine
Architecture: x86_64
Created: 2023/08/07 22:14 UTC
Last Used: 2023/08/07 23:19 UTC

Log:

qemu-system-x86_64: Issue while setting TUNSETSTEERINGEBPF: Invalid argument with fd: 83, prog_fd: -1
  • any relevant kernel output (syslog), the single GPU case
Aug  7 23:18:53 q1 kernel: [  846.086912] vfio-pci 0000:ca:00.0: vgaarb: changed VGA decodes: olddecodes=none,decodes=io+mem:owns=none
Aug  7 23:18:53 q1 snapd[2334]: udevmon.go:149: udev event error: Unable to parse uevent, err: cannot parse libudev event: invalid env data
Aug  7 23:18:54 q1 kernel: [  846.548326] xhci_hcd 0000:ca:00.2: remove, state 4
Aug  7 23:18:54 q1 kernel: [  846.548343] usb usb10: USB disconnect, device number 1
Aug  7 23:18:54 q1 kernel: [  846.549060] xhci_hcd 0000:ca:00.2: USB bus 10 deregistered
Aug  7 23:18:54 q1 kernel: [  846.549083] xhci_hcd 0000:ca:00.2: remove, state 4
Aug  7 23:18:54 q1 kernel: [  846.549091] usb usb9: USB disconnect, device number 1
Aug  7 23:18:54 q1 kernel: [  846.550896] xhci_hcd 0000:ca:00.2: USB bus 9 deregistered
Aug  7 23:18:54 q1 kernel: [  846.653021] kauditd_printk_skb: 9 callbacks suppressed
Aug  7 23:18:54 q1 kernel: [  846.653026] audit: type=1400 audit(1691450334.129:54): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxd-vm2_</var/snap/lxd/common/lxd>" pid=5316 comm="apparmor_parser"
Aug  7 23:18:53 q1 snapd[2334]: message repeated 3 times: [ udevmon.go:149: udev event error: Unable to parse uevent, err: cannot parse libudev event: invalid env data]
Aug  7 23:18:55 q1 systemd[3823]: Started snap.lxd.lxc.b9b13195-c7c3-46d4-842a-856565db2c99.scope.
Aug  7 23:19:13 q1 kernel: [  865.800363] vfio-pci 0000:ca:00.0: vfio_ecap_init: hiding ecap 0x1e@0x258
Aug  7 23:19:13 q1 kernel: [  865.800386] vfio-pci 0000:ca:00.0: vfio_ecap_init: hiding ecap 0x19@0x900
Aug  7 23:19:46 q1 systemd[3823]: Started snap.lxd.lxc.0a424bc8-95d2-4cb9-bdd0-468d3dbce737.scope.
Aug  7 23:19:51 q1 systemd[3823]: Started snap.lxd.lxc.63564057-7dd7-462c-9548-3a5153ddd1e7.scope.
Aug  7 23:19:51 q1 systemd[1]: Starting Cleanup of Temporary Directories...
Aug  7 23:19:51 q1 systemd[1]: systemd-tmpfiles-clean.service: Deactivated successfully.
Aug  7 23:19:51 q1 systemd[1]: Finished Cleanup of Temporary Directories.
Aug  7 23:19:54 q1 kernel: [  907.246377] vfio-pci 0000:ca:00.0: Relaying device request to user (#0)
Aug  7 23:20:01 q1 kernel: [  913.710624] vfio-pci 0000:ca:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
Aug  7 23:20:01 q1 kernel: [  913.711376] vfio-pci 0000:ca:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
Aug  7 23:20:01 q1 lxd.daemon[3076]: time="2023-08-07T23:20:01Z" level=error msg="Failed to stop device" device=gpu3 err="Failed probing device \"0000:ca:00.0\" via \"/sys/bus/pci/drivers_probe\": write /sys/bus/pci/drivers_probe: invalid argument" instance=vm2 instanceType=virtual-machine project=default
Aug  7 23:20:01 q1 systemd-networkd[2222]: mac6293c2ac: Link DOWN
Aug  7 23:20:01 q1 systemd-networkd[2222]: mac6293c2ac: Lost carrier
Aug  7 23:20:01 q1 kernel: [  913.898141] audit: type=1400 audit(1691450401.373:55): apparmor="STATUS" operation="profile_remove" profile="unconfined" name="lxd-vm2_</var/snap/lxd/common/lxd>" pid=10366 comm="apparmor_parser"
Aug  7 23:32:42 q1 systemd[3823]: Started snap.lxd.lxc.44a0582a-97eb-4f56-9149-a7b6f2afec5b.scope.
  • any relevant kernel output (syslog), two GPU case
Aug  7 23:45:38 q1 kernel: [ 2450.861745] vfio-pci 0000:17:00.0: vgaarb: changed VGA decodes: olddecodes=none,decodes=io+mem:owns=none
Aug  7 23:45:38 q1 snapd[2334]: udevmon.go:149: udev event error: Unable to parse uevent, err: cannot parse libudev event: invalid env data
Aug  7 23:45:38 q1 kernel: [ 2451.339448] xhci_hcd 0000:17:00.2: remove, state 4
Aug  7 23:45:38 q1 kernel: [ 2451.339464] usb usb4: USB disconnect, device number 1
Aug  7 23:45:38 q1 kernel: [ 2451.340164] xhci_hcd 0000:17:00.2: USB bus 4 deregistered
Aug  7 23:45:38 q1 kernel: [ 2451.340188] xhci_hcd 0000:17:00.2: remove, state 4
Aug  7 23:45:38 q1 kernel: [ 2451.340197] usb usb3: USB disconnect, device number 1
Aug  7 23:45:38 q1 kernel: [ 2451.341944] xhci_hcd 0000:17:00.2: USB bus 3 deregistered
Aug  7 23:45:40 q1 kernel: [ 2453.384621] vfio-pci 0000:31:00.0: vgaarb: changed VGA decodes: olddecodes=none,decodes=io+mem:owns=none
Aug  7 23:45:41 q1 kernel: [ 2453.867449] xhci_hcd 0000:31:00.2: remove, state 4
Aug  7 23:45:41 q1 kernel: [ 2453.867464] usb usb6: USB disconnect, device number 1
Aug  7 23:45:41 q1 kernel: [ 2453.868123] xhci_hcd 0000:31:00.2: USB bus 6 deregistered
Aug  7 23:45:41 q1 kernel: [ 2453.868144] xhci_hcd 0000:31:00.2: remove, state 4
Aug  7 23:45:41 q1 kernel: [ 2453.868151] usb usb5: USB disconnect, device number 1
Aug  7 23:45:41 q1 kernel: [ 2453.869683] xhci_hcd 0000:31:00.2: USB bus 5 deregistered
Aug  7 23:45:41 q1 kernel: [ 2453.966981] audit: type=1400 audit(1691451941.446:56): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lxd-vm2_</var/snap/lxd/common/lxd>" pid=11010 comm="apparmor_parser"
Aug  7 23:46:00 q1 kernel: [ 2472.883434] vfio-pci 0000:17:00.0: vfio_ecap_init: hiding ecap 0x1e@0x258
Aug  7 23:46:00 q1 kernel: [ 2472.883457] vfio-pci 0000:17:00.0: vfio_ecap_init: hiding ecap 0x19@0x900
Aug  7 23:46:00 q1 kernel: [ 2473.055433] vfio-pci 0000:31:00.0: vfio_ecap_init: hiding ecap 0x1e@0x258
Aug  7 23:46:00 q1 kernel: [ 2473.055455] vfio-pci 0000:31:00.0: vfio_ecap_init: hiding ecap 0x19@0x900
Aug  7 23:45:40 q1 snapd[2334]: message repeated 7 times: [ udevmon.go:149: udev event error: Unable to parse uevent, err: cannot parse libudev event: invalid env data]
Aug  7 23:47:16 q1 systemd[3823]: Started snap.lxd.lxc.c918eda7-03e8-4d84-9cb2-c9e1b4d6bfa2.scope.
Aug  7 23:49:01 q1 kernel: [ 2653.889634] vfio-pci 0000:31:00.0: Relaying device request to user (#0)
Aug  7 23:49:08 q1 kernel: [ 2660.602855] vfio-pci 0000:31:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
Aug  7 23:49:08 q1 kernel: [ 2660.603292] nvidia 0000:31:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=none
Aug  7 23:49:08 q1 kernel: [ 2660.690297] snd_hda_intel 0000:31:00.1: Disabling MSI
Aug  7 23:49:08 q1 kernel: [ 2660.690325] snd_hda_intel 0000:31:00.1: Handle vga_switcheroo audio client
Aug  7 23:49:08 q1 kernel: [ 2660.714786] input: HDA NVidia HDMI/DP,pcm=3 as /devices/pci0000:30/0000:30:02.0/0000:31:00.1/sound/card0/input19
Aug  7 23:49:08 q1 kernel: [ 2660.714916] input: HDA NVidia HDMI/DP,pcm=7 as /devices/pci0000:30/0000:30:02.0/0000:31:00.1/sound/card0/input20
Aug  7 23:49:08 q1 kernel: [ 2660.715088] input: HDA NVidia HDMI/DP,pcm=8 as /devices/pci0000:30/0000:30:02.0/0000:31:00.1/sound/card0/input21
Aug  7 23:49:08 q1 kernel: [ 2660.715283] input: HDA NVidia HDMI/DP,pcm=9 as /devices/pci0000:30/0000:30:02.0/0000:31:00.1/sound/card0/input22
Aug  7 23:49:08 q1 snapd[2334]: udevmon.go:149: udev event error: Unable to parse uevent, err: cannot parse libudev event: invalid env data
Aug  7 23:49:08 q1 kernel: [ 2660.726602] xhci_hcd 0000:31:00.2: xHCI Host Controller
Aug  7 23:49:08 q1 kernel: [ 2660.726615] xhci_hcd 0000:31:00.2: new USB bus registered, assigned bus number 3
Aug  7 23:49:08 q1 kernel: [ 2660.727221] xhci_hcd 0000:31:00.2: hcc params 0x0180ff05 hci version 0x110 quirks 0x0000000000000010
Aug  7 23:49:08 q1 kernel: [ 2660.727606] xhci_hcd 0000:31:00.2: xHCI Host Controller
Aug  7 23:49:08 q1 kernel: [ 2660.727610] xhci_hcd 0000:31:00.2: new USB bus registered, assigned bus number 4
Aug  7 23:49:08 q1 kernel: [ 2660.727613] xhci_hcd 0000:31:00.2: Host supports USB 3.1 Enhanced SuperSpeed
Aug  7 23:49:08 q1 kernel: [ 2660.727661] usb usb3: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 6.02
Aug  7 23:49:08 q1 kernel: [ 2660.727664] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
Aug  7 23:49:08 q1 kernel: [ 2660.727666] usb usb3: Product: xHCI Host Controller
Aug  7 23:49:08 q1 kernel: [ 2660.727668] usb usb3: Manufacturer: Linux 6.2.0-26-generic xhci-hcd
Aug  7 23:49:08 q1 kernel: [ 2660.727669] usb usb3: SerialNumber: 0000:31:00.2
Aug  7 23:49:08 q1 kernel: [ 2660.727830] hub 3-0:1.0: USB hub found
Aug  7 23:49:08 q1 kernel: [ 2660.727837] hub 3-0:1.0: 2 ports detected
Aug  7 23:49:08 q1 kernel: [ 2660.727975] usb usb4: We don't know the algorithms for LPM for this host, disabling LPM.
Aug  7 23:49:08 q1 kernel: [ 2660.727993] usb usb4: New USB device found, idVendor=1d6b, idProduct=0003, bcdDevice= 6.02
Aug  7 23:49:08 q1 kernel: [ 2660.727995] usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1
Aug  7 23:49:08 q1 kernel: [ 2660.727997] usb usb4: Product: xHCI Host Controller
Aug  7 23:49:08 q1 kernel: [ 2660.727999] usb usb4: Manufacturer: Linux 6.2.0-26-generic xhci-hcd
Aug  7 23:49:08 q1 kernel: [ 2660.728000] usb usb4: SerialNumber: 0000:31:00.2
Aug  7 23:49:08 q1 kernel: [ 2660.728175] hub 4-0:1.0: USB hub found
Aug  7 23:49:08 q1 kernel: [ 2660.728184] hub 4-0:1.0: 4 ports detected
Aug  7 23:49:08 q1 snapd[2334]: message repeated 3 times: [ udevmon.go:149: udev event error: Unable to parse uevent, err: cannot parse libudev event: invalid env data]
Aug  7 23:49:08 q1 systemd[3823]: Reached target Sound Card.
Aug  7 23:49:08 q1 kernel: [ 2660.807453] vfio-pci 0000:17:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
Aug  7 23:49:08 q1 kernel: [ 2660.807674] vfio-pci 0000:17:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
Aug  7 23:49:08 q1 lxd.daemon[3076]: time="2023-08-07T23:49:08Z" level=error msg="Failed to stop device" device=gpu0 err="Failed probing device \"0000:17:00.0\" via \"/sys/bus/pci/drivers_probe\": write /sys/bus/pci/drivers_probe: invalid argument" instance=vm2 instanceType=virtual-machine project=default
Aug  7 23:49:08 q1 systemd-networkd[2222]: mac43379c64: Link DOWN
Aug  7 23:49:08 q1 systemd-networkd[2222]: mac43379c64: Lost carrier
Aug  7 23:49:08 q1 kernel: [ 2661.011901] audit: type=1400 audit(1691452148.495:57): apparmor="STATUS" operation="profile_remove" profile="unconfined" name="lxd-vm2_</var/snap/lxd/common/lxd>" pid=13584 comm="apparmor_parser"
  • the VM configuration file
architecture: x86_64
config:
  agent.nic_config: "true"
  cloud-init.network-config: |
    version: 1
    config:
      - type: physical
        name: eth0
        subnets:
          - type: static
            ipv4: true
            address: 10.10.10.10/25
            gateway: 10.10.10.1
            control: auto
      - type: nameserver
        address:
          - 1.1.1.1
          - 1.0.0.1
  cloud-init.user-data: |
    #cloud-config
    ssh_import_id: [gh:itzsimpl]
  image.architecture: amd64
  image.description: ubuntu 22.04 LTS amd64 (release) (20230729)
  image.label: release
  image.os: ubuntu
  image.release: jammy
  image.serial: "20230729"
  image.type: disk-kvm.img
  image.version: "22.04"
  limits.cpu: "20"
  limits.memory: 64GiB
  security.secureboot: "false"
  volatile.base_image: c3a32ce371819c4fb845867e8e602ad6a636e211cfaeca448e767de4b415c038
  volatile.cloud-init.instance-id: f6fa9720-3024-4574-bbd7-e29a10e14ca0
  volatile.eth0.hwaddr: 00:16:3e:73:46:f3
  volatile.last_state.power: STOPPED
  volatile.last_state.ready: "false"
  volatile.uuid: 114bc8ad-0afb-4732-9911-f2583a3330c4
  volatile.uuid.generation: 114bc8ad-0afb-4732-9911-f2583a3330c4
  volatile.vsock_id: "1262936222"
devices:
  eth0:
    name: eth0
    nictype: macvlan
    parent: ens97f0np0
    type: nic
  gpu0:
    gputype: physical
    pci: "0000:17:00.0"
    type: gpu
  gpu1:
    gputype: physical
    pci: "0000:31:00.0"
    type: gpu
  root:
    path: /
    pool: default
    size: 128GB
    type: disk
ephemeral: false
profiles:
- default
- pub-macvlan
- gpu0
- gpu1
stateful: false
description: vm2

Build aarch64 static binaries

When building static binaries for incus and incus-migrate, also have Go cross-compile some arm64 versions for all three OS.

Remove Canonical RBAC support

Canonical RBAC is effectively a dead solution which shouldn't be brought forward into a fork.
Instead the fork will have OIDC and eventually OpenFGA support.

Release tarball needs to include the vendored tree for lxd-to-incus

Required information

  • Distribution: Gentoo
  • Distribution version: ~
  • The output of "inc info" or if that fails:
    • Kernel version: 6.1.55
    • LXC version: 5.0.3
    • Incus version: 0.1
    • Storage backend in use: -

Issue description

I downloaded incus-0.1 from https://linuxcontainers.org/incus/downloads/ which holds the vendored go modules. When making distribution package, these are often built offline so providing the vendored go mods is great! But it looks like there's some sort of a problem with the lxd-to-incus tool in this tarball. I don't understand go enough, but this is the error I'm getting:

main module (github.com/lxc/incus) does not contain package github.com/lxc/incus/cmd/lxd-to-incus

I will attach the fuild build log so you can inspect it. Building the other tools, fuidshift incus-benchmark incus-user incus lxc-to-incus incusd incus-migrate incus-agent work correctly. Only lxd-to-incus is problematic.

Steps to reproduce

  1. Download tarball from https://linuxcontainers.org/incus/downloads/
  2. Build it offline so go doesn't have a chance to download any missing modules

Add a new image restriction for privileged containers

The current NixOS image can't work in privileged containers, while that's being worked on, it would be nice to be able to publish the image but have a restriction in place that prevents it from being used for privileged containers.

Add test for incus-user

We just had a regression in incus-user which really should have been caught prior to release.
This highlighted the fact that this daemon is completely untested and entirely relies on manual testing pre-release.

We should add a basic test to the testsuite which spawns incus-user and attempts to interact with Incus from an unprivileged user to validate its behavior.

lxd-to-incus version mismatch

Issue description

lxd-to-incus in tag incus-0.1 reports version 5.16. This is because lxd-to-incus is a completely separate go package and its parsing of the incus version is locked to what go.sum has.

Steps to reproduce

  1. Clone repo
  2. cd cmd/lxd-to-incus
  3. go build ./.
  4. ./lxd-to-incus --version
    5.16

Other

I'm attempting to finalize packaging incus for nixpkgs. Am I correct to assume that this should be 0.1 to match the incus version? Would it be better to maybe version it on its own? Since it's using go packages for all the incus dependencies, does it benefit from being in this repo?

doc: unreleased Incus X.YY version

There are four instances of unreleased Incus X.YY version in the doc.

$ rg 'Incus [1-9](\.[0-9]*)?' doc/
doc/authentication.md
127:Starting with Incus 5.13, authentication through OpenID Connect is supported, but there is no user role handling in place so far.

doc/support.md
12:The upcoming LTS release will be Incus 6.0, which will supported until June 2029 and will get frequent bugfix and security updates, but won't receive any feature additions.

doc/rest-api.md
236:The documentation shows paths such as `/1.0/instances/...`, which were introduced with Incus 3.19.

doc/explanation/clustering.md
8:This feature was introduced as part of the [`clustering`](../api-extensions.md#clustering) API extension and is available since Incus 3.0.

Eliminate `core.trust_password`

core.trust_password was a pretty bad idea for how to handle initial client trust and has since been mostly replaced with the token-based workflow (incus config trust add --name foo).

The concept of trust password has been slowly phased out from documentation, init and examples in LXD.
But it's also one of those things that could never fully go away because of backward-compatibility.

With Incus, we can now fully get rid of it and also return the server config to a clean map[string]string as the map[string]interface{} was a hack done solely to avoid the trust password to be showed.

Introduce a proper `inc snapshot` subcommand

One bit of inconsistency that's always bothered me with LXD is that lxc snapshot and lxc restore are top level commands of their own. This makes it very awkward to allow for snapshot listing, renaming, ...

We've been wanting to fix this for years, but couldn't without breaking backward compatibility. As the fork doesn't pretend to be backward compatible at the CLI level, it's a good opportunity to fix that one.

Stopping a container brings down the host eth0 interface

Required information

  • Distribution: Debian
  • Distribution version: 12
  • The output of "inc info" or if that fails:
    • Kernel version: 6.x
    • LXC version: n/a
    • Incus version: 0.1
    • Storage backend in use: btrfs

Issue description

As discussed in zabbly/incus#5, stopping a container brings down the host eth0 interface.

Renaming the host eth0 to the default enp5s0 works (actually, I've removed net.ifnames=0 from the linux kernel command line); when stopping the container, it no longer shuts down the host enp5s0 network interface.

Please note that this does not happen when using a virtual machine; in this case, the host interface is not shutdown.

Steps to reproduce

With Incus running in a Debian 12 (vagrant managed) virtual machine, and configured as https://github.com/rgl/incus-playground/blob/main/provision-incus.sh#L71-L99, follow the next steps.

# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:8f:68:65 brd ff:ff:ff:ff:ff:ff
    altname enp5s0
    inet 192.168.121.125/24 brd 192.168.121.255 scope global dynamic eth0
       valid_lft 3384sec preferred_lft 3384sec
    inet6 fe80::5054:ff:fe8f:6865/64 scope link 
       valid_lft forever preferred_lft forever
3: incusbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 00:16:3e:b3:a2:02 brd ff:ff:ff:ff:ff:ff
    inet 10.2.0.1/24 scope global incusbr0
       valid_lft forever preferred_lft forever

# incus launch images:debian/12 debian-ct
Creating debian-ct
Starting debian-ct                            

# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:8f:68:65 brd ff:ff:ff:ff:ff:ff
    altname enp5s0
    inet 192.168.121.125/24 brd 192.168.121.255 scope global dynamic eth0
       valid_lft 3357sec preferred_lft 3357sec
    inet6 fe80::5054:ff:fe8f:6865/64 scope link 
       valid_lft forever preferred_lft forever
3: incusbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:16:3e:b3:a2:02 brd ff:ff:ff:ff:ff:ff
    inet 10.2.0.1/24 scope global incusbr0
       valid_lft forever preferred_lft forever
5: vethc085fdca@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master incusbr0 state UP group default qlen 1000
    link/ether c6:48:85:2f:67:d5 brd ff:ff:ff:ff:ff:ff link-netnsid 0

# incus stop debian-ct

# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000
    link/ether 52:54:00:8f:68:65 brd ff:ff:ff:ff:ff:ff
    altname enp5s0
3: incusbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 00:16:3e:b3:a2:02 brd ff:ff:ff:ff:ff:ff
    inet 10.2.0.1/24 scope global incusbr0
       valid_lft forever preferred_lft forever

# journalctl
...
Oct 08 09:14:05 incus kernel: incusbr0: port 1(vethc085fdca) entered disabled state
Oct 08 09:14:05 incus kernel: veth0ed2f97b: renamed from physpu5wB2
Oct 08 09:14:05 incus (udev-worker)[13609]: Network interface NamePolicy= disabled on kernel command line.
Oct 08 09:14:05 incus kernel: device vethc085fdca left promiscuous mode
Oct 08 09:14:05 incus kernel: incusbr0: port 1(vethc085fdca) entered disabled state
Oct 08 09:14:05 incus systemd[1]: Stopping [email protected] - ifup for eth0...
Oct 08 09:14:05 incus dhclient[13642]: Killed old client process
Oct 08 09:14:05 incus ifdown[13642]: Killed old client process
Oct 08 09:14:05 incus audit[13652]: AVC apparmor="STATUS" operation="profile_remove" profile="unconfined" name="incus-debian-ct_</var/lib/incus>" pid=13652 comm="apparmor_parser"
Oct 08 09:14:05 incus kernel: audit: type=1400 audit(1696752845.948:19): apparmor="STATUS" operation="profile_remove" profile="unconfined" name="incus-debian-ct_</var/lib/incus>" pid=13652 comm="apparmor_parser"
Oct 08 09:14:06 incus dhclient[13642]: Internet Systems Consortium DHCP Client 4.4.3-P1
Oct 08 09:14:06 incus ifdown[13642]: Internet Systems Consortium DHCP Client 4.4.3-P1
Oct 08 09:14:06 incus ifdown[13642]: Copyright 2004-2022 Internet Systems Consortium.
Oct 08 09:14:06 incus ifdown[13642]: All rights reserved.
Oct 08 09:14:06 incus ifdown[13642]: For info, please visit https://www.isc.org/software/dhcp/
Oct 08 09:14:06 incus dhclient[13642]: Copyright 2004-2022 Internet Systems Consortium.
Oct 08 09:14:06 incus dhclient[13642]: All rights reserved.
Oct 08 09:14:06 incus dhclient[13642]: For info, please visit https://www.isc.org/software/dhcp/
Oct 08 09:14:06 incus dhclient[13642]: 
Oct 08 09:14:06 incus dhclient[13642]: Listening on LPF/eth0/52:54:00:8f:68:65
Oct 08 09:14:06 incus ifdown[13642]: Listening on LPF/eth0/52:54:00:8f:68:65
Oct 08 09:14:06 incus ifdown[13642]: Sending on   LPF/eth0/52:54:00:8f:68:65
Oct 08 09:14:06 incus ifdown[13642]: Sending on   Socket/fallback
Oct 08 09:14:06 incus dhclient[13642]: Sending on   LPF/eth0/52:54:00:8f:68:65
Oct 08 09:14:06 incus dhclient[13642]: Sending on   Socket/fallback
Oct 08 09:14:06 incus dhclient[13642]: DHCPRELEASE of 192.168.121.125 on eth0 to 192.168.121.1 port 67
Oct 08 09:14:06 incus ifdown[13642]: DHCPRELEASE of 192.168.121.125 on eth0 to 192.168.121.1 port 67
Oct 08 09:14:06 incus systemd[1]: [email protected]: Deactivated successfully.
Oct 08 09:14:06 incus systemd[1]: Stopped [email protected] - ifup for eth0.

Rework the OVN syslog handler into a more generic network ACL log handler

LXD 5.18 introduced a new feature allowing LXD to act as a syslog target for OVN, the logs received are then streamed over both the events API and to LOKI.

This is pretty neat, but the reality is that we only care about log messages related to networks that we managed, we don't really want to start handling and forwarding system log messages about things like the state of the OVN cluster or its database, that should be handled separately by the system administrator.

Remove Canonical MAAS support

The MAAS integration is a nightmare to test and to my knowledge never had a real user, so it should probably be dropped from the fork.

doc: change Grafana dashboard screen shot images for Incus

Eliminate `INCUS_UNPRIVILEGED_ONLY`

This was a workaround introduced to add a needed restriction on chromebooks, but this is now possible through project limits and so should be dropped.

Remove shiftfs support

Shiftfs is an Ubuntu-specific technology which is very actively being deprecated in favor of the mainline VFS idmap feature.
There's not much reason to keep the double logic in a new fork, so let's remove it.

Create tooling to import changes from canonical/lxd

With Incus having renamed most files across the entire tree, it's near impossible to cherry-pick anything from the LXD repository.
As the fork should definitely be kept up to date, custom tooling that can programmatically handle the bulk of the file and function renames will be needed.

doc: "lxd" leftover

First of all, thanks for your hard work!

It seems a few "lxd" left in doc/ directory. I suppose these are needed to be modified too, right?

$ rg -i lxd doc/
doc/doc-cheat-sheet.md
695:    relatedlinks: https://github.com/canonical/lxd-sphinx-extensions, [RTFM](https://www.google.com)

doc/cloud-init.md
48:For more information about the configuration options, see the [`cloud-init` instance options](instance-options-cloud-init), and the documentation for the {ref}`LXD data source <cloud-init:datasource_lxd>` in the `cloud-init` documentation.
$ git rev-parse HEAD
5e7be670320b81f1eacf0406a236eed3d5fad8d5

[lxd-to-incus] Support migrating clusters

Currently lxd-to-incus refuses to migrate clusters.
That's because there's a fair bit more synchronization involved with migrating a cluster as opposed to a standalone system.

For clusters, we'll need to:

  • Confirm that all cluster members are currently online
  • Show the list of machines that will need to be migrated
  • Have the user confirm that they wish to migrate their cluster
  • Generate a single database patch to be run by the initial migrated server
  • Provide a command line flag to lxd-to-incus for the remaining servers which will bypass all validation and just do the data migration. This is needed as once we start shutting down LXD, we won't be able to talk to it again from the remaining servers.

Use tags with semver syntax

Instead of using incus-0.1 please use tags with the semver syntax like 0.1.0 or v0.1.0. That would make things easier to reason about (e.g. tools like renovate work out-of-the-box with those).

Improvements to lxd-to-incus

Prior to Incus 0.1, we need lxd-to-incus to:

  • Handle migrating Debian installations
  • Not require that LXD and Incus version match
  • Support LXD > 4.0 and <= 5.18
  • Properly rewrite storage pool paths during migration
  • Detect limits.network.priority
  • Detect fan or bridge.mode in networks

Disable raft snapshot compression

I've run a few tests and benchmarks around cowsql database raft snapshot compression using lz4, and the conclusion seems to be that they are not worth, since they save relatively little space at the price of noticeable CPU overhead.

The database of an incus deployment with 1000 instances and associated metadata (profiles, networks, snapshots, images, volumes, etc) has an uncompressed size of ~4M and a compressed size of ~3M, so lz4 doesn't really move the needle there. A deployment of 50k instances has a database size of ~181M, and a compressed size of ~150M: the storage/memory gain is modest, while the CPU overhead of running compression can be noticeable when hitting the API when the snapshot is in progress.

A much more effective measure seems to be to regularly VACUUM the database, which can reduce the size of a database dramatically (even 10x for databases that have been running for a long time and have a lot of fragmentation). I'm planning to run VACUUM transparently in cowsql.

If there's consensus around this, I'd like to remove compression from both incus and cowsql and figure the best option for upgrades (it'd basically just a matter of uncompressing the existing snapshots at upgrade time and deciding where to put that logic).

This should also simplify the on-disk format and make it easier to improve the command line tooling that we have around it.

panic: Architecture isn't supported: loong64

Debian has announced loong64 as a new port. While Incus hasn't yet been uploaded to Debian, LXD encounters a run-time panic on that architecture (buildd log), and I would expect Incus to have the same issue:

panic: Architecture isn't supported: loong64

goroutine 1 [running]:
github.com/lxc/lxd/shared/version.getUserAgent()
	github.com/lxc/lxd/shared/version/useragent.go:22 +0x548
github.com/lxc/lxd/shared/version.init()
	github.com/lxc/lxd/shared/version/useragent.go:15 +0x28

I'm not sure if this will require more than just adding loong64 to shared/osarch/architectures.go.

(Also reported to LXD as canonical/lxd#12387)

Errors in build

rpmbuild -ba incus.spec               
Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.13883
+ umask 022
+ cd /home/builder/rpmbuild/BUILD
+ '[' 1 -eq 1 ']'
+ '[' 1 -eq 1 ']'
+ '[' 1 -eq 1 ']'
+ cd /home/builder/rpmbuild/BUILD
+ rm -rf incus-incus-0.1
+ /bin/gzip -dc /home/builder/rpmbuild/SOURCES/incus-0.1.tar.gz
+ /usr/bin/tar -xf -
+ STATUS=0
+ '[' 0 -ne 0 ']'
+ cd incus-incus-0.1
+ exit 0
Executing(%build): /bin/sh -e /var/tmp/rpm-tmp.13883
+ umask 022
+ cd /home/builder/rpmbuild/BUILD
+ cd incus-incus-0.1
+ '[' 1 -eq 1 ']'
+ '[' 1 -eq 1 ']'
+ export CC=/opt/llvm16/bin/clang
+ CC=/opt/llvm16/bin/clang
+ export CXX=/opt/llvm16/bin/clang++
+ CXX=/opt/llvm16/bin/clang++
+ export CPP=/opt/llvm16/bin/clang-cpp
+ CPP=/opt/llvm16/bin/clang-cpp
+ export PATH=/opt/llvm16/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+ PATH=/opt/llvm16/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+ export GOPROXY=direct
+ GOPROXY=direct
+ /usr/bin/make -j8 build
make: command: Command not found
make: command: Command not found
make: command: Command not found
CC="/opt/llvm16/bin/clang" CGO_LDFLAGS_ALLOW="(-Wl,-wrap,pthread_create)|(-Wl,-z,now)" go install -v -tags "libsqlite3"  ./...
github.com/lxc/incus/internal/server/seccomp
# github.com/lxc/incus/internal/server/seccomp
internal/server/seccomp/seccomp.go:342:23: error: 'aligned' attribute ignored when parsing type [-Werror,-Wignored-attributes]
internal/server/seccomp/seccomp.go:254:25: note: expanded from macro 'ptr_to_u64'
/usr/include/linux/types.h:43:44: note: expanded from macro '__aligned_u64'
internal/server/seccomp/seccomp.go:347:21: error: 'aligned' attribute ignored when parsing type [-Werror,-Wignored-attributes]
internal/server/seccomp/seccomp.go:254:25: note: expanded from macro 'ptr_to_u64'
/usr/include/linux/types.h:43:44: note: expanded from macro '__aligned_u64'
internal/server/seccomp/seccomp.go:348:22: error: 'aligned' attribute ignored when parsing type [-Werror,-Wignored-attributes]
internal/server/seccomp/seccomp.go:254:25: note: expanded from macro 'ptr_to_u64'
/usr/include/linux/types.h:43:44: note: expanded from macro '__aligned_u64'
make: *** [Makefile:35: build] Error 1
error: Bad exit status from /var/tmp/rpm-tmp.13883 (%build)

%global        optflags %(echo %{optflags} | sed -e 's/-ffat-lto-objects//g' -e 's/-fvar-tracking-assignments//g' -e 's/-Wa,--compress-debug-sections//g')
%define         debug %nil
%define         debug_-package %nil

Name:           incus
Version:        0.1
Release:        1
Summary:        Powerful system container and virtual machine manager
URL:            https://linuxcontainers.org/incus
Source0:        https://github.com/lxc/incus/archive/%{name}-%{version}.tar.gz
License:        ASL 2.0
Group:          Development/Other
BuildRequires:  go
BuildRequires:  git-core
BuildRequires:  upx
BuildRequires:  pkgconfig(libuv) >= 1.8
BuildRequires:  pkgconfig(sqlite)
BuildRequires:  pkgconfig(cowsql)
BuildRequires:  pkgconfig(raft) >= 0.17.5
BuildRequires:  pkgconfig(libcap)
BuildRequires:  pkgconfig(lxc)
BuildRequires:  pkgconfig(libacl)
BuildRequires:  pkgconfig(libudev)
BuildRequires:  pkgconfig(libseccomp)
BuildRequires:  llvm16

%description
...

%prep
%setup -qn %{name}-%{name}-%{version}

%build

export CC=/opt/llvm16/bin/clang
export CXX=/opt/llvm16/bin/clang++
export CPP=/opt/llvm16/bin/clang-cpp

export PATH=/opt/llvm16/bin:$PATH

export GOPROXY=direct

%make build
lsb_release -a
LSB Version:    *
Distributor ID: RosaDesktopFresh
Description:    ROSA Desktop Fresh R11.1
Release:        2016.1
Codename:       Fresh

go version
go version go1.21.1 linux/amd64

Required information

  • Distribution:
  • Distribution version:
  • The output of "inc info" or if that fails:
    • Kernel version:
    • LXC version:
    • Incus version:
    • Storage backend in use:

Issue description

A brief description of the problem. Should include what you were
attempting to do, what you did, what happened and what you expected to
see happen.

Steps to reproduce

  1. Step one
  2. Step two
  3. Step three

Information to attach

  • Any relevant kernel output (dmesg)
  • Container log (inc info NAME --show-log)
  • Container configuration (inc config show NAME --expanded)
  • Main daemon log (at /var/log/incus/incus.log)
  • Output of the client with --debug
  • Output of the daemon with --debug (alternatively output of inc monitor while reproducing the issue)

Snapshot retention policy (instances and storage volumes)

With #9 making significant changes to the structure of the inc snapshot command (and there being more of a possibility of breaking backwards compatibility), I'd like to suggest a feature to increase snapshot retention granularity. The following use cases warrant maintaining a history of snapshots, but not at the same duration or granularity as the server where the incus container/VM is actually running:

  • Backup Server: you may want to keep a variety of snapshots going back several months (or even years) but you need a variable level of granuarlity as you go farther back in time (or storage space and the sheer number of snapshots will become unmanagable). For example, let's say you take snapshots every hour. You may want a lot of granularity in recent snapshots, say the last 10 hourly snapshots, but then you only need 2 snapshots from last week, 1 snapshot from last month, and 1 from last year. Sanoid lets you configure this type of variable-granularity-based-on-timescale and it would be great if we could do this with incus too

  • Warm Spare: it's desirable to maintain a warm spare server with a copy of the incus container/VM data: in case the active server goes down, the guest can quickly be brought back online on the warm spare. This is discussed briefly here, but utilizing syncoid and ZFS snapshots to only send the updated delta means that keeping the warm spare up-to-date is easy and doesn't require re-syncing the whole dataset. I think incus supports copying the guest with the --refresh flag, but again you have the problem of not being able to maintain a separate snapshot retention policy on the warm spare

Sanoid now supports pre-snapshot, post-snapshot, and post-prune hooks so if incus had a command like incus snapshot {create,delete} --metadata-only --snapshot-name="xxx" to only create/delete the database entry but not actually try to zfs snapshot/zfs destroy the snapshot, this would work (you could let sanoid be in charge of creating and deleting the actual ZFS snapshots).

Is it possible to add the above type of enhanced functionality for snapshots to incus, or provide some type of integration with 3rd-party tools like sanoid/syncoid?

Remove Ubuntu Fan support

The Ubuntu Fan bridge which is primarily used in the clustering case is a very convenient feature, but it's one that Canonical never tried to upstream and therefore requires custom patches to both the Linux kernel and iproute2 to function.

As Incus has proper overlay networking support through standard OVS/OVN, that should be the focus and the Ubuntu Fan can go away.

Add packager documentation

To try and get a consistent experience across all Linux distributions, we should document some recommendations around things like init script naming, unix groups and permissions.

Move the daemon commands under `incus admin`

Rather than have users invoke sub-commands on the daemon, lets move all of those over to a new incus admin sub-command in the client. All daemon commands have been API based for years, so there's no real reason for them to be under a different binary and having to keep both the daemon and client in the PATH has caused no end of confusion over the years.

With this change in place, the incusd binary should be kept out of PATH, only leaving incus to be user visible.

Rework incus-agent loader to use udev

Currently incus-agent is primarily spawnwed by a systemd unit with a fallback in case of race conditions done through a udev rule.

After some discussion on the topic, it appears that the better option would be to not rely on systemd to start the service but instead use udev with systemd tags to tell systemd when the unit needs starting.

`incus admin cluster` subcommands just call `incus admin cluster` again

Required information

  • Distribution: Debian
  • Distribution version: 12.2
  • The output of "inc info" or if that fails:
    • Kernel version: 6.1.0-13-amd64
    • LXC version: not sure. I'm only using VMs and have no packages installed specifically for lxc, nor an lxc executable in /opt/incus/bin.
    • Incus version: 0.1 (Debian package version 0.1-202310101806-debian12)
    • Storage backend in use: ZFS (on cluster)

Issue description

I ran into two issues when attempting to change the IP address of cluster members using this documentation: https://linuxcontainers.org/incus/docs/main/howto/cluster_recover/#recover-cluster-members-with-changed-addresses

The first is the focus of this issue: running any incus admin cluster <subcommand> is identical to running incus admin cluster - it just shows the help page for incus admin cluster. I was looking to run incus admin cluster edit to modify IP address settings, but the other subcommands have the same behavior.

The second is probably a packaging or Debian installation documentation issue where incus admin cluster wasn't finding dynamic libraries for (at least) liblxc or libcowsql. I temporarily fixed this by setting LD_LIBRARY_PATH=/opt/incus/lib.

Steps to reproduce

  1. Run incus admin cluster edit or any other subcommand of incus admin cluster.

doc: remove trust password

73f1896

There are still "trust password" in doc.

$ rg 'trust password'
doc/api-extensions.md
1583:without using the trust password.
1871:This adds token-based certificate addition to the trust store as a safer alternative to a trust password.

doc/authentication.md
54:   - If the client certificate is not in the server's trust store, the server prompts the user for a token or the trust password.
55:     If the provided token or trust password matches, the client certificate is added to the server's trust store and the connection is granted.
76:You can also add new clients by using tokens. This is a safer way than using the trust password, because tokens expire after a configurable time ({config:option}`server-core:core.remote_token_expiry`) or once they've been used.
79:The clients can then add their certificates to the server's trust store by providing the generated token when prompted for the trust password.

doc/howto/cluster_form.md
155:Cluster trust password:

doc/howto/projects_confine.md
36:To confine access for an existing certificate (either because the access restrictions change or because the certificate was added with a trust password), use the following command:

po/ja.po
9786:#~ "    Will set the server's trust password to blah."

doc/howto/initialize.md
52:  You can choose to add client certificates to the server (manually or through tokens, the recommended way) or set a trust password.

po/fr.po
10300:#~ "    Will set the server's trust password to blah."

po/de.po
9896:#~ "    Will set the server's trust password to blah."

Remove all references to LXD and `lxc` command line

This is going to be a big one, but the entire codebase needs to be looked at and all mentions of LXD, lxd or lxc need to go away and be replaced with their new Incus equivalent. This can't be done fully programmatically as there are cases where lxc/LXC for example refers to liblxc rather than the confusingly named LXD client, those need to be preserved.

At the end of this process, a test should be added to ensure that this doesn't regress when importing LXD changes into this repository.

using the fan-overlay network with default subnet causes system instability

Required information

  • Distribution: Ubuntu
  • Distribution version:22.04.3
  • The output of "inc info" or if that fails:
    • Kernel version:
    • LXC version: Linux hv1 6.2.0-26-generic #26~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Jul 13 16:27:29 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
    • Incus version: lxd 5.15 installed via snap 5.15-002fa0f
    • Storage backend in use: zfs

Issue description

When using lxd with the an automatically created fan-overlay network, a kernel bug triggers which causes system to almost completely lock up.

A brief description of the problem. Should include what you were
attempting to do, what you did, what happened and what you expected to
see happen.

Steps to reproduce

  1. Step one
  2. Step two
  3. Step three

Information to attach

output from journalctl:

Aug 17 13:24:48 hv1 kernel: ------------[ cut here ]------------
Aug 17 13:24:48 hv1 kernel: Voluntary context switch within RCU read-side critical section!
Aug 17 13:24:48 hv1 kernel: WARNING: CPU: 5 PID: 9611 at kernel/rcu/tree_plugin.h:318 rcu_note_context_switch+0x2a7/0x2f0
Aug 17 13:24:48 hv1 kernel: Modules linked in: veth nft_masq nft_chain_nat vxlan ip6_udp_tunnel udp_tunnel dummy bridge stp llc ebtable_filter ebtables ip6table_raw ip6table_ma>
Aug 17 13:24:48 hv1 kernel: mei_me soundcore mei intel_pch_thermal mac_hid acpi_pad sch_fq_codel msr parport_pc ppdev lp parport ramoops reed_solomon pstore_blk pstore_zone ef>
Aug 17 13:24:48 hv1 kernel: CPU: 5 PID: 9611 Comm: rsyslogd Tainted: P IO 6.2.0-26-generic #26~22.04.1-Ubuntu
Aug 17 13:24:48 hv1 kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z170 Extreme6, BIOS P7.50 10/18/2018
Aug 17 13:24:48 hv1 kernel: RIP: 0010:rcu_note_context_switch+0x2a7/0x2f0
Aug 17 13:24:48 hv1 kernel: Code: 08 f0 83 44 24 fc 00 48 89 de 4c 89 f7 e8 61 c4 ff ff e9 1e fe ff ff 48 c7 c7 98 4e 53 9d c6 05 ee b7 3f 02 01 e8 09 1b f3 ff <0f> 0b e9 bd fd>
Aug 17 13:24:48 hv1 kernel: RSP: 0018:ffffae450d4df910 EFLAGS: 00010046
Aug 17 13:24:48 hv1 kernel: RAX: 0000000000000000 RBX: ffff9c9336172e40 RCX: 0000000000000000
Aug 17 13:24:48 hv1 kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
Aug 17 13:24:48 hv1 kernel: RBP: ffffae450d4df930 R08: 0000000000000000 R09: 0000000000000000
Aug 17 13:24:48 hv1 kernel: R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
Aug 17 13:24:48 hv1 kernel: R13: ffff9c844e928000 R14: 0000000000000000 R15: 0000000000000000
Aug 17 13:24:48 hv1 kernel: FS: 00007f418098dc40(0000) GS:ffff9c9336140000(0000) knlGS:0000000000000000
Aug 17 13:24:48 hv1 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug 17 13:24:48 hv1 kernel: CR2: 00007f4180d42000 CR3: 000000017152c002 CR4: 00000000003706e0
Aug 17 13:24:48 hv1 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Aug 17 13:24:48 hv1 kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Aug 17 13:24:48 hv1 kernel: Call Trace:
Aug 17 13:24:48 hv1 kernel:
Aug 17 13:24:48 hv1 kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Aug 17 13:24:48 hv1 kernel: Call Trace:
Aug 17 13:24:48 hv1 kernel:
Aug 17 13:24:48 hv1 kernel: __schedule+0xbc/0x5f0
Aug 17 13:24:48 hv1 kernel: schedule+0x68/0x110
Aug 17 13:24:48 hv1 kernel: schedule_hrtimeout_range_clock+0x97/0x130
Aug 17 13:24:48 hv1 kernel: ? __pfx_hrtimer_wakeup+0x10/0x10
Aug 17 13:24:48 hv1 kernel: schedule_hrtimeout_range+0x13/0x30
Aug 17 13:24:48 hv1 kernel: do_poll.constprop.0+0x22a/0x3b0
Aug 17 13:24:48 hv1 kernel: do_sys_poll+0x166/0x260
Aug 17 13:24:48 hv1 kernel: ? ___sys_sendmsg+0x95/0xe0
Aug 17 13:24:48 hv1 kernel: ? __mod_lruvec_state+0x37/0x50
Aug 17 13:24:48 hv1 kernel: ? __mod_lruvec_page_state+0xa0/0x160
Aug 17 13:24:48 hv1 kernel: ? folio_memcg_unlock+0x38/0x80
Aug 17 13:24:48 hv1 kernel: ? unlock_page_memcg+0x18/0x60
Aug 17 13:24:48 hv1 kernel: ? page_add_file_rmap+0x89/0x2b0
Aug 17 13:24:48 hv1 kernel: ? __pfx_pollwake+0x10/0x10
Aug 17 13:24:48 hv1 kernel: ? __sys_sendmmsg+0x100/0x210
Aug 17 13:24:48 hv1 kernel: ? __secure_computing+0x9b/0x110
Aug 17 13:24:48 hv1 kernel: ? __seccomp_filter+0x3df/0x5e0
Aug 17 13:24:48 hv1 kernel: ? __pfx_pollwake+0x10/0x10
Aug 17 13:24:48 hv1 kernel: ? __sys_sendmmsg+0x100/0x210
Aug 17 13:24:48 hv1 kernel: ? __secure_computing+0x9b/0x110
Aug 17 13:24:48 hv1 kernel: ? __seccomp_filter+0x3df/0x5e0
Aug 17 13:24:48 hv1 kernel: ? syscall_exit_to_user_mode+0x2a/0x50
Aug 17 13:24:48 hv1 kernel: ? ktime_get_ts64+0x52/0x110
Aug 17 13:24:48 hv1 kernel: __x64_sys_poll+0xb5/0x150
Aug 17 13:24:48 hv1 kernel: do_syscall_64+0x59/0x90
Aug 17 13:24:48 hv1 kernel: ? exc_page_fault+0x92/0x1b0
Aug 17 13:24:48 hv1 kernel: entry_SYSCALL_64_after_hwframe+0x72/0xdc
Aug 17 13:24:48 hv1 kernel: RIP: 0033:0x7f4180d32d47
Aug 17 13:24:48 hv1 kernel: Code: 00 00 00 5b 49 8b 45 10 5d 41 5c 41 5d 41 5e c3 0f 1f 00 f3 0f 1e fa 64 8b 04 25 18 00 00 00 85 c0 75 10 b8 07 00 00 00 0f 05 <48> 3d 00 f0 ff>
Aug 17 13:24:48 hv1 kernel: RSP: 002b:00007ffdc5692788 EFLAGS: 00000246 ORIG_RAX: 0000000000000007
Aug 17 13:24:48 hv1 kernel: RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f4180d32d47
Aug 17 13:24:48 hv1 kernel: RDX: 0000000000001388 RSI: 0000000000000001 RDI: 00007ffdc56928b8
Aug 17 13:24:48 hv1 kernel: RBP: 0000000034c6ac4a R08: 0000000000000005 R09: 0000000000000000
Aug 17 13:24:48 hv1 kernel: R10: 0000000000000001 R11: 0000000000000246 R12: 0000000000000002
Aug 17 13:24:48 hv1 kernel: R13: 00007ffdc5692880 R14: 00007ffdc56928b8 R15: 00007f4180e3c340
Aug 17 13:24:48 hv1 kernel:
Aug 17 13:24:48 hv1 kernel: ---[ end trace 0000000000000000 ]---

output of inc monitor while reproducing the issue)

Here is how I ran lxd init:

root@hv1:/snap/lxd/25112# lxd init
Would you like to use LXD clustering? (yes/no) [default=no]: yes
What IP address or DNS name should be used to reach this server? [default=192.168.3.1]:
Are you joining an existing cluster? (yes/no) [default=no]:
What member name should be used to identify this server in the cluster? [default=hv1]:
Do you want to configure a new local storage pool? (yes/no) [default=yes]:
Name of the storage backend to use (btrfs, dir, lvm, zfs) [default=zfs]:
Would you like to create a new zfs dataset under rpool/lxd? (yes/no) [default=yes]:
Do you want to configure a new remote storage pool? (yes/no) [default=no]:
Would you like to connect to a MAAS server? (yes/no) [default=no]:
Would you like to configure LXD to use an existing bridge or host interface? (yes/no) [default=no]:
Would you like to create a new Fan overlay network? (yes/no) [default=yes]:
What subnet should be used as the Fan underlay? [default=auto]:
Would you like stale cached images to be updated automatically? (yes/no) [default=yes]:
Would you like a YAML "lxd init" preseed to be printed? (yes/no) [default=no]: yes
config:
core.https_address: 192.168.3.1:8443
networks:

  • config:
    bridge.mode: fan
    fan.underlay_subnet: auto
    description: ""
    name: lxdfan0
    type: ""
    project: default
    storage_pools:
  • config:
    source: rpool/lxd
    description: ""
    name: local
    driver: zfs
    profiles:
  • config: {}
    description: ""
    devices:
    eth0:
    name: eth0
    network: lxdfan0
    type: nic
    root:
    path: /
    pool: local
    type: disk
    name: default
    projects: []
    cluster:
    server_name: hv1
    enabled: true
    member_config: []
    cluster_address: ""
    cluster_certificate: ""
    server_address: ""
    cluster_password: ""
    cluster_certificate_path: ""
    cluster_token: ""

It's hard to be precise about exactly when the bug triggers. Just running lxd init above will not trigger the bug if there are no containers running. If I bring up a container with no network connection, the bug also does not trigger. However if I have at least one container running with an active network connection, the bug seems to trigger reliably. Within a few minutes the system will usually be so unstable it is barely usable, though at other times the system remains somewhat usable for at least 20 minutes (if not more).

What does seem to reliably happen is that the system will not cleanly reboot. I can only get it to reboot using the "sysrec magic key" , presumably this is related to this kernel message:

Aug 17 16:36:21 hv1 kernel: rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { P5846 } 243611 jiffies s: 801 root: 0x0/T
Aug 17 16:36:21 hv1 kernel: rcu: blocking rcu_node structures (internal RCU debug):

presumably the stalled tasks cannot be terminated, thus the system will not reboot.

Move commands to standard `cmd` directory

Another thing we've meant to do in LXD but never really found the time for is to re-organize the repo a bit, starting with moving the individual commands under a cmd directory.

In time, this should allow for a cleaner repository root with the structure looking like:

  • client/ => The Go client package for the API
  • cmd/ => Commands
  • internal/ => Internal packages (can't be used externally)
  • shared/ => Packages that are meant for outside users to consume

Incus fails to launch (Arch AUR)

Required information

  • Distribution: Garuda
  • Distribution version: rolling
  • The output of "inc info" or if that fails:
    • Kernel version: 6.5.7-zen2-1-zen
    • LXC version: 5.0.3
    • Incus version: 0.1
    • Storage backend in use: huh?

Issue description

Hi there. :)

I just installed incus fresh and attempted to launch an image.
This fails in a very generic error message.

Steps to reproduce

  1. I freshly installed incus from AUR
  2. incus launch images:archlinux/current/amd64 kate
  3. Error: Get "http://unix.socket/1.0": dial unix /var/lib/incus/unix.socket: connect: no such file or directory

Information to attach

dmesg

inc is not available on my system.
var/log/incus is empty

incus launch --debug images:archlinux/current/amd64 kate

DEBUG  [2023-10-21T21:29:50+02:00] Connecting to a local Incus over a Unix socket
DEBUG  [2023-10-21T21:29:50+02:00] Sending request to Incus                      etag= method=GET url="http://unix.socket/1.0"
Error: Get "http://unix.socket/1.0": dial unix /var/lib/incus/unix.socket: connect: no such file or directory

The Arch AUR package seems to do a few things weird, it does not setup incus-admin, as an example, and I don't have inc available.

Remove snap handling logic

As Incus is extremely unlikely to ever be packaged as a snap, with it being a fork of LXD and LXD itself needing a lot of special logic and exceptions to function as a snap. We can simplify the code a fair bit by removing the snap-handling codepaths.

Please make package github.com/lxc/incus/internal/cliconfig public

Required information

  • Source version:
    commit 4ee8580 (HEAD -> main, origin/main, origin/HEAD)
    Date: Wed Sep 20 07:54:35 2023 +0100

Issue description

The former package github.com/canonical/lxd/lxc/config has been renamed as github.com/lxc/incus/internal/cliconfig
Because it is under "internal", it cannot be used by Go code outside of github.com/lxc/incus.
I've been using this package in order to parse the default remote from ~/.config/incus/config.yml (formerly ~/snap/lxd/common/config/config.yml), so that my program uses the same remote as the incus client (formerly lxc command).

As it is, I cannot port my program to the incus codebase, unless I copy the cliconfig package to my own codebase, which I find counterproductive.

Steps to reproduce

Use the following include in a .go file and try to compile:

import (
	config "github.com/lxc/incus/internal/cliconfig"
)

You will get the error:

use of internal package github.com/lxc/incus/internal/cliconfig not allowed

Since the file config.yml is saved in the user's directory, its format should be considered part of the public incus interface, and any code that is associated with this file format should also be considered part of the public interface.

I suggest moving this package to github.com/lxc/incus/cliconfig, so that other programs can use it.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.