Terminus is a planet in Isaac Asimov's Foundation series. It is the capital planet of the First Foundation. It is located at the edge of the Galaxy. --Asimov Fandom Page

This repo documents the steps that I took to implement a test bed for exploring Metal-as-a-Service (MAAS) in HA mode. The overall design is illustrated in this diagram.

The objective here is to have a workable environment using just spare hardware in my home office. Thus, this environment is built more for exploration, rather than performance and reliability. In other words, don't depend on this design for production, folks!

Install Ubuntu Server LTS on all baremetals and configure them with static IPs as indicated in the logical network diagram above. We use static IPs here because MAAS, which will be the DHCP (and PXE) server, is not yet set up in the network.

For each baremetal KVM host, create a backup of its /etc/netplan/xxxxxxxxx.yaml file and then edit it like so:

network:
  version: 2
  ethernets:
    <interface-name-here>:
      dhcp4: false
      dhcp6: false

  bridges:
    br0:
      interfaces:
      - <interface-name-here>
      addresses:
      - 192.168.100.XX/24
      gateway4: 192.168.100.1
      nameservers:
        addresses:
        - 1.1.1.1
        - 8.8.8.8
        - 8.8.4.4
      dhcp4: false
      dhcp6: false
      parameters:
        forward-delay: 4
        stp:           true

Then apply the changes:

sudo netplan --debug generate
sudo netplan --debug apply

NOTE: We can't use netplan try here since we are defining a bridge and reverting that config isn't supported. In that case, make sure you got everything right or you'll have to walk over to the machine and fix it via its console! Optionally, if you have a second ethernet or maybe even a WiFi interface, keep that on. That's outside the scope of this guide though.

What we just did is create a Linux bridge (think of it as a software-defined L2 switch on steroids) that's connected to the underlying L2 network. This will allow us to instantiate VMs and connect them to this bridge thereby making them visible to all machines in the underlying L2 network.

Note that we have not configured VLANs at this point. We will do that once all baremetal and virtual machines have been set up and verified to ping each other on the 192.168.100.x subnet.

To manage KVM, we will use libvirt as our main management interface. Let's install it now:

sudo apt update
sudo apt install qemu-kvm libvirt-daemon-system

Log out and log back in to ensure that your membership to the libvirt group is in effect.

The above installation will automatically create virbr0 for us but we want to use br0 instead. Let's remove virbr0 and make br0 the default.

virsh net-destroy default
virsh net-edit default

Update the contents to:

<network>
  <name>br0</name>
  <forward mode='bridge'/>
  <bridge name='br0'/>
</network>

virsh net-start br0
sudo systemctl restart systemd-networkd
virsh net-undefine default
virsh net-list --all
brctl show
ip a

Copy the Ubuntu 20.04 (or 18.04) iso to the baremetal machines.

We will need another CLI tool to easily create a VM on KVM. Install:

sudo apt install virtinst

Then:

for i in $(seq 1 3); do
    virt-install \
        --name=infra-$i \
        --vcpus=2 \
        --memory=6144 \
        --disk size=64 \
        --cdrom=ubuntu-18.04.5-live-server-amd64.iso \
        --os-variant=ubuntu18.04 \
        --graphics vnc,listen=0.0.0.0 --noautoconsole
done

Note that --memory uses MiB as units and --disk uses GiB

To get the VNC port number that the above VMs are connected to, run:

virsh vncdisplay infra-1

You may need to run sshuttle in your localhost if the baremetals are accessible via a jumpbox:

sshuttle -v -H -r <jumpbox-hostname-or-ip> 192.168.100.0/24

Now run your VNC viewer and point it to <baremetal-ip>:<vnc-port> to continue with the installation process.

Make sure to assign the correct static IPs to the infra VMs as shown in Level 1 of the logical network diagram.

Once Ubuntu has finished installation, the VM will shutdown. To start it and make it autostart when the baremetal machine boots up, run:

virsh autostart infra-1
virsh start infra-1

Repeat the above steps for the config (configurator) VM.

Ensure you have something like this in your local ~/.ssh/config

Host terminus-jb
    ForwardAgent yes
    HostName 192.168.86.21
    User ubuntu

Host kvm-1
    ForwardAgent yes
    HostName 192.168.100.11
    User ubuntu
    ProxyCommand ssh -W %h:%p terminus-jb

Host kvm-2
    ForwardAgent yes
    HostName 192.168.100.12
    User ubuntu
    ProxyCommand ssh -W %h:%p terminus-jb

Host config
    ForwardAgent yes
    HostName 192.168.100.20
    User ubuntu
    ProxyCommand ssh -W %h:%p terminus-jb

Host infra-1
    ForwardAgent yes
    HostName 192.168.100.21
    User ubuntu
    ProxyCommand ssh -W %h:%p terminus-jb

Host infra-2
    ForwardAgent yes
    HostName 192.168.100.22
    User ubuntu
    ProxyCommand ssh -W %h:%p terminus-jb

Host infra-3
    ForwardAgent yes
    HostName 192.168.100.23
    User ubuntu
    ProxyCommand ssh -W %h:%p terminus-jb

This will save you a lot of typing down the line. Run the following commands in all the machines (baremetals and VMs)

First, test that it does the write thing:

sudo sed -E 's/^(%sudo.*) ALL$/\1 NOPASSWD:ALL/g' /etc/sudoers

Then commit to it!

sudo sed -i -E 's/^(%sudo.*) ALL$/\1 NOPASSWD:ALL/g' /etc/sudoers

At this point, your baremetal and virtual machines should be able to ping each other on the 192.168.100.x subnet. If that's not the case, double check your netplan config for all baremetal and virtual machines before proceeding.

As per this blog, if we want to have our infra nodes be VLAN-aware, we must first make sure our baremetal hosts are aware of the VLAN. It's also important to define and attach our VLANs to br0 rather than the baremetal host's physical interface, otherwise the packets will be untagged before they even reach br0. Thus, make sure to append the following to your baremetal host's /etc/netplan/xxxxxxx.yaml file:

  vlans:
    vlan10:
      id:        10
      addresses: [192.168.110.XX/24]
      link:      br0
    vlan11:
      id:        11
      addresses: [192.168.111.XX/24]
      link:      br0
    vlan12:
      id:        12
      addresses: [192.168.112.XX/24]
      link:      br0

Change the last octets above to match the baremetal's last octet in the 192.168.100.x subnet. Save the netplan file and then run:

sudo netplan --debug generate
sudo netplan --debug apply

Now the baremetal machines should be able to ping each other on all four subnets that are indicated in our netplan config.

At this point, our baremetal machines are VLAN aware, but our infra nodes have no idea of these yet. Let's add the following to their own netplan config:

  vlans:
    vlan10:
      id:        10
      addresses: [192.168.110.XX/24]
      link:      <machine's-interface-name>
    vlan11:
      id:        11
      addresses: [192.168.111.XX/24]
      link:      <machine's-interface-name>
    vlan12:
      id:        12
      addresses: [192.168.112.XX/24]
      link:      <machine's-interface-name>

Change the last octets above to match the virtual machine's last octet in the 192.168.100.x subnet.

Note how we link the VLAN to the interface that's connected to br0. Save the updated configuration and then run:

sudo netplan --debug generate
sudo netplan --debug apply

Now all 6 machines (baremetal and virtual) should be able to ping each other on all 4 subnets.

If you plan on re-installing MAAS multiple times (as one does in a lab environment), consider setting up a local image mirror in kvm-1 so that your image syncs go faster. The following steps is adapted from this guide.

ssh kvm-1
sudo apt install simplestreams
KEYRING_FILE=/usr/share/keyrings/ubuntu-cloudimage-keyring.gpg
IMAGE_SRC=https://images.maas.io/ephemeral-v3/stable
IMAGE_DIR=/var/www/html/maas/images/ephemeral-v3/stable

KEYRING_FILE points to the GPG keyring file that sstream-mirror will use to verify the integrity of the downloaded images.

sudo sstream-mirror --keyring=$KEYRING_FILE $IMAGE_SRC $IMAGE_DIR \
    'arch=amd64' 'release~(bionic|focal)' --max=1 --progress
sudo sstream-mirror --keyring=$KEYRING_FILE $IMAGE_SRC $IMAGE_DIR \
    'os~(grub*|pxelinux)' --max=1 --progress

The last two commands above will take some time depending on your bandwidth.

Once the initial sync has completed, run a simple HTTP server using Python:

cd /var/www/html
python3 -m http.server 3003

Then from each of the infra nodes, run:

curl http://192.168.100.11:3003/maas/images/ephemeral-v3/stable/

You should see something like:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<title>Directory listing for /maas/images/ephemeral-v3/stable/</title>
</head>
<body>
<h1>Directory listing for /maas/images/ephemeral-v3/stable/</h1>
<hr>
<ul>
<li><a href=".data/">.data/</a></li>
<li><a href="bionic/">bionic/</a></li>
<li><a href="focal/">focal/</a></li>
<li><a href="streams/">streams/</a></li>
</ul>
<hr>
</body>
</html>

Now let's make the HTTP server permanent via a systemd service. Write the following in /etc/systemd/system/maas-local-image-mirror.service:

[Unit]
Description=Dumb HTTP server for MAAS local image mirror
After=network.target
StartLimitIntervalSec=0

[Service]
Type=simple
Restart=always
RestartSec=3
User=ubuntu
WorkingDirectory=/var/www/html
ExecStart=/usr/bin/env python3 -m http.server 3003

[Install]
WantedBy=multi-user.target

Then enable and start it:

sudo systemctl daemon-reload
sudo systemctl enable maas-local-image-mirror
sudo systemctl start maas-local-image-mirror

You should now have a local image mirror that MAAS can connect to via http://192.168.100.11:3003/maas/images/ephemeral-v3/stable/

Optionally keep your local mirror in-sync by running the above sstream-mirror commands using a cronjob or a systemd timer.

Want to stand up a new MAAS cluster or you just messed up and need to start over? Or maybe you want to try out a different version of MAAS? No problem.

First, check which VMs are running in the baremetal:

virsh list --all

Next destroy and undefine each VM:

for i in $(seq 1 3); do
    virsh destroy infra-$i
    virsh undefine infra-$i
    virsh vol-delete infra-$i.qcow2 --pool=default
done

Now re-create the VMs as stated above

• libvirt
• netplan