Interlude: Setup an automated RabbitMQ Provider

Let’s step back a bit regarding our cloud computing undertaking and look at the results we have achieved so far. We can now create new VMs based on base-images with a single message to RabbitMQ. We can stop, restart or delete these VMs with other messages and we are able to find out the IP address of the guest VM.

This is enough to let us create a small service hosting provider. Let us assume we want to offer some hosted service to other people. They should be able to place a payment via an API payment provider and automatically receive a VM with a given service. The process should look like this:

   o      --- $$$ --->  [Payment Received]  --- {create-vm} --->        a s
 -- --                                      --- {query-ip} --->         e c
   |                                                                    t a
  / \                                                                   h l
                                                                        e e
Customer <--- https://192.168.0.123/ --- [VM Created, Return Response]  r

Our customer pays with dollars for the service. Once the payment has been received, the service can start to create a VM by issuing a RabbitMQ message. It will also start to query the guest machine for an IP. Once the guest is fully up and we received the IP address, we can return a URL to the customer.

Of course, for small scales this could also be easily achieved without a message broker. We also didn’t implement so much, yet. The main tasks up to now were finding out which technologies we can use (e.g. QEMU, systemd, QMP, QEMU Guest Agent) and writing some Python code to connect them. Similar things could have probably been achieved with just bash scripts, but at some point we wouldn’t be flexible enough with bash, anymore.

The power of the message broker approach comes with the scalability, because we can just add more machines and they will start receiving requests to setup VMs. Of course, good load distribution is still missing. At the moment we just send requests to any machine and each machine would accept the request and setup a VM, even if it is already overloaded. But these are things we could add.

Create the RabbitMQ base image

As a simple use case we will serve VMs with a RabbitMQ broker installed. Since this is only an example, we will not implement the payment integration, but I assume this would not be a difficult component to add.

Before we start writing the code, we will create a VM with a RabbitMQ server. I will use Arch Linux this time, because I do not need all the functionality that comes with Ubuntu. Let’s create an empty image, boot an Arch Linux installation ISO and then perform the installation steps.

qemu-img create -f qcow2 rabbitmq.qcow2 10G
qemu-system-x86_64 -cpu host -accel kvm -m 2048 \
    -cdrom archlinux-2020.11.01-x86_64.iso \
    -hda rabbitmq.qcow2

The following steps should be sufficient for a quick installation. Make sure to run this inside the VM, not on your host OS, as it will format partitions.

timedatectl set-ntp true
echo 'start=2048, type=83' | sfdisk /dev/sda
mkfs.ext4 /dev/sda1
mount /dev/sda1 /mnt
pacstrap /mnt base linux linux-firmware grub dhcpcd
genfstab -U /mnt >> /mnt/etc/fstab
arch-chroot /mnt
hwclock --systohc
echo "en_US.UTF-8 UTF-8" >> /etc/locale.gen
locale-gen
echo "LANG=en_US.UTF-8" > /etc/locale.conf
# This is a line that we might want to change on each host later
echo "rabbitmq" > /etc/hostname
# Set some root password, i used rabbitmq
mkinitcpio -P
passwd
grub-install --target=i386-pc /dev/sda
grub-mkconfig -o /boot/grub/grub.cfg
exit
reboot

After the reboot we can install and enable RabbitMQ and the QEMU Guest Agent. We also enable the RabbitMQ HTTP management site:

systemctl start dhcpcd && systemctl enable dhcpcd
pacman -Sy rabbitmq qemu-guest-agent
systemctl enable rabbitmq
systemctl enable qemu-guest-agent
systemctl start rabbitmq
rabbitmq-plugins enable rabbitmq_management

I experienced one problem with this: While the QEMU Guest Agent is correctly enabled and after a restart of the VM (with Guest Agent command line flags added) the file /dev/virtio-ports/org.qemu.guest_agent.0 does exist, systemd keeps telling me that the start of QEMU Guest Agent failed, because the file is missing. I assume that this is a timing problem, the file is probably created after QEMU Guest Agent tries to start.

I was able to solve this issue with a systemd path unit. A path unit can monitor paths and activate a unit on specific changes, e.g. once a file exists.

Create a file /etc/systemd/system/qemu-guest-agent.path with the following content in the VM:

[Unit]
Description=Start QEMU guest agent once required file exists

[Path]
PathExists=/dev/virtio-ports/org.qemu.guest_agent.0

[Install]
WantedBy=multi-user.target

If no Unit option is given (as in my file) it will start a service with the same name as the path unit.

Now, enable this unit and disable the service unit (as it does not work correctly on its own):

systemctl disable qemu-guest-agent.service
systemctl enable qemu-guest-agent.path

Now, the QEMU Guest Agent inside Arch Linux should start correctly.

Start new RabbitMQ VMs from Python

With that, we have a ready-to-go base system that starts RabbitMQ automatically. Now we can develop the Python code. I will put the code into examples/rabbitmq_vm_hosting.py in my project directory.

There’s not much we have to do. We can re-use the ServerCommunication class from aetherscale.client. We then send a create-vm command an a list-vms command and fetch the IP address for the created VM. We wait for 30 seconds to give the VM some time to start and to perform DHCP and then we query the IP addresses.

Then we just filter out the localhost IP addresses and the link-local IPv6 address and are ready to go.

#!/usr/bin/env python

import sys
import time
from typing import List

from aetherscale.client import ServerCommunication


def create_rabbitmq_vm(comm: ServerCommunication) -> str:
    responses = comm.send_msg({
        'command': 'create-vm',
        'options': {
            # The rabbitmq image has to be created before running this script
            'image': 'rabbitmq',
        }
    }, response_expected=True)

    if len(responses) != 1:
        raise RuntimeError(
            'Did not receive exactly one response, something went wrong')

    if responses[0]['execution-info']['status'] != 'success':
        raise RuntimeError('Execution was not successful')

    return responses[0]['response']['vm-id']


def get_vm_ips(vm_id: str, comm: ServerCommunication) -> List[str]:
    responses = comm.send_msg({
        'command': 'list-vms',
    }, response_expected=True)

    for r in responses:
        try:
            for vm in r['response']:
                if vm['vm-id'] == vm_id:
                    return vm['ip-addresses']
        except KeyError:
            pass

    return []


def is_external_ip(ip: str) -> bool:
    """Quick hack to check whether an IP returned by list-vms is an
    external IP address"""
    if ':' in ip:
        first_part = ip.split(':')[0]

        try:
            if int('fe80', 16) <= int(first_part, 16) <= int('febf', 16):
                # link-local
                return False
        except ValueError:
            # if it cannot be parsed as hex, it's not in range
            return False

    if ip == '::1' or ip == '127.0.0.1':
        # localhost
        return False

    return True


def format_ip_for_url(ip: str) -> str:
    if ':' in ip:
        return f'[{ip}]'
    else:
        return ip


def main():
    with ServerCommunication() as comm:
        try:
            vm_id = create_rabbitmq_vm(comm)
        except RuntimeError as e:
            print(str(e), file=sys.stderr)
            sys.exit(1)

    time.sleep(30)
    # TODO: There seems to be a bug in ServerCommunication so that we can only
    # exchange one message pair per context
    # Probably related to the AMQ reply-to channel
    with ServerCommunication() as comm:
        ips = get_vm_ips(vm_id, comm)

    ips = [f'http://{format_ip_for_url(ip)}:15672/' for ip in ips
           if is_external_ip(ip)]

    print(ips)


if __name__ == '__main__':
    main()

Let’s try it! Make sure that the aetherscale server is running.

$ python examples/rabbitmq_vm_hosting.py
['http://192.168.2.222:15672/', 'http://[2001:0db8::d14]:15672/', 'http://[2001:0db8::b6e7:8be3:a2da:367c]:15672/']

There’s one remaining problem. The RabbitMQ HTTP management interface only seems to listen to one IP address. To be sure to hand out the right IP addresses to customers, we would have to define an IP address in its configuration file. This is possible either by mounting the created image on the host, changing the configuration file and then booting the VM or by using the QEMU Guest Agent with filesystem operations. Both, however, are operations we did not perform in the tutorial series, yet, so we will skip this problem.

The RabbitMQ broker itself should listen to all interfaces, though.

We’re able to boot a RabbitMQ VM and hand out its IP address in only 30 seconds. I assume it could even go faster, I just wanted to have some safety margin. With just a bit more code, we could start handing out RabbitMQ VMs to customers after they have paid - ok almost, we’d have to handle support cases and so on and do better error handling.