VDE Networking for our Cloud Computing

If you’ve followed the first part of this series, you’ll remember that our server script currently requires root privileges to setup a TAP device for each QEMU instance. It would be much nicer if this was not needed and instead our script could run in user-mode. This is possible with VDE networking.

VDE networking allows us (among other things) to setup a switch in user-mode to which our QEMU VMs can connect for networking. On the other end VDE can connect to a TAP device to establish connectivity with our real network.

According to the documentation VDE will require root privileges if you use TAP networking, but I was able to run it in user-mode with TAP devices. I assume that this happens either if vde_switch is required to create the TAP device itself or if the TAP device is not setup with user permissions.

So let’s get this running! First of all, we have to install VDE. On Arch Linux this can be achieved with:

pacman -S vde2

This will install a set of executables from which we need vde_switch. On the one end of VDE we will have a TAP device to which VDE can connect. On the other end VDE will create a socket to which our QEMU VMs can connect. To create the TAP device we need root permissions, but thereafter we can continue with standard user permissions (even VDE can be started with standard user permissions). The device br0 is setup like in the previous tutorial to be a software bridge for the real ethernet device.

ip tuntap add dev tap-vde mode tap user $YOUR_USER
ip link set dev tap-vde up
ip link set tap-vde master br0

# Continue with standard user from here
vde_switch -tap tap-vde -s /tmp/vde.ctl
qemu-system-x86_64 -accel kvm -cpu host -m 4096 \
   -cdrom ubuntu-20.04.1-desktop-amd64.iso \
   -net nic,model=virtio,macaddr=<SOME-MAC> \
   -net vde,sock=/tmp/vde.ctl

This will bring up an Ubuntu Desktop CD. If you’re behind a DHCP-enabled router and everything works correctly, Ubuntu should automatically fetch an IP address from your router. Otherwise, you could listen to tap-vde and br0 with Wireshark to see which packages are sent.

When this works, let’s change the Python code for starting a machine. We will change the setup of individual tap devices for each VM (not needed anymore) and the netdev option for each VM. I also added two new global variables to allow to switch networking mode easily, because I don’t have any experience with VDE yet and do not know how well it works.

NETWORKING_MODE = 'vde'
VDE_FOLDER = '/tmp/vde.ctl'

# ...

def callback(ch, method, properties, body):
    # ...

    if NETWORKING_MODE != 'vde':
        tap_device = f'vm-{vm_id}'
        if not interfaces.create_tap_device(
                tap_device, 'br0', run_qemu_username):
            print(f'Could not create tap device for VM "{vm_id}"',
                file=sys.stderr)
            return

    mac_addr = interfaces.create_mac_address()
    print(f'Assigning MAC address "{mac_addr}" to VM "{vm_id}"')

    if NETWORKING_MODE == 'vde':
        netdev = \
            f'vde,id=pubnet,sock={VDE_FOLDER}'
    else:
        netdev = \
            f'tap,id=pubnet,ifname={tap_device},script=no,downscript=no'

    p = subprocess.Popen([
        'qemu-system-x86_64', '-m', '4096', '-hda', str(user_image),
        '-device', f'virtio-net-pci,netdev=pubnet,mac={mac_addr}',
        '-netdev', netdev,
        '-name', f'qemu-vm-{vm_id},process=vm-{vm_id}',
    ])

    # ...

If you already started VDE yourself, this should work fine. However, I want VDE to be started by the script on its own. Since I do not fancy managing all kinds of services in my script, we will be using user-mode systemd.

For this, we will create a systemd unit file for VDE at data/systemd/aetherscale-vde.service:

[Unit]
Description=aetherscale VDE networking

[Service]
ExecStart=vde_switch -tap tap-vde -s /tmp/vde.ctl

[Install]
WantedBy=default.target

Feel free to add any kind of restart behaviour, dependencies etc. This file will be copied to $HOME/.config/systemd/user/ by our server.

To manage systemd unit files, we will add some functions to execution.py to setup and start services.

def copy_systemd_unit(unit_file: Path, unit_name: str):
    if '.' not in unit_name:
        raise ValueError('Unit name must contain the suffix, e.g. .service')

    systemd_unit_dir = Path().home() / '.config/systemd/user'
    systemd_unit_dir.mkdir(parents=True, exist_ok=True)
    target_unit_file = systemd_unit_dir / unit_name

    shutil.copyfile(unit_file, target_unit_file)

    # Reload system
    subprocess.run(['systemctl', '--user', 'daemon-reload'])


def start_systemd_unit(unit_name: str) -> bool:
    return run_command_chain([
        ['systemctl', '--user', 'start', unit_name],
    ])


def enable_systemd_unit(unit_name: str) -> bool:
    return run_command_chain([
        ['systemctl', '--user', 'enable', unit_name],
    ])


def systemctl_is_running(unit_name: str) -> bool:
    result = subprocess.run([
        'systemctl', '--user', 'is-active', '--quiet', unit_name])
    return result.returncode == 0

Since the server should be started by a standard user, we require that a TAP device was created beforehand. If the TAP device does not exist the server will exit with an error message.

VDE_TAP_INTERFACE = 'tap-vde'

# ...

def run():
    channel.basic_consume(queue=QUEUE_NAME, on_message_callback=callback)

    if NETWORKING_MODE == 'vde':
        if not interfaces.check_device_existence(VDE_TAP_INTERFACE):
            print(
                f'Interface {VDE_TAP_INTERFACE} does not exist. '
                'Please create it manually and then start this service again',
                file=sys.stderr)
            sys.exit(1)

        logging.info('Bringing up VDE networking')
        execution.copy_systemd_unit(
            Path('data/systemd/aetherscale-vde.service'),
            'aetherscale-vde.service')
        execution.start_systemd_unit('aetherscale-vde.service')
        # Give systemd a bit time to start VDE
        time.sleep(0.5)
        if not execution.systemctl_is_running('aetherscale-vde.service'):
            logging.error('Failed to start VDE networking.')
            sys.exit(1)
    else:
        # ...

    channel.start_consuming()

An open issue at the moment seems to be that two Ubuntu Server instances with the same hostname, but different MAC addresses receive the same IP address (the DHCP lease for the second instance seems to overwrite the lease for the first instance). This did not happen for two Ubuntu Desktop Live-CDs.