VDE Networking for our Cloud Computing
If you’ve followed the first part of this series, you’ll remember that our server script currently requires root privileges to setup a TAP device for each QEMU instance. It would be much nicer if this was not needed and instead our script could run in user-mode. This is possible with VDE networking.
VDE networking allows us (among other things) to setup a switch in user-mode to which our QEMU VMs can connect for networking. On the other end VDE can connect to a TAP device to establish connectivity with our real network.
According to the documentation
VDE will require root privileges if you use TAP networking, but I was able to
run it in user-mode with TAP devices. I assume that this happens either if vde_switch
is required to create the TAP device itself or if the TAP device is not setup
with user permissions.
So let’s get this running! First of all, we have to install VDE. On Arch Linux this can be achieved with:
pacman -S vde2
This will install a set of executables from which we need vde_switch
.
On the one end of VDE we will have a TAP device to which VDE can connect. On
the other end VDE will create a socket to which our QEMU VMs can connect.
To create the TAP device we need root permissions, but thereafter we
can continue with standard user permissions (even VDE can be started with
standard user permissions). The device br0
is setup like in the previous
tutorial to be a software bridge for the real ethernet device.
ip tuntap add dev tap-vde mode tap user $YOUR_USER
ip link set dev tap-vde up
ip link set tap-vde master br0
# Continue with standard user from here
vde_switch -tap tap-vde -s /tmp/vde.ctl
qemu-system-x86_64 -accel kvm -cpu host -m 4096 \
-cdrom ubuntu-20.04.1-desktop-amd64.iso \
-net nic,model=virtio,macaddr=<SOME-MAC> \
-net vde,sock=/tmp/vde.ctl
This will bring up an Ubuntu Desktop CD. If you’re behind a DHCP-enabled router
and everything works correctly, Ubuntu should automatically fetch an IP
address from your router. Otherwise, you could listen to tap-vde
and
br0
with Wireshark to see which packages are sent.
When this works, let’s change the Python code for starting a machine. We
will change the setup of individual tap devices for each VM (not needed anymore)
and the netdev
option for each VM. I also added two new global variables
to allow to switch networking mode easily, because I don’t have any experience
with VDE yet and do not know how well it works.
NETWORKING_MODE = 'vde'
VDE_FOLDER = '/tmp/vde.ctl'
# ...
def callback(ch, method, properties, body):
# ...
if NETWORKING_MODE != 'vde':
tap_device = f'vm-{vm_id}'
if not interfaces.create_tap_device(
tap_device, 'br0', run_qemu_username):
print(f'Could not create tap device for VM "{vm_id}"',
file=sys.stderr)
return
mac_addr = interfaces.create_mac_address()
print(f'Assigning MAC address "{mac_addr}" to VM "{vm_id}"')
if NETWORKING_MODE == 'vde':
netdev = \
f'vde,id=pubnet,sock={VDE_FOLDER}'
else:
netdev = \
f'tap,id=pubnet,ifname={tap_device},script=no,downscript=no'
p = subprocess.Popen([
'qemu-system-x86_64', '-m', '4096', '-hda', str(user_image),
'-device', f'virtio-net-pci,netdev=pubnet,mac={mac_addr}',
'-netdev', netdev,
'-name', f'qemu-vm-{vm_id},process=vm-{vm_id}',
])
# ...
If you already started VDE yourself, this should work fine. However, I want VDE to be started by the script on its own. Since I do not fancy managing all kinds of services in my script, we will be using user-mode systemd.
For this, we will create a systemd unit file for VDE at
data/systemd/aetherscale-vde.service
:
[Unit]
Description=aetherscale VDE networking
[Service]
ExecStart=vde_switch -tap tap-vde -s /tmp/vde.ctl
[Install]
WantedBy=default.target
Feel free to add any kind of restart behaviour, dependencies etc. This file
will be copied to $HOME/.config/systemd/user/
by our server.
To manage systemd unit files, we will add some functions to execution.py
to
setup and start services.
def copy_systemd_unit(unit_file: Path, unit_name: str):
if '.' not in unit_name:
raise ValueError('Unit name must contain the suffix, e.g. .service')
systemd_unit_dir = Path().home() / '.config/systemd/user'
systemd_unit_dir.mkdir(parents=True, exist_ok=True)
target_unit_file = systemd_unit_dir / unit_name
shutil.copyfile(unit_file, target_unit_file)
# Reload system
subprocess.run(['systemctl', '--user', 'daemon-reload'])
def start_systemd_unit(unit_name: str) -> bool:
return run_command_chain([
['systemctl', '--user', 'start', unit_name],
])
def enable_systemd_unit(unit_name: str) -> bool:
return run_command_chain([
['systemctl', '--user', 'enable', unit_name],
])
def systemctl_is_running(unit_name: str) -> bool:
result = subprocess.run([
'systemctl', '--user', 'is-active', '--quiet', unit_name])
return result.returncode == 0
Since the server should be started by a standard user, we require that a TAP device was created beforehand. If the TAP device does not exist the server will exit with an error message.
VDE_TAP_INTERFACE = 'tap-vde'
# ...
def run():
channel.basic_consume(queue=QUEUE_NAME, on_message_callback=callback)
if NETWORKING_MODE == 'vde':
if not interfaces.check_device_existence(VDE_TAP_INTERFACE):
print(
f'Interface {VDE_TAP_INTERFACE} does not exist. '
'Please create it manually and then start this service again',
file=sys.stderr)
sys.exit(1)
logging.info('Bringing up VDE networking')
execution.copy_systemd_unit(
Path('data/systemd/aetherscale-vde.service'),
'aetherscale-vde.service')
execution.start_systemd_unit('aetherscale-vde.service')
# Give systemd a bit time to start VDE
time.sleep(0.5)
if not execution.systemctl_is_running('aetherscale-vde.service'):
logging.error('Failed to start VDE networking.')
sys.exit(1)
else:
# ...
channel.start_consuming()
An open issue at the moment seems to be that two Ubuntu Server instances with the same hostname, but different MAC addresses receive the same IP address (the DHCP lease for the second instance seems to overwrite the lease for the first instance). This did not happen for two Ubuntu Desktop Live-CDs.
I do not maintain a comments section. If you have any questions or comments regarding my posts, please do not hesitate to send me an e-mail to blog@stefan-koch.name.