Hosting QEMU VMs with Public IP Addresses using TAP Interfaces

Recently, I wanted to play around a bit with hosting services in VMs. Let’s imagine we want to host a service for a customer, but the service is so simple that a physical server would be overkill. Still, the server should look like a physical server to the user. For a VM to be transparent to the user, it needs to have a public IP address and should be exposed to the public internet like a real server would. If our VM would use the IP address of the host machine, we could only serve each port once and thus multiple customers would not be able to host services on the same port at the same time (except it is handled on the host in another way, e.g. by DNS name). This is of course unsatisfactory.

Luckily, QEMU guests can use TAP networking. With TAP networking the networking traffic is routed by a software bridge. The linux tools call it bridge, but according to my research basically a bridge is the same thing as a switch, it routes traffic on Layer-2 based on MAC addresses. Some documents try to distinguish switches and bridges, but for our use case it does not matter.

So what are we trying to achieve? In a standard setup of your PC, eth0 is connected to the physical link and maintains the IP address of your computer (eth0 in modern Linux often has another name, e.g. enp0s25 on my current machine, but in the following I will always refer to it as eth0). To allow VMs to have their own IP addresses while using the same physical link, we need to introduce a bridge into this setup. Both the VM’s network interface as well as the host’s interface will be connected to the bridge (commonly named br0).

        |
        | physical link
        |
        o br0
       / \
      /   \
eth0 o     o tap0
           |
- - - - - - - - - - - - - - - Host / VM boundary
           |
           o ens0 (inside VM)

How can we achieve this? It’s not very difficult actually, we only need the ip command (cheat sheet).

At first we want to create the bridge:

ip link add br0 type bridge
ip link set br0 up

Next, we want to connect the eth0 interface to the bridge and re-assign the IP address from eth0 to br0. Please note that these commands will drop internet connectivity of your machine, so either only run them on a local PC or ensure that in case of disaster you can reboot the machine somehow.

192.168.0.10 is just an example IP address, you need to use your own of course.

# According to Arch wiki eth0 needs to be up
ip link set eth0 up
ip link set eth0 master br0

# Drop existing IP from eth0
ip addr flush dev eth0

# Assign IP to br0
ip addr add 192.168.0.10/24 brd + dev br0
ip route add default via 192.168.0.1 dev br0

Now, connectivity from the host should already work again.

Next, we can create the TAP interface to be used by the VMs. $YOUR_USER is used to allow an unprivileged user to connect to the TAP device. This is important for QEMU, since QEMU VMs should be started as non-root users.

ip tuntap add dev tap0 mode tap user $YOUR_USER
ip link set dev tap0 up
ip link set tap0 master br0

And now we can start a VM. The MAC address could e.g. be generated randomly:

RAND_MAC=$(printf 'DE:AD:BE:EF:%02X:%02X\n' $((RANDOM%256)) $((RANDOM%256)))

qemu-system-x86_64 -m 1024 -cdrom archlinux-2020.05.01-x86_64.iso \
   -device virtio-net-pci,netdev=network0,mac=$RAND_MAC \
   -netdev tap,id=network0,ifname=tap0,script=no,downscript=no

Since the Arch Linux image I booted uses a DHCP client it will request an IP address from my home router automatically. It’s also listed as a newly detected machine in my router’s user interface. Yay!