Connecting servers from multiple providers to one cluster in a VPN (and cutting the AWS bill)

When I first started to use AWS services I was wondering why so many people use EC2 machines 24/7 and pay a ton of money for a server that - at least in Europe - you get for much less money (if you run your server the whole month). A bit later, I came to realize that it’s a lot easier to run a cluster inside AWS with their VPC service than it is at other providers.

I needed a 24/7 server nonetheless and was not willing to pay the high price at AWS for private projects. So I got the cheapest virtual server at Hetzner for around four euros per month - and ran out of memory.
So, there were basically two different solutions to the problem. Either scale up and buy a bigger server or scale out and buy a second server. I have not decided for one, yet, but I started thinking about how to scale out. To my knowledge, Hetzner does not offer private subnets to every customers. I read somewhere on the internet, that they might do for larger customers, but not for everybody. On the other hand, I also did not want to publish all my services to the global net and secure all of them with a reverse proxy with SSL and authentication.

I started to wonder if this problem might be solvable with a virtual private network (VPN). One of the machines can be the VPN server and others connect to it. Whenever traffic has to pass through the private network, it will go via the VPN interface, but all other traffic can still be routed through each of the machines standard eth0 interfaces.

This would also allow me to use cheap long term servers for my standard load and only when I need to scale out for a short period of time use AWS servers.

This tutorial uses OpenVPN to connect the servers, but as of 2019 it is easier and more lightweight to setup Wireguard.

Setting up an OpenVPN server on the long term host

For setting up an OpenVPN server, we can follow a DigitalOcean tutorial. First, you need to create our own Root Certificate Authority for the OpenVPN network. With this we can create certificates for both the log term server and the newly joining nodes (clients). So start by installing openvpn and easy-rsa with your package manager. easy-rsa will help in creating all the required certificates.

We should prepare the Certificate Authority:

mkdir openvpn && make-cadir openvpn/ca
cd openvpn/ca

The directory contains a file vars with default settings for the key creation. You can set all of them to something reasonable that will be used for all created certificates. The only difference between the keys will be the Common Name, which you define when you call the easy-rsa command line scripts.

export KEY_COUNTRY="US"
export KEY_PROVINCE="CA"
export KEY_CITY="SanFrancisco"
export KEY_ORG="Fort-Funston"
export KEY_EMAIL="me@myhost.mydomain"
export KEY_OU="MyOrganizationalUnit"


# X509 Subject Field
export KEY_NAME="EasyRSA"

On my host, I adjusted all of the fields except for OU which I left empty.

Now we are ready to create our own CA, a certificate for the server as well as strong Diffie-Hellman keys and an HMAC signature for more security. Just hit enter all the time and y when asked if you want to sign and commit the certificate.

source var
./clean-all
./build-ca
./build-key-server [your-server-hostname]
./build-dh
openvpn --genkey --secret keys/ta.key

You will call clean-all only once before you start setting up your certificates. Afterwards, you’ll need the data in this directory to create your client certificates. If you clean the directory and then want to create a server or a client certificate, easy-rsa will complain that CA information is missing:

Try pkitool --initca to build a root certificate/key.

Now we copy all generated files to the OpenVPN directory:

cd keys
sudo cp ca.crt ca.key [your-hostname].crt [your-hostname].key ta.key dh2048.pem /etc/openvpn

The next step is creating the OpenVPN server configuration. OpenVPN already comes with an example config from which we can derive our custom config.

gunzip -c /usr/share/doc/openvpn/examples/sample-config-files/server.conf.gz > /etc/openvpn/server.conf

I followed the DigitalOcean proposals which require the following changes (I did not include all the unchanged lines):

tls-auth ta.key 0 # This file is secret
key-direction 0

cipher AES-128-CBC   # AES
auth SHA256

user nobody
group nogroup

cert [your-hostname].crt
key [your-hostname].key  # This file should be kept secret

Connecting from the short term host

Next we need some client certificates to connect to the server. These can be created from the easy-rsa directory:

source vars
./build-key [client-hostname]

If you do not know the client hostname in advance, you can choose something else, but you need to make sure that it’s a unique CommonName which you did not use for other certificates before.

DigitalOcean proposes the neat idea to program a script that automatically creates configuration files for the clients. Again, OpenVPN ships with a base configuration which we can use and adjust to our needs:

cp /usr/share/doc/openvpn/examples/sample-config-files/client.conf ~/openvpn/client-configs/base.conf

The directory openvpn/client-configs is a new directory for the client configuration generation. But first we have to make the following adjustments to base.conf (again, I did not include all the unchanged lines):

remote [your-server-ip-or-hostname] 1194

user nobody
group nogroup

#ca ca.crt
#cert client.crt
#key client.key

cipher AES-128-CBC
auth SHA256

key-direction 1

# script-security 2
# up /etc/openvpn/update-resolv-conf
# down /etc/openvpn/update-resolv-conf

Next, they use a small script that appends all certificates and keys to this base configuration inside the right tags. Due to copyright I will not copy the full script. Yet, the idea is to create a client configuration file in the following format:

[all lines from base config]

<ca>
[the exact content of ca.crt
</ca>
<cert>
[the exact content of client.crt]
</cert>
<key>
[the exact content of client.key]
</key>
<tls-auth>
[the exact content of ta.crt]
</tls-auth>

With this script it’s as simple as typing

cd ~/openvpn/ca
./build-key client1

cd ~/openvpn/client-configs
./make_config.sh client1

to create key, certificate and a configuration file for a new client.

We can automate these steps even more in a short script to be able to add any host to the OpenVPN configuration automatically:

CLIENTNAME=$1

# This is the same as ./build-key $CLIENTNAME, but in non-interactive mode
cd ~/openvpn/ca
source vars
./pkitool $CLIENTNAME

cd ~/openvpn/client-configs
./make_config.sh $CLIENTNAME

There already is a secure communication channel between the PC and the active server and the PC and the new server, so we can just transfer the OpenVPN configuration file via the PC. This means, we send this script from the local PC to the active server, execute it, copy the generated file from the server to the PC and pass it on to the new server. Then, we can start openvpn with the client configuration there.

CLIENTNAME=$1
TARGETHOST=$2
SERVER=server

CONFIGFILE=${CLIENTNAME}.ovpn
if [ "$TARGETHOST" = "localhost" ]; then
    SSHTARGET="./"
else
    SSHTARGET=$TARGETHOST:
fi

ssh $SERVER "bash -s $CLIENTNAME" < generate-client-cert.sh
scp $SERVER:openvpn/client-configs/files/$CONFIGFILE $SSHTARGET

if [ "$TARGETHOST" = "localhost" ]; then
    su -c "openvpn --config $CONFIGFILE"
else
    ssh -t $TARGETHOST "su -c 'openvpn --config $CONFIGFILE'"
fi

Communicating between both hosts

Communication between both hosts can happen inside the OpenVPN subnet in a secure way without opening ports to other hosts which are not part of the private network.

I personally use this setup to startup Kibana on my local machine and connect it to nginx and elasticsearch on my server. If I have elasticsearch and kibana both running all the time, they take up all 1GB of my memory and make my server unstable.

Nginx always serves the domain logs.stefankoch.name, but requires a username / password combination for it. So, nobody except for me can access it. When Kibana is not running nginx would serve a gateway error, but when it is running it will redirect traffic to my local PC (or AWS instance or anything else) and Kibana on my local PC will in turn query elasticsearch on the server for data - all through the OpenVPN tunnel.

For this purpose I created another small script, which will adjust some environment variables pointing to my newly added cluster node (aka my local PC) and restart the nginx container. The nginx container will then know which IP my newly added cluster node has and can redirect traffic correctly.

KIBANA_HOST=$1
ELASTICSEARCH=http://10.8.0.1:9300

DOCKER_COMPOSE_BIN="/opt/docker-compose/bin/docker-compose"

sed -i -- "s/KIBANA_HOST=.*/KIBANA_HOST=$1/g" docker-compose.yml
scp docker-compose.yml tazaki:docker-compose-logging.yml
ssh -t tazaki "su -c '$DOCKER_COMPOSE_BIN -f docker-compose-logging.yml -f docker-compose.yml up -d'"

sudo docker run -e ELASTICSEARCH_URL=$ELASTICSEARCH -p $KIBANA_HOST:5601:5601 --rm kibana:4

With the same approach, you can also move your daily load to cheap mothly paid providers while you can start new servers for short term tasks at AWS - and connect them to the other servers via VPN.