When I first started to use AWS services I was wondering why so many people use EC2 machines 24/7 and pay a ton of money for a server that - at least in Europe - you get for much less money (if you run your server the whole month). A bit later, I came to realize that it’s a lot easier to run a cluster inside AWS with their VPC service than it is at other providers.
I needed a 24/7 server nonetheless and was not willing to pay the high price
at AWS for private projects. So I got the cheapest virtual server at Hetzner for
around four euros per month - and ran out of memory.
So, there were basically two different solutions to the problem. Either scale up and buy a bigger server or scale out and buy a second server. I have not decided for one, yet, but I started thinking about how to scale out. To my knowledge, Hetzner does not offer private subnets to every customers. I read somewhere on the internet, that they might do for larger customers, but not for everybody. On the other hand, I also did not want to publish all my services to the global net and secure all of them with a reverse proxy with SSL and authentication.
I started to wonder if this problem might be solvable with a virtual private
network (VPN). One of the machines can be the VPN server and others connect
to it. Whenever traffic has to pass through the private network, it will go
via the VPN interface, but all other traffic can still be routed through each of
the machines standard
This would also allow me to use cheap long term servers for my standard load and only when I need to scale out for a short period of time use AWS servers.
This tutorial uses OpenVPN to connect the servers, but as of 2019 it is easier and more lightweight to setup Wireguard.
Setting up an OpenVPN server on the long term host
For setting up an OpenVPN server, we can follow a
DigitalOcean tutorial. First, you need to
create our own Root Certificate Authority for the OpenVPN network. With
this we can create certificates for both the log term server and the
newly joining nodes (clients). So start by installing
easy-rsa with your package manager.
easy-rsa will help in creating all the required certificates.
We should prepare the Certificate Authority:
The directory contains a file
vars with default settings for the key
creation. You can set all of them to something reasonable that will be
used for all created certificates. The only difference between the keys
will be the
Common Name, which you define when you call the
command line scripts.
On my host, I adjusted all of the fields except for OU which I left empty.
Now we are ready to create our own CA, a certificate for the server
as well as strong Diffie-Hellman keys and an HMAC signature for more
Just hit enter all the time and
y when asked if you want to sign and
commit the certificate.
You will call
clean-all only once before you start setting up your
certificates. Afterwards, you’ll need the data in this directory to
create your client
certificates. If you clean the directory and then want to create a server
or a client certificate,
easy-rsa will complain that CA information
Try pkitool --initca to build a root certificate/key.
Now we copy all generated files to the OpenVPN directory:
The next step is creating the OpenVPN server configuration. OpenVPN already comes with an example config from which we can derive our custom config.
I followed the DigitalOcean proposals which require the following changes (I did not include all the unchanged lines):
tls-auth ta.key 0 # This file is secret key-direction 0 cipher AES-128-CBC # AES auth SHA256 user nobody group nogroup cert [your-hostname].crt key [your-hostname].key # This file should be kept secret
Connecting from the short term host
Next we need some client certificates to connect to the server. These
can be created from the
If you do not know the client hostname in advance, you can choose something else, but you need to make sure that it’s a unique CommonName which you did not use for other certificates before.
DigitalOcean proposes the neat idea to program a script that automatically creates configuration files for the clients. Again, OpenVPN ships with a base configuration which we can use and adjust to our needs:
openvpn/client-configs is a new directory for the client
configuration generation. But first we have to make the following
base.conf (again, I did not include all the unchanged
remote [your-server-ip-or-hostname] 1194 user nobody group nogroup #ca ca.crt #cert client.crt #key client.key cipher AES-128-CBC auth SHA256 key-direction 1 # script-security 2 # up /etc/openvpn/update-resolv-conf # down /etc/openvpn/update-resolv-conf
Next, they use a small script that appends all certificates and keys to this base configuration inside the right tags. Due to copyright I will not copy the full script. Yet, the idea is to create a client configuration file in the following format:
[all lines from base config] <ca> [the exact content of ca.crt </ca> <cert> [the exact content of client.crt] </cert> <key> [the exact content of client.key] </key> <tls-auth> [the exact content of ta.crt] </tls-auth>
With this script it’s as simple as typing
to create key, certificate and a configuration file for a new client.
We can automate these steps even more in a short script to be able to add any host to the OpenVPN configuration automatically:
There already is a secure communication channel between the PC and the active server and the PC and the new server, so we can just transfer the OpenVPN configuration file via the PC. This means, we send this script from the local PC to the active server, execute it, copy the generated file from the server to the PC and pass it on to the new server. Then, we can start openvpn with the client configuration there.
Communicating between both hosts
Communication between both hosts can happen inside the OpenVPN subnet in a secure way without opening ports to other hosts which are not part of the private network.
I personally use this setup to startup Kibana on my local machine and connect it to nginx and elasticsearch on my server. If I have elasticsearch and kibana both running all the time, they take up all 1GB of my memory and make my server unstable.
Nginx always serves
logs.stefankoch.name, but requires a username / password
combination for it. So, nobody except for me can access it. When Kibana
is not running nginx would serve a gateway error, but when it is running
it will redirect traffic to my local PC (or AWS instance or anything else)
and Kibana on my local PC will in turn query elasticsearch on the server
for data - all through the OpenVPN tunnel.
For this purpose I created another small script, which will adjust some environment variables pointing to my newly added cluster node (aka my local PC) and restart the nginx container. The nginx container will then know which IP my newly added cluster node has and can redirect traffic correctly.
With the same approach, you can also move your daily load to cheap mothly paid providers while you can start new servers for short term tasks at AWS - and connect them to the other servers via VPN.