A little over 2 years ago, we introduced a new Backup server for our PostgreSQL data. Check out my post on that here.
Now 2 years later, the disks got too small for our growing size of our PostgreSQL data which urged us to move on to a larger server.
We found one in Hetzners SX-line Server fleet - the SX62 with plenty of space. In fact with RAID-5 we will get 10 times the disk size for our backup storage.
Basically the same setup we used before but this time with way larger disks (GPT vs. MBR) and a software-raid and not a hardware-raid plus a newer version of Debian.
This guide is written way shorter and mostly covers the steps without much information beside of that. Read my last post to get more insights.
I don't take responsibilities for any damage that my tutorial may cause to your system. I've tested it twice on my system and it worked well all the time but the commands may be different on your system. So please keep that in mind while following my remarks but the commands should be somewhat identical on Debian derivates or older and newer versions of them.
Install the base system with initial RAID and LVM config
This setup here is a lot easier than for my old post, so boot up your new SX62 to Hetzners Rescue System.
Now let us directly invoke
installimage and configure the SW RAID-5 and LVM and let it install the minimal Debian 10 installation.
installimage -a -n myserver.com \ -b grub \ -r yes \ -l 5 \ -d sda,sdb,sdc,sdd \ -i root/.oldroot/nfs/images/Debian-105-buster-64-minimal.tar.gz \ -p /boot:ext4:512M,lvm:vg0:all \ -v vg0:swap:swap:swap:8G,vg0:root:/:ext4:16G,vg0:var-log:/var/log:ext4:8G,vg0:tmp:/tmp:ext4:8G,vg0:var-lib-postgresql:/var/lib/postgresql:ext4:500G,vg0:var-lib-barman:/var/lib/barman:ext4:10T \ -f yes -s de -K /root/.ssh/robot_user_keys
What we defined here is:
-amakes it non-interactive
-n myserver.comdefines the servers hostname
installimageto install grub as our preferred bootloader
installimageto setup a RAID
-l 5defines the RAID's level. Here it's a RAID-5, so 30/40TB usable
-d sda..defines the disks we want to use for the RAID
-i root/defines the source image for the installation
-pdefines all physical partitions we want to see on the SW-RAID
-vthen defines all logical volumes we want to see on the LVM
installimageto format the disks
-sdefines the preferred server language
installimageto install the following SSH robot keys for root
Encrypt everything and make it headless
Now that the system has been initially installed, reboot to your shiny new installation.
We will now install
dropbear so we can headlessly decrypt the server over SSH at boot time.
apt update && apt-get -y upgrade apt -y install busybox dropbear dropbear-initramfs
You will see the following warning which is ok. We will fix it in a minute.
dropbear: WARNING: Invalid authorized_keys file, remote unlocking of cryptroot via SSH won't work!
Now reboot to the Rescue System again for the next steps.
At first we will make a full backup of our new installed system for a moment.
mkdir /rootbackup mount /dev/vg0/root /mnt mount /dev/vg0/var-log /mnt/var/log mount /dev/vg0/tmp /mnt/tmp mount /dev/vg0/var-lib-postgresql /mnt/var/lib/postgresql mount /dev/vg0/var-lib/barman /mnt/var/lib/barman rsync -av /mnt/ /rootbackup/
This makes a full backup now, because we need to remove the LVM again. Rsync shouldn't take too long but you can see it's output.
Now unmount everything and disable the LVM.
umount /mnt/var/lib/postgresql && umount /mnt/var/lib/barman && umount /mnt/tmp && umount /mnt/var/log && umount /mnt vgchange -a n vg0 # Otherwise recreation of LUKS on md1 will fail later
Since we now have two RAID-5's -
md0 for boot and
md1 for current LVM, we can just overwrite
md1 with the LUKS device.
cryptsetup --cipher aes-xts-plain64 --key-size 512 --hash sha256 --iter-time 6000 luksFormat /dev/md1
Choose a long and safe password with
pwgen 64 1 for the encryption.
When you're done, mount the new LUKS device and let us recreate the LVM and put the system back in place.
cryptsetup luksOpen /dev/md1 cryptroot # Open LUKS device pvcreate /dev/mapper/cryptroot # Create physical LVM volume on LUKS vgcreate vg0 /dev/mapper/cryptroot # Create our volume group again # Create our logical volumes now lvcreate -L 8G -n swap vg0 lvcreate -L 16G -n root vg0 lvcreate -L 8G -n var-log vg0 lvcreate -L 8G -n tmp vg0 lvcreate -L 500G -n var-lib-postgresql vg0 lvcreate -L 10T -n var-lib-barman vg0 # Format the volumes and create swap again mkfs.ext4 /dev/vg0/root mkfs.ext4 /dev/vg0/var-log mkfs.ext4 /dev/vg0/tmp mkfs.ext4 /dev/vg0/var-lib-postgresql mkfs.ext4 /dev/vg0/var-lib-barman mkswap /dev/vg0/swap # Now lets mount everything and put back the system mount /dev/vg0/root /mnt mkdir -p /mnt/var/log mount /dev/vg0/var-log /mnt/var/log mkdir /mnt/tmp mount /dev/vg0/tmp /mnt/tmp mkdir -p /mnt/var/lib/postgresql mount /dev/vg0/var-lib-postgresql /mnt/var/lib/postgresql mkdir -p /mnt/var/lib/barman mount /dev/vg0/var-lib-barman /mnt/var/lib/barman rsync -av /rootbackup/ /mnt/
In the next step, let's add the boot partition to the party as well as the current devices, proc and sys to our chroot environment
mount /dev/md0 /mnt/boot mount --bind /dev /mnt/dev mount --bind /sys /mnt/sys mount --bind /proc /mnt/proc chroot /mnt
Configure LUKS for boot-time decryption
Now let's open
/etc/crypttab and add the LUKS partition, so the system knows what to mount.
# <target name> <source device> <key file> <options> cryptroot /dev/md1 none luks,initramfs
/etc/rc.local and add some content. This will ensure proper network configuration after unlocking and booting the main system.
# Content of rc.local /sbin/ifdown --force eth0 /sbin/ifup --force eth0
The last step is to add your ssh public keys to dropbear, so you can authenticate with it at boot time. For Debian 9 there was no ed25519 key support. For Debian 10 I did a short dig into the changelog but haven't found anything, so my best guess is, that there is still only RSA key support.
vim /etc/dropbear-initramfs/authorized_keys # Add your SSH keys like you would in ~/.ssh/authorized_keys
Now let's update the initramfs with the dropbear config and keys and install grub again on all 4 disks, so we could boot from any single one.
For me I had to wait for the SW-RAID recovery to finish before I could install grub. So you may need to check for
cat /proc/mdstat if it's finished already.
update-initramfs -u -k all grub-install /dev/sda grub-install /dev/sdb grub-install /dev/sdc grub-install /dev/sdd
Cleanup and reboot
We have done everything and we can cleanup our chroot environment and do the first boot into our new encrypted SX62.
exit # Exit Chroot environment umount /mnt/var/lib/barman/ umount /mnt/var/lib/postgresql umount /mnt/var/log umount /mnt/tmp umount /mnt/boot umount /mnt/dev umount /mnt/sys umount /mnt/proc umount /mnt sync shutdown -r now
After reboot, login as root and run
crypttab-unlock to mount the LUKS device and continue to boot.
I hope this guide helps some people out there to further secure their servers. Normally a LUKS encrypted server that runs 24/7 doesn't really help against usual network attacks but if you ever need to have the data save when the server is down or you need to instruct Hetzner to replace a failing disk, you can be sure that your data on the drives are safe, even without a manual erase step. This is why I recommend everyone to go this extra mile.
Also check out my last post if you want to decrease attack surface and disable USB on this server.