Hello everyone! During one of those illuminated evenings, I got the idea to move my small server in Scaleway to some more powerful server in Hetzner. If I will make the move, I am thinking of splitting the server in various VMs, to host different services that belongs to different trust boundaries, for example:
- A Lemmy/writefreely instance
- Vaultwarden/Gitea
- Wireguard tunnel to my home infrastructure
- Blogs, and other convenience services
In order to achieve the best level of separation, I was thinking of using VMs. My default choice would be Proxmox, because I used it in the past, and because I generally trust it, however I am trying to evaluate multiple options, and maybe someone has good or better experiences to share.
Other options I thought about are:
- Run everything in Docker. I am going to do this nevertheless, but Docker escapes are always possible, especially with public facing images that I did not write myself and/or that require a host volume.
- KVM directly? I am OK even without a GUI to be honest. I am not aware if there is some ansible module or even better Terraform provider for this, it would be great. (EDIT: I found https://registry.terraform.io/providers/dmacvicar/libvirt/0.7.1 which seems awesome!)
- ESxi? I have no experience with this solution.
Any idea or recommendation?
Why rent a whole server? You can run a cloud VM at a fraction of the cost.
Yeah that is true, but at the same time I always felt a bit uncomfortable with using a VM which shares resources with who knows what else. I also like the idea of having for example one VM acting as VPN, firewall, rev proxy, while other VMs behind that do not have internet connection at all (inbound). It is somewhat achievable even with VPSs, but it’s more complex IMO.
I am conflicted though, and I did consider VPSs to be clear.
The resources are shared, sure, but there’s complete logical isolation. Your VM can’t see others, and they can’t see you (barring any exploit or misconfiguration, but that can happen with physical servers just as well).
Personally I have all my services running in separate containers in one VM. Same separation, just at a different level.
Well, hypervisor bugs are rare, but not so much. A physical server is fully isolated by other tenants of the provider (or better, I can achieve that full isolation with network configuration).
I will definitely anyway run all the services in containers, but I am fully aware that containers don’t provide much isolation, especially once you start using the host network to serve native port (i.e., containerized nginx/haproxy) or mounting filesystem volumes inside them. To be honest, in my current setup, where I am the only user of both the machine and the services (made exception for a few family members), I am OK with this separation. However, if I run a lemmy/writefreely/fedisoftware instance, which is going to host other untrusted users, I am not happy if on the same box my git server is running, or my password manager. That’s mostly the reason why I was looking for full separation. I guess separate VPSs would also work, though.