Hi,
I know this is quite impossible to diagnose from afar, but I came across the posting from lemmy.world admins talking about the attacks they are facing where the database will get overwhelmed and the server doesn’t respond anymore. And something similar seemed to have happened to my own servers.
Now, I’m running my own self-hosted Lemmy and Mastodon instances (on 2 seperate VPS) and had them become completely unresponsive yesterday. Mastodon and Lemmy both showed the “there is an internal/database error” message and my other services (Nextcloud and Synapse) didn’t load or respond.
Login into my VPS console showed me that both servers ran at 100% CPU load since a couple of hours. I can’t currently SSH into these servers, as I’m away for a couple of days and forgot to bring my private SSH key on my Laptop. So, for now I just switched the servers off.
Anyway, the main question is: what should I look at in troubleshooting when I’m back home? I’m a beginner in selfhosting and I run these instances just for myself and don’t mind if I’d have to roll them back a couple days (I have backups). But I would like to learn from this and get better at running my own services.
For reference: I run everything in docker containers behind Nginx Proxy Manager as my reverse proxy. I have only ports 80, 443 and 22 open to the outside. I have fail2ban set up. The Mastodon and Lemmy instances are not open for registration and just have 2 users each (admin + my account).
I can’t help much regarding the service denial issue.
However Port 22 should never be open to the outside world. Limiting to key authentication is a really good first step.
To avoid automated scans you should also change the port to a higher number, maybe something above 10,000.
This both saves traffic and CPU. And if a security bug in sshd exists this helps, too.
Moving off from port 22 is effectively just security by obscurity. It will save you some logs but the bandwidth and CPU time saving is negligible - especially with fail2ban.
That’s not a good advice, sorry. You can bind openssh to an other port, but the only thing it changes is that you have less noise in your logs (and the real solution to that is to use fail2ban, as it also protect you for upcoming attacks on other services from those hosts). The real most important security measure is to make sure your softwares are always up to date, as old vulnerable software is the first cause of penetration (and yes, it’s better to deactivate password login to only use ssh keys, that’s a good advice).
EDIT: I’m elaborating on that because I realize it may come out as harsh without giving enough details. The main reason while changing ports is a bad idea is because it gives a false sense of security (like your last sentence makes it obvious). While it does allow to protect from automated vulnerability scanners that sweep the internet, it’s trivial to port scan your host, then to test unfamiliar ports for well known protocols. When that happens (and it will), if you think you could avoid frequent updates thanks to port change, you’re powned. The most important thing is to have a strict update policy of weekly, if not daily, updates. There is no working around it.
As long as you do not allow password logins for ssh you can let the silly idiots beat their heads against it or you could use a script to ban them. They will not brute force a properly secured ssh server.
I would be mostly annoyed about the log entries. That would be my primary motivation to ban script kiddy hack attempts.
The points I made should not be used instead of all other security precautions like prohibited password login, fail2ban and updates, I thought that is common knowledge. It’s additional steps to increase security.
I disagree that changing the port is just security by obscurity. Scanning ips on port 22 is a lot easier than probing thousands of ports for every IP.
The reason people do automated exploit attempts on port 22 is because it is fast, cheap and effective. By changing the port you avoid these automated scans. I agree with you, this does not help if someone knows your IP and is targeting you specifically. But if you’re such a valuable target you hopefully have specialized people protecting your IT infrastructure.
Edit: as soon as your sshd answers on port 22, a potential attacker knows that the IP is currently in use and might try to penetrate. As stated above, this information would most likely not be shared with the automated attacks if you used any random port.
If you do not neglect updates, then by all mean, changing ports does not hurt. :) Sorry if I may have strong reaction on that, but I’ve seen way too many people in the past couple decades counting on such anecdotal measures and not doing the obvious. I’ve seen companies doing that. I’ve seen one changing ports, forcing us to use the company certificate to log in, and then not update their servers in 6 months. I’ve seen sysadmins who considered that rotating servers every year made it useless to update them, but employees should all use Jumpcloud “for security reasons”! Beware, though, mentioning port changing without saying it’s anecdotal and the most important thing is updates, because it will encourage such behaviors. I think the reason is because changing ports sounds cool and smart, while updates just sound boring.
That being said, port scanning is not just about targeted pentesting. You can’t just run nmap on a host anymore, because IDS (intrusion detection systems) will detect it, but nowadays automated pentesting tools do distributed port scanning to bypass them : instead of flooding a host to test all their ports, they test a range of hosts for the same port, then start over with a new port. It’s half-way classic port scanning and the “let’s just test the whole IP range for a single vulnerability” that we more commonly see nowadays. But they are way harder to detect, as they scan smaller sets of hosts, and there can be hours before the same host is tested twice.
One could setup a VPN and expose the SSH port to the VPN network only. It think tailscale operates this way?
The issue with this is that if the VPN breaks, you can’t SSH in to fix it, which is a problem if it’s a remote host.
Instead, disable password authentication, use a strong (Ed25519) key, and configure two-factor auth (TOTP or FIDO2).
I’m not sure about the feasibility of this (my first thought would be that ssh on the host can be accessed directly by IP, unless maybe the VPN software creates its own network interface and sshd binds on it?), but this does not remove the need for frequent updates anyway, as openssh is not the only software that could have bugs : every software that opens a port should be protected as well, and you can’t hide your webserver on port 80 behind a VPN if you want it to be public. And it’s anyway a way more complicated setup than just doing updates weekly. :)
No, this doesn’t remove the need to stay up to date.
However, it works on my server and was very easy to setup: a few ufw rules so that port 22 is blocked everywhere, allowed only on the VPN IP range and my local network range. Nmapping from outside does not show port 22 accessible, and indeed you can’t SSH to it without the VPN.
Security is quite tough to get right eh? I tried my best to harden everything opening ports on my server, having a fail2ban, VPN for maintenance, webserver to expose some personal services…
Oh, ok, you whitelist IPs in your firewall. That certainly works, if a bit brutal. :) (then again, I blacklist everyone who is triggering a 404 on my webserver, maybe I’m not the one to speak about brutality :P ) You don’t even need a VPN, then, unless you travel frequently (or your ISP provides dynamic IP, I guess).
Well that’s a bit of both: I need to be able to get on my server from work (with my phone… Yeah not great but that works), because I often break stuff haha ; also a nice thing to have when I’m on the bus and want to add more music or movies to listen to during the travel!
Are there ISPs that don’t provide dynamic IPs? I had to setup a script and get some API keys for different services to ensure the IP is properly updated on my DNS servers.
Speaking of brutality, I considered doing the same but then I would have banned myself from testing the APIs of my services 🤧
Oh, I see. Totally makes sense. :)
I guess it depends on the country, but here in France, yes, most landline ISPs provide static IPs (maybe all? there are a couple I haven’t try ; mobile IPs are always dynamic, though). It was not always the case, but I haven’t had a dynamic IP since the 2000’. I feel you, dealing with pointing a domain to a dynamic IP is a PITA.
Ahah, yeah, I protected myself against accidentally banning my own IPs. First, my server is a Pi at home, so I can just plug a keyboard and a screen to it in case of problem. But more importantly, as I do that blacklisting through fail2ban, I just whitelisted my IPs and those of my relatives (it’s the
ignore_ipvariable in/etc/fail2ban/jail.conf)., so we never get banned even if we trigger fail2ban rules (hopefully, grandma won’t try to bruteforce my ssh!). It allowed me to do an other cool stuff : I made a script ran through cron that parses logs for 404 and checks if they were generated by one of the IPs in that list, mailing me if it’s the case. That way, I’m made aware of legit 404 that I should fix in my applications.This doesn’t really work any more. Port scanning is trivial with IPv4, and tools like masscan can scan the entire IPv4 internet (all IPv4 addresses) in less than 15 minutes.
Very interesting, thanks for sharing!
I know it’s just anecdotal evidence, however fail2ban in my one machine which does need ssh on port 22 to the open internet bans a lot of IPs every hour. All other ones with ssh on a higher port do not. Also their auth log does not show any failed attempts.
Yeah it’s probably script kiddies that are only trying port 22 because it’s easier than port scanning.
Make sure you disable
PasswordAuthenticationin the OpenSSH configuration (/etc/ssh/sshd_configon Debian at least). People won’t be able to try to brute force their way in if the only way in is without a password (ie using an SSH key) :)