Hi,

I know this is quite impossible to diagnose from afar, but I came across the posting from lemmy.world admins talking about the attacks they are facing where the database will get overwhelmed and the server doesn’t respond anymore. And something similar seemed to have happened to my own servers.

Now, I’m running my own self-hosted Lemmy and Mastodon instances (on 2 seperate VPS) and had them become completely unresponsive yesterday. Mastodon and Lemmy both showed the “there is an internal/database error” message and my other services (Nextcloud and Synapse) didn’t load or respond.

Login into my VPS console showed me that both servers ran at 100% CPU load since a couple of hours. I can’t currently SSH into these servers, as I’m away for a couple of days and forgot to bring my private SSH key on my Laptop. So, for now I just switched the servers off.

Anyway, the main question is: what should I look at in troubleshooting when I’m back home? I’m a beginner in selfhosting and I run these instances just for myself and don’t mind if I’d have to roll them back a couple days (I have backups). But I would like to learn from this and get better at running my own services.

For reference: I run everything in docker containers behind Nginx Proxy Manager as my reverse proxy. I have only ports 80, 443 and 22 open to the outside. I have fail2ban set up. The Mastodon and Lemmy instances are not open for registration and just have 2 users each (admin + my account).

  • @[email protected]
    link
    fedilink
    English
    61 year ago

    Don’t copy your private key to your laptop, generate a new one and add its public key to the same user. Also use a different key for every remote host. (or don’t, but that’s kind of like using the same password for all your accounts)

    • Mike
      link
      fedilink
      21 year ago

      @ErwinLottemann @Solvena …or take it one step further and store your private key, only(except your offline backup), on an hsm/smartcard such as; yubikey.

      • @[email protected]
        link
        fedilink
        English
        11 year ago

        Sounds great, too. But is that not also using the same key/password for everything? Or can you store multiple keys on a yubikey? I’m currently moving everything to certificate based ssh auth…