I put up a vps with nginx and the logs show dodgy requests within minutes, how do you guys deal with these?

Edit: Thanks for the tips everyone!

  • Teapot
    link
    fedilink
    English
    461 year ago

    Anything exposed to the internet will get probed by malicious traffic looking for vulnerabilities. Best thing you can do is to lock down your server.

    Here’s what I usually do:

    • Install and configure fail2ban
    • Configure SSH to only allow SSH keys
    • Configure a firewall to only allow access to public services, if a service only needs to be accessible by you then whitelist your own IP. Alternatively install a VPN
      • @[email protected]
        link
        fedilink
        English
        131 year ago

        Seconded, not only is CrowdSec a hell of a lot more resource efficient (Go vs Python IIRC), having it download a list of known bad actors for you in advance really slows down what it needs to process in the first place. I’ve had servers DDoSed just by fail2ban trying to process the requests.

        • Alfi
          link
          fedilink
          English
          3
          edit-2
          1 year ago

          Hi,

          Reading the thread I decided to give it a go, I went ahead and configured crowdsec. I have a few questions, if I may, here’s the setup:

          • I have set up the basic collections/parsers (mainly nginx/linux/sshd/base-http-scenarios/http-cve)
          • I only have two services open on the firewall, https and ssh (no root login, ssh key only)
          • I have set up the firewall bouncer.

          If I understand correctly, any attack detected will result in the ip being banned via iptables rule (for a configured duration, by default 4 hours).

          • Is there any added value to run the nginx bouncer on top of that, or any other?
          • cscli hub update/upgrade will fetch new definitions for collections if I undestand correctly. Is there any need to run this regularly, scheduled with let’s say a cron job, or does crowdsec do that automatically in the background?
          • @[email protected]
            link
            fedilink
            English
            11 year ago

            Well I was expecting some form of notification for replies, but still, seen it now.

            My understanding of this is limited having mostly gotten as far as you have and been satisfied.

            For other bouncers, there’s actually a few decisions you can apply. By default the only decision is BAN which as the name suggests just outright blocks the IP at whatever level your bouncer runs at (L4 for firewall and L7 for nginx). The nginx bouncer can do more thought with CAPTCHA or CHALLENGE decisions to allow false alerts to still access your site. I tried writing something similar for traefik but haven’t deployed anything yet to comment further.

            Wih updates, I don’t have them on automated, but I do occasionally go in and run a manual update when I remember (usually when I upgrade my OPNSense firewall that’s runs it). I don’t think it’s a bad idea at all to automate them, however the attack vectors don’t change that often. One thing to note, newer scenarios only run on the latest agent, something I discovered recently when trying to upgrade. I believe it will refuse to update them if it would cause them to break in this way, but test it yourself before enabling corn

  • h3x
    link
    fedilink
    45
    edit-2
    1 year ago

    A pentester here. Those bad looking requests are mostly random fuzzing by bots and sometimes from benign vulnerability scanners like Censys. If you keep your applications up date and credentials strong, there shouldn’t be much to worry about. Of course, you should review the risks and possible vulns of every web application and other services well before putting them up in the public. Search for general server hardening tips online if you’re unsure about your configuration hygiene.

    An another question is, do you need to expose your services to the public? If they are purely private or for a small group of people, I’d recommend putting them behind a VPN. Wireguard is probably the easiest one to set up and so transparent you wouldn’t likely even notice it’s there while using it.

    But if you really want to get rid of just those annoying requests, there’s really good tips already posted here.

    Edit. Typos

    • @[email protected]
      link
      fedilink
      English
      31 year ago

      I really wanted to use this and set it up a while ago. Works great but in the end I had to deactivate it, because my nextcloud instance would cause too many false positives (404s and such) and I would ban my own up way too often.

          • 𝓢𝓮𝓮𝓙𝓪𝔂𝓔𝓶𝓶
            link
            fedilink
            English
            21 year ago

            Ok, so I spent way too much time tonight trying to figure this out, made a mess of my npm, and fixed it.

            It is very well documented.

            Official documentation on using crowdsec with NPM is out of date and relies on a fork that’s no longer maintained. I’m trying to find any documentation on how to integrate the bouncer into the official NPM project and am really coming up empty.

              • 𝓢𝓮𝓮𝓙𝓪𝔂𝓔𝓶𝓶
                link
                fedilink
                English
                11 year ago

                I’m not sure if it’s the fact that I was up at 1am trying to figure this all out or what but it wasn’t clicking last night. So the NPM (nginx) integration would strictly be the captcha and I would need to bounce at the firewall to block? That makes way more sense to me now. Thanks.

                • @[email protected]
                  link
                  fedilink
                  English
                  21 year ago

                  No problem, crowdsec is not super simple. There is a new learning platform you need to check out! Feel free to DM me when you are stuck on something.

  • z3bra
    link
    fedilink
    English
    171 year ago

    I mean, it’s not a big deal to have crawlers and bots poking at our webserver if all you do is serving static pages (which is common for a blog).

    Now if you run code on server side (eg using PHP or python), you’ll want to retrieve multiple known lists of bad actors to block them by default, and setup fail2ban to block those that went through. The most important thing however is to keep your server up to date at all times.

  • @gobbling871
    link
    English
    161 year ago

    Nothing too fancy other than following the recommended security practices. And to be aware of and regularly monitor the potential security holes of the servers/services I have open.

    Even though semi-related, and commonly frowned upon by admins, I have unattended upgrades on my servers and my most of my services are auto-updated. If an update breaks a service, I guess its an opportunity to earn some more stripes.

      • @gobbling871
        link
        English
        41 year ago

        Mostly because stability is usually prioritized above all else on servers. There’s also a multitude of other legit reasons.

        • exu
          link
          fedilink
          English
          101 year ago

          All the legit reasons mentioned in the blog post seem to apply to badly behaved client software. Using a good and stable server OS avoids most of the negatives.

          Unattended Upgrades on Debian for example will by default only apply security updates. I see no reason why this would harm stability more than running a potentially unpatched system.

          • @gobbling871
            link
            English
            31 year ago

            Even though minimal, the risk of security patches introducing new changes to your software is still there as we all have different ideas on how/what correct software updates should look like.

            • exu
              link
              fedilink
              English
              31 year ago

              Fair, I’d just rather have a broken system than a compromised one.

          • @[email protected]
            link
            fedilink
            English
            11 year ago

            Hell, debian is usually so stable I would just run dist-upgrade on my laptop every morning.

            The difference there is that I’d be working with my laptop regularly and would notice problems more quickly

  • @orangeboats
    link
    English
    16
    edit-2
    1 year ago

    I only expose services on IPv6, for now that seems to work pretty well - very few scanners (I encounter only 1 or 2 per week, and they seem to connect to port 80/443 only).

    • 🅱🅴🅿🅿🅸
      link
      fedilink
      English
      51 year ago

      Must be nice living in a post 1995 country… theres only 1 or 2 ISPs in Australia that support ipv6…

      • @orangeboats
        link
        English
        31 year ago

        Lol, I have heard some ISP horror stories from the Down Under.

        I am fortunate enough that my country’s government has been forcing ISPs to implement IPv6 in their backbone infrastructure, so nowadays all I have to really do is to flick a switch on the router (unfortunately many routers still turn off IPv6 by default) to get an IPv6 connection.

        • 🅱🅴🅿🅿🅸
          link
          fedilink
          English
          21 year ago

          Yeah the internet services here are really stuck in the past. Hard to tell if theyre taking advantage of the scarcity of ipv4 addresses to make more money somehow, or of theyre just too fuckn lazy

    • @[email protected]
      link
      fedilink
      English
      21 year ago

      Isn’t that akin to security through obscurity… you might want one more layer of defense

      • @orangeboats
        link
        English
        2
        edit-2
        1 year ago

        I still have firewall (that blocks almost all incoming connections) and sshguard setup. I also check the firewall logs daily, blocking IPs that I find to be suspicious.

        I could probably do better, but with so few scanners connecting to my home server, I have managed to sleep way better than back when I setup a server on IPv4!

        Also, even if my home server gets attacked, at least I know that my other devices aren’t sharing the same IP with them… NAT-less is a godsend.

  • @OuiOuiOui
    link
    English
    111 year ago

    I’ve been using crowdsec with swag for quite some time. I set it up with a discord notifier. It’s very interesting to see the types of exploits that are probed and from each country. Crowdsec blocks just like fail2ban and seems to do so in a more elegant fashion.

  • takeda
    link
    fedilink
    81 year ago

    I use fail2ban and add detection (for example I noticed that after I implemented it for ssh, they started using SMTP for brute force, so had to add that one as well.

    I also have another rule that observes fail2ban log and adds repeated offenders to a long term black list.

  • Alfi
    link
    fedilink
    English
    81 year ago

    sometimes I grab popcorn and “tail -f /var/log/secure”

  • @ichbinjasokreativ
    link
    English
    71 year ago

    Ignore them, as long as your firewall is set up properly.

  • @[email protected]
    link
    fedilink
    English
    71 year ago

    Any service I have that is public facing is proxied through Cloudflare. I run a firewall on the host that only allows traffic from Cloudflare IPs. Those IPs are updated via a cron job that calls this script: https://github.com/Paul-Reed/cloudflare-ufw I also have a rule set up in Cloudflare that blocks traffic from other countries.

    For WAF, I use modsecurity with nginx. It can be a little time consuming to set up and weed out false positives, but it works really well when you get it configured properly.

    Some of my applications are set up with Cloudflare Access. I use this with Azure AD free tier and SAML, but it could be set up with self hosted solutions like authentik.

      • @PhilBro
        link
        English
        31 year ago

        Pretty much, strange in the self-hosted community to have stuff like that happen.

      • @[email protected]
        link
        fedilink
        English
        21 year ago

        cloudflare is sweet I just switched to there from Google domains and it feels like a billion options have just opened up

        also the https security radio buttons I always forget to change in new sites

  • @dinosaurdynasty
    link
    English
    61 year ago

    I use Caddy as a reverse proxy, but most of this should carry over to nginx. I used to use basic_auth at the proxy level, which worked fine(-ish) though it broke Kavita (because websockets don’t work with basic auth, go figure). I’ve since migrated to putting everything behind forward_auth/Authelia which is even more secure in some ways (2FA!) and even more painless, especially on my phone/tablet.

    Sadly reverse proxy authentication doesn’t work with most apps (though it works with PWAs, even if they’re awkward about it sometimes), so I have an exception that allows Jellyfin through if it’s on a VPN/local network (I don’t have it installed on my phone anyway):

    @notapp {
      not {
        header User-Agent *Jellyfin*
        remote_ip 192.160.0.0/24 192.168.1.0/24
      }
    }
    forward_auth @notapp authelia:9091 {
      uri /api/verify?rd=https://authelia.example
    }
    

    It’s nice being able to access everything from everywhere without needing to deal with VPNs on Android^ and not having to worry too much about security patching everything timely (just have to worry about Caddy + Authelia basically). Single sign on for those apps that support it is also a really nice touch.

    ^You can’t run multiple VPN tunnels at once without jailbreaking/rooting Android

  • @[email protected]
    link
    fedilink
    English
    51 year ago

    These requests are probably made by search/indexing bots. My personal server gets a quite a lot of these, but they rarely use any bandwidth.
    The easiest choice (probably disliked by more savvy users) is to just enable cloudflare on your server. It won’t block the requests, but will stop anything malicious.
    With how advanced modern scraping techniques are there is so much you can do. I am not an expert, so take what I say with a grain of salt.

    • Rusty
      link
      English
      21 year ago

      Fail2Ban is great and all, but Cloudflare provides such an amazing layer of protection with so little effort that it’s probably the best choice for most people.

      You press a few buttons and have a CDN, bot attack protection, DDOS protection, captcha for weird connections, email forwarding, static website hosting… It’s suspicious just how much stuff you get for free tbh.

        • @GlitzyArmrest
          link
          English
          2
          edit-2
          1 year ago

          To be fair, you can configure Cloudflare to use your own certs.

            • @[email protected]
              link
              fedilink
              English
              31 year ago

              You can use a custom origin certificate, but that’s irrelevant when CloudFlare still re-encrypt everything to analyse the request in more detail. It does leave me torn when using it, I don’t use it on anything where sensitive plain text is flying around, especially authentication data (which is annoying when that’s the most valuable place to have the protection), but I do have it on my matrix homeserver as anything remotely important is E2EE anyway so there’s little they can gain, and with the amount of requests it gets some level of mitigation is desirable