Hi all,

If you’re just now signing in for the first time in 12+ hours, you may just now be finding out that Lemmy World and other instances where hijacked. The hijackers had the full abilities of hijacked user, mod, and admin accounts. At this time, I am only aware of instance defacing and URL redirections to have been done by the hijackers.

If you were not forced to sign back in this morning, contact your instance admin to verify mitigations were completed on your instance.

How?

This occurred due to an XSS attack in the recently added custom emojis. Instance admins should follow the issue tracker on the LemmyNet GitHub, as well as the Matrix Chat. Post-Incident Activity is still on-going.

Currently, it is likely that just your session cookie was stolen, with instance admins being targeted specifically by checking for navAdmin, an HTML element only instance admins had. I do not believe this to affect users across instances, but I have yet to confirm this.

What happens next?

As I am not the developers or affected instance admins, I cannot make any guarantees. However, here is what you’ll likely see:

  1. Post Incident investigation continues. This will include inspecting code, posts, websites, and more used by the hijackers. An official incident writeup may occur. You should expect the following from that report:
  • Exactly what happened, when.
  • The incident response that occurred from instance admins
  • Information that might have helped resolve the issue sooner
  • Any issues that prevented successful resolution
  • What should have been done differently by admins
  • What should be improved by developers
  • What can be used to identify the next attack
  • What tools are needed to identify that information
  1. A CVE is created. This is an official alert of the issue, and notifies security experts (and enthusiasts), even those not using lemmy, about the issue.

  2. A code security audit is done. This will likely just be casual reviews by technical lemmy users. However, I will be reaching out to the Mozilla Foundation and Cure53 as they recently did an audit of Mastodon. If there is interest in an external audit of lemmy and the costs are affordable, I’ll look into crowdfunding this cost.

  • redcalcium
    link
    fedilink
    English
    200
    edit-2
    1 year ago

    This incident made me realize not to use an admin account for my primary lemmy account in my personal instance. I setup another account for instance admin purpose (with 2FA enabled) and keep it logged out, then remove my primary account from the instance admin list.

    • @Renacles
      link
      English
      761 year ago

      This is a good mindset in general, when working in AWS you are not supposed to use your root account unless it’s absolutely necessary even if you are the only user. Hosting a Lemmy instance should be no different.

      • @[email protected]
        link
        fedilink
        English
        111 year ago

        Yeah, even in Windows or what-have-you, you should always keep your admin account separate from your daily driver account for exactly this reason.

      • @[email protected]
        link
        fedilink
        English
        2
        edit-2
        1 year ago

        I just setup a VPS for a Minecraft server for some friends and did exactly that. I was under a bit of a time crunch, but still took the time to think through those challenges in access for everything. Created an unprivileged user to run the server as, created a seperate unprivileged user for another service. Disabled password-based SSH login, etc.

        I should probably setup a dedicated non-root admin account for administrative functions but that’s a problem for after work

    • @Crackhappy
      link
      English
      171 year ago

      I work with 2 factor, Oath, SAML, etc. all the time for work, and for the life of me I can’t get it working properly with Lemmy.

      • Notorious
        link
        fedilink
        English
        211 year ago

        Lemmy decided to go with SHA256 for TOTP seed. This is a very odd move since many 2FA apps don’t support SHA256. I actually had to write a quick python script to spit out my 2FA code since Bitwarden doesn’t support it. Hopefully either Lemmy will change to SHA-1 or Bitwarden will start to support SHA256 seeds.

        • redcalcium
          link
          fedilink
          English
          17
          edit-2
          1 year ago

          Wait, I use bitwarden and it works just fine. And yes, it’s using SHA256.

          Screenshot:

          • @subtext
            link
            English
            31 year ago

            Bitwarden definitely works with SHA256 and even SHA512, I believe. And with codes longer than 6 digits also. Oh and even Steam codes if you really want.

            • redcalcium
              link
              fedilink
              English
              11 year ago

              You can put steam OTP there too? I’m tired getting locked up from steam whenever I format my phone. If I can put steam OTP into bitwarden, that’ll be really great.

          • @Trapping5341
            link
            English
            21 year ago

            wait wait wait. how did you get lemmy 2FA into bitwarden. I can only get it to set up in authy or google auth

            • redcalcium
              link
              fedilink
              English
              21 year ago

              You’ll need to either subscribe to bitwarden premium ($10 / year), or deploy vaultwarden in your own server and have your bitwarden extension/app use that vaultwarden server instead of the official server.

              • @Trapping5341
                link
                English
                11 year ago

                Yeah I have premium and every account I have that offers 2fa is set up but Lemmy will only let me set it up in authy or Google auth. It won’t give bitwarden as an option or give me the code to manually add it to bitwarden.

                • redcalcium
                  link
                  fedilink
                  English
                  21 year ago

                  Just right click at the 2fa button and select “copy link”, then paste the link into bitwarden TOTP field. The link you copied should already be in the format otpauth://totp/xxxx which will be recognized by bitwarden.

                  • @Trapping5341
                    link
                    English
                    11 year ago

                    😂 can’t believe I didn’t think of that. Thank you so much.

        • @Renacles
          link
          English
          121 year ago

          SHA-1 has been broken already.

          • @subtext
            link
            English
            21 year ago

            Doesn’t matter too too much for a TOTP though. It just needs to be pseudorandom enough for a relatively difficult to guess 6 digit (or longer) code that changes every 30 seconds. Much more likely for someone to get phished for their TOTP than for someone to reverse engineer someone’s TOTP seed. Plus how would that even happen? The attacker would need to have the device and / or track the user’s 6 digit codes for quite a while at which point the game’s already up.

        • @[email protected]
          link
          fedilink
          English
          101 year ago

          Preferably the latter, nobody should really be using SHA1 for anything security focused and new

      • redcalcium
        link
        fedilink
        English
        71 year ago

        I use bitwarden, so it’s pretty simple: just copy the link from the 2fa button in Lemmy into the TOTP field in your account editor in bitwarden, and it’ll automatically recognize the format.

    • BarterClub
      link
      fedilink
      English
      61 year ago

      Yup. Basics of running a server for anything. Never use your admin account and make a default backup with 2 factor.