While I was asleep, apparently the site was hacked. Luckily, (big) part of the lemmy.world team is in US, and some early birds in EU also helped mitigate this.

As I am told, this was the issue:

  • There is an vulnerability which was exploited
  • Several people had their JWT cookies leaked, including at least one admin
  • Attackers started changing site settings and posting fake announcements etc

Our mitigations:

  • We removed the vulnerability
  • Deleted all comments and private messages that contained the exploit
  • Rotated JWT secret which invalidated all existing cookies

The vulnerability will be fixed by the Lemmy devs.

Details of the vulnerability are here

Many thanks for all that helped, and sorry for any inconvenience caused!

Update While we believe the admins accounts were what they were after, it could be that other users accounts were compromised. Your cookie could have been ‘stolen’ and the hacker could have had access to your account, creating posts and comments under your name, and accessing/changing your settings (which shows your e-mail).

For this, you would have had to be using lemmy.world at that time, and load a page that had the vulnerability in it.

  • @LuckyLu
    link
    32010 months ago

    Very impressed by how quickly action has been taken by this and other instances to patch the issue.

    • @Darkard
      link
      20
      edit-2
      10 months ago

      Hijacking the top comment to say I had problems with logging in to Lemmy.world today and liftoff was failing in odd ways.

      I had to go into my web browser and clear my site cookies for lemmy.world to let me log in there.

      In liftoff I had to go into the app settings in android to clear the cache and then remove and re-add my account for it to be able to log me in. (Press and hold on the account to remove it)

      • @LuckyLu
        link
        1610 months ago

        I’m on iOS with the Memmy app. It’s a work in progress that’s officially unfinished so I’m not surprised but it has also been a bit buggy. Doesn’t seem that I can log out without deleting and reinstalling the app so hopefully this doesn’t happen too often XD

        • @Carnelian
          link
          1210 months ago

          So I was actually just struggling with that myself, also in the Memmy app in case that isn’t clear

          What I did was add my account (again)

          There was no warning or anything, and it populated the list with two of me.

          At that point, a “delete account” option appeared under both of them. So I guess in normal circumstances, it wants you to keep one account around at all times?

          I deleted one of them, and the app basically reinitialized. Both were gone and it showed me the welcome screen.

          I logged back in, and now everything is back to normal

          • NaN
            link
            fedilink
            310 months ago

            I just did edit account and then saved, it seemed to trick it into logging in again (secrets on my instance were also reset).

          • @scarabic
            link
            210 months ago

            Ah interesting. I’ve had multiple accounts from the start so it was much easier for me. Just removed my main account and added it back.

          • JJROKCZ
            link
            110 months ago

            Finally I found good instructions, was about to delete and reinstall until I followed this!

          • @LuckyLu
            link
            110 months ago

            Interesting. Definitely could be made clearer, I’ll make a post on the GitHub later about some of my suggestions.

          • @Pandantic
            link
            1
            edit-2
            10 months ago

            I did this, but I just didn’t delete either accounts and it worked fine. Idk if it’s detrimental to have two of the same but it worked for me.

          • @LuckyLu
            link
            110 months ago

            Interesting. Definitely could be made clearer, I’ll make a post on the GitHub later about some of my suggestions.

            • @LuckyLu
              link
              110 months ago

              Whoops, glitched double response.

        • @Infernal_pizza
          link
          710 months ago

          I found I didn’t actually have to log out, just go into account settings and reconfirm everything without changing it

        • @scarabic
          link
          210 months ago

          No you can. You just remove the account from the accounts list. It’s labeled “delete this account” which is scary but it just removes it from Memmy. You can add it right back and that logs you back in. Not a great experience.

          I sure hope this doesn’t happen a lot. This kind of barrier hurts site growth. I’ve managed a lot of large sites and seen a lot of bugs and when everyone gets logged out there is a measurable impact, and some folks never return. Just look at all the comments here saying “thank I didn’t know to do that.” For every one of those there are 100 people going “huh… Lemmy is down… oh well… on to something else…”

        • @calaei
          link
          210 months ago

          Go into account settings, clear your password, re-enter your password, save, go to feed and pull to refresh. That’s what worked for me.

        • @No1ButtMe
          link
          110 months ago

          I was I able to upvote anything or subscribe. Seems like uninstalling and reinstalling fixed my issue

        • LazaroFilm
          link
          110 months ago

          For Memmy, I went to the accounts page in the settings. Click d on my lemmy.world account then to the page where you can change the password then navigated away. That reactivated the account. Maybe we should add a ticket on Memmy’s GitHub about reactivating cookies when there’s an issue. Or at least place à poput to double check credentials or something.

      • @Caboose12000
        link
        310 months ago

        thanks for posting this, I wouldn’t have figured that out lol

      • cookie
        link
        210 months ago

        Oh, I was wondering why it was showing me as logged in but wouldn’t let me upvote due to not being logged in. Your liftoff psa just cleared that right up for me, thanks!

      • @runswithjedi
        link
        210 months ago

        Thank you so much. I was having the same issue with the website and PWA. Clearing site settings worked perfectly.

      • deweydecibel
        link
        210 months ago

        In liftoff I had to go into the app settings in android to clear the cache and then remove and re-add my account for it to be able to log me in. (Press and hold on the account to remove it)

        Good PSA. It took me a bit to figure it out, the app doesn’t make this obvious.

      • anonymous
        link
        110 months ago

        How have I never thought of comment hijacking?!

      • @LuckyLu
        link
        210 months ago

        Negative one upvotes would mean that enough people disliked me/another poster to bring my upvote total to zero. (Upvotes and likes are effectively the same thing, it’s just a naming convention). Reddit totals them up and seemingly Lemmy does as well.

        • @[email protected]
          link
          fedilink
          110 months ago

          huh that’s weird (yes I meant negative one downvote), I already know that the total can be either positive or negative, but shouldn’t the upvote number and downvote number be either positive or zero? (for now I’ll just accept it as a lemmy bug/ inconsistencies between instances)

          • @LuckyLu
            link
            110 months ago

            Nope, just like Reddit it’s a value that ranges between negatives and positives. If I get two thousand upvotes, positive 2k. If I get two thousand downvotes, negative 1999 (because iirc you start with one by default).

            Not exactly sure I understood what you meant by “either positive or zero”.

            • @[email protected]
              link
              fedilink
              210 months ago

              see your comment rn, it has 1 upvote (from yourself by default) and 0 dislike (so it’s not shown)

              but in the screenshot I sent above you got 287 upvote and minus -1 downvote (making your total 288) which is mathematically correct but seems like an unintended behavior

              for example this comment of mine normally have 9 upvote and 2 downvote (which is shown as a positive integer 2, not negative), making my total upvote 7

              • @LuckyLu
                link
                210 months ago

                Just occurred to me that the app I use also shows separate counters. I fooled myself into thinking it was a single counter.

                That’s interesting. Remember it’s a very new platform, minor bugs aren’t out of the ordinary.