We’ve upgraded lemmy.world to 0.18.1-rc.1 and rolled back that upgrade because of issues.

(If you had posted anything in those 10 minutes between upgrade and rollback, that post is gone. Sorry!)

The main issue we saw is that users can’t login anymore. Existing sessions still worked, but new logins failed (from macos, ios and android. From linux and windows it worked)

Also new account creation didn’t work.

I’ll create an issue for the devs and retry once it’s fixed.

Edit Contacted the devs, they tell me to try again with lemmy-ui at version 0.18.0. Will try again, brace for some downtime!

Edit 2 So we upgraded again, and it seemed to work nicely! But then it slowed down so much it was unuseable. There were many locks in the database. People reported many JSON errors. Sorry, we won’t be on 0.18.1 any time soon I’m afraid…

    • RuudOPMA
      18•1 year ago

      Yeah we have a test instance, but not sure if we could test with this kind of load…

      • BitOneZero @ .world
        1•1 year ago

        Yeah we have a test instance, but not sure if we could test with this kind[ of load…

        This is the problem. The big servers have accumulated data that is not in the testing servers and smaller peers. Lemmy instances start out with no data, and joining a community from a remote instance only brings in new data. So far, the big servers have not openly published their lemmy_server logging so we don’t know what is going on inside the Rust code - we are only seeing the symptoms with front-end crashes and federation failing to get comments to other servers.

    • Ulu-Mulu-no-die
      8•1 year ago

      There are already 3 test instances where devs push updates before going live, but:

      • if not many people go there test, they won’t catch all the bugs.
      • in this specific case, admins are trying to push a new release candidate (that is not officially released yet) because some people are stuck with mobile apps that don’t work anymore on the version we have now.