We’ve upgraded lemmy.world to 0.18.1-rc.1 and rolled back that upgrade because of issues.
(If you had posted anything in those 10 minutes between upgrade and rollback, that post is gone. Sorry!)
The main issue we saw is that users can’t login anymore. Existing sessions still worked, but new logins failed (from macos, ios and android. From linux and windows it worked)
Also new account creation didn’t work.
I’ll create an issue for the devs and retry once it’s fixed.
Edit Contacted the devs, they tell me to try again with lemmy-ui at version 0.18.0. Will try again, brace for some downtime!
Edit 2 So we upgraded again, and it seemed to work nicely! But then it slowed down so much it was unuseable. There were many locks in the database. People reported many JSON errors. Sorry, we won’t be on 0.18.1 any time soon I’m afraid…
appreciate the transparency. how are things looking in the back in lemmy.world (server wise)? will we get to a point where it wont require complete rollbacks on the state when a botched update gets rolled out?
Not having to do a database rollback is a really, really hard problem to solve, and it would almost certainly need to be on the Lemmy developers side, not the server owner’s side. And if I’m them, that’s a low priority issue, and probably not something I even think about until 1.0.
Basically, they write code that says what to do in the event of a database version change. Usually this only handles upgrade cases, because that’s what happening most of the time. One example of something you might do in a db upgrade is let’s say you had a column where the data type was only numbers, but now you want to allow any alphanumeric character for some reason. You could have a line of code that converts the number to a string.
Okay, but now you need to go back to the previous version. Okay, your db change code runs, but it’s the old version of the db change code, not some new version that you wrote. You unfortunately didn’t have a crystal ball when you wrote this code and couldn’t predict that you were going to change the data to strings, so you didnt write code to change it from a string to a number.
This is why most software doesn’t support downgrades unless you wipe first. For example, if you updated your aging MacBook to the latest Mac OS version, then realized it slows down your laptop too much, you can only go back if you first wipe your laptop in the process. So it’s just easier to just take a snapshot before an upgrade and revert to the snapshot if it fails. Some folks will even do “scheduled maintenance” time during the upgrade in which the whole system goes down for a short time so they don’t have to risk losing data that happened after the snapshot.