We’ve upgraded lemmy.world to 0.18.1-rc.1 and rolled back that upgrade because of issues.
(If you had posted anything in those 10 minutes between upgrade and rollback, that post is gone. Sorry!)
The main issue we saw is that users can’t login anymore. Existing sessions still worked, but new logins failed (from macos, ios and android. From linux and windows it worked)
Also new account creation didn’t work.
I’ll create an issue for the devs and retry once it’s fixed.
Edit Contacted the devs, they tell me to try again with lemmy-ui at version 0.18.0. Will try again, brace for some downtime!
Edit 2 So we upgraded again, and it seemed to work nicely! But then it slowed down so much it was unuseable. There were many locks in the database. People reported many JSON errors. Sorry, we won’t be on 0.18.1 any time soon I’m afraid…
Thank you for the transparancy!
I love post like this, another reason of why this instance is so good
Thanks for all your hard work @ruud@lemmy.world. It’s not easy!
Both Dessalines and Nutomic have been working their butts off to get 0.18.x ready for the Reddit API changes. Huge hopes they can pull through!
Dessalines:
Nutomic:
Yeah they’ve been hard at work all month. But it’s also okay if things aren’t ready in time. Most of the people who matter are already here.
Maybe we will blow up soon, maybe later, but the quality of content here is sufficient to drive growth regardless of whether or not we get the prophesized huge migrations from reddit
Agreed. I’m here to stay with the content from the last couple weeks. 3 day poop embargoes alone is worth it. And recognizing users because the community is still small enough is SO different from they who shall not be named.
It’s adequate the way it is now. They should hold off on any major upgrades until after the next batch of new users come in. Lemmy breaking during that period would be far worse then missing some updates and new features.
Oh god, so much coffee… it’s also thrilling to grasp a sense of what’s going on under the hood of such big social networks at a development level (not like I could understand it, but it was very interesting to see twitter’s recommendation algorithm being open-sourced)
is that users can’t login anymore
Ouch, that’s bad :(
Thank you for your work, as always :)
I mean that’s a Tuesday on Reddit lmao.
As any engineer who does ops can tell you - you did the right thing - the solution is always to roll back, never force a roll forward, ever.
We should totally do pre and post update parties though. Even if the update fails we can have an excuse for drinks and a fun thread.
Oh. Good luck with the next try!
deleted by creator
Reasons I can think of:
-
The official Android client for lemmy, Jerboa, only supports 0.18 and later, unless users download older version from github and sideload manually.
-
Sorting is broken pre 0.18, new posts keeps flowing in.
-
Performance improvement by removing web socket from lemmy. (which fixes 2, which is why 1. happens)
Jerboa works fine with lemmy world, it just gives a warning and crashes on occasion. Not an issue to use it though.
Yes, if you are already logged in. If you are not logged in, or get logged out you will not be able to log back in until the server version is above 0.18
And jerboa recommended logging out. Hmm.
For me at least, I’m unable to change the community scope or sorting method. The only thing I can use Jerboa for is “local” and sorted by “active”. It isn’t totally unusable, but I would much rather be able to switch to subscribed and hot.
I had the same bug. A full restart of my phone seems to have fixed it.
Unfortunately that didn’t work for me. Thanks for the idea, though. I’m sure it will work for some. I suspect that if I wiped the app and set everything back up it might work, but that’s more than I’m willing to do at the moment and I can live with the inconvenience for now.
Alternatively, you can try this fork: https://lemmy.ca/post/899342
Édit : https://lemmy.ca/post/998253
deleted by creator
That sounds like Bethesda’s definition of “works fine”
It works fine if you apply unofficial patch!
It just works, it just works…
I tried it again and you are totally correct if you see this message.
I tried it again and you are totally correct if you see this message.
I do
That is absolutely not “fine”.
With the last couple updates jerboa has got to a point that I probably won’t even move to sync when it comes. I’m pretty happy with it. But if Lemmy Is Fun ever happens, I’m out.
As a new user I tried logging in on Jerboa and it says the server must be on 0.18 and refuses the login.
Have you tried this fork? https://lemmy.ca/post/899342
Édit : https://lemmy.ca/post/998253
Thanks for the link. I knew could side load an older version, I just wanted to be lazy and use something on the play store. I’m trying out several clients right anyway.
With the last couple updates jerboa has got to a point that I probably won’t even move to sync when it comes. I’m pretty happy with it. But if Lemmy Is Fun ever happens, I’m out.
I’m of a similar mind: I’m happy with Jerboa, yet also looking forward to Slide.
Well you’re a lucky one. Jerboa has been unusable for me and other folks post v0.34 since the first rc release of 0.18. I finally managed to downgrade to 0.34 and login after lots of black magic. It used to crash instantly, even on 0.34.
That being said, good luck to lemmy.world admins and thanks a lot for the open comms !
It won’t let me log in, so no, it doesn’t “work fine.” But I have six other apps for Lemmy, so I’m not worried. It’ll all be sorted eventually.
Alternatively, you can try this fork: https://lemmy.ca/post/899342
Édit : https://lemmy.ca/post/998253
The latest update of jerboa did not let me browse all communities. I had to sideload a previous version.
“crashes on occasion”
Unfortunately, the occasion seems to be at least once every five minutes for me. thisisfine.jpg This is not fine and definitely a major issue. Especially as we get closer to July 1. After this news from lemmy.world, I will have to switch, until Sync arrives. Update: Liftoff is fantastic, give it a try!
Have you tried this fork? https://lemmy.ca/post/899342
Try wefwef.app on mobile
(no I’m not a bot account shilling I just really like the app)
You made me try it, and I’ve just switched from Jerboa! It has more of an “Infinity” feel to it. Also tried Liftoff today, wasn’t my cup of tea either.
The design is a copy of Apollo
If there was a way to adjust the font sizes it would be 100% perfect. Damn! I’m surprised I hadn’t heard of it before. Thanks internet stranger!
I have been testing this web app today and I’m really surprised of how it good it looks and behaves… I kinda want it as a standalone app now though.
deleted by creator
It’s an open issue on their github (https://github.com/aeharding/wefwef). Will probably be available soon, the development seems to be fast - I got two updates in a single day. Also tagging @kingcake@lemmy.world
Removed by mod
I’m sold. 😊 I did need to try a couple of times to figure out I needed to swipe left to reply to a comment, though. But it looks good, works well, and I’ll get used to the quirks.
The official Android client for lemmy, Jerboa, only supports 0.18 and later, unless users download older version from github and sideload manually.
Even that won’t work, because Jerboa initially tries to connect to lemmy.ml which is running 0.18, which older versions of Jerboa aren’t compatible with. The app just crashes instantly without giving you the opportunity to log in to your instance.
The old version is working fine for me, I had the crashing issue, but reinstalled and it’s working now
Why the hell does Jerboa even limit support to 0.18? That’s so stupid.
Likely because Jebora is written as a side project of the devs of lemmy so it’s always going to be cutting edge
I understand that. I’m not complaining about the quality. I’m just confused from a technical perspective why it doesn’t support rollback or older server versions in the event of…this.
Fixing the issues with posts appearing and scrolling the page the backend needed some pretty drastic rewrites. I’m not super familiar, but from what I’ve seen of the code, it would be a decent amount more work to support both versions.
Would it have been worth it? Yes, but it wasn’t anticipated that devs would stay on 0.0.17 for more than a day or two. With the time it takes for app stores to update, servers would have been updated before lemmy 0.0.45 was updated for the vast majority of users. At most, it would be a day or two instead of a week or two.
The latest version only supports 0.18 because the backend works differently. Older versions of Jerboa support older backends of lemmy.
Connect for Lemmy works better.
Is it open source?
deleted by creator
I don’t know but I installed and older version of Jerboa from F-Droid store. I couldn’t login from the most recent version of Jerboa because of this so it was pretty much useless for me. Working fine so far.
deleted by creator
I probably disabled auto updates on my end some time in the past because my Jerboa says its version 0.0.33 and it’s working without any noticeable problems. Thanks, past me.
-
I thought because right now on 0.17 mobile access with jebora is pretty broken and since 0.18 .0 is not an option they are trying to go to 0.18.1 as soon as possible. Even when it means installing rc versions
Federation is completely borked with .18 servers. Its very difficult for us to interact with https://lemmy.ca in any way: subscribing, upvoting, comments, posting… its all bugged.
Its maybe not that big a deal because Lemmy.world “has the most users”, so in some regards its https://lemmy.ca’s loss but… we need to restore reliable federation… especially before the July 1st rush IMO.
The .17 to .18 upgrade is basically a soft-defederation event, because of whatever this bug is between the two versions.
Ah that explains it. Someone posted a cool photo to my community from lemmy.ca but didn’t interact further. Looks like my comment didn’t even show up on their end.
Anyway, thanks to everyone working on the issue. I know these things aren’t easy.
I’m subscribed to lemmy.ca and programming.dev, both of which are .18 now. Feels kinda bad losing access to those communities while this issue is getting worked out…
I hear you and it is certainly an inconvenience. My solution for now is just to create mirror accounts on a few different instances under the same name. Hopefully the version compatibility will improve as the Lemmy software matures over time.
deleted by creator
Just curious, why are we updating now instead of waiting for the proper 0.18.1 release?
A Release Candidate is supposed to be past beta testing already and in a state of no major bugs.
Cause if you are on the homepage for a little bit new post start pushing the old post down and it keeps doing it. something about websockets that 18 fixes it. im assuming the admins want to get it out now so people can stop complaining about it. i get it though just wait for the 18 instead of the rc-1 but users are impatient and can’t wait for sh*t!
Thanks for the hard work /u/ruud, we’ll get there
Running one of the largest and most vibrant Lemmy instances certainly makes this deployment a great test case for the devs and the system overall. @ruud@lemmy.world this community really appreciates you!
That doesn’t make sense. Of course you can fail. Stop filling the boy’s head with this nonsense.
The motivational post is self-explanatory
pretty brutal bug ngl. was hoping I’d be using jerboa again soon
Is the bug with Jerboa that no subscribed communities are showing up? From my point of view logged in now I only have access to lemmy.world communities
Jerboa crashes for me
Download v0.34 from F-droid or Obtainium, which both let you downgrade. It’s compatible with Lemmy v0.17.
appreciate the transparency. how are things looking in the back in lemmy.world (server wise)? will we get to a point where it wont require complete rollbacks on the state when a botched update gets rolled out?
I want to upvote, but I get this instead.
Can you clear your browser cache? Should be ok after
Oh man, for a sec there you made me thing I was getting errors 😅
Same on thunder app
I’m getting the same on my iPhone through lemmy.world
Suddenly my Jerboa won’t change from local and I can’t change the sort. Not sure if this is related, but it was working fine yesterday after I updated the app. But no longer this morning.
deleted by creator
Yeah we have a test instance, but not sure if we could test with this kind of load…
Yeah we have a test instance, but not sure if we could test with this kind[ of load…
This is the problem. The big servers have accumulated data that is not in the testing servers and smaller peers. Lemmy instances start out with no data, and joining a community from a remote instance only brings in new data. So far, the big servers have not openly published their lemmy_server logging so we don’t know what is going on inside the Rust code - we are only seeing the symptoms with front-end crashes and federation failing to get comments to other servers.
There are already 3 test instances where devs push updates before going live, but:
- if not many people go there test, they won’t catch all the bugs.
- in this specific case, admins are trying to push a new release candidate (that is not officially released yet) because some people are stuck with mobile apps that don’t work anymore on the version we have now.
